Existence of ground state solutions for a class of quasilinear elliptic systems in Orlicz-Sobolev spaces

Abstract

In this paper, we investigate the following nonlinear and non-homogeneous elliptic system:

$$\begin{aligned} \textstyle\begin{cases} {-}\operatorname{div}(a_{1}( \vert \nabla{u} \vert )\nabla{u})+V_{1}(x)a_{1}( \vert u \vert )u=F_{u}(x,u,v)\quad \mbox{in } \mathbb{R}^{N},\\ {-}\operatorname{div}(a_{2}( \vert \nabla{v} \vert )\nabla{v})+V_{2}(x)a_{2}( \vert v \vert )v=F_{v}(x,u,v) \quad\mbox{in } \mathbb{R}^{N},\\ (u, v)\in W^{1,\Phi_{1}}(\mathbb{R}^{N})\times W^{1, \Phi_{2}}(\mathbb{R}^{N}), \end{cases}\displaystyle \end{aligned}$$

where $\phi_{i}(t)=a_{i}( \vert t \vert )t (i=1,2)$ are two increasing homeomorphisms from $\mathbb{R}$ onto $\mathbb{R}$, functions $V_{i}(i=1,2)$ and F are 1-periodic in x, and F satisfies some $(\phi_{1},\phi_{2})$-superlinear Orlicz-Sobolev conditions. By using a variant mountain pass lemma, we obtain that the system has a ground state.

1 Introduction

In this paper, we consider the following nonlinear and non-homogeneous elliptic system in Orlicz-Sobolev spaces:

$$\begin{aligned} \textstyle\begin{cases} {-}\operatorname{div}(a_{1}( \vert \nabla{u} \vert )\nabla{u})+V_{1}(x)a_{1}( \vert u \vert )u=F_{u}(x,u,v) \quad\mbox{in } \mathbb{R}^{N},\\ {-}\operatorname{div}(a_{2}( \vert \nabla{v} \vert )\nabla{v})+V_{2}(x)a_{2}( \vert v \vert )v=F_{v}(x,u,v) \quad \mbox{in } \mathbb{R}^{N},\\ (u, v)\in W^{1,\Phi_{1}}(\mathbb{R}^{N})\times W^{1, \Phi_{2}}(\mathbb{R}^{N}), \end{cases}\displaystyle \end{aligned}$$

(1.1)

where $a_{i} (i=1,2):(0,+\infty)\rightarrow\mathbb{R}$ are two functions satisfying:

$(\phi_{1})$ :

$\phi_{i}(i=1,2):\mathbb{R}\rightarrow\mathbb{R}$ defined by

$$\begin{aligned} \phi_{i}(t)= \textstyle\begin{cases} a_{i}( \vert t \vert )t &\mbox{for } t\neq0,\\ 0 &\mbox{for } t=0, \end{cases}\displaystyle \end{aligned}$$

(1.2)

are two increasing homeomorphisms from $\mathbb{R}$ onto $\mathbb {R}$ ;

$(\phi_{2})$ :

$$1< l_{i}:=\inf_{t>0}\frac{t\phi_{i}(t)}{\Phi_{i}(t)}\leq\sup _{t>0}\frac{t\phi _{i}(t)}{\Phi_{i}(t)}=:m_{i}< \min\bigl\{ N,l_{i}^{*}\bigr\} , $$

where

$$\Phi_{i}(t):= \int_{0}^{t}\phi_{i}(s)\,ds,\quad t\in[0, \infty) \quad\textit{and} \quad l_{i}^{*}:=\frac{l_{i}N}{N-l_{i}}, $$

$V_{i}(i=1,2)$ satisfy

$(V_{1})$ :

$V_{i}(i=1,2)\in C(\mathbb{R}^{N}, \mathbb{R})$ are 1-periodic in $x_{1}, \ldots, x_{N}$ (called 1-periodic in x for short);

$(V_{2})$ :

there exist two constants $\alpha_{1}, \alpha_{2}>0$ such that

$$\alpha_{1}\leq\min\bigl\{ V_{1}(x), V_{2}(x)\bigr\} \leq\max\bigl\{ V_{1}(x), V_{2}(x)\bigr\} \leq\alpha _{2} \quad\textit{for all }x\in\mathbb{R}^{N}, $$

and F satisfies

$(F_{1})$ :

$F\in C^{1}(\mathbb{R}^{N}\times\mathbb{R}\times\mathbb{R})$ is 1-periodic in x, $F(x,0,0)=0$ for all $x\in\mathbb{R}^{N}$.

Set $a_{2}=a_{1}$, $v=u$, $V_{2}=V_{1}$ and $F(x,u,v)=F(x,v,u)$. Then system (1.1) reduces to the following quasilinear elliptic equation:

$$\begin{aligned} \textstyle\begin{cases} {-}\operatorname{div}(a_{1}( \vert \nabla{u} \vert )\nabla{u})+V_{1}(x)a_{1}( \vert u \vert )u=f(x,u) \quad\mbox{in } \mathbb{R}^{N},\\ u\in W^{1,\Phi_{1}}(\mathbb{R}^{N}). \end{cases}\displaystyle \end{aligned}$$

(1.3)

When $a_{1}( \vert t \vert )t= \vert t \vert ^{p-2}t (p>1)$, equation (1.3) reduces to the following well-known p-Laplacian equation:

$$\begin{aligned} \textstyle\begin{cases} {-}\operatorname{div}( \vert \nabla{u} \vert ^{p-2}\nabla{u})+V_{1}(x) \vert u \vert ^{p-2}u=f(x,u) \quad\mbox{in } \mathbb{R}^{N},\\ u\in W^{1,p}(\mathbb{R}^{N}). \end{cases}\displaystyle \end{aligned}$$

(1.4)

To investigate the solutions of p-Laplacian equations like (1.4), the variational method has become one of useful tools over the past several decades (see [1] and the references therein). In most of the references, to ensure the boundedness of the Palais-Smale ((PS) for short) sequence of the energy functional, the following growth condition due to Ambrosetti-Rabinowitz [1] was always assumed for the nonlinearity f:

(AR) there exists $\mu>p$ such that

$$0< \mu F(x,u)\leq uf(x,u) \quad\mbox{for all } u\neq0, $$

where, and in the sequel, $F(x,u)=\int_{0}^{u}f(x,s)\,ds$ . $(AR)$ implies that there exist two positive constants $c_{1}, c_{2}$ such that

$$F(x,u)\geq c_{1} \vert u \vert ^{\mu}-c_{2} \quad\mbox{for all } (x,u)\in\mathbb{R}^{N}\times \mathbb{R}, $$

which shows that $(AR)$ is a p-superlinear growth condition. Based on the fact that the (PS) condition can be replaced by the weaker Cerami condition for some deformation theorems which are the footstone for minimax methods, some new p-superlinear growth conditions were established in order to weaken $(AR)$. For example, in [2], for the case $p=2$, Ding and Szulkin replaced $(AR)$ with conditions:

$(f_{1})$ :

$\lim_{ \vert u \vert \rightarrow\infty}\frac{F(x,u)}{ \vert u \vert ^{p}}=+\infty$ uniformly in $x\in\mathbb{R}^{N}$ ;

$(f_{2})$ :

$\mathcal{F}(x,u)>0$ for all $u\neq0$ , and $\vert f(x,u) \vert ^{\tau }\leq c_{3}\mathcal{F}(x,u) \vert u \vert ^{\tau}$ for some $c_{3}>0, \tau>\max\{1,\frac {N}{2}\}$ and all $(x,u)$ with $\vert u \vert$ large enough, where $\mathcal {F}(x,u)=f(x,u)u-2F(x,u)$ .

They proved that $(f_{1})$ and $(f_{2})$ hold if the nonlinearity f satisfies $(AR)$ and a subcritical growth condition that $\vert f(x,u) \vert \leq c_{4}( \vert u \vert + \vert u \vert ^{q-1})$ for some $c_{4}>0, q\in(2, 2^{*})$ and all $(x,u)\in \mathbb{R}^{N}\times\mathbb{R}$, where $2^{*}=\frac{2N}{N-2}$ if $N\geq3$ and $2^{*}=\infty$ if $N=1$ or $N=2$. Some conditions similar to $(f_{2})$ were also introduced in [3] for the case $p=2$ and in [4] for the case $p>1$. Moreover, in [5], Liu proved the existence of ground state for equation (1.4) when the nonlinearity f satisfies $(f_{1})$, the following p-superlinear growth condition:

$(f_{3})$ :

there exists $\theta\geq1$ such that $\theta\mathcal {F}(x,u)\geq\mathcal{F}(x,su)$ for all $(x,u)\in\mathbb{R}^{N}\times \mathbb{R}$ and $s\in[0,1]$ , where $\mathcal {F}(x,u)=f(x,u)u-pF(x,u)$ ,

and some reasonable assumptions. Instead of minimizing the energy functional on the Nehari manifold, they obtained that (1.4) has a nontrivial solution by a mountain pass type argument, and then, by using a technique of Jeanjean and Tanaka in [6], they obtained that (1.4) has a ground state. $(f_{1})$ and $(f_{3})$ are different from $(AR)$. Indeed, in [5], an example which satisfies $(f_{1})$ and $(f_{3})$ but does not satisfy $(AR)$ was given, that is,

$$f(x,u)= \vert t \vert ^{p-2}\log\bigl(1+ \vert t \vert \bigr), $$

and in [3], an example which satisfies $(AR)$ but does not satisfy $(f_{3})$ was also given when $p=2$, that is,

$$\begin{aligned} f(x.u)=3 \vert u \vert ^{2} \int_{0}^{u} \vert t \vert ^{1+\sin t}t\,dt+ \vert u \vert ^{4+\sin u}u. \end{aligned}$$

(1.5)

Under assumption $(\phi_{1})$, equations like (1.3) may be allowed to possess more complicated nonlinear or non-homogeneous operator $\phi _{1}$, which can be used to model many phenomena (see [7, 8]). Based on these interesting facts, this type of equations has caused great interest among scholars in recent years. In Clément et al. [9], the authors firstly studied the existence of nontrivial solution for the following equation in Orlicz-Sobolev spaces by the variational method:

$$\begin{aligned} \textstyle\begin{cases} {-}\operatorname{div}(a_{1}( \vert \nabla{u} \vert )\nabla{u})=f(x,u) \quad\mbox{in } \Omega,\\ u=0 \quad\mbox{on } \partial\Omega, \end{cases}\displaystyle \end{aligned}$$

where Ω is a bounded domain in $\mathbb{R}^{N}$ with smooth boundary ∂Ω and f satisfies the following $(AR)$ type condition for ϕ-Laplacian operator and some reasonable assumptions:

$(\operatorname{AR})^{*}$ there exist $\mu>\limsup_{t\rightarrow+\infty }\frac{t\phi_{1}(t)}{\Phi_{1}(t)}$ and $R_{1}>0$ such that

$$0\leq\mu F(x,u)\leq uf(x,u) \quad \textit{for all } (x,u)\in\overline {\Omega}\times \mathbb{R} \textit{ with } \vert u \vert \geq R_{1}. $$

From then on, the variational method has been used widely to study the existence and multiplicity of solutions for this type of elliptic equations, and some growth conditions for the nonlinearity f in the case of p-Laplacian type were extended to the case of ϕ-Laplacian type (for example, see [8, 10, 11]). However, there are very few results regarding the existence of ground state for equations like (1.3). In [12], by using the mountain pass lemma and the Nehari manifold method, Alves and Silva proved the existence of nonnegative ground state for the following ϕ-Laplacian equation with autonomous nonlinearity f:

$$\begin{aligned} \textstyle\begin{cases} {-}\operatorname{div}(a_{1}( \vert \nabla{u} \vert )\nabla{u})+V_{1}(\epsilon x)a_{1}( \vert u \vert )u=f(u) \quad\mbox{in } \mathbb{R}^{N},\\ u\in W^{1,\Phi_{1}}(\mathbb{R}^{N}), \end{cases}\displaystyle \end{aligned}$$

where $f:\mathbb{R}\rightarrow\mathbb{R}$ is a $C^{1}$-function satisfying an $(AR)$ type condition and some reasonable assumptions, ϵ is a positive parameter, and $V_{1}:\mathbb{R}^{N}\rightarrow\mathbb{R}$ is a continuous function belonging to the autonomous case $V_{1}(x)\equiv\mu$ (see Theorem 3.4 in [12]) or the nonautonomous case

$$V_{\infty}:=\liminf_{ \vert x \vert \rightarrow\infty}V_{1}(x)>V_{0}:= \inf_{\mathbb {R}^{N}}V_{1}(x)>0 \quad\mbox{(see Theorem 4.11 in [12])}. $$

For the systems like (1.1), on the whole space $\mathbb{R}^{N}$, to the best of our knowledge, there is no paper to study the existence and multiplicity of solutions by the variational method, except for [13]. In [13], we investigated system (1.1) with $V_{i}(x)(i=1, 2): \mathbb{R}^{N}\rightarrow\mathbb{R}$ satisfying $(V_{2})$ and $a_{i} (i=1,2):(0,+\infty)\rightarrow\mathbb{R}$ satisfying $(\phi_{1})$ and $(\phi_{2})$. By using the least action principle, we obtained that system (1.1) has at least one nontrivial solution if $F: \mathbb{R}^{N}\times\mathbb{R}\times\mathbb{R}\rightarrow\mathbb {R}$ is a $C^{1}$ function, $F(x,0,0)=0$ and satisfies

• (1) there exist constants $p_{i}\in[m_{i}, l_{i}^{*})(i=1, 2)$ , $\max\{ \frac{1}{p_{1}}, \frac{1}{p_{2}}\}\leq q_{1}< q_{2}<\cdots<q_{k}<\min\{\frac {l_{1}}{p_{1}},\frac{l_{2}}{p_{2}}\}$ , and functions $a_{1j}, a_{2j}, a_{3j}, a_{4j}\in L^{\frac{1}{1-q_{j}}}(\mathbb{R}^{N}, [0,+\infty))(j=1,2,\ldots ,k)$ such that

  $$\begin{aligned} &\bigl\vert F_{u}(x,u,v) \bigr\vert \leq\sum _{j=1}^{k}a_{1j}(x) \vert u \vert ^{p_{1}q_{j}-1}+\sum_{j=1}^{k}a_{2j}(x) \vert v \vert ^{\frac{p_{2}(p_{1}q_{j}-1)}{p_{1}}}, \\ &\bigl\vert F_{v}(x,u,v) \bigr\vert \leq\sum _{j=1}^{k}a_{3j}(x) \vert u \vert ^{\frac {p_{1}(p_{2}q_{j}-1)}{p_{2}}}+\sum_{j=1}^{k}a_{4j}(x) \vert v \vert ^{p_{2}q_{j}-1} \end{aligned}$$

  for all $(x,u,v)\in\mathbb{R}^{N}\times\mathbb{R}\times\mathbb{R}$ ;

• (2) there exist an open set $\Omega\subset\mathbb{R}^{N}$ with $\vert \Omega \vert >0$ and constants $\alpha_{0}\in[1,l_{1})$ , $\beta_{0}\in[1,l_{2})$ , $\delta >0$ , $c>0$ and $\iota,\kappa\in\mathbb{R}$ with $\iota^{2}+\kappa^{2}\neq 0$ such that

  $$\begin{aligned} F(x,\iota t,\kappa t)\geq c\bigl( \vert \iota t \vert ^{\alpha_{0}}+ \vert \kappa t \vert ^{\beta_{0}}\bigr)\quad \textit{for all }(x,t)\in\Omega \times[0, \delta]. \end{aligned}$$

Moreover, when F satisfies an additional symmetric condition, by using the genus theory, we also obtained that system (1.1) has infinitely many solutions.

On the whole space $\mathbb{R}^{N}$, the main difficulty for this type of elliptic equations and systems without the $(AR)$ type conditions is the lack of compactness of the Sobolev embedding, which is crucial to ensure the boundedness of (PS) or Cerami sequence. To overcome this difficulty, the usual way is to reconstruct the compactness of the Sobolev embedding, which can be done by assuming that $V_{1}$ and f possess the radially symmetric structure (that is, $V_{1}$ and f depend on $\vert x \vert$) and then choosing a radially symmetric function subspace as the working space (see [8, 14, 15]) or by assuming that $V_{1}$ is coercive and then choosing a subspace depending on $V_{1}$ as the working space (see [3, 4, 16, 17]). Then radial and nonradical solutions can be obtained, respectively. When $V_{1}$ is bounded and $V_{1}, f$ are without the radially symmetric structure, the compactness of the Sobolev embedding will be lost. For this situation, to ensure the boundedness of (PS) or Cerami sequence, a useful way is to assume that $V_{1}$ and f satisfy some specific periodicity conditions (see [5, 8, 18]), and another useful way is to assume that the nonlinearity satisfies a sublinear growth condition such that the energy functional is coercive (see [13]).

In this paper, we study the existence of ground state for system (1.1) under the assumption that $V_{i}(i=1,2)$ and F are 1-periodic in x. Motivated by [5], we also obtain that system (1.1) has a nontrivial solution by a variant mountain pass lemma, and then by using a technique of Jeanjean and Tanaka in [6], we obtain the existence of ground state. We manage to extend the p-superlinear growth conditions $(AR)$ and $(f_{1})$ with $(f_{2})$ for p-Laplacian equations to $(\phi_{1},\phi_{2})$-superlinear growth conditions in the Orlicz-Sobolev space (called $(\phi_{1},\phi _{2})$-superlinear Orlicz-Sobolev conditions for short) for $(\phi_{1},\phi _{2})$-Laplacian systems, respectively (see $(F_{3})$-$(F_{5})$ in Section 3). Since the system case is different from the scalar case, we will come across some new difficulties, and more computing skills are needed in the process of our proofs. We point out that our results are different from those in [12] and [5] even if system (1.1) reduces to equations (1.3) and (1.4).

This paper is organized as follows. In Section 2, we recall some preliminary knowledge on Orlicz and Orlicz-Sobolev spaces. In Section 3, we give our main results and complete the proofs. In Section 4, we present some examples to illustrate our results.

2 Preliminaries

In this section, we introduce some fundamental notions and important properties about Orlicz and Orlicz-Sobolev spaces. We refer the reader for more details to the books [19, 20] and the references therein.

First of all, we recall the notion of N-function. Let $\phi: [0,+\infty)\rightarrow[0,+\infty)$ be a right-continuous, monotone increasing function with

• (1) $\phi(0)=0$;

• (2) $\lim_{t\rightarrow+\infty} \phi(t)=+\infty$;

• (3) $\phi(t)>0$ whenever $t>0$.

Then the function defined on $[0, +\infty)$ by $\Phi(t)=\int _{0}^{t}\phi(s)\,ds$ is called an N-function. It is obvious that $\Phi(0)=0$ and Φ is strictly increasing and convex in $[0, +\infty)$.

An N-function Φ satisfies a $\Delta_{2}$-condition globally (or near infinity) if

$$\sup_{t>0}\frac{\Phi(2t)}{\Phi(t)}< +\infty\quad \biggl(\mbox{or } \limsup_{t\rightarrow+\infty}\frac{\Phi(2t)}{\Phi(t)}< +\infty\biggr), $$

which implies that there exists a constant $K>0$ such that $\Phi (2t)\leq K\Phi(t)$ for all $t\geq0$ (or $t\geq t_{0}>0$). Φ satisfies a $\Delta_{2}$-condition globally (or near infinity) if and only if for any given $c\geq1$, there exists a constant $K_{c}>0$ such that $\Phi(ct)\leq K_{c}\Phi(t)$ for all $t\geq0$ (or $t\geq t_{0}>0$).

For the N-function Φ, the complement of Φ is given by

$$\widetilde{\Phi}(t)=\max_{s\geq0}\bigl\{ ts-\Phi(s)\bigr\} \quad\mbox{for } t\geq0. $$

Φ̃ is also an N-function and $\widetilde{\widetilde {\Phi}}=\Phi$. In addition, we have Young’s inequality, that is,

$$\begin{aligned} st\leq\Phi(s)+\widetilde{\Phi}(t) \quad\mbox{for all } s, t\geq0, \end{aligned}$$

(2.1)

and the following inequality (see [21], Lemma A.2):

$$\begin{aligned} \widetilde{\Phi}\bigl(\phi(t)\bigr)\leq\Phi(2t) \quad\mbox{for all } t \geq0. \end{aligned}$$

(2.2)

Now we recall the Orlicz space $L^{\Phi}(\Omega)$ associated with Φ. When Φ satisfies the $\Delta_{2}$-condition globally, the Orlicz space $L^{\Phi}(\Omega)$ is the vectorial space of measurable functions $u: \Omega\rightarrow\mathbb{R}$ satisfying

$$\int_{\Omega}\Phi\bigl( \vert u \vert \bigr)\,dx< +\infty, $$

where $\Omega\subset\mathbb{R}^{N}$ is an open set. $L^{\Phi}(\Omega)$ is a Banach space endowed with Luxemburg norm

$$\Vert u \Vert _{\Phi}:=\inf \biggl\{ \lambda>0: \int_{\Omega}\Phi \biggl(\frac { \vert u \vert }{\lambda} \biggr)\,dx\leq1 \biggr\} \quad\mbox{for } u\in L^{\Phi}(\Omega). $$

Particularly, when $\Phi(t)= \vert t \vert ^{p} (1< p<+\infty)$, the corresponding Orlicz space $L^{\Phi}(\Omega)$ is the classical Lebesgue space $L^{p}(\Omega)$ and the corresponding Luxemburg norm $\Vert u \Vert _{\Phi}$ is equal to the classical $L^{p}(\Omega)$ norm, that is,

$$\Vert u \Vert _{L^{p}(\Omega)}:= \biggl( \int_{\Omega} \bigl\vert u(x) \bigr\vert ^{p}\,dx \biggr)^{\frac{1}{p}} \quad\mbox{for } u\in L^{p}(\Omega). $$

When $\Omega=\mathbb{R}^{N}$, we denote $\Vert u \Vert _{L^{p}(\mathbb{R}^{N})}$ by $\Vert u \Vert _{p}$.

The fact that Φ satisfies the $\Delta_{2}$-condition globally implies that

$$\begin{aligned} u_{n}\rightarrow u \quad\mbox{in } L^{\Phi}(\Omega) \quad\Longleftrightarrow\quad \int _{\Omega}\Phi\bigl( \vert u_{n}-u \vert \bigr)\,dx\rightarrow0. \end{aligned}$$

(2.3)

By the above Young’s inequality (2.1), the following generalized Hölder’s inequality

$$\begin{aligned} \biggl\vert \int_{\Omega}uv\,dx \biggr\vert \leq2 \Vert u \Vert _{\Phi} \Vert v \Vert _{\widetilde{\Phi }} \quad\mbox{for all } u \in L^{\Phi}(\Omega) \mbox{ and } \mbox{all } v \in L^{\widetilde{\Phi}}(\Omega) \end{aligned}$$

(2.4)

can be obtained (see [19, 20]).

Define

$$W^{1, \Phi}(\Omega):= \biggl\{ u \in L^{\Phi}(\Omega): \frac{\partial u}{\partial x_{i}} \in L^{\Phi}(\Omega), i=1,\ldots, N \biggr\} $$

with the norm

$$\Vert u \Vert _{1, \Phi}:= \Vert u \Vert _{\Phi}+ \Vert \nabla u \Vert _{\Phi}. $$

Then $W^{1, \Phi}(\Omega)$ is a Banach space called an Orlicz-Sobolev space. Denote the closure of $C_{0}^{\infty}(\Omega)$ in $W^{1, \Phi }(\Omega)$ by $W_{0}^{1, \Phi}(\Omega)$. Then, by some basic properties in Orlicz-Sobolev spaces, we obtain that $W_{0}^{1, \Phi}(\mathbb {R}^{N})=W^{1, \Phi}(\mathbb{R}^{N})$.

Next, we recall some inequalities. For more details, we refer the reader to the references [19, 21].

Lemma 2.1

see [19, 21]

If Φ is an N-function, then the following conditions are equivalent:

• (1)
  $$\begin{aligned} 1\leq l=\inf_{t>0}\frac{t\phi(t)}{\Phi(t)}\leq\sup _{t>0}\frac{t\phi (t)}{\Phi(t)}=m< +\infty; \end{aligned}$$

  (2.5)

• (2) let $\zeta_{0}(t)=\min\{t^{l}, t^{m}\}$, $\zeta_{1}(t)=\max\{t^{l}, t^{m}\}$ for $t\geq0$. Φ satisfies

  $$\zeta_{0}(t)\Phi(\rho)\leq\Phi(\rho t)\leq\zeta_{1}(t) \Phi(\rho) \quad\textit{for all } \rho, t\geq0; $$

• (3) Φ satisfies a $\Delta_{2}$-condition globally.

Lemma 2.2

see [21]

If Φ is an N-function and (2.5) holds, then Φ satisfies

$$\zeta_{0}\bigl( \Vert u \Vert _{\Phi}\bigr)\leq \int_{\mathbb{R}^{N}}\Phi\bigl( \vert u \vert \bigr)\,dx\leq \zeta_{1}\bigl( \Vert u \Vert _{\Phi}\bigr) \quad\textit{for all }u \in L^{\Phi}\bigl(\mathbb{R}^{N}\bigr). $$

Lemma 2.3

see [21]

If Φ is an N-function and (2.5) holds with $l>1$. Let Φ̃ be the complement of Φ and $\zeta_{2}(t)=\min\{t^{\widetilde {l}},t^{\widetilde{m}}\}$, $\zeta_{3}(t)=\max\{t^{\widetilde {l}},t^{\widetilde{m}}\}$ for $t\geq0$, where $\widetilde{l}:=\frac {l}{l-1}$ and $\widetilde{m}:=\frac{m}{m-1}$. Then Φ̃ satisfies

• (1)
  $$\widetilde{m}=\inf_{t>0}\frac{t\widetilde{\Phi}^{'}(t)}{\widetilde{\Phi }(t)}\leq\sup _{t>0}\frac{t\widetilde{\Phi}^{'}(t)}{\widetilde{\Phi }(t)}=\widetilde{l}; $$
• (2)
  $$\zeta_{2}(t)\widetilde{\Phi}(\rho)\leq\widetilde{\Phi}(\rho t)\leq \zeta _{3}(t)\widetilde{\Phi}(\rho)\quad \textit{for all } \rho, t\geq0; $$
• (3)
  $$\zeta_{2}\bigl( \Vert u \Vert _{\widetilde{\Phi}}\bigr)\leq \int_{\mathbb{R}^{N}}\widetilde{\Phi }\bigl( \vert u \vert \bigr)\,dx \leq\zeta_{3}\bigl( \Vert u \Vert _{\widetilde{\Phi}}\bigr) \quad\textit{for all }u \in L^{\widetilde{\Phi}}\bigl(\mathbb{R}^{N}\bigr). $$

If

$$\begin{aligned} \int_{0}^{1}\frac{\Phi^{-1}(s)}{s^{\frac{N+1}{N}}}\,ds< +\infty\quad\mbox{and}\quad \int_{1}^{+\infty}\frac{\Phi^{-1}(s)}{s^{\frac{N+1}{N}}}\,ds=+\infty, \end{aligned}$$

(2.6)

then the Sobolev conjugate N-function, function $\Phi_{*}$ of Φ, is given in [19] by

$$\Phi_{*}^{-1}(t)= \int_{0}^{t}\frac{\Phi^{-1}(s)}{s^{\frac{N+1}{N}}}\,ds \quad\textit{for } t \geq0. $$

Lemma 2.4

see [21]

If Φ is an N-function and (2.5) holds with $l,m\in(1, N)$, then (2.6) holds. Let $\zeta_{4}(t)=\min\{t^{l^{*}},t^{m^{*}}\}$, $\zeta _{5}(t)=\max\{t^{l^{*}},t^{m^{*}}\}$ for $t\geq0$, where $l^{*}:=\frac {lN}{N-l}$, $m^{*}:=\frac{mN}{N-m}$. Then $\Phi_{*}$ satisfies

• (1)
  $$l^{*}=\inf_{t>0}\frac{t\Phi_{*}'(t)}{\Phi_{*}(t)}\leq\sup_{t>0} \frac{t\Phi _{*}'(t)}{\Phi_{*}(t)}=m^{*}; $$
• (2)
  $$\zeta_{4}(t)\Phi_{*}(\rho)\leq\Phi_{*}(\rho t)\leq\zeta_{5}(t) \Phi_{*}(\rho) \quad\textit{for all } \rho, t\geq0; $$
• (3)
  $$\zeta_{4}\bigl( \Vert u \Vert _{\Phi_{*}}\bigr)\leq \int_{\mathbb{R}^{N}}\Phi_{*}\bigl( \vert u \vert \bigr)\,dx\leq\zeta _{5}\bigl( \Vert u \Vert _{\Phi_{*}}\bigr) \quad\textit{for all }u \in L^{\Phi_{*}}\bigl(\mathbb{R}^{N}\bigr). $$

The following important embedding proposition involving the Orlicz-Sobolev spaces will be used frequently in our proofs.

Lemma 2.5

see [19, 20]

If Φ is an N-function and (2.5) holds with $l,m\in(1, N)$, then the embedding

$$W^{1, \Phi}\bigl(\mathbb{R}^{N}\bigr)\hookrightarrow L^{\Psi}\bigl(\mathbb{R}^{N}\bigr) $$

is continuous for any N-function Ψ satisfying

$$\limsup_{t\rightarrow0^{+}}\frac{\Psi(t)}{\Phi(t)}< \infty \quad\textit{and}\quad \limsup _{t\rightarrow+\infty}\frac{\Psi(t)}{\Phi_{*}(t)}< \infty. $$

Therefore, there exists a constant $C_{\Psi}$ such that

$$\begin{aligned} \Vert u \Vert _{\Psi}\leq C_{\Psi} \Vert u \Vert _{1, \Phi} \quad\textit{for all } u \in W^{1, \Phi}\bigl( \mathbb{R}^{N}\bigr). \end{aligned}$$

(2.7)

If the space $\mathbb{R}^{N}$ is replaced by a bounded domain $D\subset \mathbb{R}^{N}$ and Ψ increases essentially more slowly than $\Phi _{*}$ near infinity, that is,

$$\lim_{t\rightarrow+\infty}\frac{\Psi(ct)}{\Phi_{*}(t)}=0 $$

for any constant $c>0$, then the embedding $W^{1, \Phi }(D)\hookrightarrow L^{\Psi}(D)$ is compact.

Remark 2.6

By Lemmas 2.1, 2.3 and 2.4, $(\phi _{1})$-$(\phi_{2})$ imply that $\Phi_{i}(i=1,2)$, $\widetilde{\Phi }_{i}(i=1,2)$, $\Phi_{i*}(i=1,2)$ and ${\widetilde{\Phi}_{i*}}(i=1,2)$ are N-functions that satisfy the $\Delta_{2}$-condition globally, where and in the sequel ${\widetilde{\Phi}_{i}}$ denotes the complement of $\Phi _{i} (i=1,2)$, $\Phi_{i*}$ denotes the Sobolev conjugate N-function function of $\Phi_{i}(i=1,2)$ and ${\widetilde{\Phi}_{i*}}$ denotes the complement of $\Phi_{i*} (i=1,2)$. Moreover, the fact that $\Phi _{i}(i=1,2)$ and $\widetilde{\Phi}_{i}(i=1,2)$ satisfy the $\Delta _{2}$-condition globally implies that $L^{\Phi_{i}}(\mathbb{R}^{N})(i=1,2)$ and $W^{1, \Phi_{i}}(\mathbb{R}^{N})(i=1,2)$ are separable and reflexive Banach spaces (see [19, 20]).

Remark 2.7

Under assumptions $(\phi_{1})$ and $(\phi_{2})$, Lemmas 2.1, 2.4 and 2.5 imply that the embeddings

$$\begin{aligned} W^{1, \Phi_{i}}\bigl(\mathbb{R}^{N}\bigr) \hookrightarrow L^{\Phi_{i}}\bigl(\mathbb{R}^{N}\bigr),\qquad W^{1, \Phi_{i}}\bigl(\mathbb{R}^{N}\bigr)\hookrightarrow L^{\Phi_{i*}}\bigl(\mathbb{R}^{N}\bigr),\quad i=1,2, \end{aligned}$$

(2.8)

are continuous and the embeddings

$$\begin{aligned} W^{1, \Phi_{i}}(B_{r})\hookrightarrow L^{\Phi_{i}}(B_{r}),\quad i=1,2, \end{aligned}$$

(2.9)

are compact, where and in the sequel $B_{r}=\{x\in\mathbb{R}^{N}: \vert x \vert < r\}$ for $r>0$.

3 Main results and proofs

Theorem 3.1

Assume that $(\phi_{1})$, $(\phi_{2})$, $(V_{1})$, $(V_{2})$, $(F_{1})$ and the following conditions hold:

$(F_{2})$ :

$$\begin{aligned} &\lim_{ \vert (u,v) \vert \rightarrow0}\frac{F_{u}(x,u,v)}{\phi_{1}( \vert u \vert )+{\widetilde {\Phi}_{1}}^{-1}(\Phi_{2}( \vert v \vert ))}=0,\qquad \lim_{ \vert (u,v) \vert \rightarrow0} \frac{F_{v}(x,u,v)}{{\widetilde{\Phi }_{2}}^{-1}(\Phi_{1}( \vert u \vert ))+\phi_{2}( \vert v \vert )}=0, \\ &\lim_{ \vert (u,v) \vert \rightarrow\infty}\frac{F_{u}(x,u,v)}{\Phi _{1*}'( \vert u \vert )+{\widetilde{\Phi}_{1*}}^{-1}(\Phi_{2*}( \vert v \vert ))}=0, \qquad\lim_{ \vert (u,v) \vert \rightarrow\infty} \frac{F_{v}(x,u,v)}{{\widetilde{\Phi }_{2*}}^{-1}(\Phi_{1*}( \vert u \vert ))+\Phi_{2*}'( \vert v \vert )}=0, \end{aligned}$$

uniformly in $x\in\mathbb{R}^{N}$, where and in the sequel ${\widetilde {\Phi}_{i}}^{-1}$ denotes the inverse of $\widetilde{\Phi}_{i} (i=1,2)$, $\Phi_{i*}'$ denotes the derivative of $\Phi_{i*}(i=1,2)$ and ${\widetilde{\Phi}_{i*}}^{-1}$ denotes the inverse of $\widetilde{\Phi }_{i*}(i=1,2)$;

$(F_{3})$ :

there exist $\mu_{i}>m_{i} (i=1,2)$ such that

$$0< F(x,u,v)\leq\frac{1}{\mu_{1}}uF_{u}(x,u,v)+\frac{1}{\mu_{2}}vF_{v}(x,u,v)\quad \textit{for all }(u,v)\neq(0,0). $$

Then system (1.1) has a ground state, that is, a nontrivial solution $(u_{0},v_{0})$ such that

$$I(u_{0},v_{0})=\inf\bigl\{ I(u,v): (u,v)\in W\setminus \bigl\{ (\mathbf{0,0})\bigr\} \textit{ and } I'(u,v)=0\bigr\} , $$

where $W=W^{1, \Phi_{1}}(\mathbb{R}^{N})\times W^{1, \Phi_{2}}(\mathbb{R}^{N})$ and

$$\begin{aligned} I(u,v) ={}& \int_{\mathbb{R}^{N}}\Phi_{1}\bigl( \vert \nabla u \vert \bigr)\,dx+ \int_{\mathbb {R}^{N}}V_{1}(x)\Phi_{1}\bigl( \vert u \vert \bigr)\,dx \\ & {}+ \int_{\mathbb{R}^{N}}\Phi_{2}\bigl( \vert \nabla v \vert \bigr)\,dx+ \int_{\mathbb {R}^{N}}V_{2}(x)\Phi_{2}\bigl( \vert v \vert \bigr)\,dx- \int_{\mathbb{R}^{N}}F(x,u,v)\,dx. \end{aligned}$$

Theorem 3.2

Assume that $(\phi_{1})$, $(\phi_{2})$, $(V_{1})$, $(V_{2})$, $(F_{1})$, $(F_{2})$ and the following conditions hold:

$(\phi_{3})$ :

$$\limsup_{t\rightarrow0}\frac{ \vert t \vert ^{l_{i}}}{\Phi_{i}( \vert t \vert )}< \infty, \quad i=1,2; $$

$(F_{4})$ :

$$\lim_{ \vert (u,v) \vert \rightarrow\infty}\frac{F(x,u,v)}{\Phi_{1}( \vert u \vert )+\Phi _{2}( \vert v \vert )}=+\infty $$

uniformly in $x\in\mathbb{R}^{N}$;

$(F_{5})$ :

$\overline{F}(x,u,v)>0$ for all $(u,v)\neq(0,0)$ and there exists $k>\max\{\frac{N}{l_{1}},\frac{N}{l_{2}}\}$ such that

$$\limsup_{ \vert (u,v) \vert \rightarrow\infty} \biggl(\frac {F(x,u,v)}{ \vert u \vert ^{l_{1}}+ \vert v \vert ^{l_{2}}} \biggr)^{k} \frac{1}{\overline {F}(x,u,v)}< \infty, $$

where

$$\overline{F}(x,u,v)=\frac{1}{m_{1}}u F_{u}(x,u,v)+ \frac{1}{m_{2}}v F_{v}(x,u,v)-F(x,u,v). $$

Then system (1.1) has a ground state.

By Lemmas 2.1 and 2.4, it is easy to check that the following conditions $(F_{2})'$ and $(F_{4})'$ imply $(F_{2})$ and $(F_{4})$, respectively.

$(F_{2})'$ :

$$\begin{aligned} &\lim_{ \vert (u,v) \vert \rightarrow0}\frac{F_{u}(x,u,v)}{ \vert u \vert ^{m_{1}-1}+ \vert v \vert ^{\frac {m_{2}(m_{1}-1)}{m_{1}}}}=0,\qquad \lim_{ \vert (u,v) \vert \rightarrow0} \frac{F_{v}(x,u,v)}{ \vert u \vert ^{\frac {m_{1}(m_{2}-1)}{m_{2}}}+ \vert v \vert ^{m_{2}-1}}=0, \\ &\lim_{ \vert (u,v) \vert \rightarrow\infty}\frac {F_{u}(x,u,v)}{ \vert u \vert ^{l_{1}^{*}-1}+ \vert v \vert ^{\frac{l_{2}^{*}(l_{1}^{*}-1)}{l_{1}^{*}}}}=0,\\ & \lim_{ \vert (u,v) \vert \rightarrow\infty} \frac{F_{v}(x,u,v)}{ \vert u \vert ^{\frac {l_{1}^{*}(l_{2}^{*}-1)}{l_{2}^{*}}}+ \vert v \vert ^{l_{2}^{*}-1}}=0,\quad\textit{uniformly in }x\in \mathbb{R}^{N}; \end{aligned}$$

$(F_{4})'$ :

$$\lim_{ \vert (u,v) \vert \rightarrow\infty}\frac {F(x,u,v)}{ \vert u \vert ^{m_{1}}+ \vert v \vert ^{m_{2}}}=+\infty, \quad\textit{uniformly in }x \in \mathbb{R}^{N}. $$

Thus, we have the following corollary.

Corollary 3.3

In Theorems 3.1 and 3.2, if conditions $(F_{2})$ and $(F_{4})$ are replaced by $(F_{2})'$ and $(F_{4})'$, respectively, then the conclusions still hold.

Remark 3.4

We point out that Theorems 3.1 and 3.2 are complementary, which is based on the fact that there are functions satisfying $(F_{4})$ and $(F_{5})$ but not satisfying $(F_{3})$ (see Example 4.2 in Section 4) and there are also functions $\phi _{i}(i=1,2)$ defined by (1.2) satisfying $(\phi_{1})$ and $(\phi_{2})$ but not satisfying $(\phi_{3})$ (see Case 4 in Section 4).

When system (1.1) reduces to equation (1.3), we present the following results which correspond to Theorems 3.1 and 3.2.

Corollary 3.5

Assume that functions $a_{1}$, $V_{1}$ and f satisfy $(\phi_{1})$-$(\phi_{2})$, $(V_{1})$-$(V_{2})$ and

$(f_{1})^{*}$ :

$f\in C(\mathbb{R}^{N}, \mathbb{R})$ is 1-periodic in x,

$(f_{2})^{*}$ :

$$\lim_{ \vert u \vert \rightarrow0}\frac{f(x,u)}{\phi_{1}( \vert u \vert )}=0,\qquad \lim_{ \vert u \vert \rightarrow\infty} \frac{f(x,u)}{\Phi_{1*}'( \vert u \vert )}=0 $$

uniformly in $x\in\mathbb{R}^{N}$;

$(f_{3})^{*}$ :

there exists $\mu>m_{1}$ such that

$$0< \mu F(x,u)\leq uf(x,u) \quad\textit{for all } u\neq0. $$

Then equation (1.3) has a ground state in $W^{1,\Phi_{1}}(\mathbb{R}^{N})$.

Corollary 3.6

Assume that functions $a_{1}$, $V_{1}$ and f satisfy $(\phi_{1})$-$(\phi_{3})$, $(V_{1})$-$(V_{2})$, $(f_{1})^{*}$-$(f_{2})^{*}$ and

$(f_{4})^{*}$ :

$$\lim_{ \vert u \vert \rightarrow\infty}\frac{F(x,u)}{\Phi_{1}( \vert u \vert )}=+\infty $$

uniformly in $x\in\mathbb{R}^{N}$;

$(f_{5})^{*}$ :

$\overline{F}(x,u)>0$ for all $u\neq0$ and there exists $k>\frac{N}{l_{1}}$ such that

$$\limsup_{ \vert (u,v) \vert \rightarrow\infty} \biggl(\frac{F(x,u)}{ \vert u \vert ^{l_{1}}} \biggr)^{k} \frac{1}{\overline{F}(x,u)}< \infty, $$

where

$$\overline{F}(x,u)=u f(x,u)-m_{1}F(x,u,v). $$

Then equation (1.3) has a ground state in $W^{1,\Phi_{1}}(\mathbb{R}^{N})$.

Remark 3.7

It is easy to see that our results are different from Theorem 3.4 and Theorem 4.11 in [12].

Remark 3.8

For the nonlinearity f, our subcritical growth condition in the Orlicz-Sobolev space

$$\begin{aligned} \lim_{ \vert u \vert \rightarrow\infty}\frac{f(x,u)}{\Phi_{1*}'( \vert u \vert )}=0, \quad\textit{uniformly in } x\in\mathbb{R}^{N} \end{aligned}$$

(3.1)

in $(f_{2})^{*}$ is weaker than the following one which is usually assumed in many papers in order to consider ϕ-Laplacian problems (for example, see [9–12]):

$(SC)$ there exist a constant $C>0$ and an N-function defined by $\Psi(t):=\int_{0}^{t}\psi(s)\,ds, t\in[0,+\infty)$ satisfying

$$m_{1}< l_{\Psi}:=\inf_{t>0} \frac{t\psi(t)}{\Psi(t)}\leq\sup_{t>0}\frac {t\psi(t)}{\Psi(t)} =:m_{\Psi}< l_{1}^{*} $$

or increasing essentially more slowly than $\Phi_{1*}$ near infinity, such that

$$\limsup_{ \vert u \vert \rightarrow\infty} \biggl\vert \frac{f(x,u)}{\psi(u)} \biggr\vert < \infty,\quad \textit{uniformly in } x\in\mathbb{R}^{N}. $$

Condition (3.1) was introduced by Alves et al. [8] for the autonomous nonlinearity f in the Orlicz-Sobolev space. When $a_{1}( \vert t \vert )t= \vert t \vert ^{p-2}t (p>1)$ , (3.1) reduces to

$$\begin{aligned} \lim_{ \vert u \vert \rightarrow\infty}\frac{f(x,u)}{ \vert u \vert ^{p^{*}-1}}=0, \quad\mbox{uniformly in } x\in\mathbb{R}^{N}, \end{aligned}$$

(3.2)

which was first introduced by Liu and Wang [22] instead of the usual subcritical growth condition, that is, there exist constants $C>0$ and $q\in(p,p^{*})$ such that

$$\begin{aligned} \bigl\vert f(x,u) \bigr\vert \leq C\bigl( \vert u \vert ^{p-1}+ \vert u \vert ^{q-1}\bigr)\quad \textit{for all }(x,u)\in \mathbb {R}^{N}\times\mathbb{R}. \end{aligned}$$

(3.3)

Remark 3.9

A condition similar to $(f_{5})^{*}$ was introduced by Carvalho et al. [11] for the ϕ-Laplacian equation in the bounded domain $\Omega\subset\mathbb{R}^{N}$. In this paper, because we consider problems on the whole space $\mathbb {R}^{N}$ where the Sobolev spaces lack compactness of the Sobolev embedding, we claim $\overline{F}(x,u)>0$ for all $u\neq0$ in $(f_{5})^{*}$.

When $a_{1}( \vert t \vert )t= \vert t \vert ^{p-2}t (1< p< N)$, it is obvious that $(\phi _{1})$-$(\phi_{3})$ hold, and then we also present the corresponding results for equation (1.4).

Corollary 3.10

Assume that $N>p$ and functions $V_{1}$ and f satisfy $(V_{1})$-$(V_{2})$, $(f_{1})^{*}$, $(AR)$ and

$(f_{2})'$ :

$$\lim_{ \vert u \vert \rightarrow0}\frac{f(x,u)}{ \vert u \vert ^{p-1}}=0,\qquad \lim_{ \vert u \vert \rightarrow\infty} \frac{f(x,u)}{ \vert u \vert ^{p^{*}-1}}=0 $$

uniformly in $x\in\mathbb{R}^{N}$.

Then equation (1.4) has a ground state in $W^{1,p}(\mathbb{R}^{N})$.

Corollary 3.11

Assume that $N>p$ and functions $V_{1}$ and f satisfy $(V_{1})$-$(V_{2})$, $(f_{1})^{*}$, $(f_{2})'$ and

$(f_{4})'$ :

$$\lim_{ \vert u \vert \rightarrow\infty}\frac{F(x,u)}{ \vert u \vert ^{p}}=+\infty $$

uniformly in $x\in\mathbb{R}^{N}$;

$(f_{5})'$ :

$\overline{F}(x,u)>0$ for all $u\neq0$ and there exists $k>\frac{N}{p}$ such that

$$\limsup_{ \vert (u,v) \vert \rightarrow\infty} \biggl(\frac{F(x,u)}{ \vert u \vert ^{p}} \biggr)^{k} \frac{1}{\overline{F}(x,u)}< \infty, $$

where

$$\overline{F}(x,u)=u f(x,u)-pF(x,u,v). $$

Then equation (1.4) has a ground state in $W^{1,p}(\mathbb{R}^{N})$.

Remark 3.12

If the subcritical growth condition (3.2) in $(f_{2})'$ is replaced by (3.3), Corollary 3.10 becomes a corollary of Corollary 3.11 based on the fact that $(AR)$ and (3.3) imply $(f_{4})'$ and $(f_{5})'$ (see [3, 4]). However, we are not sure whether $(AR)$ and (3.2) imply $(f_{4})'$ and $(f_{5})'$ so that we do not know whether Corollary 3.10 is a corollary of Corollary 3.11. It is remarkable that our Corollaries 3.10 and 3.11 are different from Theorem 1.1 in [5] because there are examples satisfying $(AR)$ and $(f_{5})'$ but not satisfying $(f_{3})$ (see example (1.5) for $p=2$).

Next, we start to present our proofs. By $(\phi_{1})$ and $(\phi_{2})$, we define the space $W:=W^{1, \Phi_{1}}(\mathbb{R}^{N})\times W^{1, \Phi _{2}}(\mathbb{R}^{N})$ with the norm

$$\bigl\Vert (u, v) \bigr\Vert = \Vert u \Vert _{1,\Phi_{1}}+ \Vert v \Vert _{1,\Phi_{2}}= \Vert \nabla u \Vert _{\Phi_{1}}+ \Vert u \Vert _{\Phi_{1}}+ \Vert \nabla v \Vert _{\Phi_{2}}+ \Vert v \Vert _{\Phi_{2}}. $$

Then W is a separable and reflexive Banach space by Remark 2.6.

On W, define a functional I by

$$\begin{aligned} I(u,v):={}& \int_{\mathbb{R}^{N}}\Phi_{1}\bigl( \vert \nabla u \vert \bigr)\,dx+ \int_{\mathbb {R}^{N}}V_{1}(x)\Phi_{1}\bigl( \vert u \vert \bigr)\,dx \\ & {}+ \int_{\mathbb{R}^{N}}\Phi_{2}\bigl( \vert \nabla v \vert \bigr)\,dx+ \int_{\mathbb {R}^{N}}V_{2}(x)\Phi_{2}\bigl( \vert v \vert \bigr)\,dx- \int_{\mathbb{R}^{N}}F(x,u,v)\,dx. \end{aligned}$$

(3.4)

Standard arguments show that I is well defined and of class $C^{1}(W,\mathbb{R})$ and

$$\begin{aligned} \bigl\langle I'(u,v),(\tilde{u},\tilde{v})\bigr\rangle = {}& \int_{\mathbb {R}^{N}}a_{1}\bigl( \vert \nabla u \vert \bigr) \nabla u\nabla\tilde{u}\,dx + \int_{\mathbb{R}^{N}}V_{1}(x)a_{1}\bigl( \vert u \vert \bigr)u\tilde{u}\,dx \\ & {}+ \int_{\mathbb{R}^{N}}a_{2}\bigl( \vert \nabla v \vert \bigr) \nabla v\nabla\tilde{v}\,dx + \int_{\mathbb{R}^{N}}V_{2}(x)a_{2}\bigl( \vert v \vert \bigr)v\tilde{v}\,dx \\ & {}- \int_{\mathbb{R}^{N}}F_{u}(x,u,v)\tilde{u}\,dx- \int_{\mathbb {R}^{N}}F_{v}(x,u,v)\tilde{v}\,dx \end{aligned}$$

(3.5)

for all $(\tilde{u},\tilde{v})\in W$. For the sake of completeness, we give the proof in the Appendix. Thus, the critical points of I in W are weak solutions of system (1.1). Denote by $I_{i}(i=1,2): W\rightarrow\mathbb{R}$ the functionals

$$\begin{aligned} I_{1}(u,v)={}& \int_{\mathbb{R}^{N}}\Phi_{1}\bigl( \vert \nabla u \vert \bigr)\,dx+ \int_{\mathbb {R}^{N}}V_{1}(x)\Phi_{1}\bigl( \vert u \vert \bigr)\,dx + \int_{\mathbb{R}^{N}}\Phi_{2}\bigl( \vert \nabla v \vert \bigr)\,dx \\ &{}+ \int_{\mathbb{R}^{N}}V_{2}(x)\Phi_{2}\bigl( \vert v \vert \bigr)\,dx \end{aligned}$$

(3.6)

and

$$\begin{aligned} I_{2}(u,v)= \int_{\mathbb{R}^{N}}F(x,u,v)\,dx. \end{aligned}$$

(3.7)

Then

$$I(u,v)=I_{1}(u,v)-I_{2}(u,v). $$

Lemma 3.13

If $(F_{1})$ and $(F_{2})$ hold, then there exist positive constants $C_{i}$ $(i=1,2,3)$ such that

$$\begin{aligned} & \bigl\vert F_{u}(x,u,v) \bigr\vert \leq C_{1} \bigl(\phi_{1}\bigl( \vert u \vert \bigr)+{ \widetilde{\Phi}_{1}}^{-1}\bigl(\Phi _{2}\bigl( \vert v \vert \bigr)\bigr)+\Phi_{1*}'\bigl( \vert u \vert \bigr)+{\widetilde{\Phi}_{1*}}^{-1}\bigl(\Phi _{2*}\bigl( \vert v \vert \bigr)\bigr) \bigr), \end{aligned}$$

(3.8)

$$\begin{aligned} &\bigl\vert F_{v}(x,u,v) \bigr\vert \leq C_{2} \bigl({\widetilde{\Phi}_{2}}^{-1}\bigl( \Phi_{1}\bigl( \vert u \vert \bigr)\bigr)+\phi _{2}\bigl( \vert v \vert \bigr)+{\widetilde{\Phi}_{2*}}^{-1}\bigl( \Phi_{1*}\bigl( \vert u \vert \bigr)\bigr)+\Phi _{2*}'\bigl( \vert v \vert \bigr) \bigr), \end{aligned}$$

(3.9)

$$\begin{aligned} & \bigl\vert F(x,u,v) \bigr\vert \leq C_{3} \bigl( \Phi_{1}\bigl( \vert u \vert \bigr)+\Phi_{2}\bigl( \vert v \vert \bigr)+\Phi_{1*}\bigl( \vert u \vert \bigr)+\Phi _{1*}\bigl( \vert v \vert \bigr) \bigr) \end{aligned}$$

(3.10)

for all $(x,u,v)\in\mathbb{R}^{N}\times\mathbb{R}\times\mathbb{R}$, and

$$\begin{aligned} \lim_{ \vert (u,v) \vert \rightarrow0}\frac{F(x,u,v)}{\Phi_{1}( \vert u \vert )+\Phi_{2}( \vert v \vert )}=0, \qquad \lim _{ \vert (u,v) \vert \rightarrow\infty}\frac{F(x,u,v)}{\Phi_{1*}( \vert u \vert )+\Phi_{2*}( \vert v \vert )}=0 \end{aligned}$$

(3.11)

uniformly in $x\in\mathbb{R}^{N}$.

Proof

The proof can be easily completed by virtue of Young’s inequality (2.1) and the fact

$$F(x,u,v)= \int_{0}^{u}F_{s}(x,s,v)\,ds+ \int_{0}^{v}F_{t}(x,0,t)\,dt+F(x,0,0)\quad \mbox{for all } (x,u,v)\in\mathbb{R}^{N}\times\mathbb{R}\times \mathbb{R}. $$

We omit the details. □

Notation

$C_{a}$ denotes a positive constant which depends on the real number a.

Lemma 3.14

Assume that $(\phi_{1})$, $(\phi_{2})$, $(V_{2})$, $(F_{1})$ and $(F_{2})$ hold. Then there exist two positive constants $\rho, \eta$ such that $I(u,v)\geq\eta$ for all $(u,v)\in W$ with $\Vert (u,v) \Vert =\rho$.

Proof

By (3.11), for any given $\varepsilon\in (0,\alpha_{1})$, there exists a constant $C_{\varepsilon}>0$ such that

$$\begin{aligned} \bigl\vert F(x,u,v) \bigr\vert \leq{}&\varepsilon\bigl(\Phi_{1}\bigl( \vert u \vert \bigr)+\Phi_{2}\bigl( \vert v \vert \bigr) \bigr)+C_{\varepsilon}\bigl(\Phi _{1*}\bigl( \vert u \vert \bigr)+\Phi_{2*}\bigl( \vert v \vert \bigr)\bigr) \\ &\mbox{for all } (x,u,v)\in\mathbb{R}^{N}\times \mathbb{R}\times\mathbb{R}. \end{aligned}$$

Then, by (3.4), $(V_{2})$, Lemma 2.2, (3) in Lemma 2.4, (2.8) and (2.7), when $\Vert (u,v) \Vert = \Vert u \Vert _{1,\Phi_{1}}+ \Vert v \Vert _{1,\Phi _{2}}= \Vert \nabla u \Vert _{\Phi_{1}}+ \Vert u \Vert _{\Phi_{1}}+ \Vert \nabla v \Vert _{\Phi_{2}}+ \Vert v \Vert _{\Phi_{2}}\leq1$, we have

$$\begin{aligned} I(u,v)\geq{}& \int_{\mathbb{R}^{N}}\Phi_{1}\bigl( \vert \nabla u \vert \bigr)\,dx+ \int_{\mathbb {R}^{N}}V_{1}(x)\Phi_{1}\bigl( \vert u \vert \bigr)\,dx \\ & {} + \int_{\mathbb{R}^{N}}\Phi_{2}\bigl( \vert \nabla v \vert \bigr)\,dx+ \int_{\mathbb {R}^{N}}V_{2}(x)\Phi_{2}\bigl( \vert v \vert \bigr)\,dx- \int_{\mathbb{R}^{N}} \bigl\vert F(x,u,v) \bigr\vert \,dx \\ \geq{}& \int_{\mathbb{R}^{N}}\Phi_{1}\bigl( \vert \nabla u \vert \bigr)\,dx+\alpha_{1} \int_{\mathbb {R}^{N}}\Phi_{1}\bigl( \vert u \vert \bigr)\,dx + \int_{\mathbb{R}^{N}}\Phi_{2}\bigl( \vert \nabla v \vert \bigr)\,dx+\alpha_{1} \int_{\mathbb {R}^{N}}\Phi_{2}\bigl( \vert v \vert \bigr)\,dx \\ & {} -\varepsilon \int_{\mathbb{R}^{N}}\Phi_{1}\bigl( \vert u \vert \bigr)\,dx- \varepsilon \int _{\mathbb{R}^{N}}\Phi_{2}\bigl( \vert v \vert \bigr)\,dx\\ &{}-C_{\varepsilon} \int_{\mathbb{R}^{N}}\Phi_{1*}\bigl( \vert u \vert \bigr)\,dx-C_{\varepsilon} \int _{\mathbb{R}^{N}}\Phi_{2*}\bigl( \vert v \vert \bigr)\,dx \\ \geq{}& \Vert \nabla u \Vert _{\Phi_{1}}^{m_{1}}+( \alpha_{1}-\varepsilon) \Vert u \Vert _{\Phi _{1}}^{m_{1}}+ \Vert \nabla v \Vert _{\Phi_{2}}^{m_{2}}+(\alpha_{1}- \varepsilon) \Vert v \Vert _{\Phi _{2}}^{m_{2}} \\ & {} -C_{\varepsilon}\max\bigl\{ \Vert u \Vert _{\Phi_{1*}}^{l_{1}^{*}}, \Vert u \Vert _{\Phi _{1*}}^{m_{1}^{*}}\bigr\} -C_{\varepsilon}\max \bigl\{ \Vert v \Vert _{\Phi_{2*}}^{l_{2}^{*}}, \Vert v \Vert _{\Phi _{2*}}^{m_{2}^{*}}\bigr\} \\ \geq{}& \min\{1,\alpha_{1}-\varepsilon\}C_{m_{1}} \Vert u \Vert _{1,\Phi _{1}}^{m_{1}}+\min\{1,\alpha_{1}-\varepsilon \}C_{m_{2}} \Vert v \Vert _{1,\Phi _{2}}^{m_{2}} \\ & {}-C_{\varepsilon}C_{\Phi_{1*}}^{l_{1}^{*}} \Vert u \Vert _{1,\Phi _{1}}^{l_{1}^{*}}-C_{\varepsilon}C_{\Phi_{1*}}^{m_{1}^{*}} \Vert u \Vert _{1,\Phi_{1}}^{m_{1}^{*}} -C_{\varepsilon}C_{\Phi_{2*}}^{l_{2}^{*}} \Vert v \Vert _{1,\Phi _{2}}^{l_{2}^{*}}-C_{\varepsilon}C_{\Phi_{2*}}^{m_{2}^{*}} \Vert v \Vert _{1,\Phi_{2}}^{m_{2}^{*}}. \end{aligned}$$

Note that $m_{i}< l_{i}^{*}\leq m_{i}^{*} (i=1,2)$. It is easy to see that the foregoing inequality implies that there exist positive constants ρ and η small enough such that $I(u,v)\geq\eta$ for all $(u,v)\in W$ with $\Vert (u,v) \Vert =\rho$. □

Lemma 3.15

Assume that $(\phi_{1})$, $(\phi_{2})$, $(V_{2})$, $(F_{1})$ and $(F_{3})$ (or $(F_{4})$) hold. Then there exists $(u_{0},v_{0})\in W$ such that $I(tu_{0},tv_{0})\rightarrow-\infty$ as $t\rightarrow+\infty$.

Proof

First, we prove that under assumptions $(F_{1})$ and $(F_{3})$ (or $(F_{4})$), for any given constant $M>\alpha_{2}$, there exists a constant $C_{M}>0$ such that

$$\begin{aligned} F(x,u,0)\geq M\Phi_{1}\bigl( \vert u \vert \bigr)-C_{M} \quad\mbox{for all } (x,u)\in\mathbb {R}^{N}\times \mathbb{R}. \end{aligned}$$

(3.12)

In fact, it is obvious by $(F_{1})$ and $(F_{4})$. Let $v=0$ in $(F_{3})$. Then $(F_{3})$ reduces to

$$0< F(x,u,0)\leq\frac{1}{\mu_{1}}uF_{u}(x,u,0)\quad \mbox{for all } u \neq0, $$

where $\mu_{1}>m_{1}$, which implies that $F(x,u,0)\geq C( \vert u \vert ^{\mu_{1}}-1)$ for some $C>0$ and all $(x,u)\in\mathbb{R}^{N}\times\mathbb{R}$. Moreover, it follows from (2) in Lemma 2.1 that $\Phi_{1}( \vert u \vert )\leq\Phi _{1}(1)\max\{ \vert u \vert ^{l_{1}}, \vert u \vert ^{m_{1}}\}$ for all $u\in\mathbb{R}$. Since $\mu _{1}>m_{1}$, then for any given constant $M>\alpha_{2}$, there exists a constant $C_{M}>0$ such that (3.12) holds.

Now, choose $u_{0}\in C_{0}^{\infty}(B_{r})\setminus\{\mathbf{0}\}$ with $0\leq u_{0}(x)\leq1$, where $r>0$. Then $(u_{0},0)\in W$, and by (3.4), $(V_{2})$, $(F_{1})$, (3.12) and (2) in Lemma 2.1, when $t>0$, we have

$$\begin{aligned} I(tu_{0},\mathbf{0})= {}& \int_{\mathbb{R}^{N}}\Phi_{1}\bigl( \vert t\nabla u_{0} \vert \bigr)\,dx+ \int _{\mathbb{R}^{N}}V_{1}(x)\Phi_{1}\bigl( \vert tu_{0} \vert \bigr)\,dx- \int_{\mathbb{R}^{N}}F(x,tu_{0},0)\,dx \\ ={} & \int_{\mathbb{R}^{N}}\Phi_{1}\bigl( \vert t\nabla u_{0} \vert \bigr)\,dx+ \int_{B_{r}}V_{1}(x)\Phi _{1}\bigl( \vert tu_{0} \vert \bigr)\,dx- \int_{B_{r}}F(x,tu_{0},0)\,dx \\ \leq{}& \int_{\mathbb{R}^{N}}\Phi_{1}\bigl( \vert t\nabla u_{0} \vert \bigr)\,dx+\alpha_{2} \int_{B_{r}}\Phi _{1}\bigl( \vert tu_{0} \vert \bigr)\,dx-M \int_{B_{r}}\Phi_{1}\bigl( \vert tu_{0} \vert \bigr)+C_{M} \vert B_{r} \vert \\ \leq{}&\Phi_{1}(t) \int_{\mathbb{R}^{N}}\max\bigl\{ \vert \nabla u_{0} \vert ^{l_{1}}, \vert \nabla u_{0} \vert ^{m_{1}}\bigr\} \,dx\\ &{}-\Phi_{1}(t) (M-\alpha_{2}) \int_{B_{r}}\min\bigl\{ \vert u_{0} \vert ^{l_{1}}, \vert u_{0} \vert ^{m_{1}}\bigr\} \,dx+C_{M} \vert B_{r} \vert \\ \leq{}&\Phi_{1}(t) \bigl[ \bigl\Vert \vert \nabla u_{0} \vert \bigr\Vert _{l_{1}}^{l_{1}}+ \bigl\Vert \vert \nabla u_{0} \vert \bigr\Vert _{m_{1}}^{m_{1}}-(M- \alpha_{2}) \Vert u_{0} \Vert _{m_{1}}^{m_{1}} \bigr]+C_{M} \vert B_{r} \vert . \end{aligned}$$

Since $\lim_{t\rightarrow+\infty}\Phi_{1}(t)=+\infty$, we can choose $M>\frac{ \Vert \vert \nabla u_{0} \vert \Vert _{l_{1}}^{l_{1}}+ \Vert \vert \nabla u_{0} \vert \Vert _{m_{1}}^{m_{1}}}{ \Vert u_{0} \Vert _{m_{1}}^{m_{1}}}+\alpha_{2}$ such that $I(tu_{0},\mathbf{0})\rightarrow -\infty$ as $t\rightarrow+\infty$. □

Lemmas 3.14, 3.15 and the fact $I(\mathbf{0,0})=0$ show that I has a mountain pass geometry, that is, setting

$$\Gamma=\bigl\{ \gamma\in C\bigl([0,1],W\bigr):\gamma(0)=\mathbf{0} \mbox{ and } I \bigl(\gamma (1)\bigr)< 0\bigr\} , $$

we have $\Gamma\neq\emptyset$. By a special version of the mountain pass lemma (see [23]), for the mountain pass level

$$\begin{aligned} c=\inf_{\gamma\in\Gamma}\max_{t\in[0,1]}I \bigl(\gamma(t)\bigr), \end{aligned}$$

(3.13)

there exists a $(C)_{c}$-sequence $\{(u_{n},v_{n})\}$ of I in W. Moreover, Lemma 3.14 implies that $c>0$. We recall that $(C)_{c}$-sequence $\{u_{n}, v_{n}\}$ of I in W means

$$\begin{aligned} I(u_{n},v_{n})\rightarrow c \quad\mbox{and}\quad \bigl(1+ \bigl\Vert (u_{n},v_{n}) \bigr\Vert \bigr) \bigl\Vert I'(u_{n},v_{n}) \bigr\Vert _{W^{*}}\rightarrow0 \quad\mbox{as } n\rightarrow\infty. \end{aligned}$$

(3.14)

Lemma 3.16

Assume that $(\phi_{1})$, $(\phi_{2})$, $(V_{2})$, $(F_{1})$-$(F_{3})$ hold. Then any $(C)_{c}$-sequence of I in W is bounded for all $c\geq0$.

Proof

Let $\{(u_{n},v_{n})\}$ be a $(C)_{c}$-sequence of I in W for $c\geq0$. By (3.14), we have

$$\begin{aligned} I(u_{n},v_{n})\rightarrow c \quad\mbox{as } n \rightarrow\infty \end{aligned}$$

(3.15)

and

$$\begin{aligned} \bigl\Vert I'(u_{n},v_{n}) \bigr\Vert _{W^{*}} \bigl\Vert (u_{n},v_{n}) \bigr\Vert = \bigl\Vert I'(u_{n},v_{n}) \bigr\Vert _{W^{*}}\bigl( \Vert u_{n} \Vert _{1,\Phi_{1}}+ \Vert v_{n} \Vert _{1,\Phi_{2}}\bigr)\rightarrow0 \quad\mbox{as } n \rightarrow \infty, \end{aligned}$$

which implies

$$\begin{aligned} \begin{aligned} &\biggl\vert \biggl\langle I'(u_{n},v_{n}), \biggl(\frac{1}{\mu_{1}}u_{n},\frac{1}{\mu _{2}}v_{n} \biggr) \biggr\rangle \biggr\vert \leq \bigl\Vert I'(u_{n},v_{n}) \bigr\Vert _{W^{*}} \biggl(\frac{1}{\mu_{1}} \Vert u_{n} \Vert _{1,\Phi_{1}}+\frac {1}{\mu_{2}} \Vert v_{n} \Vert _{1,\Phi_{2}} \biggr)\rightarrow0 \\ &\quad\mbox{as } n\rightarrow\infty. \end{aligned} \end{aligned}$$

(3.16)

Then, by (3.15), (3.16), (3.4), (3.5), $(\phi _{2})$, $(V_{2})$, $(F_{3})$ and Lemma 2.2, for n large, we have

$$\begin{aligned} c+1 \geq{}&I(u_{n},v_{n})- \biggl\langle I'(u_{n},v_{n}), \biggl( \frac{1}{\mu _{1}}u_{n},\frac{1}{\mu_{2}}v_{n} \biggr) \biggr\rangle \\ = {}& \int_{\mathbb{R}^{N}} \biggl(\Phi_{1}\bigl( \vert \nabla u_{n} \vert \bigr)-\frac{1}{\mu _{1}}a_{1}\bigl( \vert \nabla u_{n} \vert \bigr) \vert \nabla u_{n} \vert ^{2} \biggr)\,dx \\ &{}+ \int_{\mathbb{R}^{N}}V_{1}(x) \biggl(\Phi_{1}\bigl( \vert u_{n} \vert \bigr)-\frac{1}{\mu _{1}}a_{1}\bigl( \vert u_{n} \vert \bigr) \vert u_{n} \vert ^{2} \biggr)\,dx \\ & {}+ \int_{\mathbb{R}^{N}} \biggl(\Phi_{2}\bigl( \vert \nabla v_{n} \vert \bigr)-\frac{1}{\mu _{2}}a_{2}\bigl( \vert \nabla v_{n} \vert \bigr) \vert \nabla v_{n} \vert ^{2} \biggr)\,dx \\ &{}+ \int_{\mathbb{R}^{N}}V_{2}(x) \biggl(\Phi_{2}\bigl( \vert v_{n} \vert \bigr)-\frac{1}{\mu _{2}}a_{2}\bigl( \vert v_{n} \vert \bigr) \vert v_{n} \vert ^{2} \biggr)\,dx \\ & {} + \int_{\mathbb{R}^{N}} \biggl(\frac{1}{\mu_{1}}u_{n}F_{u}(x,u_{n},v_{n})+ \frac {1}{\mu_{2}}v_{n}F_{v}(x,u_{n},v_{n})-F(x,u_{n},v_{n}) \biggr)\,dx \\ \geq{}& \biggl(1-\frac{m_{1}}{\mu_{1}} \biggr) \int_{\mathbb{R}^{N}}\Phi _{1}\bigl( \vert \nabla u_{n} \vert \bigr)\,dx+ \biggl(1-\frac{m_{1}}{\mu_{1}} \biggr) \alpha_{1} \int_{\mathbb {R}^{N}}\Phi_{1}\bigl( \vert u_{n} \vert \bigr)\,dx \\ & {} + \biggl(1-\frac{m_{2}}{\mu_{2}} \biggr) \int_{\mathbb{R}^{N}}\Phi_{2}\bigl( \vert \nabla v_{n} \vert \bigr)\,dx+ \biggl(1-\frac{m_{2}}{\mu_{2}} \biggr) \alpha_{1} \int_{\mathbb{R}^{N}}\Phi _{2}\bigl( \vert v_{n} \vert \bigr)\,dx \\ \geq{}& \biggl(1-\frac{m_{1}}{\mu_{1}} \biggr)\min\bigl\{ \Vert \nabla u_{n} \Vert _{\Phi _{1}}^{l_{1}}, \Vert \nabla u_{n} \Vert _{\Phi_{1}}^{m_{1}}\bigr\} + \biggl(1- \frac{m_{1}}{\mu_{1}} \biggr)\alpha_{1}\min\bigl\{ \Vert u_{n} \Vert _{\Phi _{1}}^{l_{1}}, \Vert u_{n} \Vert _{\Phi_{1}}^{m_{1}}\bigr\} \\ & {} + \biggl(1-\frac{m_{2}}{\mu_{2}} \biggr)\min\bigl\{ \Vert \nabla v_{n} \Vert _{\Phi _{2}}^{l_{2}}, \Vert \nabla v_{n} \Vert _{\Phi_{2}}^{m_{2}}\bigr\} + \biggl(1- \frac{m_{2}}{\mu_{2}} \biggr)\alpha_{1}\min\bigl\{ \Vert v_{n} \Vert _{\Phi _{2}}^{l_{2}}, \Vert v_{n} \Vert _{\Phi_{2}}^{m_{2}}\bigr\} , \end{aligned}$$

which implies that $\Vert (u_{n}, v_{n}) \Vert = \Vert \nabla u_{n} \Vert _{\Phi_{1}}+ \Vert u_{n} \Vert _{\Phi _{1}}+ \Vert \nabla v_{n} \Vert _{\Phi_{2}}+ \Vert v_{n} \Vert _{\Phi_{2}}\leq C$ for some $C>0$, that is, $\{(u_{n},v_{n})\}$ is bounded in W. □

Lemma 3.17

Assume that $(\phi_{1})$-$(\phi_{3})$, $(V_{1})$, $(V_{2})$, $(F_{1})$, $(F_{2})$, $(F_{4})$ and $(F_{5})$ hold. Then any $(C)_{c}$-sequence of I in W is bounded for all $c\geq0$.

Proof

Let $\{(u_{n},v_{n})\}$ be a $(C)_{c}$-sequence of I in W for $c\geq0$. By (3.14), we have

$$\begin{aligned} \begin{aligned} &I(u_{n},v_{n})=I_{1}(u_{n},v_{n})-I_{2}(u_{n},v_{n}) \rightarrow c\quad\mbox{and}\quad \biggl\vert \biggl\langle I'(u_{n},v_{n}), \biggl(\frac{1}{m_{1}}u_{n},\frac {1}{m_{2}}v_{n} \biggr) \biggr\rangle \biggr\vert \rightarrow0\\ &\quad \mbox{as } n\rightarrow\infty. \end{aligned} \end{aligned}$$

(3.17)

Then, by (3.4), (3.5), $(\phi_{2})$ and $(V_{2})$, for n large, we have

$$\begin{aligned} c+1 \geq{}& I(u_{n},v_{n})- \biggl\langle I'(u_{n},v_{n}), \biggl( \frac {1}{m_{1}}u_{n},\frac{1}{m_{2}}v_{n} \biggr) \biggr\rangle \\ = {}& \int_{\mathbb{R}^{N}} \biggl(\Phi_{1}\bigl( \vert \nabla u_{n} \vert \bigr)-\frac {1}{m_{1}}a_{1}\bigl( \vert \nabla u_{n} \vert \bigr) \vert \nabla u_{n} \vert ^{2} \biggr)\,dx \\ &{}+ \int_{\mathbb{R}^{N}}V_{1}(x) \biggl(\Phi_{1}\bigl( \vert u_{n} \vert \bigr)-\frac {1}{m_{1}}a_{1}\bigl( \vert u_{n} \vert \bigr) \vert u_{n} \vert ^{2} \biggr)\,dx \\ & {}+ \int_{\mathbb{R}^{N}} \biggl(\Phi_{2}\bigl( \vert \nabla v_{n} \vert \bigr)-\frac {1}{m_{2}}a_{2}\bigl( \vert \nabla v_{n} \vert \bigr) \vert \nabla v_{n} \vert ^{2} \biggr)\,dx \\ &{}+ \int_{\mathbb{R}^{N}}V_{2}(x) \biggl(\Phi_{2}\bigl( \vert v_{n} \vert \bigr)-\frac {1}{m_{2}}a_{2}\bigl( \vert v_{n} \vert \bigr) \vert v_{n} \vert ^{2} \biggr)\,dx \\ & {} + \int_{\mathbb{R}^{N}} \biggl(\frac{1}{m_{1}}u_{n}F_{u}(x,u_{n},v_{n})+ \frac {1}{m_{2}}v_{n}F_{v}(x,u_{n},v_{n})-F(x,u_{n},v_{n}) \biggr)\,dx \\ \geq{}& \int_{\mathbb{R}^{N}}\overline{F}(x,u_{n},v_{n})\,dx. \end{aligned}$$

(3.18)

To prove the boundedness of $\{(u_{n},v_{n})\}$, arguing by contradiction, we suppose that there exists a subsequence of $\{(u_{n},v_{n})\}$, still denoted by $\{(u_{n},v_{n})\}$, such that $\Vert (u_{n},v_{n}) \Vert = \Vert u_{n} \Vert _{1,\Phi _{1}}+ \Vert v_{n} \Vert _{1,\Phi_{2}}\rightarrow\infty$. Next, we discuss the problem in three cases.

Case 1. Suppose that $\Vert u_{n} \Vert _{1,\Phi_{1}}\rightarrow\infty$ and $\Vert v_{n} \Vert _{1,\Phi_{2}}\rightarrow\infty$. Let $\tilde{u}_{n}=\frac{u_{n}}{ \Vert u_{n} \Vert _{1,\Phi_{1}}}$ and $\tilde{v}_{n}=\frac{v_{n}}{ \Vert v_{n} \Vert _{1,\Phi_{2}}}$. Then $\{{\tilde{u}_{n}}\}$ and $\{{\tilde{v}_{n}}\}$ are bounded in $W^{1,\Phi_{1}}(\mathbb{R}^{N})$ and $W^{1,\Phi_{2}}(\mathbb{R}^{N})$, respectively. We claim that

$$\begin{aligned} \lambda_{1}:=\lim_{n\rightarrow\infty}\sup_{y\in\mathbb{R}^{N}} \int _{B_{2}(y)}\bigl(\Phi_{1}\bigl( \vert \tilde{u}_{n} \vert \bigr)+\Phi_{2}\bigl( \vert \tilde{v}_{n} \vert \bigr)\bigr)\,dx=0. \end{aligned}$$

Indeed, if $\lambda_{1}\neq0$, there exist a constant $\delta>0$, a subsequence of $\{(\tilde{u}_{n},\tilde{v}_{n})\}$, still denoted by $\{ (\tilde{u}_{n},\tilde{v}_{n})\}$, and a sequence $\{z_{n}\}\in\mathbb{Z}^{N}$ such that

$$\begin{aligned} \int_{B_{2}(z_{n})}\bigl(\Phi_{1}\bigl( \vert \tilde{u}_{n} \vert \bigr)+\Phi_{2}\bigl( \vert \tilde{v}_{n} \vert \bigr)\bigr)\,dx>\delta \quad\mbox{for all } n\in \mathbb{N}. \end{aligned}$$

(3.19)

Let $\bar{u}_{n}=\tilde{u}_{n}(\cdot+z_{n})$ and $\bar{v}_{n}=\tilde {v}_{n}(\cdot+z_{n})$. Then $\Vert \bar{u}_{n} \Vert _{1,\Phi_{1}}= \Vert \tilde{u}_{n} \Vert _{1,\Phi_{1}}$ and $\Vert \bar{v}_{n} \Vert _{1,\Phi_{2}}= \Vert \tilde{v}_{n} \Vert _{1,\Phi_{2}}$, that is, $\{{\bar{u}_{n}}\}$ and $\{{\bar{v}_{n}}\}$ are bounded in $W^{1,\Phi_{1}}(\mathbb{R}^{N})$ and $W^{1,\Phi_{2}}(\mathbb{R}^{N})$, respectively. Passing to a subsequence of $\{({\bar{u}_{n}},{\bar{v}_{n}})\}$, still denoted by $\{({\bar{u}_{n}},{\bar{v}_{n}})\}$, by Remark 2.7, there exists $(\bar{u},\bar{v})\in W$ such that

⋆:

$\bar{u}_{n}\rightharpoonup\bar{u}$ in $W^{1,\Phi_{1}}(\mathbb{R}^{N})$, $\bar{u}_{n}\rightarrow\bar{u}$ in $L^{\Phi_{1}}(B_{2})$ and $\bar{u}_{n}(x)\rightarrow\bar{u}(x)$ a.e. in $B_{2}$;

⋆:

$\bar{v}_{n}\rightharpoonup\bar{v}$ in $W^{1,\Phi_{2}}(\mathbb{R}^{N})$, $\bar{v}_{n}\rightarrow\bar{v}$ in $L^{\Phi_{2}}(B_{2})$ and $\bar{v}_{n}(x)\rightarrow\bar{v}(x)$ a.e. in $B_{2}$.

Since

$$\begin{aligned} \int_{B_{2}}\bigl(\Phi_{1}\bigl( \vert \bar{u}_{n} \vert \bigr)+\Phi_{2}\bigl( \vert \bar{v}_{n} \vert \bigr)\bigr)\,dx= \int _{B_{2}(z_{n})}\bigl(\Phi_{1}\bigl( \vert \tilde{u}_{n} \vert \bigr)+\Phi_{2}\bigl( \vert \tilde{v}_{n} \vert \bigr)\bigr)\,dx, \end{aligned}$$

then, by (3.19), ⋆ and (2.3), we obtain that $\bar {u}\neq\mathbf{0}$ in $L^{\Phi_{1}}(B_{2})$ or $\bar{v}\neq\mathbf{0}$ in $L^{\Phi_{2}}(B_{2})$. Without loss of generality, we can assume that $\bar {u}\neq\mathbf{0}$ in $L^{\Phi_{1}}(B_{2})$, that is, $[\bar{u}\neq0]:=\{x \in B_{2}: \bar{u}(x)\neq0\}$ has nonzero Lebesgue measure. Let $u_{n}^{*}=u_{n}(\cdot+z_{n})$ and $v_{n}^{*}=v_{n}(\cdot+z_{n})$. Then $\Vert (u_{n}^{*},v_{n}^{*}) \Vert = \Vert (u_{n},v_{n}) \Vert$, and it follows from that fact that $V_{i}(i=1,2)$ and F are 1-periodic in x that

$$\begin{aligned} I\bigl(u_{n}^{*},v_{n}^{*}\bigr)=I(u_{n},v_{n})\quad \mbox{and} \quad\bigl\Vert I'\bigl(u_{n}^{*},v_{n}^{*} \bigr) \bigr\Vert _{W^{*}}= \bigl\Vert I'(u_{n},v_{n}) \bigr\Vert _{W^{*}} \quad\mbox{for all } n\in\mathbb{N}, \end{aligned}$$

that is, $\{(u_{n}^{*},v_{n}^{*})\}$ is also a $(C)_{c}$-sequence of I. Then, by (3.18), for n large, we have

$$\begin{aligned} \int_{\mathbb{R}^{N}}\overline{F}\bigl(x,u_{n}^{*},v_{n}^{*} \bigr)\,dx\leq c+1. \end{aligned}$$

(3.20)

However, by (2) in Lemma 2.1, $(F_{4})$ and $(F_{5})$ imply

$$\begin{aligned} \lim_{ \vert (u,v) \vert \rightarrow\infty}\overline{F}(x,u,v)=+\infty\quad \mbox{uniformly in } x\in\mathbb{R}^{N}, \end{aligned}$$

(3.21)

and by ⋆, $\bar{u}_{n}=\tilde{u}_{n}(\cdot+z_{n})=\frac{u_{n}(\cdot +z_{n})}{ \Vert u_{n} \Vert _{1,\Phi_{1}}}=\frac{u_{n}^{*}}{ \Vert u_{n} \Vert _{1,\Phi_{1}}}$ implies

$$\begin{aligned} \bigl\vert u_{n}^{*}(x) \bigr\vert = \bigl\vert \bar{u}_{n}(x) \bigr\vert \Vert u_{n} \Vert _{1,\Phi_{1}}\rightarrow\infty, \quad\mbox{a.e. } x\in[\bar{u}\neq0]. \end{aligned}$$

(3.22)

Then, it follows from $(F_{5})$, (3.21), (3.22) and Fatou’s lemma that

$$\begin{aligned} \int_{\mathbb{R}^{N}}\overline{F}\bigl(x,u_{n}^{*},v_{n}^{*} \bigr)\,dx\geq \int_{[\bar{u}\neq 0]}\overline{F}\bigl(x,u_{n}^{*},v_{n}^{*} \bigr)\,dx\rightarrow+\infty, \end{aligned}$$

which contradicts (3.20). Therefore, $\lambda_{1}=0$ and

$$\begin{aligned} \lim_{n\rightarrow\infty}\sup_{y\in\mathbb{R}^{N}} \int_{B_{2}(y)}\Phi _{1}\bigl( \vert \tilde{u}_{n} \vert \bigr)\,dx =\lim_{n\rightarrow\infty}\sup _{y\in\mathbb{R}^{N}} \int_{B_{2}(y)}\Phi _{2}\bigl( \vert \tilde{v}_{n} \vert \bigr)\,dx=0. \end{aligned}$$

(3.23)

By Lemma 2.5, $(\phi_{3})$ and the fact that

$$\limsup_{t\rightarrow+\infty}\frac{t^{l_{i}}}{\Phi_{i*}(t)}\leq\limsup _{t\rightarrow+\infty}\frac{t^{l_{i}}}{\Phi_{i*}(1)\min\{ t^{l_{i}^{*}},t^{m_{i}^{*}}\}}=0, \quad i=1,2, $$

imply that the embeddings $W^{1, \Phi_{i}}(\mathbb{R}^{N})\hookrightarrow L^{l_{i}}(\mathbb{R}^{N})(i=1,2)$ are continuous. Hence, there exists a constant $M_{1}>0$ such that

$$\begin{aligned} \Vert \tilde{u}_{n} \Vert _{l_{1}}^{l_{1}}+ \Vert \tilde{v}_{n} \Vert _{l_{2}}^{l_{2}}\leq M_{1} \quad\mbox{for all } n\in\mathbb{N}. \end{aligned}$$

(3.24)

For $p_{i}\in(l_{i},l_{i}^{*})(i=1,2)$, by $(\phi_{2})$ and $(\phi_{3})$, we have

$$\begin{aligned} \lim_{t\rightarrow0^{+}}\frac{t^{p_{i}}}{\Phi_{i}(t)}=0 \quad\mbox{and}\quad \lim _{t\rightarrow+\infty}\frac{t^{p_{i}}}{\Phi_{i*}(t)}\leq\lim_{t\rightarrow +\infty} \frac{t^{p_{i}}}{\Phi_{i*}(1)\min\{t^{l_{i}^{*}},t^{m_{i}^{*}}\}}=0,\quad i=1,2. \end{aligned}$$

(3.25)

Then, by the Lions type result for Orlicz-Sobolev spaces (see Theorem 1.3 in [8]), (3.23) and (3.25) imply that

$$\begin{aligned} \begin{aligned} &\tilde{u}_{n}\rightarrow\mathbf{0} \quad\mbox{in } L^{p_{1}}\bigl(\mathbb{R}^{N}\bigr) \quad\mbox{and}\\ & \tilde{v}_{n}\rightarrow\mathbf{0} \quad\mbox{in } L^{p_{2}}\bigl( \mathbb{R}^{N}\bigr) \mbox{ for all } p_{1}\in \bigl(l_{1},l_{1}^{*}\bigr), p_{2}\in \bigl(l_{2},l_{2}^{*}\bigr). \end{aligned} \end{aligned}$$

(3.26)

Now, by (3.6), $(V_{2})$ and Lemma 2.2, we have

$$\begin{aligned} & \frac{I_{1}(u_{n},v_{n})}{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+ \Vert v_{n} \Vert _{1,\Phi _{2}}^{l_{2}}} \\ &\quad\geq \frac{\int_{\mathbb{R}^{N}}\Phi_{1}( \vert \nabla u_{n} \vert )\,dx+\alpha_{1}\int _{\mathbb{R}^{N}}\Phi_{1}( \vert u_{n} \vert )\,dx +\int_{\mathbb{R}^{N}}\Phi_{2}( \vert \nabla v_{n} \vert )\,dx+\alpha_{1}\int_{\mathbb {R}^{N}}\Phi_{2}( \vert v_{n} \vert )\,dx}{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+ \Vert v_{n} \Vert _{1,\Phi _{2}}^{l_{2}}} \\ &\quad\geq \frac{\min\{ \Vert \nabla u_{n} \Vert _{\Phi_{1}}^{l_{1}}, \Vert \nabla u_{n} \Vert _{\Phi _{1}}^{m_{1}}\}+\alpha_{1}\min\{ \Vert u_{n} \Vert _{\Phi_{1}}^{l_{1}}, \Vert u_{n} \Vert _{\Phi _{1}}^{m_{1}}\}}{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+ \Vert v_{n} \Vert _{1,\Phi_{2}}^{l_{2}}} \\ & \qquad{} +\frac{\min\{ \Vert \nabla v_{n} \Vert _{\Phi_{2}}^{l_{2}}, \Vert \nabla v_{n} \Vert _{\Phi _{2}}^{m_{2}}\}+\alpha_{1}\min\{ \Vert v_{n} \Vert _{\Phi_{2}}^{l_{2}}, \Vert v_{n} \Vert _{\Phi _{2}}^{m_{2}}\}}{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+ \Vert v_{n} \Vert _{1,\Phi_{2}}^{l_{2}}} \\ &\quad \geq \frac{ \Vert \nabla u_{n} \Vert _{\Phi_{1}}^{l_{1}}+\alpha_{1} \Vert u_{n} \Vert _{\Phi _{1}}^{l_{1}}+ \Vert \nabla v_{n} \Vert _{\Phi_{2}}^{l_{2}}+\alpha_{1} \Vert v_{n} \Vert _{\Phi _{2}}^{l_{2}}-2-2\alpha_{1}}{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+ \Vert v_{n} \Vert _{1,\Phi_{2}}^{l_{2}}} \\ &\quad \geq \frac{\min\{1,\alpha_{1}\}C_{l_{1}} \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+\min\{ 1,\alpha_{1}\}C_{l_{2}} \Vert v_{n} \Vert _{1,\Phi_{2}}^{l_{2}}-2-2\alpha_{1}}{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+ \Vert v_{n} \Vert _{1,\Phi_{2}}^{l_{2}}} \\ &\quad \geq \min\{1,\alpha_{1}\}\min\{C_{l_{1}},C_{l_{2}} \}+o_{n}(1). \end{aligned}$$

(3.27)

Moreover, (3.11) and (2) in Lemma 2.1 imply that

$$\begin{aligned} \lim_{ \vert (u,v) \vert \rightarrow0}\frac{F(x,u,v)}{ \vert u \vert ^{l_{1}}+ \vert v \vert ^{l_{2}}}=0 \end{aligned}$$

uniformly in $x\in\mathbb{R}^{N}$. Then, for any given constant $\varepsilon>0$, there exists a constant $R_{\varepsilon}>0$ such that

$$\begin{aligned} \frac{ \vert F(x,u,v) \vert }{ \vert u \vert ^{l_{1}}+ \vert v \vert ^{l_{2}}}\leq\varepsilon \quad\mbox{for all } x\in \mathbb{R}^{N}, \bigl\vert (u,v) \bigr\vert \leq R_{\varepsilon}, \end{aligned}$$

(3.28)

and by $(F_{1})$ and $(F_{5})$, for above $R_{\varepsilon}>0$, there exists a constant $C_{R}>0$ such that

$$\begin{aligned} \biggl(\frac{ \vert F(x,u,v) \vert }{ \vert u \vert ^{l_{1}}+ \vert v \vert ^{l_{2}}} \biggr)^{k}\leq C_{R}\overline {F}(x,u,v)\quad \mbox{for all } x\in\mathbb{R}^{N}, \bigl\vert (u,v) \bigr\vert > R_{\varepsilon}. \end{aligned}$$

(3.29)

Let

$$X_{n}=\bigl\{ x\in\mathbb{R}^{N}: \bigl\vert \bigl(u_{n}(x),v_{n}(x)\bigr) \bigr\vert \leq R_{\varepsilon}\bigr\} \quad \mbox{and}\quad Y_{n}=\bigl\{ x\in \mathbb{R}^{N}: \bigl\vert \bigl(u_{n}(x),v_{n}(x) \bigr) \bigr\vert > R_{\varepsilon}\bigr\} . $$

Then

$$\begin{aligned} \frac{ \vert I_{2}(u_{n},v_{n}) \vert }{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+ \Vert v_{n} \Vert _{1,\Phi_{2}}^{l_{2}}} \leq& \int_{X_{n}}\frac{ \vert F(x,u_{n},v_{n}) \vert }{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+ \Vert v_{n} \Vert _{1,\Phi_{2}}^{l_{2}}}\,dx + \int_{Y_{n}}\frac{ \vert F(x,u_{n},v_{n}) \vert }{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+ \Vert v_{n} \Vert _{1,\Phi_{2}}^{l_{2}}}\,dx. \end{aligned}$$

(3.30)

By (3.28) and (3.24), we have

$$\begin{aligned} \int_{X_{n}}\frac{ \vert F(x,u_{n},v_{n}) \vert }{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+ \Vert v_{n} \Vert _{1,\Phi_{2}}^{l_{2}}}\,dx ={}& \int_{X_{n}}\frac{ \vert F(x,u_{n},v_{n}) \vert }{\frac{ \vert u_{n} \vert ^{l_{1}}}{ \vert \tilde {u}_{n} \vert ^{l_{1}}}+\frac{ \vert v_{n} \vert ^{l_{2}}}{ \vert \tilde{v}_{n} \vert ^{l_{2}}}}\,dx \\ \leq{}& \int_{X_{n}}\frac{ \vert F(x,u_{n},v_{n}) \vert }{ \vert u_{n} \vert ^{l_{1}}+ \vert v_{n} \vert ^{l_{2}}}\bigl( \vert \tilde {u}_{n} \vert ^{l_{1}}+ \vert \tilde{v}_{n} \vert ^{l_{2}}\bigr)\,dx \leq\varepsilon M_{1}. \end{aligned}$$

(3.31)

Since $k>\max\{\frac{N}{l_{1}},\frac{N}{l_{2}}\}$, then $\frac{l_{i}k}{k-1}\in (l_{i},l_{i}^{*})(i=1,2)$. Hence, by (3.29), (3.18), (3.26) and the fact $\overline{F}(x,u,v)\geq0$, for n large, we have

$$\begin{aligned} &\int_{Y_{n}}\frac{ \vert F(x,u_{n},v_{n}) \vert }{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+ \Vert v_{n} \Vert _{1,\Phi_{2}}^{l_{2}}}\,dx \\ &\quad\leq \int_{Y_{n}}\frac{ \vert F(x,u_{n},v_{n}) \vert }{ \vert u_{n} \vert ^{l_{1}}+ \vert v_{n} \vert ^{l_{2}}}\bigl( \vert \tilde {u}_{n} \vert ^{l_{1}}+ \vert \tilde{v}_{n} \vert ^{l_{2}}\bigr)\,dx \\ &\quad\leq \biggl( \int_{Y_{n}} \biggl(\frac { \vert F(x,u_{n},v_{n}) \vert }{ \vert u_{n} \vert ^{l_{1}}+ \vert v_{n} \vert ^{l_{2}}} \biggr)^{k}\,dx \biggr)^{\frac{1}{k}} \biggl( \int_{Y_{n}} \bigl( \vert \tilde{u}_{n} \vert ^{l_{1}}+ \vert \tilde{v}_{n} \vert ^{l_{2}} \bigr)^{\frac{k}{k-1}}\,dx \biggr)^{\frac{k-1}{k}} \\ &\quad\leq \biggl( \int_{Y_{n}}C_{R}\overline{F}(x,u_{n},v_{n})\,dx \biggr)^{\frac{1}{k}} \biggl( \int_{\mathbb{R}^{N}}C_{\frac{k}{k-1}} \bigl( \vert \tilde{u}_{n} \vert ^{\frac {l_{1}k}{k-1}}+ \vert \tilde{v}_{n} \vert ^{\frac{l_{2}k}{k-1}} \bigr)\,dx \biggr)^{\frac {k-1}{k}} \\ &\quad\leq\bigl[C_{R}(c+1)\bigr]^{\frac{1}{k}} \bigl[C_{\frac{k}{k-1}} \bigl( \Vert \tilde{u}_{n} \Vert _{\frac{l_{1}k}{k-1}}^{\frac{l_{1}k}{k-1}}+ \Vert \tilde {v}_{n} \Vert _{\frac{l_{2}k}{k-1}}^{\frac{l_{2}k}{k-1}} \bigr) \bigr]^{\frac {k-1}{k}} = o_{n}(1). \end{aligned}$$

(3.32)

Since ε is arbitrary, it follows from (3.30)-(3.32) that

$$\begin{aligned} \frac{I_{2}(u_{n},v_{n})}{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+ \Vert v_{n} \Vert _{1,\Phi _{2}}^{l_{2}}}\rightarrow0 \quad\mbox{as } n\rightarrow\infty. \end{aligned}$$

(3.33)

By dividing (3.17) by $\Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+ \Vert v_{n} \Vert _{1,\Phi _{2}}^{l_{2}}$ and letting $n\rightarrow\infty$, we get a contradiction via (3.27) and (3.33).

Case 2. Suppose that $\Vert u_{n} \Vert _{1,\Phi_{1}}\rightarrow\infty$ and $\Vert v_{n} \Vert _{1,\Phi_{2}}\leq M_{2}$ for some constant $M_{2}>0$. Let $\tilde {u}_{n}=\frac{u_{n}}{ \Vert u_{n} \Vert _{1,\Phi_{1}}}$ and $\tilde{v}_{n}=\frac{v_{n}}{ \Vert u_{n} \Vert _{1,\Phi_{1}}}$. Then $\{{\tilde{u}_{n}}\}$ is bounded in $W^{1,\Phi _{1}}(\mathbb{R}^{N})$ and ${\tilde{v}_{n}}\rightarrow{\mathbf{0} }$ in $W^{1,\Phi _{2}}(\mathbb{R}^{N})$. We claim that

$$\begin{aligned} \lambda_{2}:=\lim_{n\rightarrow\infty}\sup_{y\in\mathbb{R}^{N}} \int _{B_{2}(y)}\bigl(\Phi_{1}\bigl( \vert \tilde{u}_{n} \vert \bigr)+\Phi_{2}\bigl( \vert \tilde{v}_{n} \vert \bigr)\bigr)\,dx=0. \end{aligned}$$

Indeed, if $\lambda_{2}\neq0$, there exist a constant $\delta>0$, a subsequence of $\{(\tilde{u}_{n},\tilde{v}_{n})\}$, still denoted by $\{ (\tilde{u}_{n},\tilde{v}_{n})\}$, and a sequence $\{z_{n}\}\in\mathbb{Z}^{N}$ such that

$$\begin{aligned} \int_{B_{2}(z_{n})}\bigl(\Phi_{1}\bigl( \vert \tilde{u}_{n} \vert \bigr)+\Phi_{2}\bigl( \vert \tilde{v}_{n} \vert \bigr)\bigr)\,dx>\delta \quad\mbox{for all } n\in \mathbb{N}. \end{aligned}$$

(3.34)

Let $\bar{u}_{n}=\tilde{u}_{n}(\cdot+z_{n})$ and $\bar{v}_{n}=\tilde {v}_{n}(\cdot+z_{n})$. Then $\Vert \bar{u}_{n} \Vert _{1,\Phi_{1}}= \Vert \tilde{u}_{n} \Vert _{1,\Phi_{1}}$ and $\Vert \bar{v}_{n} \Vert _{1,\Phi_{2}}= \Vert \tilde{v}_{n} \Vert _{1,\Phi_{2}}$, that is, $\{{\bar{u}_{n}}\}$ is bounded in $W^{1,\Phi_{1}}(\mathbb{R}^{N})$ and ${\bar{v}_{n}}\rightarrow\mathbf{0}$ in $W^{1,\Phi_{2}}(\mathbb {R}^{N})$. Passing to a subsequence of $\{({\bar{u}_{n}},{\bar{v}_{n}})\}$, still denoted by $\{({\bar{u}_{n}},{\bar{v}_{n}})\}$, by Remark 2.7, there exists $(\bar{u},\mathbf{0})\in W$ such that

⋆:

$\bar{u}_{n}\rightharpoonup\bar{u}$ in $W^{1,\Phi_{1}}(\mathbb{R}^{N})$, $\bar{u}_{n}\rightarrow\bar{u}$ in $L^{\Phi_{1}}(B_{2})$ and $\bar{u}_{n}(x)\rightarrow\bar{u}(x)$ a.e. in $B_{2}$;

⋆:

$\bar{v}_{n}\rightarrow\mathbf{0}$ in $W^{1,\Phi_{2}}(\mathbb{R}^{N})$, $\bar{v}_{n}\rightarrow\mathbf{0}$ in $L^{\Phi_{2}}(B_{2})$ and $\bar{v}_{n}(x)\rightarrow\mathbf{0}$ a.e. in $B_{2}$.

Since

$$\begin{aligned} \int_{B_{2}}\bigl(\Phi_{1}\bigl( \vert \bar{u}_{n} \vert \bigr)+\Phi_{2}\bigl( \vert \bar{v}_{n} \vert \bigr)\bigr)\,dx= \int _{B_{2}(z_{n})}\bigl(\Phi_{1}\bigl( \vert \tilde{u}_{n} \vert \bigr)+\Phi_{2}\bigl( \vert \tilde{v}_{n} \vert \bigr)\bigr)\,dx, \end{aligned}$$

then, by (3.34), ⋆ and (2.3), we obtain that $\bar {u}\neq\mathbf{0}$ in $L^{\Phi_{1}}(B_{2})$, that is, $[\bar{u}\neq0]:=\{x \in B_{2}: \bar{u}(x)\neq0\}$ has nonzero Lebesgue measure. Let $u_{n}^{*}=u_{n}(\cdot+z_{n})$ and $v_{n}^{*}=v_{n}(\cdot+z_{n})$. Then $\Vert (u_{n}^{*},v_{n}^{*}) \Vert = \Vert (u_{n},v_{n}) \Vert$ and

$$\begin{aligned} \bigl\vert u_{n}^{*}(x) \bigr\vert = \bigl\vert \bar{u}_{n}(x) \bigr\vert \Vert u_{n} \Vert _{1,\Phi_{1}}\rightarrow\infty, \quad\mbox{a.e. } x\in[\bar{u}\neq0]. \end{aligned}$$

(3.35)

Since $V_{i}(i=1,2)$ and F are 1-periodic in x, $\{(u_{n}^{*},v_{n}^{*})\}$ is also a $(C)_{c}$-sequence of I. Then, by (3.18), for n large, we have

$$\begin{aligned} \int_{\mathbb{R}^{N}}\overline{F}\bigl(x,u_{n}^{*},v_{n}^{*} \bigr)\,dx\leq c+1. \end{aligned}$$

(3.36)

However, it follows from $(F_{5})$, (3.35), (3.21) and Fatou’s lemma that

$$\begin{aligned} \int_{\mathbb{R}^{N}}\overline{F}\bigl(x,u_{n}^{*},v_{n}^{*} \bigr)\,dx\geq \int_{[\bar {u}\neq0]}\overline{F}\bigl(x,u_{n}^{*},v_{n}^{*} \bigr)\,dx=+\infty, \end{aligned}$$

which contradicts (3.36). Therefore, $\lambda_{2}=0$ and

$$\begin{aligned} \lim_{n\rightarrow\infty}\sup_{y\in\mathbb{R}^{N}} \int_{B_{2}(y)}\Phi _{1}\bigl( \vert \tilde{u}_{n} \vert \bigr)\,dx=0. \end{aligned}$$

(3.37)

Then, by the Lions type result for Orlicz-Sobolev spaces (see Theorem 1.3 in [8]) again, (3.37), (3.25) and the fact $\frac{l_{1}k}{k-1}\in(l_{1},l_{1}^{*})$ imply that

$$\begin{aligned} \tilde{u}_{n}\rightarrow\mathbf{0} \quad\mbox{in } L^{\frac {l_{1}k}{k-1}}\bigl(\mathbb{R}^{N}\bigr). \end{aligned}$$

(3.38)

Since the embeddings $W^{1, \Phi_{i}}(\mathbb{R}^{N})\hookrightarrow L^{l_{i}}(\mathbb{R}^{N})(i=1,2)$ are continuous, there exists a constant $M_{3}>0$ such that

$$\begin{aligned} \Vert \tilde{u}_{n} \Vert _{l_{1}}^{l_{1}}+ \Vert v_{n} \Vert _{l_{2}}^{l_{2}}\leq M_{3} \quad\mbox{for all } n\in\mathbb{N}. \end{aligned}$$

(3.39)

Moreover, $\frac{l_{2}k}{k-1}\in(l_{2},l_{2}^{*})$, (3.25) and Lemma 2.5 imply that the embedding $W^{1, \Phi_{2}}(\mathbb{R}^{N})\hookrightarrow L^{\frac{l_{2}k}{k-1}}(\mathbb{R}^{N})$ is continuous. Hence, there exists a constant $M_{4}>0$ such that

$$\begin{aligned} \Vert v_{n} \Vert _{\frac{l_{2}k}{k-1}}^{l_{2}} \leq M_{4} \quad\mbox{for all } n\in\mathbb{N}. \end{aligned}$$

(3.40)

So, for any given constant $M>1$, by (3.6), $(V_{2})$ and Lemma 2.2, we have

$$\begin{aligned} \frac{I_{1}(u_{n},v_{n})}{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+M} \geq{}& \frac{\min\{ \Vert \nabla u_{n} \Vert _{\Phi_{1}}^{l_{1}}, \Vert \nabla u_{n} \Vert _{\Phi _{1}}^{m_{1}}\}+\alpha_{1}\min\{ \Vert u_{n} \Vert _{\Phi_{1}}^{l_{1}}, \Vert u_{n} \Vert _{\Phi _{1}}^{m_{1}}\}}{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+M} \\ \geq{}& \frac{ \Vert \nabla u_{n} \Vert _{\Phi_{1}}^{l_{1}}+\alpha_{1} \Vert u_{n} \Vert _{\Phi _{1}}^{l_{1}}-1-\alpha_{1}}{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+M} \\ \geq{}& \frac{\min\{1,\alpha_{1}\}C_{l_{1}} \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}-1-\alpha_{1}}{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+M} \\ = {}& \min\{1,\alpha_{1}\}C_{l_{1}}+o_{n}(1). \end{aligned}$$

(3.41)

It is obvious that (3.28) and (3.29) still hold for this case. Based on this fact, let

$$\begin{aligned} &X_{n}=\bigl\{ x\in\mathbb{R}^{N}: \bigl\vert \bigl(u_{n}(x),v_{n}(x)\bigr) \bigr\vert \leq R_{\varepsilon}\bigr\} \quad\mbox{and}\\ & Y_{n}=\bigl\{ x\in \mathbb{R}^{N}: \bigl\vert \bigl(u_{n}(x),v_{n}(x) \bigr) \bigr\vert > R_{\varepsilon}\bigr\} . \end{aligned}$$

Then

$$\begin{aligned} \frac{ \vert I_{2}(u_{n},v_{n}) \vert }{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+M} \leq \int_{X_{n}}\frac{ \vert F(x,u_{n},v_{n}) \vert }{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+M}\,dx + \int_{Y_{n}}\frac{ \vert F(x,u_{n},v_{n}) \vert }{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+M}\,dx. \end{aligned}$$

(3.42)

By (3.28) and (3.39), we have

$$\begin{aligned} \int_{X_{n}}\frac{ \vert F(x,u_{n},v_{n}) \vert }{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+M}\,dx &\leq \int_{X_{n}}\frac{ \vert F(x,u_{n},v_{n}) \vert }{ \vert u_{n} \vert ^{l_{1}}+ \vert v_{n} \vert ^{l_{2}}}\biggl( \vert \tilde {u}_{n} \vert ^{l_{1}}+\frac{1}{M} \vert v_{n} \vert ^{l_{2}}\biggr)\,dx \\ &\leq\varepsilon \biggl( \Vert \tilde{u}_{n} \Vert _{l_{1}}^{l_{1}}+ \frac{1}{M} \Vert v_{n} \Vert _{l_{2}}^{l_{2}} \biggr) \leq\varepsilon M_{3}. \end{aligned}$$

(3.43)

Note that $\frac{l_{i}k}{k-1}\in(l_{i},l_{i}^{*})(i=1,2)$. By (3.29), (3.18), (3.38), (3.40) and the fact $\overline {F}(x,u,v)\geq0$, for n large, we have

$$\begin{aligned} &\int_{Y_{n}}\frac{ \vert F(x,u_{n},v_{n}) \vert }{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+M}\,dx \\ &\quad\leq \int_{Y_{n}}\frac{ \vert F(x,u_{n},v_{n}) \vert }{ \vert u_{n} \vert ^{l_{1}}+ \vert v_{n} \vert ^{l_{2}}}\biggl( \vert \tilde {u}_{n} \vert ^{l_{1}}+\frac{1}{M} \vert v_{n} \vert ^{l_{2}}\biggr)\,dx \\ &\quad \leq \biggl( \int_{Y_{n}} \biggl(\frac { \vert F(x,u_{n},v_{n}) \vert }{ \vert u_{n} \vert ^{l_{1}}+ \vert v_{n} \vert ^{l_{2}}} \biggr)^{k}\,dx \biggr)^{\frac{1}{k}} \biggl( \int_{Y_{n}} \biggl( \vert \tilde{u}_{n} \vert ^{l_{1}}+\frac{1}{M} \vert v_{n} \vert ^{l_{2}} \biggr)^{\frac{k}{k-1}}\,dx \biggr)^{\frac{k-1}{k}} \\ &\quad\leq \biggl( \int_{Y_{n}}C_{R}\overline{F}(x,u_{n},v_{n})\,dx \biggr)^{\frac{1}{k}} \biggl[C_{\frac{k}{k-1}} \biggl( \Vert \tilde{u}_{n} \Vert _{\frac{l_{1}k}{k-1}}^{\frac{l_{1}k}{k-1}}+ \biggl( \frac {1}{M} \biggr)^{\frac{k}{k-1}} \Vert v_{n} \Vert _{\frac{l_{2}k}{k-1}}^{\frac {l_{2}k}{k-1}} \biggr) \biggr]^{\frac{k-1}{k}} \\ &\quad \leq \bigl[C_{R}(c+1)\bigr]^{\frac{1}{k}}C_{\frac{k}{k-1}}^{\frac {k-1}{k}}C_{\frac{k-1}{k}} \biggl( \Vert \tilde{u}_{n} \Vert _{\frac{l_{1}k}{k-1}}^{l_{1}}+ \frac{1}{M} \Vert v_{n} \Vert _{\frac{l_{2}k}{k-1}}^{l_{2}} \biggr) \\ &\quad\leq \bigl[C_{R}(c+1)\bigr]^{\frac{1}{k}}C_{\frac{k}{k-1}}^{\frac {k-1}{k}}C_{\frac{k-1}{k}} \biggl(o_{n}(1)+\frac{M_{4}}{M} \biggr). \end{aligned}$$

(3.44)

Since $\varepsilon>0$ and $M>1$ are arbitrary, it follows from (3.42)-(3.44) that

$$\begin{aligned} \frac{I_{2}(u_{n},v_{n})}{ \Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+M}\rightarrow0 \quad\mbox{as } n\rightarrow\infty. \end{aligned}$$

(3.45)

By dividing (3.17) by $\Vert u_{n} \Vert _{1,\Phi_{1}}^{l_{1}}+M$ and letting $n\rightarrow\infty$, we get a contradiction via (3.41) and (3.45).

Case 3. Suppose that $\Vert \nabla u_{n} \Vert _{\Phi_{1}}\leq M_{5}$ for some constant $M_{5}>0$ and $\Vert \nabla v_{n} \Vert _{\Phi_{2}}\rightarrow\infty$. For this case, with the same discussion as Case 2, we can also get a contradiction. □

Lemma 3.18

System (1.1) has a nontrivial solution under the assumptions of Theorems 3.1 and 3.2, respectively.

Proof

For the level $c>0$ given in (3.13), there exists a $(C)_{c}$-sequence $\{(u_{n},v_{n})\}$ for I in W. Moreover, Lemmas 3.16 and 3.17 show that the sequence $\{(u_{n},v_{n})\}$ is bounded in W. We claim that

$$\begin{aligned} \lambda_{3}:=\lim_{n\rightarrow\infty}\sup_{y\in\mathbb{R}^{N}} \int _{B_{2}(y)}\bigl(\Phi_{1}\bigl( \vert u_{n} \vert \bigr)+\Phi_{2}\bigl( \vert v_{n} \vert \bigr)\bigr)\,dx>0. \end{aligned}$$

Indeed, if $\lambda_{3}=0$, then

$$\begin{aligned} \lim_{n\rightarrow\infty}\sup_{y\in\mathbb{R}^{N}} \int_{B_{2}(y)}\Phi_{1}\bigl( \vert u_{n} \vert \bigr)\,dx =\lim_{n\rightarrow\infty}\sup_{y\in\mathbb{R}^{N}} \int_{B_{2}(y)}\Phi _{2}\bigl( \vert v_{n} \vert \bigr)\,dx=0. \end{aligned}$$

By using the Lions type result for Orlicz-Sobolev spaces (see Theorem 1.3 in [8]) again, we have

$$\begin{aligned} \begin{aligned}& u_{n}\rightarrow\mathbf{0} \quad\mbox{in } L^{q_{1}}\bigl(\mathbb{R}^{N}\bigr) \quad\mbox{and}\\ &v_{n}\rightarrow\mathbf{0} \quad\mbox{in } L^{q_{2}}\bigl( \mathbb{R}^{N}\bigr), \mbox{ for all } q_{1}\in \bigl(m_{1},l_{1}^{*}\bigr), q_{2}\in \bigl(m_{2},l_{2}^{*}\bigr). \end{aligned} \end{aligned}$$

(3.46)

Given $q_{i}\in(m_{i},l_{i}^{*})(i=1,2)$, by $(F_{1})$, $(F_{2})$, $(\phi_{1})$, $(\phi_{2})$ and (2.1), for any given constant $\varepsilon>0$, there exists a constant $C_{\varepsilon}>0$ such that

$$\begin{aligned} &\bigl\vert F(x,u_{n},v_{n}) \bigr\vert \leq\varepsilon \bigl(\Phi_{1}\bigl( \vert u_{n} \vert \bigr)+\Phi_{2}\bigl( \vert v_{n} \vert \bigr)+ \Phi _{1*}\bigl( \vert u_{n} \vert \bigr)+ \Phi_{2*}\bigl( \vert v_{n} \vert \bigr) \bigr)+C_{\varepsilon}\bigl( \vert u_{n} \vert ^{q_{1}}+ \vert v_{n} \vert ^{q_{2}}\bigr), \\ &\bigl\vert u_{n}F_{u}(x,u_{n},v_{n}) \bigr\vert \leq\varepsilon \bigl(\Phi_{1}\bigl( \vert u_{n} \vert \bigr)+\Phi _{2}\bigl( \vert v_{n} \vert \bigr)+\Phi_{1*}\bigl( \vert u_{n} \vert \bigr)+\Phi_{2*}\bigl( \vert v_{n} \vert \bigr) \bigr) \\ &\phantom{\bigl\vert u_{n}F_{u}(x,u_{n},v_{n}) \bigr\vert \leq}{}+C_{\varepsilon}\bigl( \vert u_{n} \vert ^{q_{1}}+ \vert v_{n} \vert ^{q_{2}}\bigr), \\ &\bigl\vert v_{n}F_{v}(x,u_{n},v_{n}) \bigr\vert \leq\varepsilon \bigl(\Phi_{1}\bigl( \vert u_{n} \vert \bigr)+\Phi _{2}\bigl( \vert v_{n} \vert \bigr)+\Phi_{1*}\bigl( \vert u_{n} \vert \bigr)+\Phi_{2*}\bigl( \vert v_{n} \vert \bigr) \bigr) \\ &\phantom{\bigl\vert v_{n}F_{v}(x,u_{n},v_{n}) \bigr\vert \leq}{}+C_{\varepsilon}\bigl( \vert u_{n} \vert ^{q_{1}}+ \vert v_{n} \vert ^{q_{2}}\bigr), \end{aligned}$$

(3.47)

for all $x\in\mathbb{R}^{N}$. Then it follows from Lemma 2.2, (2) in Lemma 2.4, (2.8), (3.46) and the arbitrariness of $\varepsilon >0$ that

$$\begin{aligned} \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}}F(x,u_{n},v_{n})\,dx={}&\lim _{n\rightarrow\infty} \int_{\mathbb{R}^{N}}u_{n}F_{u}(x,u_{n},v_{n})\,dx \\ ={}&\lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}}v_{n}F_{v}(x,u_{n},v_{n})\,dx=0. \end{aligned}$$

(3.48)

Hence, by (3.4), (3.5), (3.14), $(\phi_{2})$, $(V_{2})$ and (3.48), we have

$$\begin{aligned} c = {}& \lim_{n\rightarrow\infty} \biggl\{ I(u_{n},v_{n})- \biggl\langle I'(u_{n},v_{n}), \biggl( \frac{1}{l_{1}}u_{n},\frac{1}{l_{2}}v_{n} \biggr) \biggr\rangle \biggr\} \\ = {}& \lim_{n\rightarrow\infty} \biggl\{ \int_{\mathbb{R}^{N}}\Phi _{1}\bigl( \vert \nabla u_{n} \vert \bigr)\,dx- \int_{\mathbb{R}^{N}}\frac{1}{l_{1}}a_{1}\bigl( \vert \nabla u_{n} \vert \bigr) \vert \nabla u_{n} \vert ^{2}\,dx \\ & {} + \int_{\mathbb{R}^{N}}V_{1}(x)\Phi_{1}\bigl( \vert u_{n} \vert \bigr)\,dx- \int_{\mathbb{R}^{N}}\frac {1}{l_{1}}V_{1}(x)a_{1} \bigl( \vert u_{n} \vert \bigr) \vert u_{n} \vert ^{2}\,dx \\ & {}+ \int_{\mathbb{R}^{N}}\Phi_{2}\bigl( \vert \nabla v_{n} \vert \bigr)\,dx- \int_{\mathbb{R}^{N}}\frac {1}{l_{2}}a_{2}\bigl( \vert \nabla v_{n} \vert \bigr) \vert \nabla v_{n} \vert ^{2}\,dx \\ & {} + \int_{\mathbb{R}^{N}}V_{2}(x)\Phi_{2}\bigl( \vert v_{n} \vert \bigr)\,dx- \int_{\mathbb{R}^{N}}\frac {1}{l_{2}}V_{2}(x)a_{2} \bigl( \vert v_{n} \vert \bigr) \vert v_{n} \vert ^{2}\,dx \\ & {}+ \int_{\mathbb{R}^{N}} \biggl(\frac {1}{l_{1}}u_{n}F_{u}(x,u_{n},v_{n})+ \frac {1}{l_{2}}v_{n}F_{v}(x,u_{n},v_{n})-F(x,u_{n},v_{n}) \biggr)\,dx \biggr\} \\ \leq{}& \lim_{n\rightarrow\infty} \biggl\{ \int_{\mathbb{R}^{N}} \biggl(\frac {1}{l_{1}}u_{n}F_{u}(x,u_{n},v_{n})+ \frac {1}{l_{2}}v_{n}F_{v}(x,u_{n},v_{n})-F(x,u_{n},v_{n}) \biggr)\,dx \biggr\} =0, \end{aligned}$$

which contradicts $c>0$. Therefore, $\lambda_{3}> 0$, which implies that there exist a constant $\delta>0$, a subsequence of $\{(u_{n},v_{n})\}$, still denoted by $\{(u_{n},v_{n})\}$, and a sequence $\{z_{n}\}\in\mathbb {Z}^{N}$ such that

$$\begin{aligned} \begin{aligned} &\int_{B_{2}(z_{n})}\bigl(\Phi_{1}\bigl( \vert u_{n} \vert \bigr)+\Phi_{2}\bigl( \vert v_{n} \vert \bigr)\bigr)\,dx= \int_{B_{2}}\bigl(\Phi _{1}\bigl( \bigl\vert u_{n}^{*} \bigr\vert \bigr)+\Phi_{2}\bigl( \bigl\vert v_{n}^{*} \bigr\vert \bigr)\bigr)\,dx>\delta\\ &\quad \mbox{for all } n\in \mathbb{N}, \end{aligned} \end{aligned}$$

(3.49)

where $u_{n}^{*}:=u_{n}(\cdot+z_{n})$ and $v_{n}^{*}:=v_{n}(\cdot+z_{n})$. Since $\Vert u_{n}^{*} \Vert _{1,\Phi_{1}}= \Vert u_{n} \Vert _{1,\Phi_{1}}$ and $\Vert v_{n}^{*} \Vert _{1,\Phi_{2}}= \Vert v_{n} \Vert _{1,\Phi_{2}}$, then $\{{u_{n}^{*}}\}$ and $\{{v_{n}^{*}}\}$ are bounded in $W^{1,\Phi_{1}}(\mathbb{R}^{N})$ and $W^{1,\Phi_{2}}(\mathbb{R}^{N})$, respectively. Passing to a subsequence of $\{({u_{n}^{*}},{v_{n}^{*}})\}$, still denoted by $\{({u_{n}^{*}},{v_{n}^{*}})\}$, there exists $(u^{*},v^{*})\in W$ such that $u_{n}^{*}\rightharpoonup u^{*}$ in $W^{1,\Phi_{1}}(\mathbb{R}^{N})$ and $v_{n}^{*}\rightharpoonup v^{*}$ in $W^{1,\Phi_{2}}(\mathbb{R}^{N})$, respectively. Moreover, for any given constant $r>0$, by Remark 2.7 and the similar arguments as those in Lemma 4.3 in [8], we can assume that

⋆:

$u_{n}^{*}\rightarrow u^{*}$ in $L^{\Phi_{1}}(B_{r})$ and $u_{n}^{*}(x)\rightarrow u^{*}(x), \nabla u_{n}^{*}(x)\rightarrow\nabla u^{*}(x)$ a.e. in $B_{r}$;

⋆:

$v_{n}^{*}\rightarrow v^{*}$ in $L^{\Phi_{2}}(B_{r})$ and $v_{n}^{*}(x)\rightarrow v^{*}(x), \nabla v_{n}^{*}(x)\rightarrow\nabla v^{*}(x)$ a.e. in $B_{r}$.

Then, by (3.49), ⋆ and (2.3), we obtain that $(u^{*},v^{*})\neq(\mathbf{0,0})$. Since $V_{i}(i=1,2)$ and F are 1-periodic in x, $\{(u_{n}^{*},v_{n}^{*})\}$ is also a $(C)_{c}$-sequence of I. Then, for any given point $(w_{1},w_{2})\in C_{0}^{\infty}(\mathbb {R}^{N})\times C_{0}^{\infty}(\mathbb{R}^{N})$ with $\operatorname{supp}\{w_{1}\}\cup \operatorname{supp}\{w_{2}\}\subset B_{r}$ for some $r>0$, we have

$$\lim_{n\rightarrow\infty}\bigl\langle I'\bigl(u_{n}^{*},v_{n}^{*} \bigr),(w_{1},w_{2})\bigr\rangle =0. $$

We claim that

$$\begin{aligned} \lim_{n\rightarrow\infty}\bigl\langle I' \bigl(u_{n}^{*},v_{n}^{*}\bigr),(w_{1},w_{2}) \bigr\rangle =\bigl\langle I'\bigl(u^{*},v^{*}\bigr),(w_{1},w_{2}) \bigr\rangle . \end{aligned}$$

(3.50)

First, we claim

$$\begin{aligned} \lim_{n\rightarrow\infty} \int_{\mathbb {R}^{N}}V_{1}(x)a_{1}\bigl( \bigl\vert u_{n}^{*} \bigr\vert \bigr)u_{n}^{*}w_{1}\,dx= \int_{\mathbb{R}^{N}}V_{1}(x)a_{1}\bigl( \bigl\vert u^{*} \bigr\vert \bigr)u^{*}w_{1}\,dx. \end{aligned}$$

(3.51)

Indeed, it follows from $(\phi_{1})$, $(\phi_{2})$, $(V_{2})$, ⋆ and the boundedness of sequence $\{u_{n}^{*}\}$ in $W^{1,\Phi_{1}}(\mathbb{R}^{N})$ that the sequence $\{V_{1}(x)a_{1}( \vert u_{n}^{*} \vert )u_{n}^{*}\}$ is bounded in $L^{\widetilde{\Phi}_{1}}(B_{r})$ and $V_{1}(x)\times a_{1}( \vert u_{n}^{*}(x) \vert )u_{n}^{*}(x)\rightarrow V_{1}(x)a_{1}( \vert u^{*}(x) \vert )u^{*}(x)$ a.e. $x\in B_{r}$. Then, by applying Lemma 2.1 in [8], we get (3.51) because $w_{1}\in L^{\Phi_{1}}(B_{r})$. Similarly, we can get

$$\begin{aligned} \lim_{n\rightarrow\infty} \int_{\mathbb {R}^{N}}V_{2}(x)a_{2}\bigl( \bigl\vert v_{n}^{*} \bigr\vert \bigr)v_{n}^{*}w_{2}\,dx= \int_{\mathbb{R}^{N}}V_{2}(x)a_{2}\bigl( \bigl\vert v^{*} \bigr\vert \bigr)v^{*}w_{2}\,dx. \end{aligned}$$

(3.52)

Next, we claim

$$\begin{aligned} \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} F_{u}\bigl(x,u_{n}^{*},v_{n}^{*} \bigr)w_{1}\,dx= \int_{\mathbb{R}^{N}}F_{u}\bigl(x,u^{*},v^{*}\bigr)w_{1}\,dx. \end{aligned}$$

(3.53)

Indeed, it follows from $(\phi_{1})$, $(\phi_{2})$, $(F_{1})$, $(F_{2})$, ⋆, the boundedness of sequence $\{(u_{n}^{*},v_{n}^{*})\}$ in W and Remark 2.7 that the sequence $\{F_{u}(x,u_{n}^{*},v_{n}^{*})\}$ is bounded in $L^{\widetilde{\Phi}_{1*}}(B_{r})$ and $F_{u}(x,u_{n}^{*}(x),v_{n}^{*}(x))\rightarrow F_{u}(x,u^{*}(x),v^{*}(x))$ a.e. $x\in B_{r}$. Then, by applying Lemma 2.1 in [8] again, we get (3.53) because $w_{1}\in L^{\Phi_{1*}}(B_{r})$. Similarly, we can get

$$\begin{aligned} \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} F_{v}\bigl(x,u_{n}^{*},v_{n}^{*} \bigr)w_{2}\,dx= \int_{\mathbb{R}^{N}}F_{v}\bigl(x,u^{*},v^{*}\bigr)w_{2}\,dx. \end{aligned}$$

(3.54)

Finally, we claim

$$\begin{aligned} \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}}a_{1}\bigl( \bigl\vert \nabla u_{n}^{*} \bigr\vert \bigr)\nabla u_{n}^{*}\nabla w_{1}\,dx= \int_{\mathbb{R}^{N}}a_{1}\bigl( \bigl\vert \nabla u^{*} \bigr\vert \bigr)\nabla u^{*}\nabla w_{1}\,dx \end{aligned}$$

(3.55)

and

$$\begin{aligned} \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}}a_{2}\bigl( \bigl\vert \nabla v_{n}^{*} \bigr\vert \bigr)\nabla v_{n}^{*}\nabla w_{2}\,dx= \int_{\mathbb{R}^{N}}a_{2}\bigl( \bigl\vert \nabla v^{*} \bigr\vert \bigr)\nabla v^{*}\nabla w_{2}\,dx. \end{aligned}$$

(3.56)

In fact, the boundedness of sequence $\{(u_{n}^{*},v_{n}^{*})\}$ implies that sequences $\{a_{1}( \vert \nabla u_{n}^{*} \vert )\frac{\partial u_{n}^{*}}{\partial x_{j}} \}\ (j=1,2,\ldots,N)$ are bounded in $L^{\widetilde{\Phi }_{1}}(B_{r})$. Moreover, $(\phi_{1})$ and ⋆ imply that $a_{1}( \vert \nabla u_{n}^{*}(x) \vert )\frac{\partial u_{n}^{*}(x)}{\partial x_{j}}\rightarrow a_{1}( \vert \nabla u^{*}(x) \vert )\frac{\partial u^{*}(x)}{\partial x_{j}}(j=1,2,\ldots,N)$ a.e. $x\in B_{r}$. Then, by applying Lemma 2.1 in [8] again, we get (3.55) because $\frac{\partial w_{1}}{\partial x_{j}}\in L^{\Phi _{1}}(B_{r})(j=1,2,\ldots,N)$. Similarly, we can get (3.56). Hence, it follows from (3.51)-(3.56) that (3.50) holds, that is, $\langle I'(u^{*},v^{*}),(w_{1},w_{2})\rangle=0$ for all $(w_{1},w_{2})\in C_{0}^{\infty}(\mathbb{R}^{N})\times C_{0}^{\infty}(\mathbb{R}^{N})$. Now, we can conclude that $I'(u^{*},v^{*})=0$ because $C_{0}^{\infty}(\mathbb {R}^{N})\times C_{0}^{\infty}(\mathbb{R}^{N})$ is dense in W. □

Proof of Theorem 3.1

Lemma 3.18 shows that system (1.1) has at least a nontrivial solution. Next, we prove that system (1.1) has a ground state. Let

$$d=\inf\bigl\{ I(u,v): (u,v)\neq(\mathbf{0,0}) \mbox{ and } I'(u,v)=0 \bigr\} . $$

First, we claim that $d\geq0$. Indeed, for any given nontrivial critical point $(u,v)$ of I, by (3.4), (3.5), $(\phi _{2})$, $(V_{2})$ and $(F_{3})$, we have

$$\begin{aligned} I(u,v)={}& I(u,v)- \biggl\langle I'(u,v), \biggl( \frac{1}{\mu_{1}}u,\frac {1}{\mu_{2}}v \biggr) \biggr\rangle \\ ={} & \int_{\mathbb{R}^{N}} \biggl(\Phi_{1}\bigl( \vert \nabla u \vert \bigr)-\frac{1}{\mu _{1}}a_{1}\bigl( \vert \nabla u \vert \bigr) \vert \nabla u \vert ^{2} \biggr)\,dx \\ &{}+ \int_{\mathbb{R}^{N}}V_{1}(x) \biggl(\Phi_{1}\bigl( \vert u \vert \bigr)-\frac{1}{\mu _{1}}a_{1}\bigl( \vert u \vert \bigr) \vert u \vert ^{2} \biggr)\,dx \\ & {}+ \int_{\mathbb{R}^{N}} \biggl(\Phi_{2}\bigl( \vert \nabla v \vert \bigr)-\frac{1}{\mu _{2}}a_{2}\bigl( \vert \nabla v \vert \bigr) \vert \nabla v \vert ^{2} \biggr)\,dx\\ &{} + \int_{\mathbb{R}^{N}}V_{2}(x) \biggl(\Phi_{2}\bigl( \vert v \vert \bigr)-\frac{1}{\mu _{2}}a_{2}\bigl( \vert v \vert \bigr) \vert v \vert ^{2} \biggr)\,dx \\ & {} + \int_{\mathbb{R}^{N}} \biggl(\frac{1}{\mu_{1}}uF_{u}(x,u,v)+ \frac{1}{\mu _{2}}vF_{v}(x,u,v)-F(x,u,v) \biggr)\,dx \\ \geq{}& \biggl(1-\frac{m_{1}}{\mu_{1}} \biggr) \int_{\mathbb{R}^{N}} \bigl(\Phi _{1}\bigl( \vert \nabla u \vert \bigr)+\alpha_{1}\Phi_{1}\bigl( \vert u \vert \bigr) \bigr)\,dx \\ &{}+ \biggl(1-\frac{m_{2}}{\mu_{2}} \biggr) \int_{\mathbb{R}^{N}} \bigl(\Phi _{2}\bigl( \vert \nabla v \vert \bigr)+\alpha_{1}\Phi_{2}\bigl( \vert v \vert \bigr) \bigr)\,dx \\ & {} + \int_{\mathbb{R}^{N}} \biggl(\frac{1}{\mu_{1}}uF_{u}(x,u,v)+ \frac{1}{\mu _{2}}vF_{v}(x,u,v)-F(x,u,v) \biggr)\,dx\geq0. \end{aligned}$$

Since the nontrivial critical point $(u,v)$ of I is arbitrary, we conclude $d\geq0$. Choose a sequence $\{(u_{n},v_{n})\}\subset\{(u,v)\in W: (u,v)\neq(\mathbf{0,0}) \mbox{ and } I'(u,v)=0\}$ such that $I(u_{n},v_{n})\rightarrow d$ as $n\rightarrow\infty$. Then it is obvious that $\{(u_{n},v_{n})\}$ is a $(C)_{d}$-sequence of I for the level $d\geq0$. Lemma 3.16 shows that $\{(u_{n},v_{n})\}$ is bounded in W. Moreover, Lemma A.3 in the Appendix implies that there exists a constant $M_{6}>0$ such that

$$\begin{aligned} \bigl\Vert (u_{n},v_{n}) \bigr\Vert \geq M_{6} \quad\mbox{for all } n\in\mathbb{N}. \end{aligned}$$

(3.57)

We claim that

$$\begin{aligned} \lambda_{4}:=\lim_{n\rightarrow\infty}\sup_{y\in\mathbb{R}^{N}} \int _{B_{2}(y)}\bigl(\Phi_{1}\bigl( \vert u_{n} \vert \bigr)+\Phi_{2}\bigl( \vert v_{n} \vert \bigr)\bigr)\,dx>0. \end{aligned}$$

Indeed, if $\lambda_{4}=0$, similar to (3.48), we get

$$\begin{aligned} \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}}u_{n}F_{u}(x,u_{n},v_{n})\,dx= \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}}v_{n}F_{v}(x,u_{n},v_{n})\,dx=0. \end{aligned}$$

(3.58)

Then, by (3.5), $(\phi_{2})$, $(V_{2})$ and (3.58), we have

$$\begin{aligned} 0 = {}& \lim_{n\rightarrow\infty} \biggl\{ \bigl\langle I'(u_{n},v_{n}),(u_{n},v_{n}) \bigr\rangle + \int_{\mathbb{R}^{N}}u_{n}F_{u}(x,u_{n},v_{n})\,dx+ \int _{\mathbb{R}^{N}}v_{n}F_{v}(x,u_{n},v_{n})\,dx \biggr\} \\ = {}& \lim_{n\rightarrow\infty} \biggl\{ \int_{\mathbb{R}^{N}}a_{1}\bigl( \vert \nabla u_{n} \vert \bigr) \vert \nabla u_{n} \vert ^{2}\,dx + \int_{\mathbb {R}^{N}}V_{1}(x)a_{1}\bigl( \vert u_{n} \vert \bigr) \vert u_{n} \vert ^{2}\,dx \\ & {}+ \int_{\mathbb{R}^{N}}a_{2}\bigl( \vert \nabla v_{n} \vert \bigr) \vert \nabla v_{n} \vert ^{2}\,dx + \int _{\mathbb{R}^{N}}V_{2}(x)a_{2}\bigl( \vert v_{n} \vert \bigr) \vert v_{n} \vert ^{2}\,dx \biggr\} \\ \geq{}& \lim_{n\rightarrow\infty} \biggl\{ l_{1} \int_{\mathbb{R}^{N}}\Phi _{1}\bigl( \vert \nabla u_{n} \vert \bigr)\,dx+l_{1}\alpha_{1} \int_{\mathbb{R}^{N}}\Phi_{1}\bigl( \vert u_{n} \vert \bigr)\,dx\\ &{} +l_{2} \int_{\mathbb{R}^{N}}\Phi_{2}\bigl( \vert \nabla v_{n} \vert \bigr)\,dx+l_{2}\alpha_{1} \int_{\mathbb {R}^{N}}\Phi_{2}\bigl( \vert v_{n} \vert \bigr)\,dx \biggr\} \\ \geq{}&0, \end{aligned}$$

which, together with (2.3), implies that $\Vert (u_{n}, v_{n}) \Vert = \Vert \nabla u_{n} \Vert _{\Phi_{1}}+ \Vert u_{n} \Vert _{\Phi_{1}}+ \Vert \nabla v_{n} \Vert _{\Phi_{2}}+ \Vert v_{n} \Vert _{\Phi_{2}}\rightarrow0$, which contradicts (3.57). Therefore, $\lambda_{4}> 0$, which implies that there exist a constant $\delta>0$, a subsequence of $\{(u_{n},v_{n})\}$, still denoted by $\{(u_{n},v_{n})\}$, and a sequence $\{z_{n}\}\in\mathbb{Z}^{N}$ such that

$$\begin{aligned} \begin{aligned} &\int_{B_{2}(z_{n})}\bigl(\Phi_{1}\bigl( \vert u_{n} \vert \bigr)+\Phi_{2}\bigl( \vert v_{n} \vert \bigr)\bigr)\,dx= \int_{B_{2}}\bigl(\Phi _{1}\bigl( \bigl\vert u_{n}^{*} \bigr\vert \bigr)+\Phi_{2}\bigl( \bigl\vert v_{n}^{*} \bigr\vert \bigr)\bigr)\,dx>\delta \\ &\quad\mbox{for all } n\in \mathbb{N}, \end{aligned} \end{aligned}$$

(3.59)

where $u_{n}^{*}:=u_{n}(\cdot+z_{n})$ and $v_{n}^{*}:=v_{n}(\cdot+z_{n})$. Since $\Vert u_{n}^{*} \Vert _{1,\Phi_{1}}= \Vert u_{n} \Vert _{1,\Phi_{1}}$ and $\Vert v_{n}^{*} \Vert _{1,\Phi_{2}}= \Vert v_{n} \Vert _{1,\Phi_{2}}$, then $\{{u_{n}^{*}}\}$ and $\{{v_{n}^{*}}\}$ are bounded in $W^{1,\Phi_{1}}(\mathbb{R}^{N})$ and $W^{1,\Phi_{2}}(\mathbb{R}^{N})$, respectively. Passing to a subsequence of $\{({u_{n}^{*}},{v_{n}^{*}})\}$, still denoted by $\{({u_{n}^{*}},{v_{n}^{*}})\}$, there exists $(u_{0},v_{0})\in W$ such that $u_{n}^{*}\rightharpoonup u_{0}$ in $W^{1,\Phi_{1}}(\mathbb{R}^{N})$ and $v_{n}^{*}\rightharpoonup v_{0}$ in $W^{1,\Phi_{2}}(\mathbb{R}^{N})$, respectively. Moreover, for any given constant $r>0$, by Remark 2.7, we can assume that

⋆:

$u_{n}^{*}\rightarrow u_{0}$ in $L^{\Phi_{1}}(B_{r})$ and $u_{n}^{*}(x)\rightarrow u_{0}(x)$ a.e. in $B_{r}$;

⋆:

$v_{n}^{*}\rightarrow v_{0}$ in $L^{\Phi_{2}}(B_{r})$ and $v_{n}^{*}(x)\rightarrow v_{0}(x)$ a.e. in $B_{r}$.

Then, by (3.59), ⋆ and (2.3), we obtain that $(u_{0},v_{0})\neq(\mathbf{0,0})$. Since $V_{i}(i=1,2)$ and F are 1-periodic in x, $\{(u_{n}^{*},v_{n}^{*})\}$ is also a $(C)_{d}$-sequence of I with $\{(u_{n}^{*},v_{n}^{*})\}\subset\{(u,v)\in W: (u,v)\neq(\textbf {0,0}) \mbox{ and } I'(u,v)=0\}$. Then similar arguments as those in Lemma 3.18 show that $I'(u_{0},v_{0})=0$, and thus $I(u_{0},v_{0})\geq d$. However, for any given constant $r>0$, it follows from (3.4), (3.5), $(\phi_{2})$, $(V_{2})$, $(F_{3})$, ⋆ and Fatou’s lemma that

$$\begin{aligned} & \int_{B_{r}} \biggl(\Phi_{1}\bigl( \vert \nabla u_{0} \vert \bigr)-\frac{1}{\mu_{1}}a_{1}\bigl( \vert \nabla u_{0} \vert \bigr) \vert \nabla u_{0} \vert ^{2} \biggr)\,dx \\ &\qquad{}+ \int_{B_{r}}V_{1}(x) \biggl(\Phi_{1}\bigl( \vert u_{0} \vert \bigr)-\frac{1}{\mu _{1}}a_{1}\bigl( \vert u_{0} \vert \bigr) \vert u_{0} \vert ^{2} \biggr)\,dx \\ & \qquad{}+ \int_{B_{r}} \biggl(\Phi_{2}\bigl( \vert \nabla v_{0} \vert \bigr)-\frac{1}{\mu_{2}}a_{2}\bigl( \vert \nabla v_{0} \vert \bigr) \vert \nabla v_{0} \vert ^{2} \biggr)\,dx \\ &\qquad{}+ \int_{B_{r}}V_{2}(x) \biggl(\Phi_{2}\bigl( \vert v_{0} \vert \bigr)-\frac{1}{\mu _{2}}a_{2}\bigl( \vert v_{0} \vert \bigr) \vert v_{0} \vert ^{2} \biggr)\,dx \\ & \qquad{} + \int_{B_{r}} \biggl(\frac{1}{\mu_{1}}u_{0}F_{u}(x,u_{0},v_{0})+ \frac{1}{\mu _{2}}v_{0}F_{v}(x,u_{0},v_{0})-F(x,u_{0},v_{0}) \biggr)\,dx \\ &\quad \leq \liminf_{n\rightarrow\infty} \biggl\{ \int_{B_{r}} \biggl(\Phi _{1}\bigl( \bigl\vert \nabla u_{n}^{*} \bigr\vert \bigr)-\frac{1}{\mu_{1}}a_{1}\bigl( \bigl\vert \nabla u_{n}^{*} \bigr\vert \bigr) \bigl\vert \nabla u_{n}^{*} \bigr\vert ^{2} \biggr)\,dx\\ &\qquad{} + \int_{B_{r}}V_{1}(x) \biggl(\Phi_{1}\bigl( \bigl\vert u_{n}^{*} \bigr\vert \bigr)-\frac{1}{\mu _{1}}a_{1} \bigl( \bigl\vert u_{n}^{*} \bigr\vert \bigr) \bigl\vert u_{n}^{*} \bigr\vert ^{2} \biggr)\,dx \\ & \qquad{} + \int_{B_{r}} \biggl(\Phi_{2}\bigl( \bigl\vert \nabla v_{n}^{*} \bigr\vert \bigr)-\frac{1}{\mu_{2}}a_{2}\bigl( \bigl\vert \nabla v_{n}^{*} \bigr\vert \bigr) \bigl\vert \nabla v_{n}^{*} \bigr\vert ^{2} \biggr)\,dx \\ &\qquad{}+ \int_{B_{r}}V_{2}(x) \biggl(\Phi_{2}\bigl( \bigl\vert v_{n}^{*} \bigr\vert \bigr)-\frac{1}{\mu _{2}}a_{2} \bigl( \bigl\vert v_{n}^{*} \bigr\vert \bigr) \bigl\vert v_{n}^{*} \bigr\vert ^{2} \biggr)\,dx \\ & \qquad{} + \int_{B_{r}} \biggl(\frac{1}{\mu _{1}}u_{n}^{*}F_{u} \bigl(x,u_{n}^{*},v_{n}^{*}\bigr)+\frac{1}{\mu _{2}}v_{n}^{*}F_{v} \bigl(x,u_{n}^{*},v_{n}^{*}\bigr)-F\bigl(x,u_{n}^{*},v_{n}^{*} \bigr) \biggr)\,dx \biggr\} \\ &\quad \leq\liminf_{n\rightarrow\infty} \biggl\{ I\bigl(u_{n}^{*},v_{n}^{*} \bigr)- \biggl\langle I'\bigl(u_{n}^{*},v_{n}^{*} \bigr), \biggl(\frac{1}{\mu_{1}}u_{n}^{*},\frac{1}{\mu_{2}}v_{n}^{*} \biggr) \biggr\rangle \biggr\} \\ &\quad=d. \end{aligned}$$

Since $r>0$ is arbitrary, then $I(u_{0},v_{0})=I(u_{0},v_{0})- \langle I'(u_{0},v_{0}), (\frac{1}{\mu_{1}}u_{0},\frac{1}{\mu_{2}}v_{0} ) \rangle\leq d$. Therefore, $I(u_{0},v_{0})=d$, that is, $(u_{0},v_{0})$ is a ground state of system (1.1). □

Proof of Theorem 3.2

Lemma 3.18 shows that system (1.1) has at least a nontrivial solution under the assumptions of Theorem 3.2. Moreover, following the same steps as in the above proof of Theorem 3.1 but replacing $\mu_{i}$ with $m_{i}(i=1,2)$, we can find a ground state of system (1.1). □

4 Examples

For system (1.1), $\phi_{i} (i=1,2)$ defined by (1.2) can be chosen from the following cases which satisfy all $(\phi_{1})$-$(\phi_{3})$ type conditions:

Case 1. Let $\phi(t)= \vert t \vert ^{p-1}t$ for $t\neq0$, $\phi(0)=0$ with $1< p+1< N$. In this case, simple computations show that $l=m=p+1$;

Case 2. Let $\phi(t)= \vert t \vert ^{p-1}t+ \vert t \vert ^{q-1}t$ for $t\neq0$, $\phi(0)=0$ with $1< p+1< q+1< N<\frac{(p+1)(q+1)}{q-p}$. In this case, simple computations show that $l=p+1, m=q+1$;

Case 3. Let $\phi(t)=\frac{ \vert t \vert ^{q-1}t}{\log(1+ \vert t \vert ^{p})}$ for $t\neq0$, $\phi(0)=0$ with $1< p+1< q+1< N<\frac{(q-p+1)(q+1)}{p}$. In this case, simple computations show that $l=q-p+1, m=q+1$.

Moreover, we also give a case that satisfies $(\phi_{1})$ and $(\phi_{2})$ but does not satisfy $(\phi_{3})$ type conditions:

Case 4. Let $\phi(t)= \vert t \vert ^{q-1}t\log(1+ \vert t \vert ^{p})$ for $t\neq0$, $\phi(0)=0$ with $p,q>0$ and $p+q+1< N<\frac{(q+1)(p+q+1)}{p}$. In this case, simple computations show that $l=q+1, m=p+q+1$.

Example 4.1

Assume that $V_{i}(i=1,2)$ and $\phi_{i}(i=1,2)$ defined by (1.2) satisfy $(V_{1})$, $(V_{2})$, $(\phi_{1})$ and $(\phi _{2})$ with $m_{i}\geq4(i=1,2)$. Let $F(x,u,v)= \vert u \vert ^{\frac{m_{1}+l_{1}^{*}}{2}}+ \vert v \vert ^{\frac {m_{2}+l_{2}^{*}}{2}}+ \vert u \vert ^{\frac{m_{1}+l_{1}^{*}}{4}} \vert v \vert ^{\frac{m_{2}+l_{2}^{*}}{4}}$. Choose $\mu_{i}=\frac{m_{i}+l_{i}^{*}}{2}(i=1,2)$. Then it is easy to check that F satisfies $(F_{1})$, $(F_{2})'$ and $(F_{3})$.

Example 4.2

Assume that $V_{i}(i=1,2)$ and $\phi_{i}(i=1,2)$ defined by (1.2) satisfy $(V_{1})$, $(V_{2})$, $(\phi_{1})$-$(\phi_{3})$ with $m_{i}\geq4(i=1,2)$, $\max\{\frac{N}{l_{1}},\frac{N}{l_{2}}\}<\min\{ \frac{m_{1}}{m_{1}-l_{1}},\frac{m_{2}}{m_{2}-l_{2}}\}$. Then $F(x,u,v)= \vert u \vert ^{m_{1}}\log(1+ \vert u \vert )+ \vert v \vert ^{m_{2}}\log(1+ \vert v \vert )+ \vert u \vert ^{\frac {m_{1}+\epsilon}{2}} \vert v \vert ^{\frac{m_{2}+\epsilon}{2}}$, where constant $\epsilon>0$ satisfying $\epsilon<\frac {2l_{1}^{*}l_{2}^{*}-m_{1}l_{2}^{*}-m_{2}l_{1}^{*}}{l_{1}^{*}+l_{2}^{*}}$ and $\max\{\frac {N}{l_{1}},\frac{N}{l_{2}}\}<\min\{\frac{m_{1}}{m_{1}-l_{1}+\epsilon},\frac {m_{2}}{m_{2}-l_{2}+\epsilon}\}$ satisfies $(F_{1})$, $(F_{2})'$, $(F_{4})'$ and $(F_{5})$. In fact,

$$\begin{aligned} &F_{u}(x,u,v)=m_{1} \vert u \vert ^{m_{1}-2}u\log \bigl(1+ \vert u \vert \bigr)+\frac{ \vert u \vert ^{m_{1}-1}u}{1+ \vert u \vert }+\frac {m_{1}+\epsilon}{2} \vert u \vert ^{\frac{m_{1}+\epsilon-4}{2}} \vert v \vert ^{\frac{m_{2}+\epsilon}{2}}u, \\ &F_{v}(x,u,v)=m_{2} \vert v \vert ^{m_{2}-2}v\log \bigl(1+ \vert v \vert \bigr)+\frac{ \vert v \vert ^{m_{2}-1}v}{1+ \vert v \vert }+\frac {m_{2}+\epsilon}{2} \vert u \vert ^{\frac{m_{2}+\epsilon}{2}} \vert v \vert ^{\frac{m_{2}+\epsilon-4}{2}}v, \end{aligned}$$

then

$$\begin{aligned} \overline{F}(x,u,v)&=\frac{ \vert u \vert ^{m_{1}+1}}{m_{1}(1+ \vert u \vert )}+\frac { \vert v \vert ^{m_{2}+1}}{m_{2}(1+ \vert v \vert )}+\frac{(m_{1}+m_{2})\epsilon}{2m_{1}m_{2}} \vert u \vert ^{\frac {m_{1}+\epsilon}{2}} \vert v \vert ^{\frac{m_{2}+\epsilon}{2}} \\ &\geq \frac{ \vert u \vert ^{m_{1}+1}}{m_{1}(1+ \vert u \vert )}+\frac{ \vert v \vert ^{m_{2}+1}}{m_{2}(1+ \vert v \vert )}. \end{aligned}$$

It is obvious that F satisfies $(F_{1})$ and $(F_{4})'$. Since $0<\epsilon <\frac{2l_{1}^{*}l_{2}^{*}-m_{1}l_{2}^{*}-m_{2}l_{1}^{*}}{l_{1}^{*}+l_{2}^{*}}$, then it is easy to check $(F_{2})'$ by Young’s inequality. Next, we check $(F_{5})$. It is obvious that $\overline{F}(x,u,v)>0$ for all $(u,v)\neq(0,0)$. Moreover, choose $\max\{\frac{N}{l_{1}},\frac{N}{l_{2}}\}< k\leq\min\{\frac {m_{1}}{m_{1}-l_{1}+\epsilon},\frac{m_{2}}{m_{2}-l_{2}+\epsilon}\}$, then

$$\begin{aligned} & \limsup_{ \vert (u,v) \vert \rightarrow\infty} \biggl(\frac { \vert F(x,u,v) \vert }{ \vert u \vert ^{l_{1}}+ \vert v \vert ^{l_{2}}} \biggr)^{k}\frac{1}{\overline {F}(x,u,v)} \\ &\quad\leq \limsup_{ \vert (u,v) \vert \rightarrow\infty}\frac{ ( \vert u \vert ^{m_{1}}\log (1+ \vert u \vert )+ \vert v \vert ^{m_{2}}\log(1+ \vert v \vert )+ \vert u \vert ^{\frac{m_{1}+\epsilon}{2}} \vert v \vert ^{\frac {m_{2}+\epsilon}{2}} )^{k}}{ ( \vert u \vert ^{l_{1}}+ \vert v \vert ^{l_{2}} )^{k} (\frac { \vert u \vert ^{m_{1}+1}}{m_{1}(1+ \vert u \vert )}+\frac{ \vert v \vert ^{m_{2}+1}}{m_{2}(1+ \vert v \vert )} )} \\ &\quad \leq C_{k}\limsup_{ \vert (u,v) \vert \rightarrow\infty}\frac{ \vert u \vert ^{km_{1}}(\log (1+ \vert u \vert ))^{k}+ \vert v \vert ^{km_{2}}(\log(1+ \vert v \vert ))^{k}+ \vert u \vert ^{k(m_{1}+\epsilon )}+ \vert v \vert ^{k(m_{2}+\epsilon)}}{ \frac{ \vert u \vert ^{kl_{1}+m_{1}+1}}{m_{1}(1+ \vert u \vert )}+\frac { \vert v \vert ^{kl_{2}+m_{2}+1}}{m_{2}(1+ \vert v \vert )}} \\ & \quad < \infty. \end{aligned}$$

Appendix

Lemma A.1

If Φ is an N-function and (2.5) holds, then for any sequence $\{u_{n}\}$ converging to u in $L^{\Phi}(\mathbb{R}^{N})$, there exist a subsequence of $\{u_{n}\}$, still denoted by $\{u_{n}\}$, and a function $h\in L^{1}(\mathbb{R}^{N})$ such that

• (a) $u_{n}(x)\rightarrow u(x)$ a.e. $x\in\mathbb{R}^{N}$;

• (b) $\widetilde{\Phi}(\phi( \vert u_{n}(x) \vert ))\leq h(x)$ for all $n\in\mathbb {N}$, a.e. $x\in\mathbb{R}^{N}$;

• (c) $\Phi( \vert u_{n}(x) \vert )\leq h(x)$ for all $n\in\mathbb{N}$, a.e. $x\in \mathbb{R}^{N}$.

Proof

Since $u_{n}\rightarrow u$ in $L^{\Phi}(\mathbb {R}^{N})$, by Lemma 2.2, we have

$$\begin{aligned} &\int_{\mathbb{R}^{N}}\Phi\bigl(4 \vert u_{n}-u \vert \bigr)\,dx\\ &\quad\leq\max\bigl\{ 4^{l} \Vert u_{n}-u \Vert _{\Phi }^{l},4^{m} \Vert u_{n}-u \Vert _{\Phi}^{m}\bigr\} \rightarrow0 \quad\mbox{as } n\rightarrow \infty, \end{aligned}$$

which implies $\Phi(4 \vert u_{n}-u \vert )\rightarrow0$ in $L^{1}(\mathbb{R}^{N})$. Hence, by ([24], Theorem 4.9), there exist a subsequence of $\{\Phi(4 \vert u_{n}-u \vert )\}$, still denoted by $\{\Phi(4 \vert u_{n}-u \vert )\}$, and functions $h_{1}\in L^{1}(\mathbb{R}^{N})$ such that

$$\begin{aligned} \Phi\bigl(4 \bigl\vert u_{n}(x)-u(x) \bigr\vert \bigr)\rightarrow0,\quad \mbox{a.e. } x\in\mathbb{R}^{N} \end{aligned}$$

(A.1)

and

$$\begin{aligned} \Phi\bigl(4 \bigl\vert u_{n}(x)-u(x) \bigr\vert \bigr)\leq h_{1}(x)\quad \mbox{for all } n\in\mathbb{N}, \mbox{ a.e. } x\in \mathbb{R}^{N}. \end{aligned}$$

(A.2)

Then, by (2.2), the monotonicity and convexity of Φ, (A.2) and the fact $4u\in L^{\Phi}(\mathbb{R}^{N})$, for all $n\in\mathbb {N}$, a.e. $x\in\mathbb{R}^{N}$, we have

$$\begin{aligned} \widetilde{\Phi}\bigl(\phi\bigl( \bigl\vert u_{n}(x) \bigr\vert \bigr)\bigr)&\leq\Phi\bigl(2 \bigl\vert u_{n}(x) \bigr\vert \bigr) \leq\frac{1}{2}\Phi \bigl(4 \bigl\vert u_{n}(x)-u(x) \bigr\vert \bigr)+\frac{1}{2}\Phi\bigl(4 \bigl\vert u(x) \bigr\vert \bigr)\\ &\leq\frac{1}{2}h_{1}(x)+\frac {1}{2}\Phi\bigl(4 \bigl\vert u(x) \bigr\vert \bigr)\in L^{1}\bigl( \mathbb{R}^{N}\bigr) \end{aligned}$$

and

$$\begin{aligned} \Phi\bigl( \bigl\vert u_{n}(x) \bigr\vert \bigr)\leq\Phi\bigl(2 \bigl\vert u_{n}(x) \bigr\vert \bigr)\leq\frac{1}{2}h_{1}(x)+ \frac{1}{2}\Phi \bigl(4 \bigl\vert u(x) \bigr\vert \bigr)\in L^{1}\bigl(\mathbb{R}^{N}\bigr). \end{aligned}$$

Moreover, (A.1) implies that $u_{n}(x)\rightarrow u(x)$ a.e. $x\in \mathbb{R}^{N}$. □

Lemma A.2

Suppose that $(\phi_{1})$, $(\phi_{2})$, $(V_{2})$, $(F1)$ and $(F_{2})$ hold. Then $I: W\rightarrow\mathbb{R}$ is well defined and of class $C^{1}(W, \mathbb{R})$ and

$$\begin{aligned} \bigl\langle I'(u,v),(\tilde{u},\tilde{v})\bigr\rangle = {}& \int_{\mathbb {R}^{N}}a_{1}\bigl( \vert \nabla u \vert \bigr) \nabla u\nabla\tilde{u}\,dx + \int_{\mathbb{R}^{N}}V_{1}(x)a_{1}\bigl( \vert u \vert \bigr)u\tilde{u}\,dx \\ & {}+ \int_{\mathbb{R}^{N}}a_{2}\bigl( \vert \nabla v \vert \bigr) \nabla v\nabla\tilde{v}\,dx + \int_{\mathbb{R}^{N}}V_{2}(x)a_{2}\bigl( \vert v \vert \bigr)v\tilde{v}\,dx \\ & {}- \int_{\mathbb{R}^{N}}F_{u}(x,u,v)\tilde{u}\,dx- \int_{\mathbb {R}^{N}}F_{v}(x,u,v)\tilde{v}\,dx \end{aligned}$$

for all $(\tilde{u},\tilde{v})\in W$.

Proof

Under assumptions $(\phi_{1})$, $(\phi_{2})$ and $(V_{2})$, by similar arguments as those in [25], we can prove that $I_{1}: W\rightarrow\mathbb{R}$ is well defined and of class $C^{1}(W,\mathbb{R})$ and

$$\begin{aligned} \bigl\langle I_{1}'(u,v),(\tilde{u}, \tilde{v})\bigr\rangle ={}& \int_{\mathbb {R}^{N}}a_{1}\bigl( \vert \nabla u \vert \bigr) \nabla u\nabla\tilde{u}\,dx + \int_{\mathbb {R}^{N}}V_{1}(x)a_{1}\bigl( \vert u \vert \bigr)u\tilde{u}\,dx \\ & {}+ \int_{\mathbb{R}^{N}}a_{2}\bigl( \vert \nabla v \vert \bigr) \nabla v\nabla\tilde{v}\,dx+ \int _{\mathbb{R}^{N}}V_{2}(x)a_{2}\bigl( \vert v \vert \bigr)v\tilde{v}\,dx \end{aligned}$$

(A.3)

for all $(\tilde{u},\tilde{v})\in W$. So, it is sufficient to prove that $I_{2}: W\rightarrow\mathbb{R}$ is well defined and of class $C^{1}(W, \mathbb{R})$ and

$$\begin{aligned} \bigl\langle I_{2}'(u,v),(\tilde{u}, \tilde{v})\bigr\rangle = \int_{\mathbb {R}^{N}}F_{u}(x,u,v)\tilde{u}\,dx+ \int_{\mathbb{R}^{N}}F_{v}(x,u,v)\tilde{v}\,dx \end{aligned}$$

(A.4)

for all $(\tilde{u},\tilde{v})\in W$.

By (3.7) and (3.10), we have

$$\begin{aligned} I_{2}(u,v) &\leq \int_{\mathbb{R}^{N}} \bigl\vert F(x,u,v) \bigr\vert \,dx\\ & \leq C_{3} \int_{\mathbb{R}^{N}}\bigl(\Phi_{1}\bigl( \vert u \vert \bigr)+\Phi_{2}\bigl( \vert v \vert \bigr)+\Phi _{1*} \bigl( \vert u \vert \bigr)+\Phi_{2*}\bigl( \vert v \vert \bigr)\bigr)\,dx, \end{aligned}$$

which, together with (2.8), implies that $I_{2}$ is well defined in W.

We now prove that (A.4) holds. For any given $(u, v), (\tilde {u}, \tilde{v})\in W$, we have

$$\begin{aligned} &\bigl\langle I_{2}'(u,v),(\tilde{u}, \tilde{v})\bigr\rangle \\ &\quad = \lim_{h\rightarrow0}\frac{1}{h}\bigl(I_{2}(u+h \tilde{u}, v+h\tilde {v})-I_{2}(u, v)\bigr) \\ &\quad = \lim_{h\rightarrow0} \int_{\mathbb{R}^{N}}\frac{F(x,u+h\tilde {u},v+h\tilde{v})-F(x,u,v+h\tilde{v})}{h}\,dx \\ &\qquad{} +\lim_{h\rightarrow0} \int_{\mathbb{R}^{N}}\frac{F(x,u,v+h\tilde {v})-F(x,u,v)}{h}\,dx \\ &\quad = \lim_{h\rightarrow0} \int_{\mathbb{R}^{N}}F_{u}\bigl(x, u+\theta_{1}(x)h \tilde {u}, v+h\tilde{v}\bigr)\tilde{u}\,dx +\lim_{h\rightarrow0} \int_{\mathbb{R}^{N}}F_{v}\bigl(x, u, v+\theta_{2}(x)h \tilde {v}\bigr)\tilde{v}\,dx, \end{aligned}$$

(A.5)

where $\theta_{1}, \theta_{2}: \mathbb{R}^{N}\rightarrow(0, 1)$. By the continuity of $F_{u}$ and $F_{v}$, we have that

$$\begin{aligned} F_{u}\bigl(x, u+\theta_{1}(x)h\tilde{u}, v+h\tilde{v}\bigr)\tilde{u}\rightarrow F_{u}(x,u,v)\tilde{u} \end{aligned}$$

(A.6)

and

$$F_{v}\bigl(x, u, v+\theta_{2}(x)h\tilde{v}\bigr)\tilde{v} \rightarrow F_{v}(x,u,v)\tilde{v} $$

as $h\rightarrow0$ for a.e. $x \in\mathbb{R}^{N}$. Moreover, for all $h\in(-1, 1)$, by (3.8), the monotonicity of functions, (2.1), $(\phi_{2})$, (1) in Lemma 2.4 and (2.8), we have

$$\begin{aligned} & \bigl\vert F_{u}\bigl(x, u+\theta_{1}(x)h \tilde{u}, v+h\tilde{v}\bigr)\tilde{u} \bigr\vert \\ &\quad \leq C_{1} \bigl(\phi_{1}\bigl( \bigl\vert u+ \theta_{1}(x)h\tilde{u} \bigr\vert \bigr)+{\widetilde{\Phi }_{1}}^{-1}\bigl(\Phi_{2}\bigl( \vert v+h \tilde{v} \vert \bigr)\bigr)+\Phi_{1*}'\bigl( \bigl\vert u+\theta_{1}(x)h\tilde{u} \bigr\vert \bigr) \\ &\qquad{}+{\widetilde{ \Phi}_{1*}}^{-1}\bigl(\Phi_{2*}\bigl( \vert v+h \tilde{v} \vert \bigr)\bigr) \bigr) \vert \tilde {u} \vert \\ &\quad \leq C_{1} \bigl(\bigl( \vert u \vert + \vert \tilde{u} \vert \bigr)\phi_{1}\bigl( \vert u \vert + \vert \tilde{u} \vert \bigr)+ \vert \tilde {u} \vert {\widetilde{\Phi}_{1}}^{-1} \bigl(\Phi_{2}\bigl( \vert v \vert + \vert \tilde{v} \vert \bigr)\bigr) \\ &\qquad{} +\bigl( \vert u \vert + \vert \tilde{u} \vert \bigr) \Phi_{1*}'\bigl( \vert u \vert + \vert \tilde{u} \vert \bigr)+ \vert \tilde {u} \vert {\widetilde{\Phi}_{1*}}^{-1} \bigl(\Phi_{2*}\bigl( \vert v \vert + \vert \tilde{v} \vert \bigr)\bigr) \bigr) \\ &\quad \leq C_{1} \bigl(m_{1}\Phi_{1}\bigl( \vert u \vert + \vert \tilde{u} \vert \bigr)+{\Phi_{1}}\bigl( \vert \tilde{u} \vert \bigr)+\Phi _{2}\bigl( \vert v \vert + \vert \tilde{v} \vert \bigr)+m_{1}^{*}\Phi_{1*}\bigl( \vert u \vert + \vert \tilde{u} \vert \bigr) \\ &\qquad{}+{\Phi_{1*}}\bigl( \vert \tilde{u} \vert \bigr)+\Phi_{2*}\bigl( \vert v \vert + \vert \tilde{v} \vert \bigr) \bigr) \\ &\quad =: g_{1}(x)\in L^{1}\bigl(\mathbb{R}^{N}\bigr). \end{aligned}$$

(A.7)

Then it follows from (A.6), (A.7) and Lebesgue’s dominated convergence theorem that

$$\begin{aligned} \lim_{h\rightarrow0} \int_{\mathbb{R}^{N}}F_{u}\bigl(x, u+\theta_{1}(x)h \tilde {u}, v+h\tilde{v}\bigr)\tilde{u}\,dx= \int_{\mathbb{R}^{N}}F_{u}(x,u,v)\tilde {u}\,dx. \end{aligned}$$

(A.8)

Similarly, we can obtain that

$$\begin{aligned} \lim_{h\rightarrow0} \int_{\mathbb{R}^{N}}F_{v}\bigl(x, u, v+\theta_{2}(x)h \tilde {v}\bigr)\tilde{v}\,dx= \int_{\mathbb{R}^{N}}F_{v}(x,u,v)\tilde{v}\,dx. \end{aligned}$$

(A.9)

Combining (A.8) and (A.9) with (A.5), we can conclude that (A.4) holds.

Next, we prove the continuity of $I_{2}'$. Let $(u_{n}, v_{n})\rightarrow (u, v)$ in W. We claim that $I_{2}'(u_{n},v_{n})\rightarrow I_{2}'(u,v)$ in $W^{*}$ (the dual space of W). Otherwise, there exist a constant $\varepsilon_{0}>0$ and a subsequence of $\{(u_{n}, v_{n})\}$, denoted by $\{ (u_{ni}, v_{ni})\}$, such that

$$\begin{aligned} \bigl\Vert I_{2}'(u_{ni},v_{ni})-I_{2}'(u,v) \bigr\Vert _{W^{*}}\geq\varepsilon_{0}>0 \quad\mbox{for all } i \in\mathbb{N}. \end{aligned}$$

(A.10)

Since $(u_{ni}, v_{ni})\rightarrow(u,v)$ in W, then $u_{ni}\rightarrow u$ in $W^{1,\Phi_{1}}(\mathbb{R}^{N})$ and $v_{ni}\rightarrow v$ in $W^{1,\Phi_{2}}(\mathbb{R}^{N})$, respectively. It follows from (2.8) that $u_{ni}\rightarrow u$ in $L^{\Phi _{1*}}(\mathbb{R}^{N})$ and $v_{ni}\rightarrow v$ in $L^{\Phi _{2*}}(\mathbb{R}^{N})$, respectively. By Lemma A.1, there exist a subsequence of $\{(u_{ni}, v_{ni})\}$, still denoted by $\{(u_{ni}, v_{ni})\}$, and a function $h\in L^{1}(\mathbb{R}^{N})$ such that

$$\begin{aligned} u_{ni}(x)\rightarrow u(x), \qquad v_{ni}(x) \rightarrow v(x), \quad\mbox{a.e. } x\in\mathbb{R}^{N} \end{aligned}$$

(A.11)

and

$$\begin{aligned} \begin{aligned}&\widetilde{\Phi}_{1}\bigl(\phi_{1}\bigl( \bigl\vert u_{ni}(x) \bigr\vert \bigr)\bigr)\leq h(x), \qquad\widetilde{ \Phi }_{1*}\bigl(\Phi_{1*}'\bigl( \bigl\vert u_{ni}(x) \bigr\vert \bigr)\bigr)\leq h(x),\\ & \Phi_{1}\bigl( \bigl\vert u_{ni}(x) \bigr\vert \bigr)\leq h(x),\qquad \Phi_{1*}\bigl( \bigl\vert u_{ni}(x) \bigr\vert \bigr) \leq h(x), \\ &\widetilde{\Phi}_{2}\bigl(\phi_{2}\bigl( \bigl\vert v_{ni}(x) \bigr\vert \bigr)\bigr)\leq h(x),\qquad \widetilde{\Phi }_{2*}\bigl(\Phi_{2*}'\bigl( \bigl\vert v_{ni}(x) \bigr\vert \bigr)\bigr)\leq h(x),\\ & \Phi_{2}\bigl( \bigl\vert v_{ni}(x) \bigr\vert \bigr)\leq h(x),\qquad \Phi_{2*}\bigl( \bigl\vert v_{ni}(x) \bigr\vert \bigr) \leq h(x) \end{aligned} \end{aligned}$$

(A.12)

for all $i\in\mathbb{N}$, a.e. $x\in\mathbb{R}^{N}$. For this subsequence $\{(u_{n}, v_{n})\}$ and all $(\tilde{u}, \tilde{v}) \in W$, by (A.4) we have

$$\begin{aligned} & \bigl\vert \bigl\langle I_{2}'(u_{ni},v_{ni})-I_{2}'(u,v),( \tilde{u},\tilde{v})\bigr\rangle \bigr\vert \\ &\quad = \biggl\vert \int_{\mathbb{R}^{N}}F_{u}(x,u_{ni},v_{ni}) \tilde{u}\,dx+ \int _{\mathbb{R}^{N}}F_{v}(x,u_{ni},v_{ni}) \tilde{v}\,dx \\ &\qquad{} - \int_{\mathbb{R}^{N}}F_{u}(x,u,v)\tilde{u}\,dx- \int_{\mathbb {R}^{N}}F_{v}(x,u,v)\tilde{v}\,dx \biggr\vert \\ &\quad \leq \int_{\mathbb{R}^{N}} \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \vert \tilde {u} \vert \,dx \\ &\qquad{}+ \int_{\mathbb{R}^{N}} \bigl\vert F_{v}(x,u_{ni},v_{ni})-F_{v}(x,u,v) \bigr\vert \vert \tilde {v} \vert \,dx. \end{aligned}$$

(A.13)

Firstly, we claim that

$$\begin{aligned} \int_{\mathbb{R}^{N}} \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \vert \tilde {u} \vert \,dx=o_{i}(1) \bigl\Vert ( \tilde{u}, \tilde{v}) \bigr\Vert . \end{aligned}$$

(A.14)

In fact,

$$\begin{aligned} & \int_{\mathbb{R}^{N}} \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \vert \tilde {u} \vert \,dx \\ &\quad = \int_{\Omega_{1}} \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \vert \tilde{u} \vert \,dx \\ &\qquad{}+ \int _{\Omega_{2}} \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \vert \tilde{u} \vert \,dx, \end{aligned}$$

(A.15)

where $\Omega_{1}=\{x\in\mathbb{R}^{N}: \vert u(x) \vert \leq1, \vert v(x) \vert \leq1 \mbox{ and } h(x)\leq1\}$, $\Omega_{2}=\mathbb{R}^{N}\setminus\Omega_{1}$. It is obvious that $\vert \Omega_{1} \vert =\infty$ and $\vert \Omega_{2} \vert <\infty$. Then, by (2.4) and (2.8), we have

$$\begin{aligned} & \int_{\Omega_{1}} \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \vert \tilde{u} \vert \,dx+ \int _{\Omega_{2}} \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \vert \tilde{u} \vert \,dx \\ &\quad = \int_{\mathbb{R}^{N}} \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \vert \tilde {u} \vert \chi\{\Omega_{1}\}\,dx \\ &\qquad{}+ \int_{\mathbb{R}^{N}} \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \vert \tilde{u} \vert \chi\{ \Omega_{2}\}\,dx \\ &\quad \leq 2 \bigl\Vert \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \chi\{\Omega_{1}\} \bigr\Vert _{\widetilde{\Phi}_{1}} \Vert \tilde{u} \Vert _{\Phi_{1}} \\ &\qquad{}+2 \bigl\Vert \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \chi\{\Omega_{2}\} \bigr\Vert _{\widetilde {\Phi}_{1*}} \Vert \tilde{u} \Vert _{\Phi_{1*}} \\ &\quad \leq 2(1+C_{\Phi_{1*}}) \bigl( \bigl\Vert \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \chi \{\Omega_{1}\} \bigr\Vert _{\widetilde{\Phi}_{1}} \\ & \qquad + \bigl\Vert \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \chi\{\Omega_{2}\} \bigr\Vert _{\widetilde{\Phi}_{1*}} \bigr) \bigl\Vert (\tilde{u},\tilde{v}) \bigr\Vert , \end{aligned}$$

where χ denotes the characteristic function. Then, to get (A.14), by (2.3) it is sufficient to prove

$$\begin{aligned} & \int_{\mathbb{R}^{N}}\widetilde{\Phi }_{1}\bigl( \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \chi\{\Omega_{1}\}\bigr)\,dx \\ &\qquad{}+ \int_{\mathbb {R}^{N}}\widetilde{\Phi}_{1*}\bigl( \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \chi\{ \Omega_{2}\}\bigr)\,dx \\ &\quad = \int_{\Omega_{1}}\widetilde{\Phi}_{1}\bigl( \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \bigr)\,dx \\ &\qquad{}+ \int_{\Omega_{2}}\widetilde{\Phi }_{1*}\bigl( \bigl\vert F_{u}(x,u_{ni},v_{ni})-F_{u}(x,u,v) \bigr\vert \bigr)\,dx= o_{i}(1). \end{aligned}$$

(A.16)

By (A.11), the continuity of $F_{u}$, $\widetilde{\Phi}_{1}$, $\widetilde{\Phi}_{1*}$ and the fact $\widetilde{\Phi}_{1}(0)=\widetilde {\Phi}_{1*}(0)=0$, we have

$$\begin{aligned} \widetilde{\Phi }_{1}\bigl( \bigl\vert F_{u}\bigl(x,u_{ni}(x),v_{ni}(x) \bigr)-F_{u}\bigl(x,u(x),v(x)\bigr) \bigr\vert \bigr)\rightarrow0,\quad \mbox{a.e. } x\in\Omega_{1} \end{aligned}$$

(A.17)

and

$$\begin{aligned} \widetilde{\Phi }_{1*}\bigl( \bigl\vert F_{u}\bigl(x,u_{ni}(x),v_{ni}(x) \bigr)-F_{u}\bigl(x,u(x),v(x)\bigr) \bigr\vert \bigr)\rightarrow0,\quad \mbox{a.e. } x\in\Omega_{2}. \end{aligned}$$

(A.18)

By (A.12) we have

$$\Phi_{1}\bigl( \bigl\vert u_{ni}(x) \bigr\vert \bigr) \leq h(x)\leq1, \qquad\Phi_{2}\bigl( \bigl\vert v_{ni}(x) \bigr\vert \bigr)\leq h(x)\leq1 \quad\mbox{for all } i\in\mathbb{N},\mbox{ a.e. } x\in \Omega_{1}, $$

which, together with the monotonicity of $\Phi_{1}$ and $\Phi_{2}$, implies that

$$\bigl\vert u_{ni}(x) \bigr\vert \leq\Phi_{1}^{-1}(1),\qquad \bigl\vert v_{ni}(x) \bigr\vert \leq\Phi_{2}^{-1}(1) \quad\mbox{for all } i\in\mathbb{N},\mbox{ a.e. } x\in\Omega_{1}. $$

Then, by $(F_{2})$, there exists a constant $M_{7}>0$ such that

$$\bigl\vert F_{u}\bigl(x,u_{ni}(x),v_{ni}(x) \bigr) \bigr\vert \leq M_{7} \bigl(\phi _{1}\bigl( \bigl\vert u_{ni}(x) \bigr\vert \bigr)+{\widetilde{\Phi}_{1}}^{-1} \bigl(\Phi_{2}\bigl( \bigl\vert v_{ni}(x) \bigr\vert \bigr)\bigr) \bigr)\quad \mbox{for all } i\in\mathbb{N},\mbox{ a.e. } x\in \Omega_{1} $$

and

$$\bigl\vert F_{u}\bigl(x,u(x),v(x)\bigr) \bigr\vert \leq M_{7} \bigl(\phi_{1}\bigl( \bigl\vert u(x) \bigr\vert \bigr)+{\widetilde{\Phi }_{1}}^{-1}\bigl(\Phi_{2} \bigl( \bigl\vert v(x) \bigr\vert \bigr)\bigr) \bigr)\quad \mbox{for all } x\in \Omega_{1}. $$

Then, by the monotonicity and convexity of $\widetilde{\Phi}_{1}$, the fact that $\widetilde{\Phi}_{1}$ satisfies the $\Delta_{2}$-condition globally, (A.12) and (2.2), for all $i\in\mathbb{N}$, a.e. $x\in\Omega_{1}$, we have

$$\begin{aligned} &\widetilde{\Phi }_{1}\bigl( \bigl\vert F_{u}\bigl(x,u_{ni}(x),v_{ni}(x) \bigr)-F_{u}\bigl(x,u(x),v(x)\bigr) \bigr\vert \bigr) \\ &\quad \le \widetilde{\Phi }_{1}\bigl( \bigl\vert F_{u} \bigl(x,u_{ni}(x),v_{ni}(x)\bigr) \bigr\vert + \bigl\vert F_{u}\bigl(x,u(x),v(x)\bigr) \bigr\vert \bigr) \\ &\quad \le \widetilde{\Phi}_{1} \bigl[M_{7} \bigl( \phi_{1}\bigl( \bigl\vert u_{ni}(x) \bigr\vert \bigr)+{ \widetilde {\Phi}_{1}}^{-1}\bigl(\Phi_{2}\bigl( \bigl\vert v_{ni}(x) \bigr\vert \bigr)\bigr)+\phi_{1} \bigl( \bigl\vert u(x) \bigr\vert \bigr) +{\widetilde{\Phi}_{1}}^{-1} \bigl(\Phi_{2}\bigl( \bigl\vert v(x) \bigr\vert \bigr)\bigr) \bigr) \bigr] \\ &\quad \le C \bigl(\widetilde{\Phi}_{1}\bigl(\phi_{1}\bigl( \bigl\vert u_{ni}(x) \bigr\vert \bigr)\bigr)+\Phi _{2}\bigl( \bigl\vert v_{ni}(x) \bigr\vert \bigr)+\widetilde{ \Phi}_{1}\bigl(\phi_{1}\bigl( \bigl\vert u(x) \bigr\vert \bigr)\bigr)+\Phi_{2}\bigl( \bigl\vert v(x) \bigr\vert \bigr) \bigr) \\ &\quad \le C \bigl(h(x)+\Phi_{1}\bigl(2 \bigl\vert u(x) \bigr\vert \bigr)+\Phi_{2}\bigl( \bigl\vert v(x) \bigr\vert \bigr) \bigr)=: g_{2}(x)\in L^{1}(\Omega_{1}), \end{aligned}$$

(A.19)

where C is a positive constant. Moreover, by $(F_{2})$, there exists a constant $M_{8}>0$ such that

$$\begin{aligned} & \bigl\vert F_{u}\bigl(x,u_{ni}(x),v_{ni}(x) \bigr) \bigr\vert \leq M_{8}+\Phi_{1*}' \bigl( \bigl\vert u_{ni}(x) \bigr\vert \bigr)+{\widetilde { \Phi}_{1*}}^{-1}\bigl(\Phi_{2*}\bigl( \bigl\vert v_{ni}(x) \bigr\vert \bigr)\bigr)\\ &\quad \mbox{for all } i\in\mathbb {N}, \mbox{ a.e. } x\in\Omega_{2} \end{aligned}$$

and

$$\bigl\vert F_{u}\bigl(x,u(x),v(x)\bigr) \bigr\vert \leq M_{8}+\Phi_{1*}'\bigl( \bigl\vert u(x) \bigr\vert \bigr)+{\widetilde{\Phi }_{1*}}^{-1}\bigl( \Phi_{2*}\bigl( \bigl\vert v(x) \bigr\vert \bigr)\bigr) \quad\mbox{for all } x\in\Omega_{2}. $$

Then, by the monotonicity and convexity of $\widetilde{\Phi}_{1*}$, the fact that $\widetilde{\Phi}_{1*}$ satisfies the $\Delta_{2}$-condition globally, (A.12) and (2.2), for all $i\in\mathbb{N}$, a.e. $x\in\Omega_{2}$, we have

$$\begin{aligned} &\widetilde{\Phi }_{1*}\bigl( \bigl\vert F_{u}\bigl(x,u_{ni}(x),v_{ni}(x) \bigr)-F_{u}\bigl(x,u(x),v(x)\bigr) \bigr\vert \bigr) \\ &\quad \le\widetilde{\Phi }_{1*}\bigl( \bigl\vert F_{u} \bigl(x,u_{ni}(x),v_{ni}(x)\bigr) \bigr\vert + \bigl\vert F_{u}\bigl(x,u(x),v(x)\bigr) \bigr\vert \bigr) \\ &\quad \le\widetilde{\Phi}_{1*} \bigl(2M_{8}+\Phi _{1*}'\bigl( \bigl\vert u_{ni}(x) \bigr\vert \bigr)+{\widetilde{\Phi}_{1*}}^{-1}\bigl(\Phi _{2*}\bigl( \bigl\vert v_{ni}(x) \bigr\vert \bigr) \bigr)+\Phi_{1*}'\bigl( \bigl\vert u(x) \bigr\vert \bigr) +{\widetilde{\Phi}_{1*}}^{-1}\bigl( \Phi_{2*}\bigl( \bigl\vert v(x) \bigr\vert \bigr)\bigr) \bigr) \\ &\quad \le C \bigl(1+\widetilde{\Phi}_{1*}\bigl(\Phi_{1*}' \bigl( \bigl\vert u_{ni}(x) \bigr\vert \bigr)\bigr)+\Phi _{2*}\bigl( \bigl\vert v_{ni}(x) \bigr\vert \bigr)+ \widetilde{\Phi}_{1*}\bigl(\Phi_{1*}'\bigl( \bigl\vert u(x) \bigr\vert \bigr)\bigr)+\Phi _{2*}\bigl( \bigl\vert v(x) \bigr\vert \bigr) \bigr) \\ &\quad \le C \bigl(1+h(x)+\Phi_{1*}\bigl(2 \bigl\vert u(x) \bigr\vert \bigr)+\Phi_{2*}\bigl( \bigl\vert v(x) \bigr\vert \bigr) \bigr)=: g_{3}(x)\in L^{1}(\Omega_{2}), \end{aligned}$$

(A.20)

where C is a positive constant. Combining (A.17)-(A.20) with Lebesgue’s dominated convergence theorem, we can conclude that (A.16) holds. Then (A.14) holds. Similarly, we can obtain that

$$\begin{aligned} \int_{\mathbb{R}^{N}} \bigl\vert F_{v}(x,u_{ni},v_{ni})-F_{v}(x,u,v) \bigr\vert \vert \tilde {v} \vert \,dx=o_{i}(1) \bigl\Vert ( \tilde{u}, \tilde{v}) \bigr\Vert . \end{aligned}$$

(A.21)

Therefore, combining (A.14) and (A.21) with (A.13), we can conclude that $I_{2}'(u_{ui},v_{ni})\rightarrow I_{2}'(u_{,}v)$ in $W^{*}$, which contradicts (A.10). □

Lemma A.3

Assume that $(\phi_{1})$, $(\phi_{2})$, $(V_{2})$, $(F_{1})$ and $(F_{2})$ hold. Then

$$\bigl\langle I'(u,v),(u,v)\bigr\rangle =\bigl\langle I_{1}'(u,v),(u,v)\bigr\rangle -o\bigl(\bigl\langle I_{1}'(u,v),(u,v)\bigr\rangle \bigr) \quad\textit{as } \bigl\Vert (u,v) \bigr\Vert \rightarrow0. $$

Proof

Since $\langle I'(u,v),(u,v)\rangle=\langle I_{1}'(u,v),(u,v)\rangle-\langle I_{2}'(u,v),(u,v)\rangle$ and $\langle I_{i}'(u,v),(u,v)\rangle=o(1)(i=1,2)$ as $\Vert (u,v) \Vert \rightarrow0$, we need to prove $\langle I_{2}'(u,v),(u,v)\rangle=o(\langle I_{1}'(u,v),(u,v)\rangle)$ as $\Vert (u,v) \Vert \rightarrow0$. By $(F_{1})$, $(F_{2})$, $(\phi_{2})$, 1) in Lemma 2.4 and (2.1), for any given constant $\varepsilon>0$, there exists a constant $C_{\varepsilon}>0$ such that

$$\bigl\vert uF_{u}(x,u,v) \bigr\vert + \bigl\vert vF_{v}(x,u,v) \bigr\vert \leq\varepsilon \bigl(\Phi_{1} \bigl( \vert u \vert \bigr)+\Phi _{2}\bigl( \vert v \vert \bigr) \bigr)+C_{\varepsilon}\bigl(\Phi_{1*}\bigl( \vert u \vert \bigr)+\Phi_{2*}\bigl( \vert v \vert \bigr) \bigr) $$

for all $(x,u,v)\in\mathbb{R}^{N}\times\mathbb{R}\times\mathbb{R}$. Then, by (A.4), 3) in Lemma 2.4, (2.8) and (2.7), we have

$$\begin{aligned} & \bigl\vert \bigl\langle I_{2}'(u,v),(u,v) \bigr\rangle \bigr\vert \\ &\quad\leq \int_{\mathbb {R}^{N}}\bigl( \bigl\vert F_{u}(x,u,v) \bigr\vert \vert u \vert + \bigl\vert F_{v}(x,u,v) \bigr\vert \vert v \vert \bigr)\,dx \\ &\quad \leq \varepsilon \int_{\mathbb{R}^{N}} \bigl(\Phi_{1}\bigl( \vert u \vert \bigr)+\Phi _{2}\bigl( \vert v \vert \bigr) \bigr)\,dx +C_{\varepsilon} \int_{\mathbb{R}^{N}} \bigl(\Phi_{1*}\bigl( \vert u \vert \bigr)+\Phi _{2*}\bigl( \vert v \vert \bigr) \bigr)\,dx \\ &\quad \leq \varepsilon \int_{\mathbb{R}^{N}} \bigl(\Phi_{1}\bigl( \vert u \vert \bigr)+\Phi _{2}\bigl( \vert v \vert \bigr) \bigr)\,dx +C_{\varepsilon} \bigl( \Vert u \Vert _{\Phi_{1*}}^{l_{1}^{*}}+ \Vert u \Vert _{\Phi_{1*}}^{m_{1}^{*}} + \Vert v \Vert _{\Phi_{2*}}^{l_{2}^{*}}+ \Vert v \Vert _{\Phi_{2*}}^{m_{2}^{*}} \bigr) \\ &\quad \leq \varepsilon \int_{\mathbb{R}^{N}} \bigl(\Phi_{1}\bigl( \vert u \vert \bigr)+\Phi _{2}\bigl( \vert v \vert \bigr) \bigr)\,dx +C \bigl( \Vert u \Vert _{1,\Phi_{1}}^{l_{1}^{*}}+ \Vert u \Vert _{1,\Phi_{1}}^{m_{1}^{*}}+ \Vert v \Vert _{1,\Phi_{2}}^{l_{2}^{*}}+ \Vert v \Vert _{1,\Phi_{2}}^{m_{2}^{*}} \bigr), \end{aligned}$$

(A.22)

where $C=C_{\varepsilon}\max\{C_{\Phi_{1*}}^{l_{1}^{*}},C_{\Phi _{1*}}^{m_{1}^{*}},C_{\Phi_{2*}}^{l_{2}^{*}},C_{\Phi_{2*}}^{m_{2}^{*}}\}$. Moreover, by (A.3), $(\phi_{2})$, $(V_{2})$ and Lemma 2.2, when $\Vert (u,v) \Vert = \Vert \nabla u \Vert _{\Phi_{1}}+ \Vert u \Vert _{\Phi_{1}}+ \Vert \nabla v \Vert _{\Phi_{2}}+ \Vert v \Vert _{\Phi_{2}}\leq1$, we have

$$\begin{aligned} & \bigl\langle I_{1}'(u,v),(u,v)\bigr\rangle \\ &\quad = \int_{\mathbb{R}^{N}}a_{1}\bigl( \vert \nabla u \vert \bigr) \vert \nabla u \vert ^{2}\,dx + \int_{\mathbb {R}^{N}}V_{1}(x)a_{1}\bigl( \vert u \vert \bigr) \vert u \vert ^{2}\,dx \\ &\qquad{}+ \int_{\mathbb{R}^{N}}a_{2}\bigl( \vert \nabla v \vert \bigr) \vert \nabla v \vert ^{2}\,dx+ \int_{\mathbb {R}^{N}}V_{2}(x)a_{2}\bigl( \vert v \vert \bigr) \vert v \vert ^{2}\,dx \\ &\quad \geq l_{1} \int_{\mathbb{R}^{N}}\Phi_{1}\bigl( \vert \nabla u \vert \bigr)\,dx+\alpha_{1}l_{1} \int _{\mathbb{R}^{N}}\Phi_{1}\bigl( \vert u \vert \bigr)\,dx \\ &\qquad{}+l_{2} \int_{\mathbb{R}^{N}}\Phi_{2}\bigl( \vert \nabla v \vert \bigr)\,dx+\alpha_{1}l_{2} \int_{\mathbb {R}^{N}}\Phi_{2}\bigl( \vert v \vert \bigr)\,dx \\ &\quad \geq \min\{l_{1},l_{2}\}\min\{1,\alpha_{1}\} \bigl( \Vert \nabla u \Vert _{\Phi _{1}}^{m_{1}}+ \Vert u \Vert _{\Phi_{1}}^{m_{1}}+ \Vert \nabla v \Vert _{\Phi_{2}}^{m_{2}}+ \Vert v \Vert _{\Phi _{2}}^{m_{2}} \bigr) \\ &\quad \geq\min\{l_{1},l_{2}\}\min\{1,\alpha_{1}\} \bigl(C_{m_{1}} \Vert u \Vert _{1,\Phi _{1}}^{m_{1}}+C_{m_{2}} \Vert v \Vert _{1,\Phi_{2}}^{m_{2}} \bigr). \end{aligned}$$

(A.23)

Then (A.22), (A.23) and the fact that $1< m_{i}< l_{i}^{*}\leq m_{i}^{*}(i=1,2)$ imply that

$$\begin{aligned} &\lim_{ \Vert (u,v) \Vert \rightarrow0}\frac{ \vert \langle I_{2}'(u,v),(u,v)\rangle \vert }{\langle I_{1}'(u,v),(u,v)\rangle} \\ &\quad \leq \lim _{ \Vert (u,v) \Vert \rightarrow0}\frac{\varepsilon\int_{\mathbb {R}^{N}}(\Phi_{1}( \vert u \vert ) +\Phi_{2}( \vert v \vert ))\,dx}{\alpha_{1}\min\{l_{1},l_{2}\}\int_{\mathbb{R}^{N}}(\Phi _{1}( \vert u \vert )+\Phi_{2}( \vert v \vert ))\,dx} \\ &\qquad{} +\lim_{ \Vert (u,v) \Vert \rightarrow0}\frac{C( \Vert u \Vert _{1,\Phi_{1}}^{l_{1}^{*}}+ \Vert u \Vert _{1,\Phi_{1}}^{m_{1}^{*}}+ \Vert v \Vert _{1,\Phi_{2}}^{l_{2}^{*}}+ \Vert v \Vert _{1,\Phi_{2}}^{m_{2}^{*}})}{ \min\{l_{1},l_{2}\}\min\{1,\alpha_{1}\}(C_{m_{1}} \Vert u \Vert _{1,\Phi _{1}}^{m_{1}}+C_{m_{2}} \Vert v \Vert _{1,\Phi_{2}}^{m_{2}})} \\ &\quad = \frac{\varepsilon}{\alpha_{1}\min\{l_{1},l_{2}\}}. \end{aligned}$$

Since ε is arbitrary, we conclude that $\vert \langle I_{2}'(u,v),(u,v)\rangle \vert =o(\langle I_{1}'(u,v),(u,v)\rangle)$ as $\Vert (u,v) \Vert \rightarrow0$. Hence, $\langle I_{2}'(u,v),(u,v)\rangle=o(\langle I_{1}'(u,v),(u,v)\rangle)$ as $\Vert (u,v) \Vert \rightarrow0$. □