On the existence of nontrivial solutions for quasilinear Schrödinger systems

Abstract

In this paper, by using a change of variable and the mountain-pass theorem, we show that the following quasilinear Schrödinger systems

$$ \textstyle\begin{cases} -\triangle u+V_{1}(x)u+\frac{\kappa}{2}[\triangle \vert u \vert ^{2}]u=\lambda f(x,u, v), & x\in \mathbb{R}^{N}, \\ -\triangle v+V_{2}(x)v+\frac{\kappa}{2}[\triangle \vert v \vert ^{2}]v=\lambda h(x,u, v), & x\in \mathbb{R}^{N} \end{cases} $$

have a nontrivial solution $(u, v)$ for all $\lambda >\lambda _{1}(\kappa )$, where $N\geq 3, V_{1}(x), V_{2}(x)$ are positive continuous functions, κ, λ are positive parameters, and nonlinear terms $f, h\in C(\mathbb{R}^{N}\times \mathbb{R}^{2}, \mathbb{R})$. Our main contribution is that we can deal with the case when $\kappa >0$ is large for the above systems.

1 Introduction

In this paper, we consider the following quasilinear Schrödinger systems of the form

$$ \textstyle\begin{cases} -\triangle u+V_{1}(x)u+\frac{\kappa}{2}[\triangle \vert u \vert ^{2}]u=\lambda f(x, u, v), & x\in \mathbb{R}^{N}, \\ -\triangle v+V_{2}(x)v+\frac{\kappa}{2}[\triangle \vert v \vert ^{2}]v=\lambda h(x, u, v), & x\in \mathbb{R}^{N}, \end{cases} $$

(1.1)

where $N\geq 3, V_{1}(x), V_{2}(x)$ are positive continuous functions, κ, λ are positive parameters and the nonlinear term $f, h\in C(\mathbb{R}^{N}\times \mathbb{R}^{2}, \mathbb{R})$.

Quasilinear Schrödinger systems like (1.1) are in part motivated by the following quasilinear Schrödinger equations

$$ iz_{t}=-\triangle z+W(x)z-k(x, z)+ \frac{\kappa}{2}\bigl[\triangle (l\bigl( \vert z \vert ^{2} \bigr)l'\bigl( \vert z \vert ^{2}\bigr)\bigr]z, \quad x\in \mathbb{R}^{N}, $$

(1.2)

where $z: \mathbb{R}\times \mathbb{R}^{N}\rightarrow \mathbb{C}$, $W: \mathbb{R}^{N}\rightarrow \mathbb{R}$ is a given potential, κ is a positive parameter and $l: \mathbb{R}\rightarrow \mathbb{R}$, $k: \mathbb{R}^{N}\times \mathbb{R}\rightarrow \mathbb{R}$ are continuous functions. The quasilinear Schrödinger equations (1.2) describe several physical phenomena with different l and k, see [3, 4, 10, 12, 14] and the references therein. In this paper, we are interested in the existence of standing-wave solutions for (1.2), that is, solutions of the form $z(t, x)=\exp (-iEt)u(x)$, where $E\in \mathbb{R}$ and u is a real function. Submitting $z(t,x)$ into (1.2) and denoting $k(x, z)=\zeta (z^{2})z$, we obtain

$$ -\triangle u+V(x)u+\frac{\kappa}{2}\bigl[\triangle \bigl(l \bigl( \vert u \vert ^{2}\bigr)\bigr)\bigr]l'\bigl( \vert u \vert ^{2}\bigr)u= \zeta \bigl(u^{2}\bigr)u, \quad x\in \mathbb{R}^{N}. $$

(1.3)

If we take $l(t)=t$, $\zeta (u^{2})u=\omega (u)$, then (1.3) can be reduced to the following equations:

$$ -\triangle u+V(x)u+\frac{\kappa}{2}\bigl[\triangle \vert u \vert ^{2}\bigr]u=\omega (u), \quad x\in \mathbb{R}^{N}. $$

(1.4)

Letting $\kappa \neq 0$, equations (1.4) are quasilinear Schrödinger equations and $[\triangle |u|^{2}]u$ is a quasilinear term. Compared to the semilinear case, quasilinear equations become more complicated due to the quasilinear and nonconvex term $[\triangle |u|^{2}]u$. One of the main difficulties of (1.4) is that the functional $\int _{\mathbb{R}^{N}}u^{2}|\nabla u|^{2}\,dx$ of the quasilinear term $[\triangle |u|^{2}]u$ is not smooth in usual Sobolev space $H^{1}(\mathbb{R}^{N})$. By using the change of variable (dual approach) $s=G^{-1}(t)$ for $t\in [0, +\infty )$, where

$$ G(s)= \int _{0}^{s}\sqrt{1-\kappa t^{2}} \,dt, $$

(1.5)

and $G^{-1}(t)=-G^{-1}(-t)$ for $t\in (-\infty , 0)$, quasilinear equations (1.4) can be reduced to the semilinear one

$$ -\triangle v+V(x)\frac{G^{-1}(v)}{g(G^{-1}(v))}= \frac{\omega (G^{-1}(v))}{g(G^{-1}(v))}, \quad x\in \mathbb{R}^{N}. $$

Then, an Orlicz-space framework can be used to prove the existence of nontrivial solutions via minimax methods. Since $1-\kappa t^{2}$ may be negative with $\kappa >0$, a change of variable (1.5) is not adequate to study the existence of nontrivial solutions for these quasilinear equations. Letting $1-\kappa t^{2}>0$, integral (1.5) makes sense and the inverse function $G^{-1}(t)$ exists. When $\kappa >0$ is small enough, many papers have studied the existence results of nontrivial solutions or multiple solutions for (1.4) via dual-approach techniques and variational methods, see [1, 2, 6, 7, 16, 18–20].

Moreover, when $\kappa >0$ is large, letting $1-\kappa t^{2}>0$, a change of variable (1.5) is also used to study the existence of solutions for quasilinear equations, but now the nonlinearity $\omega (u)$ needs to be multiplied by a large constant λ. For example, Huang and Jia [11] ($\kappa =2$), Li and Huang [13], and Liang, Gao and Li [15] obtained the existence of nontrivial solutions for the following equations

$$ -\triangle u+V(x)u+\frac{\kappa}{2}\bigl[\triangle \vert u \vert ^{2}\bigr]u=\lambda \omega (u), \quad x\in \mathbb{R}^{N}. $$

For quasilinear Schrödinger systems like (1.1), when $\kappa >0$ is small enough, Chen and Zhang proved the existence of a positive ground-state solution [8] and a nonradially symmetrical nodal solution [9] for the following quasilinear Schrödinger systems, respectively,

$$ \textstyle\begin{cases} -\triangle u+u+\frac{\kappa}{2}[\triangle \vert u \vert ^{2}]u= \frac{2\alpha}{\alpha +\beta} \vert u \vert ^{\alpha -2}u \vert v \vert ^{\beta}, & x\in \mathbb{R}^{N}, \\ -\triangle v+v+\frac{\kappa}{2}[\triangle \vert v \vert ^{2}]v= \frac{2\beta}{\alpha +\beta} \vert u \vert ^{\alpha} \vert v \vert ^{\beta -2}v, & x\in \mathbb{R}^{N}, \end{cases} $$

and

$$ \textstyle\begin{cases} -\triangle u+A(x)u+\frac{\kappa}{2}[\triangle \vert u \vert ^{2}]u= \frac{2\alpha}{\alpha +\beta} \vert u \vert ^{\alpha -2}u \vert v \vert ^{\beta}, & x\in \mathbb{R}^{N}, \\ -\triangle v+Bv+\frac{\kappa}{2}[\triangle \vert v \vert ^{2}]v= \frac{2\beta}{\alpha +\beta} \vert u \vert ^{\alpha} \vert v \vert ^{\beta -2}v, & x\in \mathbb{R}^{N}, \end{cases} $$

where $\alpha , \beta >1$, $2<\alpha +\beta <2^{*}$, potential function $A(x)$ is radially symmetric and $B>0$. However, these papers did not consider the existence of the nontrivial solutions for systems (1.1) when $\kappa >0$ is large. To the best of our knowledge, there are no results for this problem in the literature. In this paper, we will be concerned with this problem.

Throughout this paper, we assume that $V_{1}(x), V_{2}(x)\in C(\mathbb{R}^{N}, \mathbb{R})$ and satisfy the following conditions

($\mathcal{V}_{1}$):

$0< V_{0}:=\min \{\inf_{x\in \mathbb{R}^{N}}V_{1}(x), \inf_{x\in \mathbb{R}^{N}}V_{2}(x) \}$;

($\mathcal{V}_{2}$):

there exists $M_{0}>0$ such that for all $M\geq M_{0}$

$$ \mu \bigl(\bigl\{ x\in \mathbb{R}^{N}: V_{i}(x)\leq M \bigr\} \bigr)< +\infty ,\quad i=1, 2, $$

where μ denotes the Lebesgue measure in $\mathbb{R}^{N}$. Moreover, suppose the nonlinearities f, h satisfy the following conditions

• (h1) $\lim_{(s,t)\to (0, 0)}\frac{f(x, s, t)}{|(s, t)|}=\lim_{(s,t)\to (0, 0)}\frac{h(x, s, t)}{|(s, t)|}=0$;

• (h2) There is a constant $C>0$, such that $\langle \nabla \eta (x, s, t), (s, t)\rangle \leq C(|(s, t)|+|(s, t)|^{q})$, $\forall t\in \mathbb{R}$, $2< q<\frac{2^{*}+2}{2}$;

• (h3) There is $\theta \in (2, 2^{*})$ such that $0<\theta \eta (x, s, t)\leq (s, t)\nabla \eta (x, s, t)$ and $uf(x, u, v)\geq 0$, $vh(x, u, v)\geq 0$, where $\nabla \eta (x, s, t)=(f(x, s, t), h(x, s, t))$.

Our main results in this paper are as follows.

Theorem 1.1

Assume that ($\mathcal{V}_{1}$), ($\mathcal{V}_{2}$), and (h1)–(h3) hold. Then, for given $\kappa >0$, there exists $\lambda _{1}(\kappa )>0$ such that for all $\lambda >\lambda _{1}(\kappa )$, systems (1.1) have a nontrivial solution $(u, v)\in \mathcal{H}$ satisfying $\max_{x\in \mathbb{R}^{N}}|(u(x), v(x)|\leq \sqrt{ \frac{1}{2\kappa}}$.

Remark 1.1

In [17], Sever and Silva obtained the existence of nontrivial solutions for (1.1) with $\kappa =-2$, $\lambda =1$ under conditions (h1)–(h3). $f(x, u, v)=\frac{2\alpha}{\alpha +\beta}|u|^{\alpha -2}u|v|^{\beta}$, $h(x, u, v)=\frac{2\beta}{\alpha +\beta}|u|^{\alpha}|v|^{\beta -2}v$ satisfy (h1)–(h3), where $\alpha , \beta >1$.

The remainder of this paper is organized as follows. In Sect. 2, we give some preliminaries. In Sect. 3, we show the existence of a nontrivial solution $(z_{\kappa}, w_{\kappa})$ for the modified problem via the mountain-pass theorem. In Sect. 4, we use the Morse iteration technique to obtain $L^{\infty}$-estimate for $(z_{\kappa}, w_{\kappa})$ and finally we obtain the solutions for the original systems (1.1).

Throughout this paper, we use the standard notations. We use $\|\cdot \|_{q}$ ($1< q\leq \infty$) that is a standard norm in the usual Lebesgue space $L^{q}({\mathbb{R}^{N}})$. $o_{n}(1)$ will always denote the quantities tending to 0 as $n\to \infty$. ⇀ and → denote weak and strong convergence. $B_{R}(0)$ denotes a ball centered at the origin with radius $R>0$. $C, C_{0}, C_{1}, \ldots$ denote positive constants.

2 Preliminaries

The energy functional associated with (1.1) is

$$ \begin{aligned} I_{\kappa}(u, v)={}&\frac{1}{2} \int _{\mathbb{R}^{N}}\bigl(1-\kappa u^{2}\bigr) \vert \nabla u \vert ^{2}\,dx+\frac{1}{2} \int _{\mathbb{R}^{N}}V_{1}(x) \vert u \vert ^{2}\,dx \\ &{}+\frac{1}{2} \int _{\mathbb{R}^{N}}\bigl(1-\kappa v^{2}\bigr) \vert \nabla v \vert ^{2}\,dx+ \frac{1}{2} \int _{\mathbb{R}^{N}}V_{2}(x) \vert v \vert ^{2}\,dx-\lambda \int _{\mathbb{R}^{N}}\eta (x, u, v)\,dx, \end{aligned} $$

where $\nabla \eta (x, s, t)=(f(x, s, t), h(x, s, t))$. Since the terms $\int _{\mathbb{R}^{N}}u^{2}|\nabla u|^{2}\,dx$ and $\int _{\mathbb{R}^{N}}v^{2}|\nabla v|^{2}\,dx$ are not well defined in the usual Sobolev spaces, the functional $I_{\kappa}$ may not be smooth. Hence, we cannot directly apply variational methods to obtain the critical points of $I_{\kappa}$.

We define $\mathcal{H}=\mathcal{H}_{1}\times \mathcal{H}_{2}$ with the norm

$$ \bigl\Vert (u, v) \bigr\Vert ^{2}= \Vert u \Vert ^{2}+ \Vert v \Vert ^{2}, $$

where $\mathcal{H}_{i}, i=1, 2$ are Banach spaces and

$$ \mathcal{H}_{i}= \biggl\{ u\in H^{1}\bigl( \mathbb{R}^{N}\bigr): \int _{ \mathbb{R}^{N}}V_{i}(x)u^{2}\,dx< +\infty \biggr\} ,\quad i=1,2 $$

endowed with the norm

$$ \Vert u \Vert = \biggl[ \int _{\mathbb{R}^{N}}\bigl( \vert \nabla u \vert ^{2}+V_{i}(x)u^{2}\bigr)\,dx \biggr]^{1/2},\quad i=1,2, $$

$H^{1}(\mathbb{R}^{N})$ is the usual Sobolev space.

We say $(u, v): \mathbb{R}^{N}\times \mathbb{R}^{N}\to \mathbb{R}\times \mathbb{R}$ is a (weak) solution of (1.1) if $(u, v)\in \mathcal{H}$ and it holds that

$$ \begin{aligned} & \int _{\mathbb{R}^{N}}\bigl[\bigl(1-\kappa u^{2}\bigr)\nabla u\nabla \varphi +\bigl(1-\kappa v^{2}\bigr)\nabla v\nabla \psi \bigr] \,dx-\kappa \int _{\mathbb{R}^{N}} \bigl( \vert \nabla u \vert ^{2} u \varphi + \vert \nabla v \vert ^{2} v \psi \bigr) \,dx \\ &\quad + \int _{\mathbb{R}^{N}}\bigl[V_{1}(x)u\varphi +V_{2}(x)v\psi \bigr] \,dx= \lambda \int _{\mathbb{R}^{N}}\bigl[f(x, u, v)\varphi +h(x, u, v) \psi \bigr]\,dx \end{aligned} $$

for all $\varphi , \psi \in C^{\infty}_{0}(\mathbb{R}^{N})$.

In order to obtain nontrivial (weak) solutions of (1.1), we assume that $1-\kappa u^{2}>0$. Moreover, we define $g: \mathbb{R}\to \mathbb{R}^{+}$ as follows

$$ g(t)=\textstyle\begin{cases} g(-t), &t< 0, \\ \sqrt{1-\kappa t^{2}}, &0\leq t< \sqrt{\frac{1}{2\kappa}}, \\ \frac{\sqrt{2}}{8\kappa t^{2}}+\frac{\sqrt{2}}{4}, &t\geq \sqrt{ \frac{1}{2\kappa}}. \end{cases} $$

(2.1)

Then, $g\in C^{1} (\mathbb{R}, (\frac{\sqrt{2}}{4}, 1 ] )$, g is even, increasing in $(-\infty , 0)$ and decreasing in $[0, +\infty )$.

Motivated by [2], we consider the existence of nontrival solutions for the following modified quasilinear Schrödinger systems:

$$ \textstyle\begin{cases} -{\mathrm{div}}(g^{2}(u)\nabla u)+g(u)g'(u) \vert \nabla u \vert ^{2}+V_{1}(x)u= \lambda f(x, u, v),\quad x\in \mathbb{R}^{N}, \\ -{\mathrm{div}}(g^{2}(v)\nabla v)+g(v)g'(v) \vert \nabla v \vert ^{2}+V_{2}(x)v= \lambda h(x, u, v),\quad x\in \mathbb{R}^{N}, \end{cases} $$

(2.2)

where $g(t)$ is defined in (2.1). Also, we say $(u, v)$ is a (weak) solution of (2.2) if $(u, v)\in \mathcal{H}$ and

$$ \begin{aligned}[b] & \int _{\mathbb{R}^{N}}\bigl[g^{2}(u)\nabla u\nabla \varphi +g(u)g'(u) \vert \nabla u \vert ^{2}\varphi +g^{2}(v)\nabla v\nabla \psi +g(v)g'(v) \vert \nabla v \vert ^{2} \psi \bigr]\,dx \\ &\quad {}+ \int _{\mathbb{R}^{N}}\bigl[V_{1}(x)u\varphi +V_{2}(x)v\psi \bigr]\,dx= \lambda \int _{\mathbb{R}^{N}}\bigl[f(x, u, v)\varphi +h(x, u, v) \psi \bigr]\,dx \end{aligned} $$

(2.3)

for $\varphi , \psi \in C^{\infty}_{0}(\mathbb{R}^{N})$.

Clearly, if we obtain a solution u of (2.2) that satisfies $\|(u, v)\|_{\infty}<1/\sqrt{2\kappa}$, then $(u, v)$ is a solution of (1.1). By using the following change of variable

$$ z=G(u)= \int _{0}^{u}g(t)\,dt,\qquad w=G(v)= \int _{0}^{v}g(t)\,dt, $$

then we see that the problem (2.2) can be reduced to the following semilinear Schrödinger systems:

$$ \textstyle\begin{cases} -\triangle z+V_{1}(x)\frac{G^{-1}(z)}{g(G^{-1}(z))}=\lambda \frac{f(x, G^{-1}(z), G^{-1}(w))}{g(G^{-1}(z))}, \quad x\in \mathbb{R}^{N}, \\ -\triangle w+V_{2}(x)\frac{G^{-1}(w)}{g(G^{-1}(w))}=\lambda \frac{h(x, G^{-1}(z), G^{-1}(w))}{g(G^{-1}(w))}, \quad x\in \mathbb{R}^{N}, \end{cases} $$

(2.4)

where $G^{-1}(z)$, $G^{-1}(w)$ are the inverse of $G(u)$, $G(v)$. The energy functional associated with (2.4) is

$$ \begin{aligned}[b] J_{\kappa}(z, w)={}&I \bigl(G^{-1}(z), G^{-1}(w)\bigr) \\ ={}&\frac{1}{2} \int _{ \mathbb{R}^{N}} \vert \nabla z \vert ^{2}\,dx+ \frac{1}{2} \int _{\mathbb{R}^{N}}V_{1}(x) \bigl\vert G^{-1}(z) \bigr\vert ^{2}\,dx \\ &{} +\frac{1}{2} \int _{\mathbb{R}^{N}} \vert \nabla w \vert ^{2}\,dx+ \frac{1}{2} \int _{\mathbb{R}^{N}}V_{2}(x) \bigl\vert G^{-1}(w) \bigr\vert ^{2}\,dx\\ &{} - \lambda \int _{\mathbb{R}^{N}}\eta \bigl(x, G^{-1}(z), G^{-1}(w)\bigr)\,dx. \end{aligned} $$

(2.5)

It is easy to prove that $J_{\kappa}$ is well defined in $\mathcal{H}$ and $J_{\kappa}\in C^{1}(\mathcal{H}, \mathbb{R})$ under our assumptions and the following lemma (cf. [19, Lemma 2.1]).

Lemma 2.1

The functions $g(t)$, $G(t)$ enjoy the following properties:

• (i) G is inverse, $G(t)$ and the inverse $G^{-1}(t)$ are odd;

• (ii) $-1\leq \frac{t}{g(t)}g'(t)\leq 0$ for all $t\in \mathbb{R}$;

• (iii) $|t|\leq |G^{-1}(t)|\leq 2\sqrt{2}|t|$ for all $t\in \mathbb{R}$;

• (iv) $\lim_{t\rightarrow 0}\frac{G^{-1}(t)}{t}=1$, $\lim_{t \rightarrow \infty}\frac{G^{-1}(t)}{t}=2\sqrt{2}$;

• (v) $g(G^{-1}(t))\leq \frac{t}{G^{-1}(t)}$ for all $t\in \mathbb{R}$.

The following lemma shows that any critical point $(z, w)\in \mathcal{H}$ of $J_{\kappa}$ is a (weak) solution of (2.2).

Lemma 2.2

Assume that $(\mathcal{V}_{1})$, $(\mathcal{V}_{2})$, and (h1)–(h3) hold. If $(z, w)\in \mathcal{H}$ is a critical point of $J_{\kappa}$, then $(u, v)=(G^{-1}(z), G^{-1}(w))$ is a weak solution of (2.2).

Proof

Since $(z, w)\in \mathcal{H}$ is a critical point of $J_{\kappa}$, we have

$$ \begin{aligned}[b] & \int _{\mathbb{R}^{N}}\nabla z\nabla \varphi \,dx+ \int _{\mathbb{R}^{N}}V_{1}(x)\frac{G^{-1}(z)}{g(G^{-1}(z))} \varphi \,dx\\ &\qquad {}+ \int _{\mathbb{R}^{N}}\nabla w\nabla \psi \,dx+ \int _{\mathbb{R}^{N}}V_{2}(x)\frac{G^{-1}(w)}{g(G^{-1}(w))}\psi \,dx \\ &\quad =\lambda \int _{\mathbb{R}^{N}} \biggl[ \frac{f(x, G^{-1}(z), G^{-1}(w))}{g(G^{-1}(z))}\varphi + \frac{h(x, G^{-1}(z), G^{-1}(w))}{g(G^{-1}(w))}\psi \biggr] \,dx, \end{aligned} $$

(2.6)

for all $(\varphi , \psi )\in \mathcal{H}$. It also implies from $(\mathcal{V}_{1})$ and Lemma 2.1(iii) that $(u, v):=(G^{-1}(z), G^{-1}(w))\in \mathcal{H}$. For each $\varphi _{1}, \psi _{1}\in C^{\infty}_{0}(\mathbb{R}^{N})$ and taking $(\varphi , \psi ):=(g(u)\varphi _{1}, g(v)\psi _{1})\in \mathcal{H}$ in (2.6), we obtain that

$$ \begin{aligned} &\int _{\mathbb{R}^{N}}\nabla z\bigl(g'(u)\varphi _{1}\nabla u+g(u) \nabla \varphi _{1}\bigr)\,dx+ \int _{\mathbb{R}^{N}}\nabla w\bigl(g'(v) \psi _{1}\nabla v+g(v)\nabla \psi _{1}\bigr)\,dx \\ &\qquad{} + \int _{\mathbb{R}^{N}}V_{1}(x)\frac{u}{g(u)}g(u) \varphi _{1} \,dx+ \int _{\mathbb{R}^{N}}V_{2}(x)\frac{v}{g(v)}g(v) \psi _{1} \,dx \\ &\quad =\lambda \int _{\mathbb{R}^{N}}\frac{f(x, u, v)}{g(u)}g(u) \varphi _{1} \,dx+\lambda \int _{\mathbb{R}^{N}} \frac{h(x, u, v)}{g(v)}g(v)\psi _{1} \,dx. \end{aligned} $$

Note that $z=G(u)$, $w=G(v)$ and $\nabla z=g(u)\nabla u$, $\nabla w=g(v)\nabla v$, then we obtain (2.3). Therefore, $(u, v)$ is a weak solution of (2.2). □

Denote $[L^{r}(\mathbb{R}^{N}) ]^{2}=L^{r}(\mathbb{R}^{N}) \times L^{r}(\mathbb{R}^{N})$ with the norm

$$ \bigl\Vert (u, v) \bigr\Vert _{r}= \bigl( \Vert u \Vert ^{2}_{r}+ \Vert v \Vert ^{2}_{r} \bigr)^{1/2},\quad 1 \leq r< +\infty , $$

where $L^{r}(\mathbb{R}^{N})$ is the Lebesgue function space with the norm

$$ \Vert u \Vert _{r}= \biggl( \int _{\mathbb{R}^{N}} \vert u \vert ^{r}\,dx \biggr)^{1/r},\quad 1\leq r< +\infty . $$

Now, we state the Sobolev embedding Lemma.

Lemma 2.3

Assume that $(\mathcal{V}_{1})$ and $(\mathcal{V}_{2})$ hold. Let $\{z_{n}\}$ and $\{w_{n}\}$ be bounded in $\mathcal{H}$. Then, there exist $z, w\in \mathcal{H}\cap L^{r}(\mathbb{R}^{N})$ such that up to a subsequence, $z_{n}\to z$, $w_{n}\to w$ in $L^{r}(\mathbb{R}^{N}), r\in [2, 2^{*})$.

Proof

It is analogous to the proof of [5]. □

3 The modified systems

In this section, we shall prove the existence of nontrivial solutions for the modified systems (2.4) via the mountain-pass theorem.

Lemma 3.1

Assume that (h1)–(h3) hold, then

• (i) there exist $\rho , \alpha >0$ such that $J_{\kappa}(z, w)\geq \alpha$ for all $(z, w)$ satisfying $\|(z, w)\|=\rho$;

• (ii) there is $(z, w)\in \mathcal{H}\setminus \{(0, 0)\}$ such that $J_{\kappa}(z, w)\leq 0$.

Proof

(i) By (h1) and (h2), for each $\varepsilon >0$ there exists a constant $C=C(\varepsilon )>0$ such that

$$ \bigl\vert \bigl\langle \nabla \eta (x, s, t), (s, t)\bigr\rangle \bigr\vert = \bigl\vert f(x, s, t)s+h(x, s, t)t \bigr\vert \leq \varepsilon \bigl\vert (s, t) \bigr\vert +C \bigl\vert (s, t) \bigr\vert ^{q-1}, $$

(3.1)

where $2< q<2^{*}$. Then,

$$ \eta (x, s, t)\leq \frac{\varepsilon}{2} \bigl\vert (s, t) \bigr\vert ^{2}+\frac{C}{q} \bigl\vert (s, t) \bigr\vert ^{q}. $$

(3.2)

Letting $\varepsilon =\min \{\frac{V_{1}(x)}{16\lambda}, \frac{V_{2}(x)}{16\lambda} \}$, it follows from Lemma 2.1 (iii), (3.2), and Lemma 2.3 that there exists a constant $C_{r}>0$ such that

$$ \begin{aligned} J_{\kappa}(z, w)\geq {}&\frac{1}{2} \int _{\mathbb{R}^{N}}\bigl[ \vert \nabla z \vert ^{2}+V_{1}(x) \vert v \vert ^{2}\bigr] \,dx+\frac{1}{2} \int _{\mathbb{R}^{N}}\bigl[ \vert \nabla w \vert ^{2}+V_{2}(x) \vert w \vert ^{2}\bigr] \,dx \\ &{} - 4\lambda \varepsilon \int _{\mathbb{R}^{N}} \bigl[ \vert z \vert ^{2}+ \vert w \vert ^{2} \bigr]\,dx - (2\sqrt{2} )^{q} \frac{C}{q}\lambda \int _{\mathbb{R}^{N}}\bigl[ \vert z \vert ^{q}+ \vert w \vert ^{q}\bigr]\,dx \\ \geq {}&\frac{1}{4} \bigl[ \Vert z \Vert ^{2}+ \Vert w \Vert ^{2} \bigr]- (2\sqrt{2} )^{q} \frac{C}{q}\lambda \int _{\mathbb{R}^{N}}\bigl[ \vert z \vert ^{q}+ \vert w \vert ^{q}\bigr]\,dx \\ \geq{}& \frac{1}{4} \bigl\Vert (z, w) \bigr\Vert ^{2}- (2\sqrt{2} )^{q}\frac{C}{q}C^{q}_{q} \lambda \bigl\Vert (z, w) \bigr\Vert ^{q}, \end{aligned} $$

hence, we may choose $\|(z, w)\|=\rho$ so small that

$$ J_{\kappa , \lambda}(z, w)\geq \alpha :=\frac{1}{4}\rho ^{2}- (2 \sqrt{2} )^{q}\frac{C}{q}C^{q}_{q} \lambda \rho ^{q}>0. $$

(ii) Choose $(\tau _{1}, \tau _{2})\in \mathcal{H}$ with $\tau _{1}, \tau _{2}>0$. Then, by Lemma 2.1 (iii) we have $|\tau _{i}|^{2}\leq \frac{|G^{-1}(t\tau _{i})|^{2}}{t^{2}}\leq 8| \tau _{i}|^{2}$, $i=1, 2$. It follows from (h3) that $\lim_{|(\nu _{1}, \nu _{2})|\to +\infty} \frac{\eta (x, \nu _{1}, \nu _{2})}{|(\nu _{1}, \nu _{2})|^{2}}=+ \infty$. Hence, we have

$$ \begin{aligned} \frac{J_{\kappa}(t\tau _{1}, t\tau _{2})}{t^{2}}\leq{}& \frac{1}{2} \int _{\mathbb{R}^{N}} \vert \nabla \tau _{1} \vert ^{2}\,dx+4 \int _{ \mathbb{R}^{N}}V_{1}(x) \vert \tau _{1} \vert ^{2}\,dx\\ &{}+ \int _{\mathbb{R}^{N}} \vert \nabla \tau _{2} \vert ^{2}\,dx+4 \int _{\mathbb{R}^{N}}V_{2}(x) \vert \tau _{2} \vert ^{2}\,dx \\ &{} -\lambda \int _{\mathbb{R}^{N}} \frac{\eta (x, G^{-1}(t\tau _{1}), G^{-1}(t\tau _{2}))}{t^{2}}\,dx \\ ={}&\frac{1}{2} \int _{\mathbb{R}^{N}} \vert \nabla \tau _{1} \vert ^{2}\,dx+4 \int _{\mathbb{R}^{N}}V_{1}(x) \vert \tau _{1} \vert ^{2}\,dx\\ &{}+ \int _{ \mathbb{R}^{N}} \vert \nabla \tau _{2} \vert ^{2}\,dx+4 \int _{\mathbb{R}^{N}}V_{2}(x) \vert \tau _{2} \vert ^{2}\,dx \\ &{} -\lambda \int _{\mathbb{R}^{N}} \frac{\eta (x, G^{-1}(t\tau _{1}), G^{-1}(t\tau _{2}))}{ \vert (G^{-1}(t\tau _{1}), G^{-1}(t\tau _{2})) \vert ^{2}} \frac{ \vert (G^{-1}(t\tau _{1}), G^{-1}(t\tau _{2})) \vert ^{2}}{t^{2}}\,dx \\ \to{}& {-}\infty \quad \text{as } t\to +\infty , \end{aligned} $$

therefore, there is a sufficiently large $t_{0}>0$; let $(z, w)=(t_{0}\tau _{1}, t_{0}\tau _{2})$ with $\|(z, w)\|>\rho$ such that $J_{\kappa}(z, w)\leq 0$. □

It follows from Lemma 3.1 that $J_{\kappa}$ has a (PS)_c sequence $\{(z_{n}, w_{n})\}\subset \mathcal{H}$ such that

$$ J_{\kappa}(z_{n}, w_{n}) \rightarrow c,\qquad J'_{\kappa}(z_{n}, w_{n}) \to 0\quad \text{as } n\rightarrow \infty , $$

(3.3)

where

$$\begin{aligned}& c=\inf_{\gamma \in \Gamma}\sup_{t\in [0,1]} J_{\kappa}(z_{t}, w_{t}), \\& \Gamma =\bigl\{ (z_{t}, w_{t})\in C\bigl([0,1]\times [0, 1], \mathcal{H}\bigr):(z_{0}, w_{0})=(0, 0), (z_{1}, w_{1})\neq (0, 0), J_{\kappa}(z_{1}, w_{1})< 0 \bigr\} . \end{aligned}$$

Lemma 3.2

Assume that (h3) holds, then any (PS)_c sequence $\{(z_{n}, w_{n})\}\subset \mathcal{H}$ obtained in (3.3) is bounded.

Proof

By (3.3), Lemma 2.1(ii), (iii), and (h3), one has

$$ \begin{aligned}[b] &c+1+o_{n}(1) \bigl\Vert (z_{n}, w_{n}) \bigr\Vert \\ &\quad \geq J_{\kappa}(z_{n}, w_{n})- \frac{1}{\theta}\bigl\langle J'_{\kappa}(z_{n}, w_{n}), \bigl(G^{-1}(z_{n})g \bigl(G^{-1}(z_{n})\bigr), G^{-1}(w_{n})g \bigl(G^{-1}(w_{n})\bigr)\bigr) \bigr\rangle \\ &\quad = \biggl(\frac{1}{2}-\frac{1}{\theta} \biggr) \int _{ \mathbb{R}^{N}} \vert \nabla z_{n} \vert ^{2}\,dx-\frac{1}{\theta} \int _{ \mathbb{R}^{N}}\frac{G^{-1}(z_{n})}{g(G^{-1}(z_{n}))}g' \bigl(G^{-1}(z_{n})\bigr) \vert \nabla z_{n} \vert ^{2}\,dx \\ &\qquad {}+ \biggl(\frac{1}{2}-\frac{1}{\theta} \biggr) \int _{ \mathbb{R}^{N}}V_{1}(x) \bigl\vert G^{-1}(z_{n}) \bigr\vert ^{2}\,dx \\ &\qquad {}+ \biggl(\frac{1}{2}-\frac{1}{\theta} \biggr) \int _{ \mathbb{R}^{N}} \vert \nabla w_{n} \vert ^{2}\,dx-\frac{1}{\theta} \int _{ \mathbb{R}^{N}}\frac{G^{-1}(w_{n})}{g(G^{-1}(w_{n}))}g' \bigl(G^{-1}(w_{n})\bigr) \vert \nabla w_{n} \vert ^{2}\,dx \\ &\qquad {}+ \biggl(\frac{1}{2}-\frac{1}{\theta} \biggr) \int _{ \mathbb{R}^{N}}V_{2}(x) \bigl\vert G^{-1}(w_{n}) \bigr\vert ^{2}\,dx \\ &\qquad {}+\lambda \int _{\mathbb{R}^{N}} \biggl[\frac{1}{\theta} \bigl\langle \nabla \eta \bigl(x, G^{-1}(z_{n}), G^{-1}(w_{n}) \bigr), \bigl(G^{-1}(z_{n})g\bigl(G^{-1}(z_{n}) \bigr), G^{-1}(w_{n})g\bigl(G^{-1}(w_{n}) \bigr)\bigr)\bigr\rangle \\ &\qquad {}-\eta \bigl(x, \bigl(G^{-1}(z_{n})\bigr), G^{-1}(w_{n})\bigr) \biggr]\,dx \\ &\quad \geq \biggl(\frac{1}{2}-\frac{1}{\theta} \biggr) \int _{ \mathbb{R}^{N}} \bigl[ \vert \nabla z_{n} \vert ^{2}+V_{1}(x) \vert z_{n} \vert ^{2}+ \vert \nabla w_{n} \vert ^{2}+V_{2}(x) \vert w_{n} \vert ^{2} \bigr]\,dx \\ &\quad = \biggl(\frac{1}{2}-\frac{1}{\theta} \biggr) \bigl\Vert (z_{n}, w_{n}) \bigr\Vert ^{2}, \end{aligned} $$

(3.4)

which implies that $\{(z_{n}, w_{n})\}\subset \mathcal{H}$ is bounded. □

Since the sequence $\{(z_{n}, w_{n})\}$ given by Lemma 3.2 is bounded in $\mathcal{H}$, there exists $(z, w)\in \mathcal{H}$ and a subsequence of $\{(z_{n}, w_{n})\}$, still denoted by $\{(z_{n}, w_{n})\}$, such that

$$ \begin{aligned} &(z_{n}, w_{n})\rightharpoonup (z, w)\quad \text{in } \mathcal{H}, \\ &(z_{n}, w_{n})\to (z, w) \quad \text{in } \bigl[L^{q}_{\mathrm{loc}}\bigl( \mathbb{R}^{N}\bigr) \bigr]^{2}, 2\leq q< 2^{*}, \\ &\bigl(z_{n}(x), w_{n}(x)\bigr)\to \bigl(z(x), w(x) \bigr) \quad \text{a.e. in } \mathbb{R}^{N}. \end{aligned} $$

(3.5)

Now, we denote the functional $J_{\kappa}$ given in (2.5) by

$$ \begin{aligned} J_{\kappa}(z_{n}, w_{n})={}&\frac{1}{2} \int _{\mathbb{R}^{N}} \vert \nabla z_{n} \vert ^{2}\,dx+\frac{1}{2} \int _{\mathbb{R}^{N}}V_{1}(x) \vert z_{n} \vert ^{2}\,dx \\ &{}+\frac{1}{2} \int _{\mathbb{R}^{N}} \vert \nabla w_{n} \vert ^{2}\,dx+ \frac{1}{2} \int _{\mathbb{R}^{N}}V_{2}(x) \vert w_{n} \vert ^{2}\,dx - \int _{\mathbb{R}^{N}}\xi (x, z_{n}, w_{n}), \end{aligned} $$

where

$$ \begin{aligned} \xi (x, z_{n}, w_{n})={}& \frac{1}{2}V_{1}(x) \bigl[ \vert z_{n} \vert ^{2}- \bigl\vert G^{-1}(z_{n}) \bigr\vert ^{2} \bigr] \\ &{} +\frac{1}{2}V_{2}(x) \bigl[ \vert w_{n} \vert ^{2}- \bigl\vert G^{-1}(w_{n}) \bigr\vert ^{2} \bigr]+\lambda \eta \bigl(x, G^{-1}(z_{n}), G^{-1}(w_{n})\bigr), \end{aligned} $$

and

$$ \begin{aligned} &\bigl\langle \nabla \xi (x, z_{n}, w_{n}), (z_{n}, w_{n})\bigr\rangle \\ &\quad =V_{1}(x) \biggl[ \vert z_{n} \vert ^{2}-\frac{G^{-1}(z_{n})}{g(G^{-1}(z_{n}))}z_{n} \biggr]+ \lambda \frac{f(x, G^{-1}(z_{n}), G^{-1}(w_{n}))}{g(G^{-1}(z_{n}))}z_{n} \\ &\qquad {} +V_{2}(x) \biggl[ \vert w_{n} \vert ^{2}- \frac{G^{-1}(w_{n})}{g(G^{-1}(w_{n}))}w_{n} \biggr]+\lambda \frac{h(x, G^{-1}(z_{n}), G^{-1}(w_{n}))}{g(G^{-1}(w_{n}))}w_{n}. \end{aligned} $$

Lemma 3.3

Assume that (h1), (h2), $(\mathcal{V}_{1})$, $\textit{and} (\mathcal{V}_{2})$ hold, $\{(z_{n}, w_{n})\}\subset \mathcal{H}$ is a (PS)_c sequence such that $(z_{n}, w_{n})\rightharpoonup (z, w)$ in $\mathcal{H}, n\to \infty$, then

$$\begin{aligned}& \lim_{n\to \infty} \int _{\mathbb{R}^{N}}\bigl\langle \nabla \xi (x, z_{n}, w_{n}), (z_{n}, w_{n})\bigr\rangle \,dx= \int _{ \mathbb{R}^{N}}\bigl\langle \nabla \xi (x, z, w), (z, w)\bigr\rangle \,dx, \end{aligned}$$

(3.6)

$$\begin{aligned}& \lim_{n\to \infty} \int _{\mathbb{R}^{N}}\bigl\langle \nabla \xi (x, z_{n}, w_{n}), (z, w)\bigr\rangle \,dx= \int _{ \mathbb{R}^{N}}\bigl\langle \nabla \xi (x, z, w), (z, w)\bigr\rangle \,dx. \end{aligned}$$

(3.7)

Proof

By Lemma 2.3, since $z_{n}\to z$, $w_{n}\to w$ in $L^{r}(\mathbb{R}^{N}), r\in [2, 2^{*})$, for every small $\varepsilon _{1}>0$, there exists $R_{1}>0$ such that

$$ \begin{aligned} &\int _{B_{R_{1}}^{C}} \vert z_{n} \vert ^{2}\,dx\leq \varepsilon _{1}, \qquad \int _{B_{R_{1}}^{C}} \vert z \vert ^{2}\,dx\leq \varepsilon _{1},\\ &\int _{B_{R_{1}}^{C}} \vert w_{n} \vert ^{2}\,dx\leq \varepsilon _{1}, \qquad \int _{B_{R_{1}}^{C}} \vert w \vert ^{2}\,dx\leq \varepsilon _{1}, \quad n\geq 1. \end{aligned} $$

(3.8)

Then,

$$ \begin{aligned} \int _{B_{R_{1}}^{C}}V_{1}(x) \vert z_{n} \vert ^{2}\,dx\leq C_{1} \varepsilon _{1}, \qquad \int _{B_{R_{1}}^{C}}V_{1}(x) \vert z \vert ^{2}\,dx \leq C_{1}\varepsilon _{1}, \\ \int _{B_{R_{1}}^{C}}V_{2}(x) \vert w_{n} \vert ^{2}\,dx\leq C_{2} \varepsilon _{1}, \qquad \int _{B_{R_{1}}^{C}}V_{2}(x) \vert w \vert ^{2}\,dx \leq C_{2}\varepsilon _{1}. \end{aligned} $$

(3.9)

We derive from (3.5) that

$$ \begin{aligned}& \lim_{n\to \infty} \int _{B_{R_{1}}}V_{1}(x) \vert z_{n} \vert ^{2}\,dx= \int _{B_{R_{1}}}V_{1}(x) \vert z \vert ^{2}\,dx, \\ &\lim_{n\to \infty} \int _{B_{R_{1}}}V_{2}(x) \vert w_{n} \vert ^{2}\,dx= \int _{B_{R_{1}}}V_{2}(x) \vert w \vert ^{2}\,dx. \end{aligned} $$

(3.10)

By (3.9) and (3.10), we have

$$ \begin{aligned} &\lim_{n\to \infty} \int _{\mathbb{R}^{N}}V_{1}(x) \vert z_{n} \vert ^{2}\,dx= \int _{\mathbb{R}^{N}}V_{1}(x) \vert z \vert ^{2}\,dx, \\ &\lim_{n\to \infty} \int _{\mathbb{R}^{N}}V_{2}(x) \vert w_{n} \vert ^{2}\,dx= \int _{\mathbb{R}^{N}}V_{2}(x) \vert w \vert ^{2}\,dx. \end{aligned} $$

(3.11)

By Lemma 2.1(iii), since $\vert \frac{G^{-1}(t)}{g(G^{-1}(t))}t \vert \leq 8|t|^{2}$, it follows from (3.11) that

$$ \begin{aligned}& \lim_{n\to \infty} \int _{\mathbb{R}^{N}}V_{1}(x) \frac{G^{-1}(z_{n})}{g(G^{-1}(z_{n}))}z_{n} \,dx= \int _{ \mathbb{R}^{N}}V_{1}(x)\frac{G^{-1}(z)}{g(G^{-1}(z))}z\,dx, \\ &\lim_{n\to \infty} \int _{\mathbb{R}^{N}}V_{2}(x) \frac{G^{-1}(w_{n})}{g(G^{-1}(w_{n}))}w_{n} \,dx= \int _{ \mathbb{R}^{N}}V_{2}(x) \frac{G^{-1}(w)}{g(G^{-1}(w))}w\,dx. \end{aligned} $$

(3.12)

Together with Lemma 2.1 (iii), (3.1), and the Hölder inequality, we obtain

$$ \begin{aligned} & \biggl\vert \frac{f(x, G^{-1}(z_{n}), G^{-1}(w_{n}))}{g(G^{-1}(z_{n}))}z_{n}+ \frac{h(x, G^{-1}(z_{n}), G^{-1}(w_{n}))}{g(G^{-1}(w_{n}))}w_{n} \biggr\vert \\ &\quad = \bigl\vert \bigl\langle \nabla \eta \bigl(x, G^{-1}(z_{n}), G^{-1}(w_{n})\bigr), (z_{n}, w_{n})\bigr\rangle \bigr\vert \\ &\quad \leq \varepsilon \bigl\vert \bigl(G^{-1}(z_{n}), G^{-1}(w_{n})\bigr) \bigr\vert \bigl\vert (z_{n}, w_{n}) \bigr\vert +C \bigl\vert \bigl(G^{-1}(z_{n}), G^{-1}(w_{n}) \bigr) \bigr\vert ^{q-1} \bigl\vert (z_{n}, w_{n}) \bigr\vert \\ &\quad \leq 2\sqrt{2}\varepsilon \bigl\vert (z_{n}, w_{n}) \bigr\vert ^{2}+C (2 \sqrt{2} )^{q-1} \bigl\vert (z_{n}, w_{n}) \bigr\vert ^{q}. \end{aligned} $$

By (3.8), we have

$$ \int _{B_{R_{1}}^{C}} \bigl\vert (z_{n}, w_{n}) \bigr\vert ^{2}\,dx\leq C_{1} \varepsilon _{1}, \int _{B_{R_{1}}^{C}} \bigl\vert (z, w) \bigr\vert ^{2} \,dx\leq C_{1} \varepsilon _{1} $$

and

$$ \int _{B_{R_{1}}^{C}} \bigl\vert (z_{n}, w_{n}) \bigr\vert ^{q}\,dx\leq C_{2} \varepsilon _{1}, \int _{B_{R_{1}}^{C}} \bigl\vert (z, w) \bigr\vert ^{q} \,dx\leq C_{2} \varepsilon _{1}. $$

Hence,

$$ \begin{aligned}[b] & \int _{B_{R_{1}}^{C}} \bigl\vert \bigl\langle \nabla \eta \bigl(x, G^{-1}(z_{n}), G^{-1}(w_{n}) \bigr), (z_{n}, w_{n})\bigr\rangle \bigr\vert \,dx \\ &\quad \leq 2\sqrt{2}\varepsilon \int _{B_{R_{1}}^{C}} \bigl\vert (z_{n}, w_{n}) \bigr\vert ^{2}\,dx+C (2\sqrt{2} )^{q-1} \int _{B_{R_{1}}^{C}} \bigl\vert (z_{n}, w_{n}) \bigr\vert ^{q}\,dx \\ &\quad \leq \bigl[2\sqrt{2}\varepsilon +C (2\sqrt{2} )^{q-1} \bigr] \max \{C_{1}, C_{2}\}\varepsilon _{1}. \end{aligned} $$

(3.13)

It follows from (3.5) that

$$ \begin{aligned}[b] &\lim_{n\to \infty} \int _{B_{R_{1}}}\bigl\langle \nabla \eta \bigl(x, G^{-1}(z_{n}), G^{-1}(w_{n}) \bigr), (z_{n}, w_{n})\bigr\rangle \,dx \\ &\quad = \int _{B_{R_{1}}}\bigl\langle \nabla \eta \bigl(x, G^{-1}(z), G^{-1}(w)\bigr), (z, w)\bigr\rangle \,dx. \end{aligned} $$

(3.14)

Combining (3.13) and (3.14), we have

$$ \begin{aligned}[b] &\lim_{n\to \infty} \int _{\mathbb{R}^{N}}\bigl\langle \nabla \eta \bigl(x, G^{-1}(z_{n}), G^{-1}(w_{n}) \bigr), (z_{n}, w_{n})\bigr\rangle \,dx \\ &\quad =\lim_{n\to \infty} \int _{\mathbb{R}^{N}} \biggl[ \frac{f(x, G^{-1}(z_{n}), G^{-1}(w_{n}))}{g(G^{-1}(z_{n}))}z_{n}+ \frac{h(x, G^{-1}(z_{n}), G^{-1}(w_{n}))}{g(G^{-1}(w_{n}))}w_{n} \biggr]\,dx \\ &\quad = \int _{\mathbb{R}^{N}} \biggl[ \frac{f(x, G^{-1}(z), G^{-1}(w))}{g(G^{-1}(z))}z+ \frac{h(x, G^{-1}(z), G^{-1}(w))}{g(G^{-1}(w))}w \biggr]\,dx. \end{aligned} $$

(3.15)

Then, we conclude (3.6) from (3.11), (3.12), and (3.15). (3.7) can be proved similarly. □

Lemma 3.4

Suppose that $(\mathcal{V}_{1})$, $(\mathcal{V}_{2})$, (h1) and (h2) hold, then any (PS)_c sequence $\{(z_{n}, w_{n})\}\subset \mathcal{H}$ obtained in (3.3) has a strong convergence subsequence.

Proof

By Lemma 3.2, $\{(z_{n}, w_{n})\}$ is bounded in $\mathcal{H}$, up to a subsequence, we may assume that $(z_{n}, w_{n})\rightharpoonup (z, w)\in \mathcal{H}$ as $n\to \infty$, and the fact that $\langle J'_{\kappa}(z_{n}, w_{n}), (z_{n}, w_{n})\rangle =o_{n}(1)$ and Lemma 3.3 imply

$$ \lim_{n\to \infty} \bigl\Vert (z_{n}, w_{n}) \bigr\Vert ^{2}=\lim_{n \to \infty} \int _{\mathbb{R}^{N}}\bigl\langle \nabla \xi (x, z_{n}, w_{n}), (z_{n}, w_{n})\bigr\rangle \,dx= \int _{\mathbb{R}^{N}} \bigl\langle \nabla \xi (x, z, w), (z, w)\bigr\rangle \,dx. $$

On the other hand, it follows from $\langle J'_{\kappa}(z_{n}, w_{n}), (z, w)\rangle =o_{n}(1)$ that

$$ \bigl\langle (z_{n}, w_{n}), (z, w)\bigr\rangle = \int _{\mathbb{R}^{N}} \bigl\langle \nabla \xi (x, z_{n}, w_{n}), (z, w)\bigr\rangle \,dx+o_{n}(1). $$

Then,

$$ \lim_{n\to \infty}\bigl\langle (z_{n}, w_{n}), (z, w)\bigr\rangle = \lim_{n\to \infty} \int _{\mathbb{R}^{N}}\bigl\langle \nabla \xi (x, z_{n}, w_{n}), (z, w)\bigr\rangle \,dx= \int _{ \mathbb{R}^{N}}\bigl\langle \nabla \xi (x, z, w), (z, w)\bigr\rangle \,dx, $$

that is,

$$ \lim_{n\to \infty} \bigl\Vert (z_{n}, w_{n}) \bigr\Vert ^{2}= \bigl\Vert (z, w) \bigr\Vert ^{2}. $$

Hence, $(z_{n}, w_{n})\to (z, w)$ in $\mathcal{H}$. □

By Lemmas 3.1–3.4, we have the following result.

Theorem 3.5

Suppose that $(\mathcal{V}_{1})$, $(\mathcal{V}_{2})$, and (h1)–(h3) hold, then the problem (2.4) has a nontrivial solution.

Proof

By Lemmas 3.1 and 3.2, $J_{\kappa}$ has a bounded (PS)_c sequence $\{(z_{n}, w_{n})\}\subset \mathcal{H}$. It follows from Lemma 3.4 that there is a sequence of $\{(z_{n}, w_{n})\}$, up to a subsequence, such that $(z_{n}, w_{n})\to (z_{\kappa}, w_{\kappa})$ in $\mathcal{H}$ as $n\to \infty$ satisfying $J_{\kappa}(z_{\kappa}, w_{\kappa})=c\geq \rho >0$, which means that $(z_{\kappa}, w_{\kappa})$ is a nontrivial solution of (2.4). □

4 Proof of the main results

In this section, we will prove the main results. We note that the solution $(u_{\kappa}, v_{\kappa})=(G^{-1}(z_{\kappa}), G^{-1}(w_{\kappa}))$ may not be a solution of the systems (1.1). In order to find a solution of the original systems (1.1), we establish a result of $L^{\infty}$-estimate for $(z_{\kappa}, w_{\kappa})$.

Lemma 4.1

Assume that $(z_{\kappa}, w_{\kappa})$ (denoted by $(z, w)$) is a nontrivial critical point of $J_{\kappa}$ and $J_{\kappa}(z, w)=c$, then there exists a positive constant C independent of λ such that

$$ \bigl\Vert (z, w) \bigr\Vert ^{2}\leq Cc. $$

Proof

By (h3), Lemma 2.1 (ii) and (iii), and (3.4), we obtain

$$ \begin{aligned} \theta c&=\theta J_{\kappa}(z, w)-\bigl\langle J'_{\kappa}(z, w), \bigl(G^{-1}(z)g \bigl(G^{-1}(z)\bigr), G^{-1}(w)g\bigl(G^{-1}(w) \bigr)\bigr)\bigr\rangle \\ &\geq \biggl(\frac{\theta}{2}-1 \biggr) \bigl\Vert (z, w) \bigr\Vert ^{2}. \end{aligned} $$

Hence,

$$ \bigl\Vert (z, w) \bigr\Vert ^{2}\leq \frac{2\theta c}{\theta -2}=Cc. $$

(4.1)

This completes the proof. □

Lemma 4.2

Suppose that $(z, w)$ is a positive solution of (2.4), then there exists a constant $C_{1}>0$ that is independent of λ such that

$$ \bigl\Vert (z, w) \bigr\Vert _{\infty}\leq C_{1}\lambda ^{\frac{1}{2^{*}-q}} \bigl\Vert (z, w) \bigr\Vert _{2^{*}}. $$

Proof

For each $m\in \mathbb{N}$, let $\beta >1$ be a constant to be determined, we set

$$ A_{m}=\bigl\{ x\in \mathbb{R}^{N}: \vert z \vert ^{\beta -1}\leq m, \vert w \vert ^{\beta -1} \leq m\bigr\} ,\qquad B_{m}=\mathbb{R}^{N}\backslash A_{m}, $$

$$ (u_{m}, v_{m})= \textstyle\begin{cases} (z \vert z \vert ^{2(\beta -1)}, w \vert w \vert ^{2(\beta -1)}), &x\in A_{m}, \\ m^{2}(z, w),&x\in B_{m} \end{cases} $$

and

$$ (z_{m}, w_{m})= \textstyle\begin{cases} (z \vert z \vert ^{\beta -1}, w \vert w \vert ^{\beta -1}),&x\in A_{m}, \\ m(z, w),&x\in B_{m}. \end{cases} $$

Clearly, $(u_{m}, v_{m}), (z_{m}, w_{m})\in \mathcal{H}$. Since $(z, w)$ is a nontrivial solution of Eq. (2.4), we obtain

$$ \begin{aligned} & \int _{\mathbb{R}^{N}} \biggl[\nabla z\nabla u_{m}+V_{1}(x) \frac{G^{-1}(z)}{g(G^{-1}(z))}u_{m}+\nabla w\nabla v_{m}+V_{2}(x) \frac{G^{-1}(w)}{g(G^{-1}(w))}v_{m} \biggr]\,dx \\ &\quad =\lambda \int _{\mathbb{R}^{N}} \biggl[ \frac{f(x, G^{-1}(z), G^{-1}(w))}{g(G^{-1}(z))}u_{m}+ \frac{h(x, G^{-1}(z), G^{-1}(w))}{g(G^{-1}(w))}v_{m} \biggr]\,dx. \end{aligned} $$

Furthermore, we have

$$\begin{aligned}& \begin{aligned}[b] \int _{\mathbb{R}^{N}} (\nabla z\nabla u_{m}+\nabla w \nabla v_{m} ) \,dx={}&(2\beta -1) \int _{A_{m}} \bigl( \vert z \vert ^{2( \beta -1)} \vert \nabla z \vert ^{2}+ \vert w \vert ^{2(\beta -1)} \vert \nabla w \vert ^{2} \bigr)\,dx \\ &{} +m^{2} \int _{B_{m}} \bigl( \vert \nabla z \vert ^{2}+ \vert \nabla w \vert ^{2} \bigr)\,dx, \end{aligned} \end{aligned}$$

(4.2)

$$\begin{aligned}& \begin{aligned}[b] \int _{\mathbb{R}^{N}} \bigl( \vert \nabla z_{m} \vert ^{2}+ \vert \nabla w_{m} \vert ^{2} \bigr)\,dx={}&\beta ^{2} \int _{A_{m}} \bigl( \vert z \vert ^{2(\beta -1)} \vert \nabla z \vert ^{2}+ \vert w \vert ^{2(\beta -1)} \vert \nabla w \vert ^{2} \bigr)\,dx \\ &{} +m^{2} \int _{B_{m}} \bigl( \vert \nabla z \vert ^{2}+ \vert \nabla w \vert ^{2} \bigr)\,dx. \end{aligned} \end{aligned}$$

(4.3)

It follows from (4.2) and (4.3) that

$$\begin{aligned}& \begin{aligned}[b] \int _{A_{m}} \bigl( \vert z \vert ^{2(\beta -1)} \vert \nabla z \vert ^{2}+ \vert w \vert ^{2( \beta -1)} \vert \nabla w \vert ^{2} \bigr)\,dx={}&\frac{1}{2\beta -1} \int _{ \mathbb{R}^{N}} (\nabla z\nabla u_{m}+\nabla w\nabla v_{m} )\,dx \\ &{} -\frac{m^{2}}{2\beta -1} \int _{B_{m}} \bigl( \vert \nabla z \vert ^{2}+ \vert \nabla w \vert ^{2} \bigr)\,dx, \end{aligned} \end{aligned}$$

(4.4)

$$\begin{aligned}& \begin{aligned}[b] \int _{\mathbb{R}^{N}} \bigl( \vert \nabla z_{m} \vert ^{2}+ \vert \nabla w_{m} \vert ^{2} \bigr)\,dx={}&(\beta -1)^{2} \int _{A_{m}} \bigl( \vert z \vert ^{2(\beta -1)} \vert \nabla z \vert ^{2}+ \vert w \vert ^{2(\beta -1)} \vert \nabla w \vert ^{2} \bigr)\,dx \\ &{} + \int _{\mathbb{R}^{N}} (\nabla z\nabla u_{m}+ \nabla w\nabla v_{m} )\,dx. \end{aligned} \end{aligned}$$

(4.5)

By (4.4), (4.5), and the fact of $\beta >1$, we deduce that

$$\begin{aligned} & \int _{\mathbb{R}^{N}} \bigl( \vert \nabla z_{m} \vert ^{2}+ \vert \nabla w_{m} \vert ^{2} \bigr)\,dx+\beta ^{2} \int _{\mathbb{R}^{N}} \biggl(V_{1}(x) \frac{G^{-1}(z)}{g(G^{-1}(z))}u_{m} +V_{2}(x) \frac{G^{-1}(w)}{g(G^{-1}(w))}v_{m} \biggr)\,dx \\ &\quad =\frac{(\beta -1)^{2}}{2\beta -1} \int _{\mathbb{R}^{N}} (\nabla z\nabla u_{m}+\nabla w\nabla v_{m} )\,dx- \frac{m^{2}(\beta -1)^{2}}{2\beta -1} \int _{B_{m}} \bigl( \vert \nabla z \vert ^{2}+ \vert \nabla w \vert ^{2} \bigr)\,dx \\ &\qquad {} + \int _{\mathbb{R}^{N}} (\nabla z\nabla u_{m}+ \nabla w\nabla v_{m} )\,dx \\ &\qquad {} +\beta ^{2} \int _{\mathbb{R}^{N}} \biggl(V_{1}(x)\frac{G^{-1}(z)}{g(G^{-1}(z))}u_{m} +V_{2}(x) \frac{G^{-1}(w)}{g(G^{-1}(w))}v_{m} \biggr)\,dx \\ &\quad \leq \frac{\beta ^{2}}{2\beta -1} \int _{\mathbb{R}^{N}} (\nabla z\nabla u_{m}+\nabla w\nabla v_{m} )\,dx \\ &\qquad {} +\beta ^{2} \int _{\mathbb{R}^{N}} \biggl(V_{1}(x) \frac{G^{-1}(z)}{g(G^{-1}(z))}u_{m} +V_{2}(x) \frac{G^{-1}(w)}{g(G^{-1}(w))}v_{m} \biggr)\,dx \\ &\quad \leq \beta ^{2} \int _{\mathbb{R}^{N}} (\nabla z \nabla u_{m}+\nabla w\nabla v_{m} )\,dx \\ &\qquad {} +\beta ^{2} \int _{ \mathbb{R}^{N}} \biggl(V_{1}(x)\frac{G^{-1}(z)}{g(G^{-1}(z))}u_{m} +V_{2}(x) \frac{G^{-1}(w)}{g(G^{-1}(w))}v_{m} \biggr)\,dx \\ &\quad =\beta ^{2}\lambda \int _{\mathbb{R}^{N}} \biggl( \frac{f(x, G^{-1}(z), G^{-1}(w))}{g(G^{-1}(z))}u_{m}+ \frac{h(x, G^{-1}(z), G^{-1}(w))}{g(G^{-1}(w))}v_{m} \biggr)\,dx \\ &\quad \leq \beta ^{2}\lambda \int _{\mathbb{R}^{N}} \biggl[ \varepsilon \biggl(\frac{G^{-1}(z)}{g(G^{-1}(z))}u_{m} + \frac{G^{-1}(w)}{g(G^{-1}(w))}v_{m} \biggr) \\ &\qquad {}+ C \biggl( \frac{ \vert G^{-1}(z) \vert ^{q-1}}{g(G^{-1}(z))}u_{m}+ \frac{ \vert G^{-1}(w) \vert ^{q-1}}{g(G^{-1}(w))}v_{m} \biggr) \biggr]\,dx \\ &\quad \leq \beta ^{2} \int _{\mathbb{R}^{N}} \biggl(V_{1}(x) \frac{G^{-1}(z)}{g(G^{-1}(z))}u_{m} +V_{2}(x) \frac{G^{-1}(w)}{g(G^{-1}(w))}v_{m} \biggr)\,dx \\ &\qquad {} + \beta ^{2}\lambda C \int _{\mathbb{R}^{N}} \biggl(\frac{ \vert G^{-1}(z) \vert ^{q-1}}{g(G^{-1}(z))}u_{m}+ \frac{ \vert G^{-1}(w) \vert ^{q-1}}{g(G^{-1}(w))}v_{m} \biggr)\,dx, \end{aligned}$$

(4.6)

for $0<\varepsilon <\min \{\frac{V_{1}(x)}{\lambda}, \frac{V_{2}(x)}{\lambda} \}$. It follows from (iii) of Lemma 2.1, $1<\frac{1}{g(t)}<2\sqrt{2}$, and $zu_{m}=z^{2}_{m}$, $wv_{m}=w^{2}_{m}$ that

$$ \begin{aligned}[b] \int _{\mathbb{R}^{N}} \bigl( \vert \nabla z_{m} \vert ^{2}+ \vert \nabla w_{m} \vert ^{2} \bigr)\,dx&\leq \beta ^{2}C\lambda \int _{\mathbb{R}^{N}} \biggl( \frac{ \vert G^{-1}(z) \vert ^{q-1}}{g(G^{-1}(z))}u_{m}+ \frac{ \vert G^{-1}(w) \vert ^{q-1}}{g(G^{-1}(w))}v_{m} \biggr)\,dx \\ &\leq \beta ^{2}\lambda (2\sqrt{2} )^{q}C \int _{ \mathbb{R}^{N}} \bigl( \vert z \vert ^{q-2}z^{2}_{m}+ \vert w \vert ^{q-2}w^{2}_{m} \bigr)\,dx. \end{aligned} $$

(4.7)

If $\mu (a)+\mu (b)\leq \nu (a)+\nu (b)$, we have $\mu (a)\leq \nu (a)$, $\mu (b)\leq \nu (b)$. By (4.7), we can obtain

$$\begin{aligned}& \int _{\mathbb{R}^{N}} \vert \nabla z_{m} \vert ^{2}\,dx\leq \beta ^{2} \lambda (2\sqrt{2} )^{q}C \int _{\mathbb{R}^{N}} \vert z \vert ^{q-2}z^{2}_{m} \,dx, \\& \int _{\mathbb{R}^{N}} \vert \nabla w_{m} \vert ^{2}\,dx\leq \beta ^{2} \lambda (2\sqrt{2} )^{q}C \int _{\mathbb{R}^{N}} \vert w \vert ^{q-2}w^{2}_{m} \,dx. \end{aligned}$$

By the Sobolev inequality, there is $S>0$ such that

$$ \biggl( \int _{A_{m}} \vert z_{m} \vert ^{2^{*}}\,dx \biggr)^{\frac{N-2}{N}} \leq S \int _{\mathbb{R}^{N}} \vert \nabla z_{m} \vert ^{2}\,dx, $$

combining (4.6) and the Hölder inequality, we know that

$$ \biggl( \int _{A_{m}} \vert z_{m} \vert ^{2^{*}}\,dx \biggr)^{\frac{N-2}{N}} \leq \beta ^{2}\lambda (2\sqrt{2} )^{q} SC \Vert z \Vert ^{q-2}_{2^{*}} \Vert z_{m} \Vert _{2q_{1}}^{2}, $$

where $\frac{1}{q_{1}}+\frac{q-2}{2^{*}}=1$. Note that $|z_{m}|=|z|^{\beta}$ in $A_{m}$ and $|z_{m}|\leq |z|^{\beta}$ in $\mathbb{R}^{N}$, hence

$$ \biggl( \int _{A_{m}} \vert z \vert ^{2^{*}\beta}\,dx \biggr)^{\frac{N-2}{N}} \leq \beta ^{2}\lambda (2\sqrt{2} )^{q} SC \Vert z \Vert _{2^{*}}^{q-2} \Vert z \Vert _{2\beta q_{1}}^{2\beta}. $$

Letting $m\rightarrow \infty$ in the above inequality, we have

$$ \Vert z \Vert _{\beta 2^{*}}\leq \beta ^{\frac{1}{\beta}} \bigl(\lambda (2 \sqrt{2} )^{q} SC \Vert z \Vert ^{q-2}_{2^{*}} \bigr)^{\frac{1}{2\beta}} \Vert z \Vert _{2\beta q_{1}}. $$

(4.8)

Denote $\sigma =\frac{2^{*}}{2q_{1}}$. Now, taking $\beta =\sigma$ in (4.8), we see that

$$ \Vert z \Vert _{\sigma 2^{*}}\leq \sigma ^{\frac{1}{\sigma}} \bigl(\lambda (2\sqrt{2} )^{q} SC \Vert z \Vert ^{q-2}_{2^{*}} \bigr)^{ \frac{1}{2\sigma}} \Vert z \Vert _{2^{*}}. $$

(4.9)

Taking $\beta =\sigma ^{2}$ in (4.8), we obtain that

$$ \Vert z \Vert _{\sigma ^{2}2^{*}}\leq \sigma ^{\frac{2}{\sigma ^{2}}} \bigl( \lambda (2\sqrt{2} )^{q} SC \Vert z \Vert ^{q-2}_{2^{*}} \bigr)^{ \frac{1}{2\sigma ^{2}}} \Vert z \Vert _{\sigma 2^{*}}. $$

(4.10)

It follows from (4.9) and (4.10) that

$$ \Vert z \Vert _{\sigma ^{2}2^{*}}\leq \sigma ^{(\frac{1}{\sigma}+ \frac{2}{\sigma ^{2}})} \bigl(\lambda (2\sqrt{2} )^{q} SC \Vert z \Vert ^{q-2}_{2^{*}} \bigr)^{\frac{1}{2}(\frac{1}{\sigma}+ \frac{1}{\sigma ^{2}})} \Vert z \Vert _{2^{*}}. $$

Continuing in this way by taking $\beta =\sigma ^{i}$ ($i=1, 2, \ldots$) in (4.8), we obtain

$$ \Vert z \Vert _{\sigma ^{j}2^{*}}\leq \sigma ^{\sum _{i=1}^{j} \frac{i}{\sigma ^{i}}} \bigl(\lambda (2\sqrt{2} )^{q} SC \Vert z \Vert ^{q-2}_{2^{*}} \bigr)^{\frac{1}{2}\sum _{i=1}^{j} \frac{1}{\sigma ^{i}}} \Vert z \Vert _{2^{*}},\quad j=1, 2, \ldots . $$

It follows from the Sobolev inequality and letting $j\rightarrow +\infty$, we obtain

$$ \begin{aligned}[b] \Vert z \Vert _{\infty}&\leq \sigma ^{\frac{1}{(\sigma -1)^{2}}} \bigl( \lambda (2\sqrt{2} )^{q} SCS^{\frac{q}{2}}C_{0}^{ \frac{q-2}{2}} \bigr) ^{\frac{1}{2(\sigma -1)}} \Vert z \Vert _{2^{*}} \\ &=C_{1}\lambda ^{\frac{1}{2(\sigma -1)}} \Vert z \Vert _{2^{*}} \\ &=C_{1}\lambda ^{\frac{1}{2^{*}-q}} \Vert z \Vert _{2^{*}}. \end{aligned} $$

(4.11)

Similarly, we may obtain

$$ \Vert w \Vert _{\infty}\leq C_{2} \lambda ^{\frac{1}{2^{*}-q}} \Vert w \Vert _{2^{*}}. $$

(4.12)

Therefore, by (4.11) and (4.12), we obtain

$$ \Vert z \Vert _{\infty}+ \Vert w \Vert _{\infty}\leq \lambda ^{\frac{1}{2^{*}-q}}\bigl(C_{1} \Vert z \Vert _{2^{*}}+C_{2} \Vert w \Vert _{2^{*}} \bigr), $$

(4.13)

where $C_{1}, C_{2}>0$ are independent of λ. □

Proof of Theorem 1.1

Let $\delta >0$ be such that the set

$$ T=\bigl\{ x\in \mathbb{R}^{N}: \phi (x)\geq \delta \bigr\} \cap\bigl\{ x\in \mathbb{R}^{N}: \psi (x)\geq \delta \bigr\} $$

is nonempty. By (h3), for $x\in T$, there exists $C_{1}>0$ such that

$$ \eta (x, s, t)\geq C_{1} \bigl\vert (s, t) \bigr\vert ^{q}. $$

(4.14)

By Theorem 3.5, let $(z, w)$ be a critical point of $J_{\kappa}$ and $J_{\kappa}(z, w)=c$, together with Lemma 3.1 (3) and (4.14), one has

$$ \begin{aligned} c\leq {}&\max_{t>0}J_{\kappa}(t \phi , t\psi ) \\ \leq{}& \max_{t>0} \biggl[t^{2} \int _{\mathbb{R}^{N}} \biggl(\frac{1}{2}\bigl( \vert \nabla \phi \vert ^{2}+ \vert \nabla \psi \vert ^{2} \bigr)+4\bigl(V_{1}(x) \vert \phi \vert ^{2}+V_{2}(x) \vert \psi \vert ^{2}\bigr) \biggr)\,dx \\ &{} -C_{1}\lambda t^{q} \int _{T} \bigl( \vert \phi \vert ^{q}+ \vert \psi \vert ^{q} \bigr)\,dx \biggr] \\ \leq {}&C\lambda ^{-\frac{2}{q-2}}. \end{aligned} $$

By (4.1), (4.13), and the continuous embedding $\mathcal{H}\hookrightarrow L^{r}_{K}{(\mathbb{R}^{N})}$, $r\in [2, 2^{*}]$, we have

$$ \begin{aligned} \Vert z \Vert _{\infty}+ \Vert w \Vert _{\infty}&\leq C\lambda ^{\frac{1}{2^{*}-q}} \bigl( \Vert z \Vert _{2^{*}}+ \Vert w \Vert _{2^{*}} \bigr)\leq C\lambda ^{ \frac{1}{2^{*}-q}}\overline{C}\bigl( \Vert z \Vert + \Vert w \Vert \bigr) \\ &\leq C\lambda ^{\frac{1}{2^{*}-q}}\overline{C}(C_{1}c)^{\frac{1}{2}} \leq C\lambda ^{\frac{1}{2^{*}-q}}\overline{C} \bigl(C_{1}C_{2} \lambda ^{-\frac{2}{q-2}} \bigr)^{\frac{1}{2}} \\ &=C_{3}\lambda ^{-\frac{2^{*}-2q+2}{(2^{*}-q)(q-2)}}. \end{aligned} $$

Since $2< q<(2^{*}+2)/2$, for fixing $\kappa >0$, there is $\lambda _{1}(\kappa )= (16C^{2}_{3}\kappa )^{ \frac{(2^{*}-q)(q-2)}{2(2^{*}-2q+2)}}$ such that for any $\lambda >\lambda _{1}(\kappa )$, it holds that

$$ \begin{aligned} \Vert u \Vert _{\infty}+ \Vert v \Vert _{\infty}&= \bigl\Vert G^{-1}(z) \bigr\Vert _{\infty}+ \bigl\Vert G^{-1}(w) \bigr\Vert _{ \infty} \\ &\leq 2\sqrt{2} \bigl( \Vert z \Vert _{\infty}+ \Vert w \Vert _{\infty} \bigr)\leq 2 \sqrt{2}C_{3}\lambda ^{-\frac{2^{*}-2q+2}{(2^{*}-q)(q-2)}}\leq \sqrt{ \frac{1}{2\kappa}}, \end{aligned} $$

thus, $(u, v)=(G^{-1}(z), G^{-1}(w))$ is a nontrivial solution of systems (1.1). □