Existence of solutions for perturbed elliptic system with critical exponents

Abstract

In this paper, the perturbed elliptic system with critical exponents $-\varepsilon^{2}\Delta u+V(x)u= f(x,u)+\frac{\alpha}{\alpha+\beta} K(x)|u|^{\alpha-2}u|v|^{\beta}$, $x\in{\mathbb{R}^{N}}$, $-\varepsilon^{2}\Delta v+V(x)v=g(x,v)+\frac{\beta}{\alpha+\beta }K(x)|u|^{\alpha}|v|^{\beta-2}v$, $x\in{\mathbb{R}^{N}}$, is considered, where $\alpha>1$, $\beta>1$ satisfy $\alpha+\beta =2^{\ast}$, and $2^{\ast}=2N/(N-2)$ ($N\geq3$) is the Sobolev critical exponent. Under proper conditions on V, f, g, and K, the existence result is obtained by using variational methods.

Introduction

In this paper, we obtain the results of nontrivial solutions of the following perturbed elliptic system:

$$ \left \{ \textstyle\begin{array}{l} -\varepsilon^{2}\Delta u+V(x)u=f(x,u)+\frac{\alpha}{\alpha+\beta }K(x)|u|^{\alpha-2}u|v|^{\beta},\quad x\in\mathbb{R}^{N}, \\ -\varepsilon^{2}\Delta v+V(x)v=g(x,v)+\frac{\beta}{\alpha+\beta }K(x)|u|^{\alpha}|v|^{\beta-2}v,\quad x\in\mathbb{R}^{N}, \\ u(x), v(x)\rightarrow0 \quad \mbox{as } |x|\rightarrow\infty, \end{array}\displaystyle \right . $$

(1.1)

where $\alpha>1$, $\beta>1$ satisfy $\alpha+\beta=2^{\ast}$, $2^{\ast }=2N/(N-2)$ ($N\geq3$) is the critical Sobolev exponent, and $V(x)$, $K(x)$, $f(x,u)$, $g(x,v)$ satisfy the following conditions:

(V₀):

$V\in C(\mathbb{R}^{N},\mathbb{R})$, $V(0)=\inf_{x\in \mathbb{R}^{N}} V(x)=0$, and there exists $b>0$ such that the set $\nu ^{b}:=\{x\in\mathbb{R}^{N}:V(x)< b\}$ has a finite Lebesgue measure;

(K₀):

$K(x)\in C(\mathbb{R}^{N},\mathbb{R})$, $0<\inf K\leq\sup K<\infty$;

(H₁):

$f,g\in C(\mathbb{R}^{N}\times\mathbb{R})$, $f(x,u)=o(|u|)$, $g(x,v)=o(|v|)$ uniformly in x as $u\rightarrow 0$, $v\rightarrow0$;

(H₂):

there exist $2< q<2^{\ast}$ and $c_{0}>0$ such that

$$\bigl\vert f(x,u)\bigr\vert \leq c_{0}\bigl(1+|u|^{q-1} \bigr) \quad \mbox{for all } (x,u) $$

and

$$\bigl\vert g(x,v)\bigr\vert \leq c_{0}\bigl(1+|v|^{q-1} \bigr)\quad \mbox{for all } (x,v); $$

(H₃):

there exist $a_{0}>0$, $p>2$, and $2<\mu<2^{\ast}$ such that $F(x,u)\geq a_{0}|u|^{p}$, $G(x,v)\geq a_{0}|v|^{p}$, $\mu F(x,u)\leq uf(x,u)$ for all $(x,u)$, and $\mu G(x,v)\leq vg(x,v)$ for all $(x,v)$, where $F(x,u)=\int_{0}^{u} f(x,s)\, ds$, $G(x,v)=\int_{0}^{v} g(x,s)\, ds$.

Recall that there have been many papers devoted to the study of the scalar equation

$$ -\varepsilon^{2}\Delta u+V(x)u=g(x,u),\quad x\in H^{1}\bigl({\mathbb{R}^{N}}\bigr), $$

(1.2)

where one seeks for the standing waves solutions for the following nonlinear Schrödinger equation:

$$ i\hbar\frac{\partial\varphi}{\partial t}=-\frac{{\hbar }^{2}}{2m}\Delta\varphi+W(x) \varphi-f\bigl(x,\vert \varphi \vert \bigr)\varphi . $$

(1.3)

A standing wave of (1.3) is a solution of the form $\varphi (x,t)=u(x)\exp(-iEt/\hbar)$. Equation (1.2) has been studied extensively by many authors. We would like to cite the works of [1–23], and references therein.

For elliptic systems, there are a lot of works. Han [24] established the existence of positive solutions of the following elliptic system:

$$ \left \{ \textstyle\begin{array}{l} -\Delta u=\frac{2\alpha}{\alpha+\beta}u^{\alpha-1}v^{\beta}+\lambda u \quad \mbox{in } \Omega, \\ -\Delta v=\frac{2\beta}{\alpha+\beta}u^{\alpha}v^{\beta-1}+\mu v \quad \mbox{in } \Omega, \\ u(x)>0, \qquad v(x)>0 \quad \mbox{in } \Omega, \\ u=v=0 \quad \mbox{on } \partial\Omega, \end{array}\displaystyle \right . $$

where $\lambda>0$, $\mu>0$ are parameters, and $\alpha>1$, $\beta>1$ satisfy $\alpha+\beta=2^{\ast}$; $2^{\ast}=2N/(N-2)$ ($N\geq3$) denotes the critical Sobolev exponent.

For a semilinear elliptic system involving subcritical exponents, there are a lot of results. Lin [25] obtained a multiplicity of positive solutions of the following semilinear elliptic system:

$$ \left \{ \textstyle\begin{array}{l} -\varepsilon^{2}\Delta u+u=\lambda g(x)|u|^{q-2}u+\frac{\alpha}{\alpha +\beta}f(x)u|u|^{\alpha-2}|v|^{\beta} \quad \mbox{in } \mathbb{R}^{N}, \\ -\varepsilon^{2}\Delta v+v=\mu h(x)|v|^{q-2}v+\frac{\beta}{\alpha +\beta}f(x)v|u|^{\alpha}|v|^{\beta-2}\quad \mbox{in } \mathbb{R}^{N}, \\ u,v \in H^{1}({\mathbb{R}^{N}}), \end{array}\displaystyle \right . $$

where $\alpha>1$, $\beta>1$, $2< q< p=\alpha+\beta<2^{\ast}=2N/(N-2)$.

However, as far as we know, there are almost no results on the problem (1.1) involving critical exponents in the whole space. In our work, the coupled terms of the system (1.1) are the critical nonlinearities $K(x)|u|^{\alpha-2}u|v|^{\beta}$ and $K(x)|u|^{\alpha}|v|^{\beta-2}v$ ($\alpha+\beta=p^{\ast}$). We consider the problem (1.1) and use variational methods to get positive solutions. The main difficulty is that the corresponding functional to the problem (1.1) lacks compactness because of the unbounded domain $\mathbb{R}^{N}$ and the critical exponent. We can prove the functional associated to (1.1) obeys the $(\mathrm{PS})_{c}$ condition at some energy level c. To overcome the difficulty, we follow some ideas explored in [15] and also use arguments developed in [26, 27].

The main result of this paper reads as follows.

Theorem 1

Assume that (V₀), (K₀), and (H₁)-(H₃) hold. Then, for any $\sigma>0$, there is $\varepsilon_{\sigma}>0$ such that if $\varepsilon\leq\varepsilon _{\sigma}$, the problem (1.1) has at least one solution $(u_{\varepsilon}, v_{\varepsilon})$ which satisfies

$$ \frac{\mu-2}{2\mu}\int_{\mathbb{R}^{N}} \varepsilon^{2}\bigl(|\nabla u_{\varepsilon}|^{2}+|\nabla v_{\varepsilon }|^{2}\bigr)+V(x) \bigl(|u_{\varepsilon}|^{2}+|v_{\varepsilon}|^{2} \bigr)\leq\sigma \varepsilon^{N}. $$

(1.4)

This paper is organized as follows. In Section 2, we describe the analytic setting where we restate the problem in an equivalent form by replacing $\varepsilon^{-2}$ with λ other than the usual scaling. In Section 3, we show the corresponding energy functional satisfies the $(\mathrm{PS})_{c}$ condition at the level c. Section 4 gives the fact that the energy functional possesses a mountain-pass geometry structure. The last section contains the proof of the main result.

An equivalent variational problem

Let $\lambda=\varepsilon^{-2}$. The problem (1.1) reads

$$ \left \{ \textstyle\begin{array}{l} -\Delta u+\lambda V(x)u=\lambda f(x,u)+\frac{\lambda\alpha}{\alpha +\beta}K(x)|u|^{\alpha-2}u|v|^{\beta},\quad x\in\mathbb{R}^{N}, \\ -\Delta v+\lambda V(x)v=\lambda g(x,v)+\frac{\lambda\beta}{\alpha +\beta}K(x)|u|^{\alpha}|v|^{\beta-2}v, \quad x\in\mathbb{R}^{N}, \\ u(x), v(x)\rightarrow0 \quad \mbox{as } |x|\rightarrow\infty \end{array}\displaystyle \right . $$

(2.1)

for λ sufficiently large. We are going to prove the following result.

Theorem 2

Assume that (V₀), (K₀), and (H₁)-(H₃) hold. Then for any $\sigma>0$, there is $\Lambda _{\sigma}>0$ such that if $\lambda\geq\Lambda_{\sigma}$, the problem (2.1) has at least one solution $(u_{\lambda}, v_{\lambda})$ which satisfies

$$ \frac{\mu-2}{2\mu}\int_{\mathbb{R}^{N}}\bigl(|\nabla u_{\lambda }|^{2}+|\nabla v_{\lambda}|^{2}+\lambda V(x) \bigl(|u_{\lambda}|^{2}+|v_{\lambda }|^{2} \bigr)\bigr)\leq\sigma\lambda^{1-\frac{N}{2}}. $$

(2.2)

In order to prove Theorem 2, we introduce the necessary notations. The space

$$E_{\lambda}=\biggl\{ u\in H^{1}\bigl(\mathbb{R}^{N} \bigr):\int_{\mathbb{R}^{N}}\lambda V(x)u^{2}< \infty , \lambda>0 \biggr\} $$

is a Hilbert space equipped with the inner product

$$(u,v)_{E_{\lambda}}=\int_{\mathbb{R}^{N}}\bigl(\nabla u\nabla v+\lambda V(x)uv\bigr) $$

and the associated norm $\|u\|_{\lambda}^{2}=(u,u)_{E_{\lambda}}$. Set $E=E_{\lambda} \times E_{\lambda}$, the Hilbert space with the norm

$$\bigl\Vert (u,v)\bigr\Vert _{E}^{2}=\|u \|_{\lambda}^{2}+\|v\|_{\lambda}^{2}=\int _{\mathbb {R}^{N}}\bigl(|\nabla u|^{2}+ \lambda V(x)u^{2}+|\nabla v|^{2}+ \lambda V(x)v^{2}\bigr) $$

for any $(u,v)\in E$. It is obvious that, for each $2< p<2^{\ast}$, there exists $c_{p}>0$ such that if $\lambda\geq1$,

$$\|u\|_{p}\leq c_{p}\|u\|_{\lambda} \quad \mbox{for all } u\in E_{\lambda}, $$

where the $L^{p}(\mathbb{R}^{N})$, $1 \leq p< \infty$, denote Lebesgue spaces and the norm for $L^{p}$ is denoted by $\|\cdot\|_{p}$ for $1 \leq p< \infty$.

We will show the existence results of nontrivial solutions of (2.1) by looking for critical points of the associated functional

$$\begin{aligned} I_{\lambda}(u,v) =&\frac{1}{2}\int_{\mathbb{R}^{N}}\bigl(| \nabla u|^{2}+\lambda V(x)u^{2}+|\nabla v|^{2}+ \lambda V(x)v^{2}\bigr) \\ &{} -\lambda\int_{\mathbb{R}^{N}}\bigl(F(x,u)+G(x,v)\bigr)- \frac{\lambda }{\alpha+\beta} \int_{\mathbb{R}^{N}}K(x)|u|^{\alpha}|v|^{\beta}. \end{aligned}$$

In fact, the critical points of the functional $I_{\lambda}$ are the weak solutions of (2.1). By a weak solution $(u,v)$ of (2.1), we mean that $(u,v)\in E$ satisfies

$$\begin{aligned}& \int_{\mathbb{R}^{N}}\bigl(\nabla u\nabla\varphi+\lambda V(x)u\varphi+ \nabla v\nabla\psi+\lambda V(x)v\psi\bigr) \\& \quad = \lambda\int_{\mathbb{R}^{N}}\bigl(f(x,u)\varphi+g(x,v)\psi\bigr) +\frac{\lambda\alpha}{\alpha+\beta} \int_{\mathbb {R}^{N}}K(x)|u|^{\alpha-2}u|v|^{\beta} \varphi \\& \qquad {} + \frac{\lambda\beta}{\alpha+\beta} \int_{\mathbb {R}^{N}}K(x)|u|^{\alpha}|v|^{\beta-2}v \psi \end{aligned}$$

for all $(\varphi,\psi)\in E$.

Compactness condition

In this section, we will find the range of c where the $(\mathrm{PS})_{c}$ condition holds for the functional $I_{\lambda}$. For convenience, we give some notations.

Notations

• The dual space of a Banach space E will be denoted by $E^{\ast}$.

• $B_{r}:=\{x\in\mathbb{R}^{N}:|x|\leq r\}$ is the ball in $\mathbb{R}^{N}$.

• c, $c_{i}$ represent various positive constants, the exact values of which are not important.

• Let $C_{0}^{\infty}(\mathbb{R}^{N})$ denote the collection of smooth functions with compact support.

• $o(1)$ denotes $o(1)\rightarrow0$ as $n\rightarrow\infty$.

• $S_{\alpha,\beta}$ is the best Sobolev embedding constant defined by

  $$ S_{\alpha,\beta}= \inf_{u,v \in H^{1}(\mathbb{R}^{N})} \frac{\int_{\mathbb{R}^{N}}(|\nabla u|^{2}+|\nabla v|^{2})}{(\int_{\mathbb {R}^{N}}|u|^{\alpha}|v|^{\beta})^{\frac{2}{\alpha+\beta}}}. $$

  (3.1)

We can obtain

$$ S_{\alpha,\beta}= \biggl(\biggl(\frac{\alpha}{\beta}\biggr)^{\frac{\beta}{\alpha +\beta}}+\biggl( \frac{\beta}{\alpha}\biggr)^{\frac{\alpha}{\alpha+\beta}}\biggr)S, $$

where S is the best Sobolev embedding constant defined by

$$ S= \inf_{u \in H^{1}(\mathbb{R}^{N})} \frac{\int_{\mathbb {R}^{N}}|\nabla u|^{2}}{(\int_{\mathbb{R}^{N}}|u|^{2^{*}})^{\frac{2}{2^{*}}}}. $$

Based on the assumptions of Theorem 2 in [27], we can show that $I_{\lambda}\in C^{1}(E,\mathbb{R})$ and the critical points of $I_{\lambda}$ are weak solutions of the problem (2.1).

Recall that we say that a sequence $\{(u_{n},v_{n})\} \subset E$ is a $(\mathrm{PS})$ sequence at level c ($(\mathrm{PS})_{c}$ sequence, for short) if $I_{\lambda}(u_{n},v_{n})\rightarrow c$ and $I_{\lambda}'(u_{n},v_{n})\rightarrow0$. $I_{\lambda}$ is said to satisfy the $(\mathrm{PS})_{c}$ condition if any $(\mathrm{PS})_{c}$ sequence contains a convergent subsequence.

Lemma 3.1

If the sequence $\{(u_{n},v_{n})\}\subset E$ is a $(\mathrm{PS})_{c}$ sequence for $I_{\lambda}$, then we find that $c\geq0$ and $\{(u_{n},v_{n})\}$ is bounded in the space E.

Proof

We have

$$\begin{aligned}& I_{\lambda}(u_{n},v_{n})-\frac{1}{\mu}I_{\lambda }'(u_{n},v_{n}) (u_{n},v_{n}) \\& \quad = \frac{1}{2}\bigl\Vert (u_{n},v_{n})\bigr\Vert _{E}^{2}-\lambda\int_{\mathbb {R}^{N}} \bigl(F(x,u)+G(x,v)\bigr)-\frac{\lambda}{\alpha+\beta} \int_{\mathbb {R}^{N}}K(x)|u_{n}|^{\alpha}|v_{n}|^{\beta} \\& \qquad {}-\frac{1}{\mu}\biggl[\bigl\Vert (u_{n},v_{n}) \bigr\Vert _{E}^{2}-\lambda\int_{\mathbb {R}^{N}} \bigl(f(x,u_{n})u_{n}+g(x,v_{n})v_{n} \bigr)-\lambda\int_{\mathbb {R}^{N}}K(x)|u_{n}|^{\alpha}|v_{n}|^{\beta} \biggr] \\& \quad = \biggl(\frac{1}{2}-\frac{1}{\mu}\biggr)\bigl\Vert (u_{n},v_{n})\bigr\Vert _{E}^{2}+ \lambda\int_{\mathbb{R}^{N}}\biggl(\frac{1}{\mu }\bigl(f(x,u_{n})u_{n}+g(x,v_{n})v_{n} \bigr)-F(x,u_{n})-G(x,v_{n})\biggr) \\& \qquad {}+\biggl(\frac{1}{\mu}-\frac{1}{\alpha+\beta}\biggr)\lambda\int _{\mathbb {R}^{N}}K(x)|u_{n}|^{\alpha}|v_{n}|^{\beta}. \end{aligned}$$

Together with (K₀), (H₃), and $2<\mu<2^{\ast}$, we get

$$I_{\lambda}(u_{n},v_{n})-\frac{1}{\mu}I_{\lambda }'(u_{n},v_{n}) (u_{n},v_{n})\geq\biggl(\frac{1}{2}- \frac{1}{\mu}\biggr)\bigl\Vert (u_{n},v_{n})\bigr\Vert _{E}^{2}. $$

By the fact that $I_{\lambda}(u_{n},v_{n})\rightarrow c$ and $I_{\lambda }'(u_{n},v_{n})\rightarrow0$, we easily obtain the desired conclusion. □

Lemma 3.2

There exists a subsequence $\{ (u_{n_{j}},v_{n_{j}})\}$ such that for any $\varepsilon>0$, there is $r_{\varepsilon}>0$ with $r\geq r_{\varepsilon}$,

$$\lim_{j\rightarrow\infty}\sup\int_{B_{j}\setminus B_{r}} \bigl(|u_{n_{j}}|^{d}+|v_{n_{j}}|^{d}\bigr)\leq \varepsilon, $$

where $2\leq d<2^{\ast}$.

Proof

Together with Lemma 3.1, the $(\mathrm{PS})_{c}$ sequence $\{(u_{n},v_{n})\}$ for $I_{\lambda}$ is bounded in E. So, we assume $(u_{n},v_{n})\rightharpoonup (u,v)$ in E, $u_{n}\rightarrow u, v_{n}\rightarrow v$ a.e. in $\mathbb {R}^{N}$ and $(u_{n},v_{n})\rightarrow(u,v)$ in $L_{\mathrm{loc}}^{d}(\mathbb {R}^{N})\times L_{\mathrm{loc}}^{d}(\mathbb{R}^{N})$ for any $2\leq d<2^{\ast}$. Note that, for each $j\in\mathbb{N}$, we have

$$ \int_{B_{j}}\bigl(|u_{n}|^{d}+|v_{n}|^{d} \bigr)\rightarrow\int_{B_{j}}\bigl(|u|^{d}+|v|^{d} \bigr). $$

Thus, there exists $n_{0}\in\mathbb{N}$ such that

$$ \int_{B_{j}}\bigl(|u_{n}|^{d}+|v_{n}|^{d}-|u|^{d}-|v|^{d} \bigr)< \frac{1}{j} $$

for all $n\geq n_{0}+1$. Without loss of generality, we may choose $n_{j}=n_{0}+j$ such that

$$ \int_{B_{j}}\bigl(|u_{n_{j}}|^{d}+|v_{n_{j}}|^{d}-|u|^{d}-|v|^{d} \bigr)< \frac{1}{j}. $$

It is easy to see there is $r_{\varepsilon}$ satisfying

$$ \int_{\mathbb{R}^{N}\setminus B_{r}}\bigl(|u|^{d}+|v|^{d}\bigr)< \varepsilon \quad \mbox {for all } r \geq r_{\varepsilon}. $$

Since

$$ \int_{B_{j}\setminus B_{r}}\bigl(|u_{n_{j}}|^{d}+|v_{n_{j}}|^{d} \bigr)< \frac{1}{j}+\int_{\mathbb{R}^{N}\setminus B_{r}}\bigl(|u|^{d}+|v|^{d} \bigr) +\int_{B_{r}}\bigl(|u|^{d}-|u_{n_{j}}|^{d}+|v|^{d}-|v_{n_{j}}|^{d} \bigr) $$

and

$$ (u_{n},v_{n})\rightarrow(u,v) \quad \mbox{in } L_{\mathrm{loc}}^{d}\bigl(\mathbb {R}^{N}\bigr)\times L_{\mathrm{loc}}^{d}\bigl(\mathbb{R}^{N}\bigr), $$

and the lemma follows. □

Let $\eta\in C^{\infty}(\mathbb{R}^{+},[0,1])$ be a smooth function satisfying $0\leq\eta(t)\leq1$, $t\geq0$. $\eta(t)=1$ if $t\leq1$ and $\eta(t)=0$ if $t\geq2$. Define $\tilde{u}_{j}(x)=\eta (2|x|/j)u(x)$ and $\tilde{v}_{j}(x)=\eta(2|x|/j)v(x)$, then

$$ \tilde{u}_{j}\rightarrow u,\qquad \tilde{v}_{j} \rightarrow v \quad \mbox{in } E_{\lambda}\mbox{ as } j\rightarrow\infty. $$

(3.2)

Lemma 3.3

One has

$$\lim_{j\rightarrow\infty}\biggl\vert \int_{\mathbb {R}^{N}} \bigl(f(x,u_{n_{j}})-f(x,u_{n_{j}}-\tilde{u}_{j})-f(x, \tilde {u}_{j})\bigr)\varphi\biggr\vert =0 $$

and

$$\lim_{j\rightarrow\infty}\biggl\vert \int_{\mathbb {R}^{N}} \bigl(g(x,v_{n_{j}})-g(x,v_{n_{j}}-\tilde{v}_{j})-g(x, \tilde{v}_{j})\bigr)\psi\biggr\vert =0 $$

uniformly in $(\varphi,\psi)\in E$ with $\|(\varphi,\psi)\|_{E}\leq1$.

Proof

Note that (3.2) and local compactness of the Sobolev embedding imply that for any $r>0$,

$$\lim_{j\rightarrow\infty}\biggl\vert \int_{\mathbb {B}^{r}} \bigl(f(x,u_{n_{j}})-f(x,u_{n_{j}}-\tilde{u}_{j})-f(x, \tilde {u}_{j})\bigr)\varphi\biggr\vert =0 $$

uniformly in $\|\varphi\|\leq1$. For any $\varepsilon>0$, it follows from

$$\int_{\mathbb{R}^{N}\setminus B_{r}}\bigl(|u|^{d}+|v|^{d}\bigr)< \varepsilon $$

that

$$\lim_{j\rightarrow\infty}\sup\int_{B_{j}\setminus B_{r}}|\tilde {u_{j}}|^{d}\leq\int_{\mathbb{R}^{N}\setminus B_{r}}|u|^{d} \leq\varepsilon \quad \mbox{for all } r \geq r_{\varepsilon}. $$

By using Lemma 3.2 and the assumption (H₂), we get

$$\begin{aligned}& \lim_{j\rightarrow\infty}\sup\biggl\vert \int_{\mathbb {R}^{N}} \bigl(f(x,u_{n_{j}})-f(x,u_{n_{j}}-\tilde{u}_{j})-f(x, \tilde {u}_{j})\bigr)\varphi\biggr\vert \\& \quad = \lim_{j\rightarrow\infty}\sup\biggl\vert \int_{B_{j}\setminus B_{r}} \bigl(f(x,u_{n_{j}})-f(x,u_{n_{j}}-\tilde{u}_{j})-f(x, \tilde{u}_{j})\bigr)\varphi \biggr\vert \\& \quad \leq c_{2}\lim_{j\rightarrow\infty}\sup\biggl\vert \int _{B_{j}\setminus B_{r}}\bigl(\vert u_{n_{j}}\vert +| \tilde{u}_{j}|\bigr)|\varphi|\biggr\vert \\& \qquad {} +c_{3}\lim_{j\rightarrow\infty}\sup\biggl\vert \int _{B_{j}\setminus B_{r}}\bigl(|u_{n_{j}}|^{q-1}+| \tilde{u}_{j}|^{q-1}\bigr)|\varphi|\biggr\vert \\& \quad \leq c_{2}\lim_{j\rightarrow\infty}\sup\bigl(\Vert u_{n_{j}}\Vert _{L^{2}(B_{j}\setminus B_{r})}+\|\tilde{u}_{j} \|_{L^{2}(B_{j}\setminus B_{r})}\bigr)\| \varphi\|_{2} \\& \qquad {}+c_{3}\lim_{j\rightarrow\infty}\sup\bigl( \|u_{n_{j}}\|_{L^{q}(B_{j}\setminus B_{r})}^{q-1}+\|\tilde{u}_{j} \|_{L^{q}(B_{j}\setminus B_{r})}^{q-1}\bigr)\|\varphi\| _{q} \\& \quad \leq c_{4}\varepsilon^{\frac{1}{2}}+c_{5} \varepsilon^{\frac{q-1}{q}}, \end{aligned}$$

which implies that

$$\lim_{j\rightarrow\infty}\biggl\vert \int_{\mathbb {R}^{N}} \bigl(f(x,u_{n_{j}})-f(x,u_{n_{j}}-\tilde{u}_{j})-f(x, \tilde {u}_{j})\bigr)\varphi\biggr\vert =0. $$

Similar to this proof, we can prove that the other result is correct. □

Lemma 3.4

Passing to a subsequence, we have

$$I_{\lambda}(u_{n}-\tilde{u}_{n},v_{n}- \tilde{v}_{n})\rightarrow c-I_{\lambda}(u,v) $$

and

$$I_{\lambda}'(u_{n}-\tilde{u}_{n},v_{n}- \tilde{v}_{n})\rightarrow0\quad \textit{in } E^{\ast}. $$

Proof

Together with the fact that $(u_{n},v_{n})\rightharpoonup(u,v)$, $(\tilde {u}_{n},\tilde{v}_{n})\rightarrow(u,v)$ in E, we get

$$\begin{aligned}& I_{\lambda}(u_{n}-\tilde{u}_{n},v_{n}- \tilde{v}_{n}) \\& \quad = I_{\lambda}(u_{n},v_{n})-I_{\lambda}( \tilde{u}_{n},\tilde{v}_{n}) \\& \qquad {} +\frac{\lambda}{\alpha+\beta}\int_{\mathbb {R}^{N}}K(x) \bigl(|u_{n}|^{\alpha}|v_{n}|^{\beta}-|u_{n}- \tilde{u}_{n}|^{\alpha }|v_{n}-\tilde{v}_{n}|^{\beta}-| \tilde{u}_{n}|^{\alpha}|\tilde {v}_{n}|^{\beta} \bigr) \\& \qquad {}+\lambda\int_{\mathbb{R}^{N}}\bigl(F(x,u_{n})-F(x,u_{n}- \tilde {u}_{n})-F(x,\tilde{u}_{n})\bigr) \\& \qquad {}+\lambda\int_{\mathbb{R}^{N}}\bigl(G(x,v_{n})-G(x,v_{n}- \tilde {v}_{n})-G(x,\tilde{v}_{n})\bigr)+o(1). \end{aligned}$$

Similar to the proof of the Brézis-Lieb lemma [28], we easily get

$$\begin{aligned}& \lim_{n\rightarrow\infty}\int_{\mathbb{R}^{N}}K(x) \bigl(|u_{n}|^{\alpha }|v_{n}|^{\beta}-|u_{n}- \tilde{u}_{n}|^{\alpha}|v_{n}-\tilde{v}_{n}|^{\beta }-| \tilde{u}_{n}|^{\alpha}|\tilde{v}_{n}|^{\beta} \bigr)=0, \\& \lim_{n\rightarrow\infty}\int_{\mathbb {R}^{N}}\bigl(F(x,u_{n})-F(x,u_{n}- \tilde{u}_{n})-F(x,\tilde{u}_{n})\bigr)=0 \end{aligned}$$

and

$$\lim_{n\rightarrow\infty}\int_{\mathbb {R}^{N}}\bigl(G(x,v_{n})-G(x,v_{n}- \tilde{v}_{n})-G(x,\tilde{v}_{n})\bigr)=0. $$

Observing the fact that $I_{\lambda}(u_{n},v_{n})\rightarrow c$ and $I_{\lambda}(\tilde{u}_{n},\tilde{v}_{n})\rightarrow I_{\lambda}(u,v)$, we obtain

$$I_{\lambda}(u_{n}-\tilde{u}_{n},v_{n}- \tilde{v}_{n})\rightarrow c-I_{\lambda}(u,v). $$

In addition, for any $(\varphi,\psi)\in E$, we get

$$\begin{aligned}& I_{\lambda}'(u_{n}-\tilde{u}_{n},v_{n}- \tilde{v}_{n}) (\varphi,\psi) \\& \quad = I_{\lambda}'(u_{n},v_{n}) ( \varphi,\psi)-I_{\lambda}'(\tilde {u}_{n}, \tilde{v}_{n}) (\varphi,\psi) \\& \qquad {} +\frac{\lambda\alpha}{\alpha+\beta}\int_{\mathbb {R}^{N}}K(x) \bigl(|u_{n}|^{\alpha-2}u_{n}|v_{n}|^{\beta}-|u_{n}- \tilde {u}_{n}|^{\alpha-2}(u_{n}-\tilde{u}_{n})|v_{n}- \tilde{v}_{n}|^{\beta} \\& \qquad {}-|\tilde {u}_{n}|^{\alpha-2} \tilde{u}_{n}|\tilde{v}_{n}|^{\beta}\bigr)\varphi \\& \qquad {} +\frac{\lambda\beta}{\alpha+\beta}\int_{\mathbb {R}^{N}}K(x) \bigl(|u_{n}|^{\alpha}|v_{n}|^{\beta-2}v_{n}-|u_{n}- \tilde {u}_{n}|^{\alpha}|v_{n}-\tilde{v}_{n}|^{\beta-2}(v_{n}- \tilde{v}_{n}) \\& \qquad {}-|\tilde {u}_{n}|^{\alpha}| \tilde{v}_{n}|^{\beta-2}\tilde{v}_{n}\bigr)\psi \\& \qquad {} +\lambda\int_{\mathbb{R}^{N}}\bigl(f(x,u_{n})-f(x,u_{n}- \tilde {u}_{n})-f(x,\tilde{u}_{n})\bigr)\varphi \\& \qquad {} +\lambda\int_{\mathbb{R}^{N}}\bigl(g(x,v_{n})-g(x,v_{n}- \tilde {v}_{n})-g(x,\tilde{v}_{n})\bigr)\psi. \end{aligned}$$

It is standard to check

$$\lim_{n\rightarrow\infty}\int_{\mathbb{R}^{N}}K(x) \bigl(|u_{n}|^{\alpha -2}u_{n}|v_{n}|^{\beta}-|u_{n}- \tilde{u}_{n}|^{\alpha-2}(u_{n}-\tilde {u}_{n})|v_{n}-\tilde{v}_{n}|^{\beta}-| \tilde{u}_{n}|^{\alpha-2}\tilde {u}_{n}| \tilde{v}_{n}|^{\beta}\bigr)\varphi=0 $$

and

$$\lim_{n\rightarrow\infty}\int_{\mathbb{R}^{N}}K(x) \bigl(|u_{n}|^{\alpha }|v_{n}|^{\beta-2}v_{n}-|u_{n}- \tilde{u}_{n}|^{\alpha}|v_{n}-\tilde {v}_{n}|^{\beta-2}(v_{n}-\tilde{v}_{n})-| \tilde{u}_{n}|^{\alpha}|\tilde {v}_{n}|^{\beta-2} \tilde{v}_{n}\bigr)\psi=0 $$

uniformly in $\|(\varphi,\psi)\|_{E}\leq1$. By the fact of Lemma 3.3 and $I_{\lambda}'(u_{n},v_{n})\rightarrow0$, we complete the proof of Lemma 3.4. □

Set $u_{n}^{1}=u_{n}-\tilde{u}_{n}$ and $v_{n}^{1}=v_{n}-\tilde{v}_{n}$, then $u_{n}-u=u_{n}^{1}+(\tilde{u}_{n}-u)$ and $v_{n}-v=v_{n}^{1}+(\tilde{v}_{n}-v)$. We easily get $(u_{n},v_{n})\rightarrow(u,v)$ in E if and only if $(u_{n}^{1},v_{n}^{1})\rightarrow(0,0)$ in E.

Observe that

$$\begin{aligned}& I_{\lambda}\bigl(u_{n}^{1},v_{n}^{1} \bigr)-\frac{1}{2}I_{\lambda }'\bigl(u_{n}^{1},v_{n}^{1} \bigr) \bigl(u_{n}^{1},v_{n}^{1}\bigr) \\& \quad = \biggl(\frac{1}{2}-\frac{1}{\alpha+\beta}\biggr)\lambda\int _{\mathbb {R}^{N}}K(x)\bigl\vert u_{n}^{1}\bigr\vert ^{\alpha}\bigl\vert v_{n}^{1}\bigr\vert ^{\beta} \\& \qquad {} +\lambda\int_{\mathbb{R}^{N}}\biggl(\frac {1}{2}\bigl(f \bigl(x,u_{n}^{1}\bigr)u_{n}^{1}+g \bigl(x,v_{n}^{1}\bigr)v_{n}^{1}\bigr)-F \bigl(x,u_{n}^{1}\bigr)-G\bigl(x,v_{n}^{1} \bigr)\biggr) \\& \quad \geq \frac{\lambda}{N}K_{0}\int_{\mathbb{R}^{N}}\bigl\vert u_{n}^{1}\bigr\vert ^{\alpha }\bigl\vert v_{n}^{1}\bigr\vert ^{\beta}, \end{aligned}$$

where $K_{0}=\inf_{x\in\mathbb{R}^{N}}K(x)>0$. In connection with $I_{\lambda}(u_{n}^{1},v_{n}^{1})\rightarrow c-I_{\lambda}(u,v)$ and $I_{\lambda}'(u_{n}^{1},v_{n}^{1})\rightarrow0$ in $E^{\ast}$, we get

$$ \int_{\mathbb{R}^{N}}\bigl\vert u_{n}^{1} \bigr\vert ^{\alpha}\bigl\vert v_{n}^{1}\bigr\vert ^{\beta}\leq\frac {N(c-I_{\lambda}(u,v))}{\lambda K_{0}}+o(1). $$

(3.3)

In addition, by (K₀) and (H₂), for any $b>0$, there is a constant $C_{b}>0$ such that

$$\begin{aligned}& \int_{\mathbb{R}^{N}}\bigl(K(x)\bigl\vert u_{n}^{1} \bigr\vert ^{\alpha}\bigl\vert v_{n}^{1}\bigr\vert ^{\beta }+f\bigl(x,u_{n}^{1} \bigr)u_{n}^{1}+g\bigl(x,v_{n}^{1} \bigr)v_{n}^{1}\bigr) \\& \quad \leq b\bigl(\bigl\Vert u_{n}^{1}\bigr\Vert _{2}^{2}+\bigl\Vert v_{n}^{1}\bigr\Vert _{2}^{2}\bigr)+C_{b}\int _{\mathbb {R}^{N}}\bigl\vert u_{n}^{1}\bigr\vert ^{\alpha}\bigl\vert v_{n}^{1}\bigr\vert ^{\beta}. \end{aligned}$$

Let $V_{b}(x):=\max\{V(x),b\}$, where b is the positive constant in the assumption (V₀). Since the set $\nu^{b}:=\{x\in\mathbb{R}^{N}:V(x)< b\}$ has a finite Lebesgue measure and $(u_{n}^{1},v_{n}^{1})\rightarrow(0,0)$ in $L_{\mathrm{loc}}^{2}(\mathbb{R}^{N})\times L_{\mathrm{loc}}^{2}(\mathbb{R}^{N})$, we have

$$ \int_{\mathbb{R}^{N}}V(x) \bigl(\bigl\vert u_{n}^{1}\bigr\vert ^{2}+\bigl\vert v_{n}^{1}\bigr\vert ^{2}\bigr)=\int _{\mathbb {R}^{N}}V_{b}(x) \bigl(\bigl\vert u_{n}^{1}\bigr\vert ^{2}+\bigl\vert v_{n}^{1}\bigr\vert ^{2}\bigr)+o(1). $$

(3.4)

Thus

$$\begin{aligned}& S_{\alpha,\beta}\biggl(\int_{\mathbb{R}^{N}}\bigl\vert u_{n}^{1}\bigr\vert ^{\alpha }\bigl\vert v_{n}^{1}\bigr\vert ^{\beta}\biggr)^{\frac{2}{\alpha+\beta}} \\& \quad \leq \int_{\mathbb{R}^{N}}\bigl(\bigl\vert \nabla u_{n}^{1}\bigr\vert ^{2}+\bigl\vert \nabla v_{n}^{1}\bigr\vert ^{2}\bigr) \\& \quad = \int_{\mathbb{R}^{N}}\bigl(\bigl\vert \nabla u_{n}^{1}\bigr\vert ^{2}+\bigl\vert \nabla v_{n}^{1}\bigr\vert ^{2}+\lambda V(x)\bigl\vert u_{n}^{1}\bigr\vert ^{2}+\lambda V(x) \bigl\vert v_{n}^{1}\bigr\vert ^{2}\bigr)-\int _{\mathbb {R}^{N}}\lambda V(x) \bigl(\bigl\vert u_{n}^{1} \bigr\vert ^{2}+\bigl\vert v_{n}^{1}\bigr\vert ^{2}\bigr) \\& \quad = \lambda\int_{\mathbb{R}^{N}}\bigl(K(x)\bigl\vert u_{n}^{1}\bigr\vert ^{\alpha}\bigl\vert v_{n}^{1}\bigr\vert ^{\beta }+f\bigl(x,u_{n}^{1} \bigr)u_{n}^{1}+g\bigl(x,v_{n}^{1} \bigr)v_{n}^{1}\bigr) \\& \qquad {}-\lambda\int_{\mathbb {R}^{N}}V_{b}(x) \bigl(\bigl\vert u_{n}^{1}\bigr\vert ^{2}+ \bigl\vert v_{n}^{1}\bigr\vert ^{2}\bigr)+o(1) \\& \quad \leq \lambda C_{b}\int_{\mathbb{R}^{N}}\bigl\vert u_{n}^{1}\bigr\vert ^{\alpha }\bigl\vert v_{n}^{1}\bigr\vert ^{\beta}+\lambda b\bigl(\bigl\Vert u_{n}^{1}\bigr\Vert ^{2}+\bigl\Vert v_{n}^{1}\bigr\Vert ^{2}\bigr) -\lambda V_{b}(x) \bigl(\bigl\Vert u_{n}^{1}\bigr\Vert ^{2}+\bigl\Vert v_{n}^{1}\bigr\Vert ^{2}\bigr)+o(1) \\& \quad \leq \lambda C_{b}\int_{\mathbb{R}^{N}}\bigl\vert u_{n}^{1}\bigr\vert ^{\alpha }\bigl\vert v_{n}^{1}\bigr\vert ^{\beta}+o(1). \end{aligned}$$

Together with (3.3), we have

$$\begin{aligned} S_{\alpha,\beta}&\leq\lambda C_{b}\biggl(\int_{\mathbb {R}^{N}} \bigl\vert u_{n}^{1}\bigr\vert ^{\alpha}\bigl\vert v_{n}^{1}\bigr\vert ^{\beta} \biggr)^{1-\frac{2}{\alpha+\beta }}+o(1) \\ &\leq \lambda C_{b}\biggl(\frac{N(c-I_{\lambda}(u,v))}{\lambda K_{0}} \biggr)^{\frac {2}{N}}+o(1) \\ & = \lambda^{1-\frac{2}{N}}C_{b}\biggl(\frac{N}{K_{0}} \biggr)^{\frac {2}{N}}\bigl(c-I_{\lambda}(u,v)\bigr)^{\frac{2}{N}}+o(1). \end{aligned}$$

Set $\alpha_{0}=S_{\alpha,\beta}^{\frac{N}{2}}C_{b}^{-\frac {N}{2}}N^{-1}K_{0}$. This implies $\alpha_{0}\lambda^{1-\frac {N}{2}}\leq c-I_{\lambda}(u,v)+o(1)$.

Lemma 3.5

Assume that (V₀), (K₀), and (H₁)-(H₃) are satisfied. Then, for any $(\mathrm{PS})_{c}$, the sequence $\{(u_{n},v_{n})\}$ for $I_{\lambda}$, there exists a constant $\alpha _{0}>0$ (independent of λ) such that the functional $I_{\lambda}(u,v)$ satisfies the $(\mathrm{PS})_{c}$ condition for all $c< \alpha _{0}\lambda^{1-\frac{N}{2}}$.

Proof

We can check that, for any $(\mathrm{PS})_{c}$ sequence $\{(u_{n},v_{n})\}\subset E$ with $(u_{n},v_{n})\rightharpoonup(u,v)$, either $(u_{n},v_{n})\rightarrow (u,v)$ or $c-I_{\lambda}(u,v)\geq\alpha_{0}\lambda^{1-\frac{N}{2}}$.

On the contrary, if $(u_{n},v_{n})\nrightarrow(u,v)$, this shows

$$\lim\inf_{n\rightarrow\infty}\bigl\Vert (u_{n},v_{n}) \bigr\Vert _{E}>0 $$

and

$$c-I_{\lambda}(u,v)>0. $$

Based on the above mentioned conclusion, we easily find that the functional $I_{\lambda}(u,v)$ satisfies the $(\mathrm{PS})_{c}$ condition for all $c<\alpha_{0}\lambda^{1-\frac{N}{2}}$. □

Mountain-pass structure

We consider $\lambda\geq1$ and check that the functional $I_{\lambda}$ possesses the mountain-pass structure.

Lemma 4.1

Assume that (V₀), (K₀), and (H₁)-(H₃) are satisfied. There exist $\alpha_{\lambda}, \rho _{\lambda}>0$ such that

$$I_{\lambda}(u,v)>0 \quad \textit{if } 0< \bigl\Vert (u,v)\bigr\Vert _{E}< \rho_{\lambda}\quad \textit {and}\quad I_{\lambda}(u,v) \geq\alpha_{\lambda}\quad \textit{if } \bigl\Vert (u,v)\bigr\Vert _{E}=\rho_{\lambda}. $$

Proof

Note that

$$\begin{aligned} I_{\lambda}(u,v) =&\frac{1}{2}\int_{\mathbb{R}^{N}}\bigl(| \nabla u|^{2}+\lambda V(x)u^{2}+|\nabla v|^{2}+ \lambda V(x)v^{2}\bigr) \\ &{} -\lambda\int_{\mathbb{R}^{N}}\bigl(F(x,u)+G(x,v)\bigr)- \frac{\lambda }{\alpha+\beta} \int_{\mathbb{R}^{N}}K(x)|u|^{\alpha}|v|^{\beta }. \end{aligned}$$

It is clear that, for each $s\in[2,2^{\ast}]$, there is $c_{s}$ such that if $\lambda\geq1$,

$$\|u\|_{s}\leq c_{s}\|u\|_{E_{\lambda}} \quad \mbox{for all } u\in E_{\lambda}. $$

By the Young inequality, we have

$$ |u|^{\alpha}|v|^{\beta}\leq\frac{\alpha}{\alpha+\beta }|u|^{\alpha+\beta}+ \frac{\beta}{\alpha+\beta}|v|^{\alpha+\beta}. $$

Furthermore, we get

$$ \int_{\mathbb{R}^{N}}K(x)|u|^{\alpha}|v|^{\beta} \leq c_{1}\bigl(\|u\| _{2^{\ast}}^{2^{\ast}}+\|v \|_{2^{\ast}}^{2^{\ast}}\bigr)\leq c_{1}c_{2^{\ast}}\bigl\Vert (u,v)\bigr\Vert _{E}^{2^{\ast}}. $$

(4.1)

Combining (H₃) and (4.1), there is a constant $c_{\delta}$ such that

$$ I_{\lambda}(u,v)\geq\frac{1}{4}\bigl\Vert (u,v)\bigr\Vert _{E}^{2}-c_{\delta}\bigl\Vert (u,v)\bigr\Vert _{E}^{2^{\ast}}=\frac{1}{4}\bigl\Vert (u,v)\bigr\Vert _{E}^{2}\bigl(1-4c_{\delta}\bigl\Vert (u,v)\bigr\Vert _{E}^{2^{\ast}-2}\bigr). $$

Setting $\rho_{\lambda}=(\frac{1}{8c_{\delta}})^{\frac{1}{2^{\ast }-2}}$ implies

$$I_{\lambda}(u,v)\geq\frac{1}{8}\rho_{\lambda}^{2} \triangleq\alpha _{\lambda}>0\quad \mbox{if } \bigl\Vert (u,v)\bigr\Vert _{E}=\rho_{\lambda}. $$

The proof is completed. □

Lemma 4.2

For any finite-dimensional subspace $F\subset E$, we have

$$I_{\lambda}(u,v)\rightarrow-\infty \quad \textit{as } \bigl\Vert (u,v) \bigr\Vert _{E}\rightarrow\infty \textit{ for } (u,v)\in F. $$

Proof

By the assumptions (K₀) and (H₃), it follows that

$$I_{\lambda}(u,v)\leq\frac{1}{2}\bigl\Vert (u,v)\bigr\Vert _{E}^{2}-\lambda a_{0}\bigl\Vert (u,v)\bigr\Vert _{p}^{p} \quad \mbox{for all } (u,v)\in F. $$

In connection with the fact that all norms in a finite-dimensional space are equivalent and $p>2$, we easily get the desired conclusion. □

Lemma 4.3

For any $\sigma>0$, there is $\Lambda _{\sigma}>0$ such that for each $\lambda\geq\Lambda_{\sigma}$, there exists $\bar{e}_{\lambda}\in E$ with $\|\bar{e}_{\lambda}\|_{E}>\rho _{\lambda}$, and we have

$$I_{\lambda}(\bar{e}_{\lambda})\leq0 $$

and

$$\max_{t\geq0}I_{\lambda}(t\bar{e}_{\lambda})\leq \sigma\lambda ^{1-\frac{N}{2}}, $$

where $\rho_{\lambda}$ is defined in Lemma  4.1.

Proof

Define the functionals

$$\begin{aligned} \Phi_{\lambda}(u,v) =&\frac{1}{2}\int_{\mathbb{R}^{N}}\bigl(| \nabla u|^{2}+\lambda V(x)|u|^{2}+|\nabla v|^{2}+ \lambda V(x)|v|^{2}\bigr) \\ &{}-\lambda a_{0}\int _{\mathbb{R}^{N}}\bigl(|u|^{p}+|v|^{p}\bigr) \end{aligned}$$

and

$$\begin{aligned} \Psi_{\lambda}(u,v) =&\frac{1}{2}\int_{\mathbb{R}^{N}}\bigl(| \nabla u|^{2}+|\nabla v|^{2}+V\bigl(\lambda^{-\frac{1}{2}}x \bigr) \bigl(|u|^{2}+|v|^{2}\bigr)\bigr) \\ &{}-a_{0}\int _{\mathbb{R}^{N}}\bigl(|u|^{p}+|v|^{p}\bigr). \end{aligned}$$

We obtain $\Phi_{\lambda}\in C^{1}(E)$ and $I_{\lambda}(u,v)\leq\Phi _{\lambda}(u,v)$ for all $(u,v)\in E$.

Observe that

$$\inf\biggl\{ \int_{\mathbb{R}^{N}}|\nabla\phi|^{2}:\phi\in C_{0}^{\infty }\bigl(\mathbb{R}^{N},\mathbb{R}\bigr), \|\phi\|_{p}=1\biggr\} =0. $$

For any $\delta>0$, there are $\phi_{\delta}, \psi_{\delta}\in C_{0}^{\infty}(\mathbb{R}^{N},\mathbb{R})$ with $\|\phi_{\delta}\| _{p}=\|\psi_{\delta}\|_{p}=1$ such that

$$\operatorname{supp}(\phi_{\delta},\psi_{\delta})\subset B_{r_{\delta}}(0)\quad \mbox{and}\quad \|\nabla\phi_{\delta}\|_{2}^{2}, \|\nabla\psi_{\delta}\| _{2}^{2}< \delta. $$

Let $e_{\lambda}(x)=(\phi_{\delta}(\lambda^{\frac{1}{2}}x),\psi _{\delta}(\lambda^{\frac{1}{2}}x))$, then $\operatorname{supp}e_{\lambda }\subset B_{\lambda^{-\frac{1}{2}}r_{\delta}}(0)$. Furthermore, we get

$$\Phi_{\lambda}(te_{\lambda})=\lambda^{1-\frac{N}{2}} \Psi_{\lambda }(t\phi_{\delta},t\psi_{\delta}). $$

It is clear that

$$\begin{aligned} \begin{aligned} \max_{t\geq0}\Psi_{\lambda}(t\phi_{\delta},t \psi_{\delta})\leq {}&\frac{p-2}{2p(p a_{0})^{\frac{2}{p-2}}}\biggl\{ \int_{\mathbb {R}^{N}} \bigl(|\nabla\phi_{\delta}|^{2}+V\bigl(\lambda^{-\frac{1}{2}}x \bigr)|\phi _{\delta}|^{2}\bigr)\biggr\} ^{\frac{p}{p-2}} \\ &{}+\frac{p-2}{2p(p a_{0})^{\frac{2}{p-2}}}\biggl\{ \int_{\mathbb {R}^{N}}\bigl(|\nabla \psi_{\delta}|^{2}+V\bigl(\lambda^{-\frac{1}{2}}x\bigr)|\psi _{\delta}|^{2}\bigr)\biggr\} ^{\frac{p}{p-2}}. \end{aligned} \end{aligned}$$

Combining $V(0)=0$ and $\operatorname{supp}(\phi_{\delta},\psi_{\delta})\subset B_{r_{\delta}}(0)$, there is $\Lambda_{\delta}>0$ such that, for all $\lambda\geq\Lambda_{\delta}$, we have

$$ \max_{t\geq0}\Phi_{\lambda}(t\phi_{\delta},t \psi_{\delta})\leq \lambda^{1-\frac{N}{2}}\frac{(p-2)}{p(p a_{0})^{\frac {2}{p-2}}}(2 \delta)^{\frac{p}{p-2}}. $$

Thus, for all $\lambda\geq\Lambda_{\delta}$,

$$ \max_{t\geq0}I_{\lambda}(te_{\lambda}) \leq\lambda^{1-\frac {N}{2}}\frac{(p-2)}{p(p a_{0})^{\frac{2}{p-2}}}(2\delta)^{\frac{p}{p-2}}. $$

(4.2)

For any $\sigma>0$, we can choose $\delta>0$ so small that

$$\frac{(p-2)}{p(p a_{0})^{\frac{2}{p-2}}}(2\delta)^{\frac{p}{p-2}}\leq \sigma $$

and $e_{\lambda}(x)=(\phi_{\delta}(\lambda^{\frac{1}{2}}x),\psi _{\delta}(\lambda^{\frac{1}{2}}x))$. Taking $\Lambda_{\delta}=\Lambda _{\sigma}$, there is $\bar{t}_{\lambda}>0$ such that $\|\bar{t}_{\lambda}e_{\lambda}\|_{E}>\rho_{\lambda}$ and $I_{\lambda}(te_{\lambda})\leq0$ for all $t\geq\bar{t}_{\lambda}$. By (4.2), $\bar{e}_{\lambda}=\bar{t}_{\lambda}e_{\lambda}$ satisfies the requirements. □

Proof of main theorem

Proof of Theorem 2

Define

$$c_{\lambda}=\inf_{\gamma\in\Gamma_{\lambda}}\max_{t\in [0,1]}I_{\lambda}\bigl(\gamma(t)\bigr), $$

where $\Gamma_{\lambda}=\{\gamma\in C([0,1],E):\gamma(0)=0,\gamma (1)=\bar{e}_{\lambda}\}$.

In addition, for any $\sigma>0$ with $\sigma<\alpha_{0}$, there is $\Lambda_{\sigma}>0$ such that for $\lambda\geq\Lambda_{\sigma}$, we can choose $c_{\lambda}$ which satisfies $c_{\lambda}\leq\sigma\lambda ^{1-\frac{N}{2}}$.

From the above mentioned results, the functional $I_{\lambda}$ satisfies the $(\mathrm{PS})_{c_{\lambda}}$ condition if $c_{\lambda}\leq\sigma\lambda^{1-\frac{N}{2}}$ and has the mountain-pass structure. Hence, there is $(u_{\lambda},v_{\lambda})\in E$ such that

$$I_{\lambda}(u_{\lambda},v_{\lambda})=c_{\lambda}\quad \mbox{and} \quad I_{\lambda}'(u_{\lambda},v_{\lambda})=0. $$

That is to say, $(u_{\lambda},v_{\lambda})$ is a weak solution of (2.1). Similar to the arguments in [4], we also find that $(u_{\lambda},v_{\lambda})$ is a positive least energy solution.

Furthermore,

$$\begin{aligned} I_{\lambda}(u_{\lambda},v_{\lambda}) =&I_{\lambda}(u_{\lambda},v_{\lambda})-\frac{1}{\mu}I_{\lambda}'(u_{\lambda},v_{\lambda}) (u_{\lambda},v_{\lambda}) \\ \geq&\biggl(\frac{1}{2}-\frac{1}{\mu}\biggr)\bigl\Vert (u_{\lambda},v_{\lambda})\bigr\Vert _{E}^{2}. \end{aligned}$$

This shows that

$$\frac{\mu-2}{2\mu}\bigl\Vert (u_{\lambda},v_{\lambda})\bigr\Vert _{E}^{2} \leq I_{\lambda}(u_{\lambda},v_{\lambda})=c_{\lambda}\leq\sigma\lambda^{1-\frac{N}{2}}. $$

We complete the proof of Theorem 2. □