Existence of ground state solutions to a class of fractional Schrödinger system with linear and nonlinear couplings

Abstract

In this paper, we study the existence of ground state solutions to the following fractional Schrödinger system with linear and nonlinear couplings:

$$ \textstyle\begin{cases} (-\triangle )^{s}u+(\lambda _{1}+V(x))u+kv=\mu _{1}u^{3}+\beta uv^{2}, \quad \text{in } R^{3},\\ (-\triangle )^{s}v+(\lambda _{2}+V(x))v+ku=\mu _{2}v^{3}+ \beta u^{2}v, \quad \text{in } R^{3},\\ u, v\in H^{s}(R^{3}), \end{cases} $$

where $(-\triangle )^{s}$ denotes the fractional Laplacian of order $s\in (\frac{3}{4},1)$. Under some assumptions of the potential $V(x)$ and the linear and nonlinear coupling constants k, β, we prove some results for the existence of ground state solutions for the fractional Laplacian systems by using variational methods.

1 Introduction

The aim of this paper is to consider the existence of ground state solutions to the following fractional Schrödinger system with linear and nonlinear couplings:

$$ \textstyle\begin{cases} (-\triangle )^{s}u+(\lambda _{1}+V(x))u+kv=\mu _{1}u^{3}+\beta uv^{2}, \quad \text{in } R^{3},\\ (-\triangle )^{s}v+(\lambda _{2}+V(x))v+ku=\mu _{2}v^{3}+ \beta u^{2}v, \quad \text{in } R^{3},\\ u, v\in H^{s}(R^{3}), \end{cases} $$

(1.1)

where $(-\triangle )^{s}$ denotes the fractional Laplacian of order $s\in (\frac{3}{4},1)$, $\lambda _{1}$, $\lambda _{2}$, $\mu _{1}$, $\mu _{2}$ are positive constants, k, β are linear and nonlinear coupling constants respectively. The potential function $V(x)$ will always be assumed to satisfy

$$ (V)\quad \sup_{x\in R^{3}}V(x)=\lim_{\vert x\vert \rightarrow +\infty }V(x)=\varLambda >0, \quad\quad \inf_{x\in R^{3}}V(x)\geq 0. $$

If $V(x)\equiv 0$, then (1.1) becomes

$$ \textstyle\begin{cases} (-\triangle )^{s}u+\lambda _{1}u+kv=\mu _{1}u^{3}+\beta uv^{2}, \quad \text{in } R^{3},\\ (-\triangle )^{s}v+\lambda _{2}v+ku=\mu _{2}v^{3}+\beta u^{2}v, \quad \text{in } R^{3},\\ u, v\in H^{s}(R^{3}). \end{cases} $$

(1.2)

It is well known, but not completely trivial, that $(-\triangle )^{s}$ reduces to the standard Laplacian −△ as $s\rightarrow 1$. In the local case, that is, when $s=1$, the system (1.2) reduces to the following system:

$$ \textstyle\begin{cases} -\triangle u+\lambda _{1}u+kv=\mu _{1}u^{3}+\beta uv^{2}, \quad \text{in } R^{3},\\ -\triangle v+\lambda _{2}v+ku=\mu _{2}v^{3}+\beta u^{2}v, \quad \text{in } R^{3},\\ u, v\in H^{1}(R^{3}). \end{cases} $$

(1.3)

System (1.3) appears in several physical situations such as in nonlinear optics, in double Bose–Einstein condensates and in plasma physics, and it has been extensively studied by many authors in the past ten years; see, for example, [1,2,3] and the references therein.

In the nonlocal case, that is, when $s\in (0,1)$, there are very few results for the fractional Laplacian systems. If $\beta =0$, i.e., only linear coupling terms exist, in [4], Dengfeng Lv and Shuangjie Peng studied the problem

$$ \textstyle\begin{cases} (-\triangle )^{s}u+u=f(u)+k v, \quad \text{in } R^{N},\\ (-\triangle )^{s} v+v=g(v)+k u, \quad \text{in } R^{N}. \end{cases} $$

(1.4)

They obtained the existence of positive vector solutions and vector ground state solutions via variational methods. They also proved the asymptotic behavior of these solutions as the coupling parameter k tends to zero. If $k=0$, i.e., only nonlinear coupling terms exist, in [5], Q. Guo and X. He studied the problem

$$ \textstyle\begin{cases} (-\triangle )^{s}u+u=( \vert u \vert ^{2p}+b \vert u \vert ^{p-1} \vert v \vert ^{p-1})u, \quad \text{in } R^{N},\\ (-\triangle )^{s}v+\omega ^{2s}v=( \vert v \vert ^{2p}+b \vert v \vert ^{p-1} \vert u \vert ^{p-1})v, \quad \text{in } R^{N},\\ u, v\in H^{s}(R^{N}), \end{cases} $$

(1.5)

where $\omega >0$, $b>0$ are constants and p satisfies $2<2p+2<2_{s} ^{*}$. They obtained the existence of a least energy solution via Nehari manifold method. They also proved that if b is large enough, system (1.5) has a positive least energy solution with both nontrivial components. When $k\neq0$ and $\beta \neq0$, i.e., linear coupling terms and nonlinear coupling terms both exist, to the best of our knowledge, there has been almost no research on this problem. For the other work on the fractional Laplacian system, we refer the reader to [6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23] and the references therein. We would also like to mention [24,25,26,27,28,29,30,31,32,33,34,35,36] and the references therein for the information of fractional ordinary differential equations.

A solution $(u,v)\in H^{s}(R^{3})\times H^{s}(R^{3})$ to (1.1) is called a nontrivial solution if $(u,v)\neq(0,0)$. A solution $(u,v)$ with $u>0$ and $v>0$ is called a positive solution. A solution $(u,v)$ is called a ground state solution if $(u,v)\neq(0,0)$ and its energy is minimal among the energy of all nontrivial solutions to (1.1).

Our main theorems of this work read as follows.

Theorem 1.1

Suppose that $\lambda _{1}, \lambda _{2}, \mu _{1}, \mu _{2}>0$, $\beta \in R$, $k\in (-\sqrt{\lambda _{1}\lambda _{2}},0)\cup (0,\sqrt{\lambda _{1}\lambda _{2}})$. Then problem (1.2) possesses a ground state solution $(u,v)$. Moreover, $u>0$, $v>0$ as $k\in (-\sqrt{\lambda _{1}\lambda _{2}},0)$; $u>0$, $v<0$ or $u<0$, $v>0$ as $k\in (0, \sqrt{\lambda _{1}\lambda _{2}})$.

Theorem 1.2

Suppose that $\lambda _{1}, \lambda _{2}, \mu _{1}, \mu _{2}>0$, $\beta \in (-\sqrt{\mu _{1}\mu _{2}},+\infty )$, $k \in (-\sqrt{\lambda _{1}\lambda _{2}},0)\cup (0,\sqrt{\lambda _{1} \lambda _{2}})$ and that $(V)$ holds. Then problem (1.1) possesses a ground state solution $(u,v)$. Moreover, $u>0$, $v>0$ as $k\in (-\sqrt{ \lambda _{1}\lambda _{2}},0)$; $u>0$, $v<0$ or $u<0$, $v>0$ as $k\in (0,\sqrt{ \lambda _{1}\lambda _{2}})$.

The remainder of this paper is organized as follows. In Sect. 2, some notations and preliminaries are presented. The proofs of Theorem 1.1 and Theorem 1.2 are given in Sects. 3 and 4 respectively.

2 Preliminary

In this section, we outline the variational framework for the problem (1.1) and give some preliminary lemmas which will be used later.

Throughout this paper, C, $C_{i}$ will denote various positive constants; the strong convergence is denoted by →, and the weak convergence is denoted as ⇀; $2_{s}^{*}= \frac{2N}{N-2s}$ is the fractional Sobolev critical exponent; $o_{n}(1)$ denotes $o_{n}(1)\rightarrow 0$ as $n\rightarrow \infty$; For any $\rho >0$ and $z\in R^{3}$, $B_{\rho }(z):=\{x\in R^{3}: \vert x-z\vert \leq \rho \}$; For $1\leq p\leq +\infty$, and $f\in L^{p}(R^{3})$, let $f^{+}=\max \{f,0\}$, $f^{-}=\min \{f,0\}$, and $\vert f\vert _{L^{p}(R^{3})}$ denotes the usual $L^{p}$ norm of f. Let $L^{p}(R^{3})\times L^{p}(R ^{3})$ be the Cartesian product of two $L^{p}(R^{3})$ spaces, and for $(f,g)\in L^{p}(R^{3})\times L^{p}(R^{3})$, $\vert (f,g)\vert _{L^{p}\times L ^{p}}=\vert (f,g)\vert _{L^{p}(R^{3})\times L^{p}(R^{3})}:=(\vert f\vert _{L^{p}(R^{3})} ^{p}+\vert g\vert _{L^{p}(R^{3})}^{p})^{\frac{1}{p}}$.

For any $s\in (0,1)$, the fractional Sobolev space $H^{s}(R^{3})$ is defined by

$$ H^{s}\bigl(R^{3}\bigr)= \biggl\{ u\in L^{2} \bigl(R^{3}\bigr):\frac{ \vert u(x)-u(y) \vert }{ \vert x-y \vert ^{ \frac{3+2s}{2}}}\in L^{2} \bigl(R^{3}\times R^{3}\bigr) \biggr\} . $$

Let us consider a Hilbert space endowed with the scalar product given by

$$ (u,v)= \int _{R^{3}}uv \,dx+ \int _{R^{3}} \int _{R^{3}} \frac{(u(x)-u(y))(v(x)-v(y))}{ \vert x-y \vert ^{3+2s}}\,dx\,dy. $$

The corresponding norm is therefore

$$ \Vert u \Vert _{H^{s}}= \biggl( \int _{R^{3}} \vert u \vert ^{2}\,dx+ \int _{R^{3}} \int _{R^{3}} \frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy \biggr)^{\frac{1}{2}}. $$

It is well known that the fractional Laplacian $(-\triangle )^{s}$ of a function $u: R^{3}\rightarrow R$ is defined by

$$ (-\triangle )^{s}u(x)={\mathcal{F}}^{-1}\bigl( \vert \xi \vert ^{2s}({\mathcal{F}}u)\bigr) (x), \quad \forall \xi \in R^{3}, $$

where $\mathcal{F}$ is the Fourier transform, i.e.,

$$ {\mathcal{F}}u(\xi )=\frac{1}{(2\pi )^{\frac{3}{2}}} \int _{R^{3}}\exp (-2 \pi i\xi \cdot x)u(x)\,dx, $$

i is the imaginary unit. If u is smooth enough, it can be computed by the following singular integral:

$$ (-\triangle )^{s}u(x)=c_{s} \mathit{P.V.} \int _{R^{3}} \frac{u(x)-u(y)}{ \vert x-y \vert ^{3+2s}}\,dy, \quad x\in R^{3}, $$

where $c_{s}$ is normalization constant and P.V. stands for the principal value. Now one can get an alternative definition of the fractional Sobolev space $H^{s}(R^{3})$ via the Fourier transform as follows:

$$ H^{s}\bigl(R^{3}\bigr)= \biggl\{ u\in L^{2} \bigl(R^{3}\bigr): \int _{R^{3}} \vert \xi \vert ^{2s} \vert \hat{u} \vert ^{2}\,d\xi < \infty \biggr\} , $$

endowed with the norm

$$ \Vert u \Vert _{s}=\biggl( \int _{R^{3}} \bigl(1+ \vert \xi \vert ^{2s}\bigr) \vert \hat{u} \vert ^{2}\,d\xi \biggr) ^{\frac{1}{2}}, $$

where $\hat{u}={\mathcal{F}}(u)$ denotes the Fourier transform of u. It is easy to see that $\Vert \cdot \Vert _{H^{s}}$ is equivalent to $\Vert \cdot \Vert _{s}$.

The homogeneous Sobolev space $D^{s,2}(R^{3})$ is defined by

$$ D^{s}\bigl(R^{3}\bigr)= \bigl\{ u\in L^{2_{s}^{*}} \bigl(R^{3}\bigr): \vert \xi \vert ^{s}\hat{u} \in L^{2}\bigl(R^{3}\bigr) \bigr\} , $$

which is the completion of $C_{0}^{\infty }(R^{3})$ under the norm

$$ \Vert u \Vert _{D^{s,2}}= \biggl( \int _{R^{3}} \bigl\vert (-\triangle )^{s/2}u \bigr\vert ^{2}\,dx \biggr) ^{\frac{1}{2}}= \biggl( \int _{R^{3}} \vert \xi \vert ^{2s} \vert \hat{u} \vert ^{2}\,d\xi \biggr) ^{\frac{1}{2}}. $$

Now we introduce the following lemmata.

Lemma 2.1

(see [13])

For any $s\in (0,1)$, $H^{s}(R^{3})$ is continuously embedded into $L^{p}(R^{3})$ for $p\in [2,2_{s}^{*}]$ and compactly embedded into $L_{\mathrm{loc}}^{p}(R^{3})$ for $p\in [1,2_{s}^{*})$.

Lemma 2.2

(see [14])

For any $s\in (0,1)$, $D^{s,2}(R^{3})$ is continuously embedded into $L^{2_{s}^{*}}(R^{3})$ and we define

$$ S_{s}:= \inf_{u\in D^{s,2}(R^{3})\setminus \{0\}}\frac{\int _{R^{3}} \vert (-\triangle )^{s/2}u \vert ^{2}\,dx}{(\int _{R^{3}}u^{2_{s}^{*}})^{2/2_{s}^{*}}}. $$

Lemma 2.3

(see [15])

If $\{u_{n}\}$ is bounded in $H^{s}(R ^{3})$ with $s\in (0,1)$ and

$$ \lim_{n\rightarrow \infty }\sup_{y\in R^{3}} \int _{B_{r}(y)} \vert u_{n} \vert ^{2} \,dx=0, $$

where $r>0$, then we have $u_{n}\rightarrow 0$ in $L^{\nu }(R^{3})$ for $\nu \in (2,2_{s}^{*})$.

Remark 1

Similarly, in the case that the sequence $\{\vert u_{n}\vert ^{2_{s} ^{*} }\}$ is vanishing, we can prove that $u_{n}\rightarrow 0$ in $L^{\nu }(R^{3})$ for $\nu \in (2,2_{s}^{*}]$.

Lemma 2.4

(see [37]; Ekeland variational principle)

Let X be a Banach space and let $G\in C^{2}(X,R)$ be such that, for every $v\in V:=\{v\in X: G(v)=1\}$, $G'(v)\neq0$. Let $F\in C^{1}(X,R)$ be bounded from below on V, $v\in V$ and $\varepsilon , \delta >0$. If

$$ F(v)\leq \inf_{V}F+\varepsilon , $$

then there exists $u\in V$ such that

$$ F(u)\leq \inf_{V}F+2\varepsilon , \quad \quad \min _{\lambda \in R} \bigl\Vert F'(u)-\lambda G'(u) \bigr\Vert \leq \frac{8\varepsilon }{ \delta }, \quad\quad \Vert u-v \Vert \leq 2\delta . $$

3 Proof of Theorem 1.1

In this section, we shall study system (1.2). Suppose that $\lambda _{1}, \lambda _{2}, \mu _{1}, \mu _{2}>0$, $\beta \in R$, $k \in (-\sqrt{\lambda _{1}\lambda _{2}},0)\cup (0,\sqrt{\lambda _{1} \lambda _{2}})$. Let $H:=H^{s}(R^{3})\times H^{s}(R^{3})$. We define an inner product on H as follows:

$$ \begin{aligned} \bigl((u_{1},v_{1}),(u_{2},v_{2}) \bigr)&= \int _{R^{3}}(-\triangle )^{s/2}u_{1}(- \triangle )^{s/2}u_{2}+\lambda _{1}u_{1}(x)u_{2}(x) \,dx \\ & \quad {} + \int _{R^{3}}(- \triangle )^{s/2}v_{1}(- \triangle )^{s/2}v_{2}+\lambda _{2}v_{1}(x)v _{2}(x)\,dx \\ & \quad {}+k \int _{R^{3}}u_{1}(x)v_{1}(x)\,dx+k \int _{R^{3}}u_{2}(x)v_{2}(x)\,dx, \end{aligned} $$

for $(u_{1},v_{1}), (u_{2},v_{2})\in H$. $\Vert (u,v)\Vert =((u,v),(u,v))^{ \frac{1}{2}}$ is the corresponding norm if $\vert k\vert <\sqrt{\lambda _{1} \lambda _{2}}$. This is equivalent to the standard product norm on the product space $H^{s}(R^{3})\times H^{s}(R^{3})$.

For $(u,v)\in H$, the energy functional associated with (1.2) is

$$ \begin{aligned} I(u,v) &=\frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}u \bigr\vert ^{2}+ \lambda _{1}u^{2}(x)\,dx+ \frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}v \bigr\vert ^{2}+\lambda _{2}v^{2}(x)\,dx \\ & \quad {} +k \int _{R^{3}}u(x)v(x)\,dx- \frac{1}{4}\mu _{1} \int _{R^{3}}u^{4}(x)\,dx-\frac{1}{4}\mu _{2} \int _{R ^{3}}v^{4}(x)\,dx \\ & \quad {}-\frac{1}{2}\beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx \\ &= \frac{1}{2} \bigl\Vert (u,v) \bigr\Vert ^{2}- \frac{1}{4}\mu _{1} \int _{R^{3}}u^{4}(x)\,dx- \frac{1}{4}\mu _{2} \int _{R^{3}}v^{4}(x)\,dx-\frac{1}{2}\beta \int _{R^{3}}u ^{2}(x)v^{2}(x)\,dx. \end{aligned} $$

Then, under our assumptions, I is well defined on H and $I\in C^{1}(H,R)$.

Let us define the Nehari manifold

$$ N:=\bigl\{ (u,v)\in H\setminus \bigl\{ (0,0)\bigr\} : F(u,v)=\bigl\langle I'(u,v),(u,v) \bigr\rangle =0\bigr\} . $$

Now we state some properties of N.

Lemma 3.1

If the assumptions of Theorem 1.1 hold, then the following statements hold.

• (1) $N\neq\emptyset$;

• (2) N is a $C^{1}$ manifold;

• (3) there exists a positive constant $\rho _{0}>0$ such that $\Vert (u,v)\Vert \geq \rho _{0}$ for all $(u,v)\in N$;

• (4) the critical points of $I\vert _{N}$ are the critical points of I in H;

• (5) if $\{(u_{n},v_{n})\}\subseteq H$ is a P.S. sequence for $I\vert _{N}$, then $\{(u_{n},v_{n})\}$ is a P.S. sequence for I;

• (6) if $\beta \in (-\sqrt{\mu _{1}\mu _{2}},+\infty )$ then, for any $(u,v)\in H\setminus \{(0,0)\}$, there exists a unique $t_{u,v}>0$ such that $(t_{u,v}u,t_{u,v}v)\in N$.

Proof

(1) The proof of (1) is simple, so we omit the details here.

(2) Since $F(u,v)$ is a $C^{1}$ functional, in order to prove that N is a $C^{1}$ manifold, it is sufficient to prove that $F'(u,v) \neq0$ for all $(u,v)\in N$. For $(u,v)\in N$

$$ \begin{aligned} \bigl\langle F'(u,v),(u,v)\bigr\rangle &=2 \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}u \bigr\vert ^{2}+\lambda _{1}u^{2}\,dx+2 \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}v \bigr\vert ^{2}+\lambda _{2}v^{2}\,dx+4k \int _{R^{3}}uv \,dx \\ &\quad {} -4\mu _{1} \int _{R^{3}}u^{4}\,dx-4\mu _{2} \int _{R^{3}}v^{4}\,dx-8\beta \int _{R^{3}}u ^{2}v^{2}\,dx \\ &=-2 \bigl\Vert (u,v) \bigr\Vert ^{2}< 0. \end{aligned} $$

(3) Notice that if $(u,v)\in N$, then

$$ \bigl\Vert (u,v) \bigr\Vert ^{2}=\mu _{1} \int _{R^{3}}u^{4}\,dx+\mu _{2} \int _{R^{3}}v^{4}\,dx-2 \beta \int _{R^{3}}u^{2}v^{2}\,dx. $$

Therefore, using the fact that $(u,v)\in N$, we obtain

$$ \begin{aligned} I(u,v) &=\frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}u \bigr\vert ^{2}+ \lambda _{1}u^{2}(x)\,dx+ \frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}v \bigr\vert ^{2}+\lambda _{2}v^{2}(x)\,dx \\ &\quad {} +k \int _{R^{3}}u(x)v(x)\,dx- \frac{1}{4}\mu _{1} \int _{R^{3}}u^{4}(x)\,dx-\frac{1}{4}\mu _{2} \int _{R ^{3}}v^{4}(x)\,dx \\ &\quad {}-\frac{1}{2}\beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx \\ &= \frac{1}{2} \bigl\Vert (u,v) \bigr\Vert ^{2}- \frac{1}{4}\mu _{1} \int _{R^{3}}u^{4}(x)\,dx- \frac{1}{4}\mu _{2} \int _{R^{3}}v^{4}(x)\,dx-\frac{1}{2}\beta \int _{R^{3}}u ^{2}(x)v^{2}(x)\,dx \\ &=\frac{1}{2} \bigl\Vert (u,v) \bigr\Vert ^{2}- \frac{1}{4} \bigl\Vert (u,v) \bigr\Vert ^{2}= \frac{1}{4} \bigl\Vert (u,v) \bigr\Vert ^{2}. \end{aligned} $$

On the other hand, from $(u,v)\in N$ and the Sobolev embedded theorem we get

$$\begin{aligned} \mu _{1} \int _{R^{3}}u^{4}(x)\,dx+\mu _{2} \int _{R^{3}}v^{4}(x)\,dx+2\beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx&\leq C \bigl\vert (u,v) \bigr\vert _{L^{4}(R^{3})\times L^{4}(R ^{3})}^{4}\\&\leq C \bigl\Vert (u,v) \bigr\Vert ^{4}, \end{aligned}$$

and hence

$$ \bigl\Vert (u,v) \bigr\Vert ^{2}\leq C \bigl\Vert (u,v) \bigr\Vert ^{4}, $$

which implies that the conclusion (3) holds.

(4) Assume that $(u,v)\in N$ is a critical point of $I\vert _{N}$, then there exists $\gamma \in R$ such that

$$ I'(u,v)=\gamma F'(u,v). $$

Multiplying the above equality by $(u,v)$, we have

$$ \bigl\langle I'(u,v),(u,v)\bigr\rangle =\gamma \bigl\langle F'(u,v),(u,v)\bigr\rangle . $$

Now $F(u,v)=\langle I'(u,v),(u,v)\rangle =0$ and $\langle F'(u,v),(u,v) \rangle =-2\Vert (u,v)\Vert ^{2}\leq -2\rho _{0}$ for any $(u,v)\in N$ imply that $\gamma =0$, which means $I'(u,v)=0$.

(5) Let $\{(u_{n},v_{n})\}\subset H$ be a Palais–Smale sequence of $I\vert _{N}$, that is, $\{I(u_{n},v_{n})\}$ is bounded and $I'\vert _{N}(u_{n},v _{n})\rightarrow 0$. In the following, we claim that $I'(u_{n},v_{n}) \rightarrow 0$.

The proof of (3) shows $\{(u_{n},v_{n})\}$ is bounded in H. Hence after passing to a subsequence if necessary, we may assume that $(u_{n},v_{n})\rightharpoonup (u,v)$ in H. Since $\{(u_{n},v_{n})\}\subset H$ be a Palais–Smale sequence of $I\vert _{N}$, then there exists a sequence $\{\gamma _{n}\} \subseteq R$ such that

$$ I\vert '_{N}(u_{n},v_{n})=I'(u_{n},v_{n})- \gamma _{n} F'(u_{n},v_{n}). $$

Multiplying the above equality by $(u_{n},v_{n})$, we have

$$ o_{n}(1)=\bigl\langle I'(u_{n},v_{n}),(u_{n},v_{n}) \bigr\rangle -\gamma _{n} \bigl\langle F'(u_{n},v_{n}),(u_{n},v_{n}) \bigr\rangle =-\gamma _{n}\bigl\langle F'(u _{n},v_{n}),(u_{n},v_{n})\bigr\rangle . $$

It follows from $\langle F'(u_{n},v_{n}),(u_{n},v_{n})\rangle \leq -2\rho _{0}$ that $\gamma _{n}\rightarrow 0$ as $n\rightarrow +\infty$. Notice the fact that $\{(u_{n},v_{n})\}$ is bounded in N implies $F'(u_{n},v_{n})$ is bounded. Hence

$$ I'(u_{n},v_{n})=\gamma _{n} F'(u_{n},v_{n})+o(1)=o(1), \quad \text{as } n \rightarrow \infty . $$

(6) For fixed $(u,v)\in H\setminus \{(0,0)\}$ and $t>0$, we consider the map $h: t\mapsto I(tu,tv)$ defined by

$$ \begin{aligned} h(t): &=\frac{t^{2}}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}u \bigr\vert ^{2}+ \lambda _{1}u^{2}(x)\,dx+ \frac{t^{2}}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}v \bigr\vert ^{2}+\lambda _{2}v^{2}(x)\,dx \\ &\quad {} +kt^{2} \int _{R^{3}}u(x)v(x)\,dx-t ^{4} \int _{R^{3}}\frac{\mu _{1}}{4}u^{4}(x)+ \frac{\mu _{2}}{4}v^{4}(x)+ \frac{1}{2}\beta u^{2}(x)v^{2}(x)\,dx \\ &=\frac{1}{2}t^{2} \bigl\Vert (u,v) \bigr\Vert ^{2}-t ^{4} \int _{R^{3}}\frac{\mu _{1}}{4}u^{4}(x)+ \frac{\mu _{2}}{4}v^{4}(x)+ \frac{1}{2}\beta u^{2}(x)v^{2}(x)\,dx. \end{aligned} $$

Using the condition $\beta \in (-\sqrt{\mu _{1}\mu _{2}},+\infty )$, one can easily get $h(t)\rightarrow -\infty$ as $t\rightarrow +\infty$. Now we claim that $h(t)>0$ for $t>0$ small enough. Indeed, by the Sobolev embedding theorem, we have

$$ h(t)\geq \frac{1}{2}t^{2} \bigl\Vert (u,v) \bigr\Vert ^{2}-Ct^{4} \bigl\Vert (u,v) \bigr\Vert ^{4}, $$

which implies that $h(t)>0$ if $t>0$ is small enough. Hence there exists $t_{u,v}>0$ such that $h(t)$ has a positive maximum and $h'(t_{u,v})=0$. Notice that $F(tu,tv)=th'(t)$, so we have $F(t_{u,v}u,t_{u,v}v)=0$. Moreover, if $F(t_{u,v}u,t_{u,v}v)=0$, then we get

$$ t_{u,v}^{2} \bigl\Vert (u,v) \bigr\Vert ^{2}=t_{u,v}^{4}\biggl( \int _{R^{3}}\mu _{1} u^{4}(x)+\mu _{2} v^{4}(x)+2\beta u^{2}(x)v^{2}(x) \,dx\biggr). $$

Hence, $t_{u,v}=(\frac{\Vert (u,v)\Vert ^{2}}{\int _{R^{3}}\mu _{1} u^{4}(x)+\mu _{2} v^{4}(x)+2\beta u^{2}(x)v^{2}(x)\,dx})^{\frac{1}{2}}$, which is the unique critical point of $h(t)$ corresponding to its maximum, that is, $I(t_{u,v}u,t_{u,v}v)= \max_{t>0}I(tu,tv)$. □

Now we give the proof of Theorem 1.1.

Proof of Theorem 1.1

We divide the proof into three steps.

Step 1. Existence. We set $c= \inf_{(u,v)\in N}I(u,v)$. Then, by the proof of Lemma 3.1, we infer that $c\geq \frac{1}{4}\rho _{0}>0$. In view of Lemma 2.4, we can find a sequence $\{(u_{n},v_{n})\}\subseteq N$ such that $\{(u_{n},v_{n})\}$ is a $(\mathit{P.S.})_{c}$ sequence for $I\vert _{N}$. And consequently $\{(u_{n},v _{n})\}$ is a $(\mathit{P.S.})_{c}$ sequence for I by (5) of Lemma 3.1. Moreover, $\{(u_{n},v_{n})\}$ is a bounded sequence in H.

Suppose

$$ \liminf_{n\rightarrow +\infty }\sup_{x\in R^{3}} \int _{B_{1}(x)}u_{n} ^{2}(y)\,dy=0 \quad \text{and} \quad \liminf_{n\rightarrow +\infty }\sup_{x\in R^{3}} \int _{B_{1}(x)}v_{n} ^{2}(y)\,dy=0, $$

then, by Lemma 2.3, we get

$$ (u_{n},v_{n})\rightarrow (0,0) \quad \text{in } L^{4}\bigl(R^{3}\bigr)\times L^{4} \bigl(R^{3}\bigr). $$

Hence, we have

$$ \bigl\Vert (u_{n},v_{n}) \bigr\Vert ^{2}=\mu _{1} \int _{R^{3}}u_{n}^{4}(x)\,dx+\mu _{2} \int _{R^{3}}v_{n}^{4}(x)\,dx+2\beta \int _{R^{3}}u_{n}^{2}(x)v_{n}^{2}(x) \,dx \rightarrow 0, $$

contrary to $\Vert (u_{n},v_{n})\Vert >\rho _{0}$. Thus, without loss of generality, there is a constant $a>0$ such that

$$ \liminf_{n\rightarrow +\infty }\sup_{x\in R^{3}} \int _{B_{1}(x)}u_{n} ^{2}(y)\,dy=a. $$

Consequently, going to a subsequence if necessary, we can find a sequence $\{x_{n}\}\subseteq R^{3}$ that satisfies

$$ \int _{B_{1}(x_{n})}u_{n}^{2}(y)\,dy\geq \frac{a}{2}. $$

(3.1)

Since

$$ I\bigl(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n})\bigr)=\frac{1}{4} \bigl\Vert u_{n}(\cdot +x_{n}),v_{n}(\cdot +x _{n}) \bigr\Vert ^{2}=\frac{1}{4} \bigl\Vert (u_{n},v_{n}) \bigr\Vert ^{2}\rightarrow c, $$

we see that $\{(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x _{n}))\}$ is bounded. Hence, up to a subsequence, there exists $(u,v)\in H$ such that $(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\rightharpoonup (u,v)$ in H, $(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\rightarrow (u,v)$ in $L^{2}_{\mathrm{loc}}(R^{3})\times L^{2}_{\mathrm{loc}}(R^{3})$, $u_{n}(\cdot +x_{n})\rightarrow u$, $v_{n}(\cdot+x_{n})\rightarrow v$ for $\textit{a.e. }x\in R^{3}$. We pass to the limit in (3.1) and we get

$$ \int _{B_{1}(0)}u^{2}(y)\,dy\geq \frac{a}{2}, $$

(3.2)

which implies $u\neq0$. We use the invariance of N and I by translation to conclude that $\{(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\} \in N$ and $\Vert I'(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\Vert =\Vert I'(u_{n},v_{n})\Vert$, which shows that $\{(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\}$ is also a $(\mathit{P.S.})_{c}$ sequence of I. Consequently

$$ I'(u,v)=0, \quad\quad F(u,v)=0. $$

(3.3)

It follows from $u\neq0$ that $(u,v)\in N$. Thus, by the weakly lower semi-continuity of $\Vert \cdot \Vert$ we obtain

$$ \begin{aligned}[b] c &\leq I(u,v)=\frac{1}{4} \bigl\Vert (u,v) \bigr\Vert ^{2}\leq \frac{1}{4} \liminf _{n\rightarrow \infty } \bigl\Vert \bigl(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n})\bigr) \bigr\Vert ^{2} \\ &= \liminf_{n\rightarrow \infty }I\bigl(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n})\bigr)=c, \end{aligned} $$

(3.4)

which implies that $I(u,v)=c$.

Step 2. If $k\in (-\sqrt{\lambda _{1}\lambda _{2}},0)$, then $u>0$, $v>0$.

It is not difficult to see that

$$ \begin{gathered} \int _{R^{3}}u(x)v(x)\,dx\leq \int _{R^{3}} \bigl\vert u(x) \bigr\vert \bigl\vert v(x) \bigr\vert \,dx, \\k \int _{R^{3}}u(x)v(x)\,dx\geq k \int _{R^{3}} \bigl\vert u(x) \bigr\vert \bigl\vert v(x) \bigr\vert \,dx.\end{gathered} $$

(3.5)

Thus, combining (3.5) and $F(u,v)=0$ shows

$$ \begin{aligned}[b] &\bigl\Vert \bigl( \vert u \vert , \vert v \vert \bigr) \bigr\Vert ^{2} \\ &\quad = \int _{R^{3}} \int _{R^{3}} \frac{( \vert u(x) \vert - \vert u(y) \vert )^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy+ \int _{R^{3}} \int _{R^{3}} \frac{( \vert v(x) \vert - \vert v(y) \vert )^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy \\ &\quad \quad {} + \int _{R^{3}}\lambda _{1}u^{2}(x)+\lambda _{2}v^{2}(x)+2k \bigl\vert u(x) \bigr\vert \bigl\vert v(x) \bigr\vert \,dx \\ &\quad \leq \int _{R^{3}} \int _{R^{3}}\frac{(u(x)-u(y))^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy+ \int _{R^{3}} \int _{R^{3}}\frac{(v(x)-v(y))^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy \\ &\quad \quad {} + \int _{R^{3}} \lambda _{1}u^{2}(x)+\lambda _{2}v^{2}(x)+2ku(x)v(x)\,dx \\ &\quad = \bigl\Vert (u,v) \bigr\Vert ^{2}=\mu _{1} \int _{R^{3}}u^{4}(x)\,dx+\mu _{2} \int _{R^{3}}v^{4}(x)\,dx+2 \beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx. \end{aligned} $$

(3.6)

Consider

$$\begin{aligned} \varphi (t)&=I\bigl(t \vert u \vert ,t \vert v \vert \bigr)\\&= \frac{t^{2}}{2} \bigl\Vert \bigl( \vert u \vert , \vert v \vert \bigr) \bigr\Vert ^{2}- \frac{t^{4}}{4}\biggl(\mu _{1} \int _{R^{3}}u^{4}(x)\,dx+\mu _{2} \int _{R^{3}}v ^{4}(x)\,dx+2\beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx\biggr). \end{aligned}$$

Let $t^{*}=\frac{\Vert (\vert u\vert ,\vert v\vert )\Vert }{\Vert (u,v)\Vert }\in (0,1]$. It is easy to check that $\varphi (t)$ is strictly increasing in $(0,t^{*})$ and is strictly decreasing in $(t^{*},+ \infty )$, which shows that $\varphi (t)$ has a unique critical point $t^{*}>0$. So $(t^{*}\vert u\vert ,t^{*}\vert v\vert )\in N$. Then it follows from $c\leq I(t^{*}\vert u\vert ,t^{*}\vert v\vert )=\frac{(t^{*})^{2}}{4}\Vert (\vert u\vert ,\vert v\vert ) \Vert ^{2}\leq \frac{1}{4}\Vert (u,v)\Vert ^{2}=c$ that

$$ \bigl\Vert \bigl( \vert u \vert , \vert v \vert \bigr) \bigr\Vert = \bigl\Vert (u,v) \bigr\Vert , \quad \bigl( \vert u \vert , \vert v \vert \bigr)\in N, \quad\quad I\bigl(\bigl( \vert u \vert , \vert v \vert \bigr)\bigr)=c. $$

Therefore, we may assume without loss of generality that $u\geq 0$, $v \geq 0$. By Lemma 3.1, we know that $(u,v)$ is a critical point of I and hence is a ground state solution.

Through a similar argument in [17], one can show that $\vert u\vert _{L^{\infty }}<+\infty$, $\vert u\vert _{L^{\infty }}<+\infty$. By using the strong maximum principle to each single equation in (1.2), we obtain

$$ u>0 \quad \text{and} \quad v>0. $$

(3.7)

Hence, Step 2 is proved.

Step 3. If $k\in (0,\sqrt{\lambda _{1}\lambda _{2}})$, then $u>0$, $v<0$ or $u<0$, $v>0$.

It is not difficult to see that

$$ k \int _{R^{3}}u(x)v(x)\,dx\geq -k \int _{R^{3}} \bigl\vert u(x) \bigr\vert \bigl\vert v(x) \bigr\vert \,dx. $$

(3.8)

Thus, combining (3.8) and $F(u,v)=0$ shows

$$ \begin{aligned}[b] &\bigl\Vert \bigl( \vert u \vert ,- \vert v \vert \bigr) \bigr\Vert ^{2} \\ &\quad = \int _{R^{3}} \int _{R^{3}} \frac{( \vert u(x) \vert - \vert u(y) \vert )^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy+ \int _{R^{3}} \int _{R^{3}} \frac{(- \vert v(x) \vert + \vert v(y) \vert )^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy \\ &\quad \quad {} + \int _{R^{3}}\lambda _{1}u^{2}(x)+\lambda _{2}v^{2}(x)-2k \bigl\vert u(x) \bigr\vert \bigl\vert v(x) \bigr\vert \,dx \\ &\quad \leq \int _{R^{3}} \int _{R^{3}}\frac{(u(x)-u(y))^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy+ \int _{R^{3}} \int _{R^{3}}\frac{(v(x)-v(y))^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy \\ &\quad \quad {} + \int _{R^{3}} \lambda _{1}u^{2}(x)+\lambda _{2}v^{2}(x)+2ku(x)v(x)\,dx \\ &\quad = \bigl\Vert (u,v) \bigr\Vert ^{2}=\mu _{1} \int _{R^{3}}u^{4}(x)\,dx+\mu _{2} \int _{R^{3}}v^{4}(x)\,dx+2 \beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx. \end{aligned} $$

(3.9)

Consider

$$\begin{aligned} \psi (s)&=I\bigl(s \vert u \vert ,-s \vert v \vert \bigr)\\&= \frac{s^{2}}{2} \bigl\Vert \bigl( \vert u \vert ,- \vert v \vert \bigr) \bigr\Vert ^{2}- \frac{s^{4}}{4}\biggl(\mu _{1} \int _{R^{3}}u^{4}(x)\,dx+\mu _{2} \int _{R^{3}}v ^{4}(x)\,dx+2\beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx\biggr). \end{aligned}$$

Let $s^{*}=\frac{\Vert (\vert u\vert ,-\vert v\vert )\Vert }{\Vert (u,v)\Vert }\in (0,1]$. It is easy to check that $\psi (s)$ is strictly increasing in $(0,s^{*})$ and is strictly decreasing in $(s^{*},+\infty )$, which shows that $\psi (s)$ has a unique critical point $s^{*}>0$. So $(s^{*}\vert u\vert ,-s ^{*}\vert v\vert )\in N$. Then it follows from $c\leq I(s^{*}\vert u\vert ,-s^{*}\vert v\vert )=\frac{(s ^{*})^{2}}{4}\Vert (\vert u\vert ,-\vert v\vert )\Vert ^{2}\leq \frac{1}{4}\Vert (u,v)\Vert ^{2}=c$ that

$$ \bigl\Vert \bigl( \vert u \vert ,- \vert v \vert \bigr) \bigr\Vert = \bigl\Vert (u,v) \bigr\Vert , \quad \bigl( \vert u \vert ,- \vert v \vert \bigr)\in N, \quad\quad I\bigl(\bigl( \vert u \vert ,- \vert v \vert \bigr)\bigr)=c. $$

Therefore, we may assume without loss of generality that $u\geq 0$, $v \leq 0$. By Lemma 3.1, we know that $(u,v)$ is a critical point of I and hence is a ground state solution. Similar to the proof of (3.7), we can prove that $u>0$, $v<0$.

The proof of Theorem 1.1 is completed. □

4 Proof of Theorem 1.2

Now we turn to the system (1.1). Let $\lambda _{1}, \lambda _{2}, \mu _{1}, \mu _{2}>0$, $\beta \in (-\sqrt{\mu _{1}\mu _{2}},+\infty )$, $k\in (-\sqrt{\lambda _{1}\lambda _{2}},0)\cup (0,\sqrt{\lambda _{1}\lambda _{2}})$ and $(V)$ holds. We only consider the case that $V(x)\not \equiv \varLambda$, otherwise Theorem 1.2 comes down to Theorem 1.1.

We shall search solutions to the system (1.1) as critical points for the functional

$$ \begin{aligned} I_{V}(u,v) &=\frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}u \bigr\vert ^{2}+\bigl( \lambda _{1}+V(x)\bigr)u^{2}(x) \,dx \\ &\quad {}+\frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}v \bigr\vert ^{2}+\bigl(\lambda _{2}+V(x)\bigr)v^{2}(x) \,dx \\ &\quad {} +k \int _{R^{3}}u(x)v(x)\,dx- \frac{1}{4}\mu _{1} \int _{R^{3}}u^{4}(x)\,dx\\ &\quad {}-\frac{1}{4}\mu _{2} \int _{R ^{3}}v^{4}(x)\,dx-\frac{1}{2}\beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx, \end{aligned} $$

which is well defined on the Hilbert space $H^{s}(R^{3})\times H^{s}(R ^{3})$, equipped with the inner product as follows:

$$ \begin{aligned} \bigl((u_{1},v_{1}),(u_{2},v_{2}) \bigr)_{V}&= \int _{R^{3}}(-\triangle )^{s/2}u _{1}(- \triangle )^{s/2}u_{2}+\bigl(\lambda _{1}+V(x) \bigr)u_{1}(x)u_{2}(x)\,dx \\ &\quad {} + \int _{R^{3}}(-\triangle )^{s/2}v_{1}(- \triangle )^{s/2}v_{2}+\bigl(\lambda _{2}+V(x) \bigr)v_{1}(x)v_{2}(x)\,dx \\ &\quad {}+k \int _{R^{3}}u_{1}(x)v_{1}(x)\,dx+k \int _{R^{3}}u_{2}(x)v_{2}(x)\,dx, \end{aligned} $$

for $(u_{1},v_{1}), (u_{2},v_{2})\in H^{s}(R^{3})\times H^{s}(R^{3})$. $\Vert (u,v)\Vert _{V}=((u,v),(u,v))_{V}^{\frac{1}{2}}$ is the corresponding norm if $\vert k\vert <\sqrt{\lambda _{1}\lambda _{2}}$, this is equivalent to the standard product norm on the product space $H^{s}(R^{3})\times H^{s}(R^{3})$.

The limit system of (1.1) is

$$ \textstyle\begin{cases} (-\triangle )^{s}u+(\lambda _{1}+\varLambda )u+kv=\mu _{1}u^{3}+\beta uv ^{2}, \quad \text{in } R^{3},\\ (-\triangle )^{s}v+(\lambda _{2}+\varLambda )v+ku=\mu _{2}v ^{3}+\beta u^{2}v, \quad \text{in } R^{3},\\ u, v\in H^{s}(R^{3}). \end{cases} $$

(4.1)

The energy functional of the limit system (4.1) is given by

$$ \begin{aligned} I_{\varLambda }(u,v) &=\frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}u \bigr\vert ^{2}+( \lambda _{1}+\varLambda )u^{2}(x) \,dx+\frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}v \bigr\vert ^{2}+(\lambda _{2}+\varLambda )v^{2}(x)\,dx \\ &\quad {} +k \int _{R^{3}}u(x)v(x)\,dx- \frac{1}{4}\mu _{1} \int _{R^{3}}u^{4}(x)\,dx-\frac{1}{4}\mu _{2} \int _{R ^{3}}v^{4}(x)\,dx \\ &\quad {}-\frac{1}{2}\beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx. \end{aligned} $$

Let

$$\begin{aligned}& N_{V}:=\bigl\{ u\in H^{s}\bigl(R^{3}\bigr)\times H^{s}\bigl(R^{3}\bigr)\setminus \bigl\{ (0,0)\bigr\} : F _{V}(u,v)=\big\langle I'_{V}(u,v),(u,v)\big\rangle =0\bigr\} , \\& c_{V}= \inf_{(u,v)\in N_{V}}I_{V}(u,v), \end{aligned}$$

and define $c_{\varLambda }$ to be the constant which corresponds to the c in Sect. 2 when $\lambda _{1}$, $\lambda _{2}$ is replaced by $\lambda _{1}+\varLambda$, $\lambda _{2}+\varLambda$. The Nehari manifold $N_{V}$ shares the characteristic with N that has been defined in Sect. 3. In order to prove Theorem 1.2, we need the following lemma.

Lemma 4.1

If the condition V holds, then $c_{V}< c_{\varLambda }$.

Proof

Since $\beta \in (-\sqrt{\mu _{1}\mu _{2}},+\infty )$, we have

$$ c_{V}= \inf_{\eta \in \varGamma }\sup _{t\in [0,1]}I_{V}\bigl(\eta (t)\bigr), $$

(4.2)

where $\varGamma :=\{\eta \in C([0,1],H^{s}(R^{3})\times H^{s}(R^{3}): \eta (0)=(0,0), I_{V}(\eta (1))<0)\}$. Let $(\bar{u},\bar{v})$ be the ground state solution to (4.1) which was given in Theorem 1.1. In view of

$$ V(x)\leq \varLambda ,\quad\quad V(x)\neq \varLambda , \quad\quad \bar{u}\neq0, \quad\quad \bar{v} \neq0, $$

there is a positive constant $t_{*}>0$ such that

$$ \max_{t>0}I_{V}(t\bar{u},t \bar{v})=I_{V}(t_{*}\bar{u},t_{*}\bar{v})< I _{\varLambda }(t_{*}\bar{u},t_{*}\bar{v})\leq \max _{t>0}I_{\varLambda }(t\bar{u},t\bar{v})=c_{\varLambda }. $$

(4.3)

Hence

$$ c_{V}< c_{\varLambda }. $$

The proof of Lemma 4.1 is complete. □

Now we give the proof of Theorem 1.2.

Proof of Theorem 1.2

Our arguments are similar to the ones developed in Theorem 1.4 of [1] but we give the details for the reader’s convenience.

The definition of $c_{V}$ shows that $c_{V}>0$. In view of Lemma 2.4, we can find a sequence $\{(u_{n},v_{n})\}\subseteq N_{V}$ such that $\{(u_{n},v_{n})\}$ is a $(\mathit{P.S.})_{c_{V}}$ sequence for $I_{V}\vert _{N_{V}}$. And consequently $\{(u_{n},v_{n})\}$ is a $(\mathit{P.S.})_{c_{V}}$ sequence for I by (5) of Lemma 3.1. Moreover, $\{(u_{n},v_{n})\}$ is a bounded sequence in $H^{s}(R^{3})\times H^{s}(R^{3})$. Thus, up to a subsequence, there exists $(u,v)\in H^{s}(R^{3})\times H^{s}(R^{3})$ such that $(u_{n},v_{n})\rightharpoonup (u,v)$ in $H^{s}(R^{3})\times H^{s}(R^{3})$. In the following, we claim that $(u,v)\neq(0,0)$.

Indeed, if $(u,v)=(0,0)$, then $u_{n}\rightharpoonup 0$, $v_{n}\rightharpoonup 0$ in $H^{s}(R^{3})$. From $\lim_{\vert x\vert \rightarrow +\infty }V(x)=\varLambda$ we see that, for any $\varepsilon >0$, there exists R large enough, such that

$$ \int _{B_{R}^{c}(0)} \bigl\vert V(x)-\varLambda \bigr\vert u_{n}^{2}(x)\,dx< \frac{\varepsilon }{2}. $$

Since $V(x)\in L^{\infty }$ and $(u_{n},v_{n})\rightharpoonup (0,0)$ we obtain

$$ \int _{B_{R}(0)} \bigl\vert V(x)-\varLambda \bigr\vert u_{n}^{2}(x)\,dx< \frac{\varepsilon }{2}, $$

for large n, so that

$$ \int _{R^{3}} \bigl\vert V(x)-\varLambda \bigr\vert u_{n}^{2}(x)\,dx=o_{n}(1), \quad n\rightarrow + \infty . $$

(4.4)

With the same computation we see that $\forall \varphi \in H^{s}(R ^{3})$

$$\begin{aligned}& \int _{R^{3}} \bigl\vert V(x)-\varLambda \bigr\vert v_{n}^{2}(x)\,dx=o_{n}(1),\quad n\rightarrow + \infty , \end{aligned}$$

(4.5)

$$\begin{aligned}& \int _{R^{3}} \bigl\vert V(x)-\varLambda \bigr\vert u_{n}(x)\varphi (x)\,dx=o_{n}(1),\quad n\rightarrow +\infty , \end{aligned}$$

(4.6)

$$\begin{aligned}& \int _{R^{3}} \bigl\vert V(x)-\varLambda \bigr\vert v_{n}(x)\varphi (x)\,dx=o_{n}(1),\quad n\rightarrow +\infty . \end{aligned}$$

(4.7)

Consequently,

$$\begin{aligned}& I_{\varLambda }(u_{n},v_{n})=I_{V}(u_{n},v_{n})+o_{n}(1)=c_{V}+o_{n}(1), \end{aligned}$$

(4.8)

$$\begin{aligned}& I'_{\varLambda }(u_{n},v_{n})=I'_{V}(u_{n},v_{n})+o_{n}(1)=o_{n}(1). \end{aligned}$$

(4.9)

Assume

$$ \liminf_{n\rightarrow +\infty }\sup_{x\in R^{3}} \int _{B_{1}(x)}u_{n} ^{2}(y)\,dy=0 \quad \text{and} \quad \liminf_{n\rightarrow +\infty }\sup_{x\in R^{3}} \int _{B_{1}(x)}v_{n} ^{2}(y)\,dy=0, $$

then, by Lemma 2.3, we get

$$ (u_{n},v_{n})\rightarrow (0,0) \quad \text{in } L^{4}\bigl(R^{3}\bigr)\times L^{4} \bigl(R^{3}\bigr). $$

Hence, we have

$$ \bigl\Vert (u_{n},v_{n}) \bigr\Vert _{V}^{2}=\mu _{1} \int _{R^{3}}u_{n}^{4}(x)\,dx+\mu _{2} \int _{R^{3}}v_{n}^{4}(x)\,dx+2\beta \int _{R^{3}}u_{n}^{2}(x)v_{n}^{2}(x) \,dx \rightarrow 0, $$

which is contrary to the fact that $\Vert (u_{n},v_{n})\Vert _{V}^{2}>C>0$. Thus, without loss of generality, there is a constant such that $\alpha _{V}>0$

$$ \liminf_{n\rightarrow +\infty }\sup_{x\in R^{3}} \int _{B_{1}(x)}u_{n} ^{2}(y)\,dy=\alpha _{V}. $$

Consequently, going if necessary to a subsequence, we can find a sequence $\{x_{n}\}\subseteq R^{3}$ that satisfies

$$ \int _{B_{1}(x_{n})}u_{n}^{2}(y)\,dy\geq \frac{\alpha _{V}}{2}. $$

(4.10)

Since $V(x)\leq \varLambda$, we deduce from (4.4) and (4.5) that

$$\begin{aligned} &\bigl\Vert \bigl(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n})\bigr) \bigr\Vert _{V}^{2} \\ &\quad \leq \int _{R^{3}} \bigl\vert (- \triangle )^{s/2}u_{n} \bigr\vert ^{2}+(\lambda _{1}+\varLambda )u_{n}^{2}(x)\,dx \\ &\quad\quad {} + \int _{R^{3}} \bigl\vert (-\triangle )^{s/2}v_{n} \bigr\vert ^{2}+(\lambda _{2}+\varLambda )v _{n}^{2}(x)\,dx+2k \int _{R^{3}}u_{n}(x)v_{n}(x)\,dx \\ &\quad = \int _{R^{3}} \bigl\vert (- \triangle )^{s/2}u_{n} \bigr\vert ^{2}+\bigl(\lambda _{1}+V(x) \bigr)u_{n}^{2}(x)\,dx \\ &\quad\quad {} + \int _{R^{3}} \bigl\vert (-\triangle )^{s/2}v_{n} \bigr\vert ^{2}+\bigl(\lambda _{2}+V(x) \bigr)v_{n} ^{2}(x)\,dx+2k \int _{R^{3}}u_{n}(x)v_{n}(x)\,dx+o(1) \\ &\quad =4I_{V}(u_{n},v _{n})+o_{n}(1)=4c_{V}+o_{n}(1), \quad \text{as } n\rightarrow \infty , \end{aligned}$$

so that $\{(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\}$ is bounded. Thus, up to a subsequence, there exists $(u_{*},v_{*})\in H^{s}(R^{3})\times H ^{s}(R^{3})$ such that $(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\rightharpoonup (u_{*},v_{*})$ in $H^{s}(R^{3})\times H^{s}(R^{3})$, $(u_{n}(\cdot +x_{n}),v _{n}(\cdot +x_{n}))\rightarrow (u_{*},v_{*})$ in $L^{2}_{\mathrm{loc}}(R^{3})\times L^{2}_{\mathrm{loc}}(R^{3})$, $u_{n}(\cdot +x_{n})\rightarrow u_{*}$, $v_{n}(\cdot +x_{n}) \rightarrow v_{*}$ for $\textit{a.e. }x\in R^{3}$. We pass to the limit in (4.10) and we get

$$ \int _{B_{1}(0)}u_{*}^{2}(y)\,dy\geq \frac{\alpha _{V}}{2}, $$

(4.11)

which implies $u_{*}\neq0$. From (4.8) and (4.9), we derive that $\{(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\}$ is a $(\mathit{P.S.})_{c_{V}}$ sequence for $I_{\varLambda }$. We also know that $I'_{\varLambda }(u_{*},v _{*})=0$. Therefore

$$ \begin{aligned} I_{\varLambda }(u_{*},v_{*})={} &\frac{1}{4} \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}u_{*} \bigr\vert ^{2}+(\lambda _{1}+\varLambda )u_{*}^{2}(x)\,dx+ \frac{1}{4} \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}v_{*} \bigr\vert ^{2}+(\lambda _{2}+\varLambda )v_{*}^{2}(x)\,dx \\ & {} +\frac{1}{2}k \int _{R^{3}}u_{*}(x)v _{*}(x)\,dx. \end{aligned} $$

Let $(\bar{u_{n}},\bar{v_{n}})=(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))$. Now $(\bar{u_{n}},\bar{v_{n}})\rightharpoonup (u_{*},v_{*})$ implies that

$$\begin{aligned}& \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}u_{*} \bigr\vert ^{2}+(\lambda _{1}+ \varLambda )u_{*}^{2}(x)\,dx+ \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}v_{*} \bigr\vert ^{2}+( \lambda _{2}+\varLambda )v_{*}^{2}(x)\,dx \\& \quad \quad {}+2k \int _{R^{3}}u_{*}(x)v_{*}(x)\,dx \\& \quad \leq \liminf_{n\rightarrow \infty } \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}\bar{u _{n}} \bigr\vert ^{2}+(\lambda _{1}+\varLambda ){\bar{u_{n}}}^{2}\,dx+ \int _{R^{3}} \bigl\vert (- \triangle )^{\frac{s}{2}} \bar{v_{n}} \bigr\vert ^{2}+(\lambda _{2}+ \varLambda ){\bar{v _{n}}}^{2}\,dx \\& \quad \quad {}+2k \int _{R^{3}}\bar{u_{n}}\bar{v_{n}}\,dx \\& \quad = \liminf_{n\rightarrow \infty } \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}u _{n} \bigr\vert ^{2}+(\lambda _{1}+\varLambda )u_{n}^{2}\,dx+ \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}v_{n} \bigr\vert ^{2}+(\lambda _{2}+\varLambda )v_{n}^{2}\,dx \\& \quad \quad {}+2k \int _{R^{3}}u_{n}v_{n}\,dx \\& \quad = \liminf_{n\rightarrow \infty } \bigl\Vert (u_{n},v_{n}) \bigr\Vert _{V}^{2} \end{aligned}$$

so that

$$ c_{\varLambda }\leq I_{\varLambda }(u_{*},v_{*})\leq \liminf_{n\rightarrow \infty }\frac{1}{4} \bigl\Vert (u_{n},v_{n}) \bigr\Vert _{V}^{2}= \liminf_{n\rightarrow \infty }I_{V}(u_{n},v_{n})=c_{V}, $$

this contradicts Lemma 4.1. Hence $u\neq0$ or $v\neq0$.

Then we can prove Theorem 1.2 similarly by using the same method as has been used in Theorem 1.1. □