Skip to main content

# Existence of ground state solutions to a class of fractional Schrödinger system with linear and nonlinear couplings

## Abstract

In this paper, we study the existence of ground state solutions to the following fractional Schrödinger system with linear and nonlinear couplings:

$$\textstyle\begin{cases} (-\triangle )^{s}u+(\lambda _{1}+V(x))u+kv=\mu _{1}u^{3}+\beta uv^{2}, \quad \text{in } R^{3},\\ (-\triangle )^{s}v+(\lambda _{2}+V(x))v+ku=\mu _{2}v^{3}+ \beta u^{2}v, \quad \text{in } R^{3},\\ u, v\in H^{s}(R^{3}), \end{cases}$$

where $$(-\triangle )^{s}$$ denotes the fractional Laplacian of order $$s\in (\frac{3}{4},1)$$. Under some assumptions of the potential $$V(x)$$ and the linear and nonlinear coupling constants k, β, we prove some results for the existence of ground state solutions for the fractional Laplacian systems by using variational methods.

## 1 Introduction

The aim of this paper is to consider the existence of ground state solutions to the following fractional Schrödinger system with linear and nonlinear couplings:

$$\textstyle\begin{cases} (-\triangle )^{s}u+(\lambda _{1}+V(x))u+kv=\mu _{1}u^{3}+\beta uv^{2}, \quad \text{in } R^{3},\\ (-\triangle )^{s}v+(\lambda _{2}+V(x))v+ku=\mu _{2}v^{3}+ \beta u^{2}v, \quad \text{in } R^{3},\\ u, v\in H^{s}(R^{3}), \end{cases}$$
(1.1)

where $$(-\triangle )^{s}$$ denotes the fractional Laplacian of order $$s\in (\frac{3}{4},1)$$, $$\lambda _{1}$$, $$\lambda _{2}$$, $$\mu _{1}$$, $$\mu _{2}$$ are positive constants, k, β are linear and nonlinear coupling constants respectively. The potential function $$V(x)$$ will always be assumed to satisfy

$$(V)\quad \sup_{x\in R^{3}}V(x)=\lim_{\vert x\vert \rightarrow +\infty }V(x)=\varLambda >0, \quad\quad \inf_{x\in R^{3}}V(x)\geq 0.$$

If $$V(x)\equiv 0$$, then (1.1) becomes

$$\textstyle\begin{cases} (-\triangle )^{s}u+\lambda _{1}u+kv=\mu _{1}u^{3}+\beta uv^{2}, \quad \text{in } R^{3},\\ (-\triangle )^{s}v+\lambda _{2}v+ku=\mu _{2}v^{3}+\beta u^{2}v, \quad \text{in } R^{3},\\ u, v\in H^{s}(R^{3}). \end{cases}$$
(1.2)

It is well known, but not completely trivial, that $$(-\triangle )^{s}$$ reduces to the standard Laplacian − as $$s\rightarrow 1$$. In the local case, that is, when $$s=1$$, the system (1.2) reduces to the following system:

$$\textstyle\begin{cases} -\triangle u+\lambda _{1}u+kv=\mu _{1}u^{3}+\beta uv^{2}, \quad \text{in } R^{3},\\ -\triangle v+\lambda _{2}v+ku=\mu _{2}v^{3}+\beta u^{2}v, \quad \text{in } R^{3},\\ u, v\in H^{1}(R^{3}). \end{cases}$$
(1.3)

System (1.3) appears in several physical situations such as in nonlinear optics, in double Bose–Einstein condensates and in plasma physics, and it has been extensively studied by many authors in the past ten years; see, for example, [1,2,3] and the references therein.

In the nonlocal case, that is, when $$s\in (0,1)$$, there are very few results for the fractional Laplacian systems. If $$\beta =0$$, i.e., only linear coupling terms exist, in [4], Dengfeng Lv and Shuangjie Peng studied the problem

$$\textstyle\begin{cases} (-\triangle )^{s}u+u=f(u)+k v, \quad \text{in } R^{N},\\ (-\triangle )^{s} v+v=g(v)+k u, \quad \text{in } R^{N}. \end{cases}$$
(1.4)

They obtained the existence of positive vector solutions and vector ground state solutions via variational methods. They also proved the asymptotic behavior of these solutions as the coupling parameter k tends to zero. If $$k=0$$, i.e., only nonlinear coupling terms exist, in [5], Q. Guo and X. He studied the problem

$$\textstyle\begin{cases} (-\triangle )^{s}u+u=( \vert u \vert ^{2p}+b \vert u \vert ^{p-1} \vert v \vert ^{p-1})u, \quad \text{in } R^{N},\\ (-\triangle )^{s}v+\omega ^{2s}v=( \vert v \vert ^{2p}+b \vert v \vert ^{p-1} \vert u \vert ^{p-1})v, \quad \text{in } R^{N},\\ u, v\in H^{s}(R^{N}), \end{cases}$$
(1.5)

where $$\omega >0$$, $$b>0$$ are constants and p satisfies $$2<2p+2<2_{s} ^{*}$$. They obtained the existence of a least energy solution via Nehari manifold method. They also proved that if b is large enough, system (1.5) has a positive least energy solution with both nontrivial components. When $$k\neq0$$ and $$\beta \neq0$$, i.e., linear coupling terms and nonlinear coupling terms both exist, to the best of our knowledge, there has been almost no research on this problem. For the other work on the fractional Laplacian system, we refer the reader to [6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23] and the references therein. We would also like to mention [24,25,26,27,28,29,30,31,32,33,34,35,36] and the references therein for the information of fractional ordinary differential equations.

A solution $$(u,v)\in H^{s}(R^{3})\times H^{s}(R^{3})$$ to (1.1) is called a nontrivial solution if $$(u,v)\neq(0,0)$$. A solution $$(u,v)$$ with $$u>0$$ and $$v>0$$ is called a positive solution. A solution $$(u,v)$$ is called a ground state solution if $$(u,v)\neq(0,0)$$ and its energy is minimal among the energy of all nontrivial solutions to (1.1).

Our main theorems of this work read as follows.

### Theorem 1.1

Suppose that $$\lambda _{1}, \lambda _{2}, \mu _{1}, \mu _{2}>0$$, $$\beta \in R$$, $$k\in (-\sqrt{\lambda _{1}\lambda _{2}},0)\cup (0,\sqrt{\lambda _{1}\lambda _{2}})$$. Then problem (1.2) possesses a ground state solution $$(u,v)$$. Moreover, $$u>0$$, $$v>0$$ as $$k\in (-\sqrt{\lambda _{1}\lambda _{2}},0)$$; $$u>0$$, $$v<0$$ or $$u<0$$, $$v>0$$ as $$k\in (0, \sqrt{\lambda _{1}\lambda _{2}})$$.

### Theorem 1.2

Suppose that $$\lambda _{1}, \lambda _{2}, \mu _{1}, \mu _{2}>0$$, $$\beta \in (-\sqrt{\mu _{1}\mu _{2}},+\infty )$$, $$k \in (-\sqrt{\lambda _{1}\lambda _{2}},0)\cup (0,\sqrt{\lambda _{1} \lambda _{2}})$$ and that $$(V)$$ holds. Then problem (1.1) possesses a ground state solution $$(u,v)$$. Moreover, $$u>0$$, $$v>0$$ as $$k\in (-\sqrt{ \lambda _{1}\lambda _{2}},0)$$; $$u>0$$, $$v<0$$ or $$u<0$$, $$v>0$$ as $$k\in (0,\sqrt{ \lambda _{1}\lambda _{2}})$$.

The remainder of this paper is organized as follows. In Sect. 2, some notations and preliminaries are presented. The proofs of Theorem 1.1 and Theorem 1.2 are given in Sects. 3 and 4 respectively.

## 2 Preliminary

In this section, we outline the variational framework for the problem (1.1) and give some preliminary lemmas which will be used later.

Throughout this paper, C, $$C_{i}$$ will denote various positive constants; the strong convergence is denoted by →, and the weak convergence is denoted as ; $$2_{s}^{*}= \frac{2N}{N-2s}$$ is the fractional Sobolev critical exponent; $$o_{n}(1)$$ denotes $$o_{n}(1)\rightarrow 0$$ as $$n\rightarrow \infty$$; For any $$\rho >0$$ and $$z\in R^{3}$$, $$B_{\rho }(z):=\{x\in R^{3}: \vert x-z\vert \leq \rho \}$$; For $$1\leq p\leq +\infty$$, and $$f\in L^{p}(R^{3})$$, let $$f^{+}=\max \{f,0\}$$, $$f^{-}=\min \{f,0\}$$, and $$\vert f\vert _{L^{p}(R^{3})}$$ denotes the usual $$L^{p}$$ norm of f. Let $$L^{p}(R^{3})\times L^{p}(R ^{3})$$ be the Cartesian product of two $$L^{p}(R^{3})$$ spaces, and for $$(f,g)\in L^{p}(R^{3})\times L^{p}(R^{3})$$, $$\vert (f,g)\vert _{L^{p}\times L ^{p}}=\vert (f,g)\vert _{L^{p}(R^{3})\times L^{p}(R^{3})}:=(\vert f\vert _{L^{p}(R^{3})} ^{p}+\vert g\vert _{L^{p}(R^{3})}^{p})^{\frac{1}{p}}$$.

For any $$s\in (0,1)$$, the fractional Sobolev space $$H^{s}(R^{3})$$ is defined by

$$H^{s}\bigl(R^{3}\bigr)= \biggl\{ u\in L^{2} \bigl(R^{3}\bigr):\frac{ \vert u(x)-u(y) \vert }{ \vert x-y \vert ^{ \frac{3+2s}{2}}}\in L^{2} \bigl(R^{3}\times R^{3}\bigr) \biggr\} .$$

Let us consider a Hilbert space endowed with the scalar product given by

$$(u,v)= \int _{R^{3}}uv \,dx+ \int _{R^{3}} \int _{R^{3}} \frac{(u(x)-u(y))(v(x)-v(y))}{ \vert x-y \vert ^{3+2s}}\,dx\,dy.$$

The corresponding norm is therefore

$$\Vert u \Vert _{H^{s}}= \biggl( \int _{R^{3}} \vert u \vert ^{2}\,dx+ \int _{R^{3}} \int _{R^{3}} \frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy \biggr)^{\frac{1}{2}}.$$

It is well known that the fractional Laplacian $$(-\triangle )^{s}$$ of a function $$u: R^{3}\rightarrow R$$ is defined by

$$(-\triangle )^{s}u(x)={\mathcal{F}}^{-1}\bigl( \vert \xi \vert ^{2s}({\mathcal{F}}u)\bigr) (x), \quad \forall \xi \in R^{3},$$

where $$\mathcal{F}$$ is the Fourier transform, i.e.,

$${\mathcal{F}}u(\xi )=\frac{1}{(2\pi )^{\frac{3}{2}}} \int _{R^{3}}\exp (-2 \pi i\xi \cdot x)u(x)\,dx,$$

i is the imaginary unit. If u is smooth enough, it can be computed by the following singular integral:

$$(-\triangle )^{s}u(x)=c_{s} \mathit{P.V.} \int _{R^{3}} \frac{u(x)-u(y)}{ \vert x-y \vert ^{3+2s}}\,dy, \quad x\in R^{3},$$

where $$c_{s}$$ is normalization constant and P.V. stands for the principal value. Now one can get an alternative definition of the fractional Sobolev space $$H^{s}(R^{3})$$ via the Fourier transform as follows:

$$H^{s}\bigl(R^{3}\bigr)= \biggl\{ u\in L^{2} \bigl(R^{3}\bigr): \int _{R^{3}} \vert \xi \vert ^{2s} \vert \hat{u} \vert ^{2}\,d\xi < \infty \biggr\} ,$$

endowed with the norm

$$\Vert u \Vert _{s}=\biggl( \int _{R^{3}} \bigl(1+ \vert \xi \vert ^{2s}\bigr) \vert \hat{u} \vert ^{2}\,d\xi \biggr) ^{\frac{1}{2}},$$

where $$\hat{u}={\mathcal{F}}(u)$$ denotes the Fourier transform of u. It is easy to see that $$\Vert \cdot \Vert _{H^{s}}$$ is equivalent to $$\Vert \cdot \Vert _{s}$$.

The homogeneous Sobolev space $$D^{s,2}(R^{3})$$ is defined by

$$D^{s}\bigl(R^{3}\bigr)= \bigl\{ u\in L^{2_{s}^{*}} \bigl(R^{3}\bigr): \vert \xi \vert ^{s}\hat{u} \in L^{2}\bigl(R^{3}\bigr) \bigr\} ,$$

which is the completion of $$C_{0}^{\infty }(R^{3})$$ under the norm

$$\Vert u \Vert _{D^{s,2}}= \biggl( \int _{R^{3}} \bigl\vert (-\triangle )^{s/2}u \bigr\vert ^{2}\,dx \biggr) ^{\frac{1}{2}}= \biggl( \int _{R^{3}} \vert \xi \vert ^{2s} \vert \hat{u} \vert ^{2}\,d\xi \biggr) ^{\frac{1}{2}}.$$

Now we introduce the following lemmata.

### Lemma 2.1

(see [13])

For any $$s\in (0,1)$$, $$H^{s}(R^{3})$$ is continuously embedded into $$L^{p}(R^{3})$$ for $$p\in [2,2_{s}^{*}]$$ and compactly embedded into $$L_{\mathrm{loc}}^{p}(R^{3})$$ for $$p\in [1,2_{s}^{*})$$.

### Lemma 2.2

(see [14])

For any $$s\in (0,1)$$, $$D^{s,2}(R^{3})$$ is continuously embedded into $$L^{2_{s}^{*}}(R^{3})$$ and we define

$$S_{s}:= \inf_{u\in D^{s,2}(R^{3})\setminus \{0\}}\frac{\int _{R^{3}} \vert (-\triangle )^{s/2}u \vert ^{2}\,dx}{(\int _{R^{3}}u^{2_{s}^{*}})^{2/2_{s}^{*}}}.$$

### Lemma 2.3

(see [15])

If $$\{u_{n}\}$$ is bounded in $$H^{s}(R ^{3})$$ with $$s\in (0,1)$$ and

$$\lim_{n\rightarrow \infty }\sup_{y\in R^{3}} \int _{B_{r}(y)} \vert u_{n} \vert ^{2} \,dx=0,$$

where $$r>0$$, then we have $$u_{n}\rightarrow 0$$ in $$L^{\nu }(R^{3})$$ for $$\nu \in (2,2_{s}^{*})$$.

### Remark 1

Similarly, in the case that the sequence $$\{\vert u_{n}\vert ^{2_{s} ^{*} }\}$$ is vanishing, we can prove that $$u_{n}\rightarrow 0$$ in $$L^{\nu }(R^{3})$$ for $$\nu \in (2,2_{s}^{*}]$$.

### Lemma 2.4

(see [37]; Ekeland variational principle)

Let X be a Banach space and let $$G\in C^{2}(X,R)$$ be such that, for every $$v\in V:=\{v\in X: G(v)=1\}$$, $$G'(v)\neq0$$. Let $$F\in C^{1}(X,R)$$ be bounded from below on V, $$v\in V$$ and $$\varepsilon , \delta >0$$. If

$$F(v)\leq \inf_{V}F+\varepsilon ,$$

then there exists $$u\in V$$ such that

$$F(u)\leq \inf_{V}F+2\varepsilon , \quad \quad \min _{\lambda \in R} \bigl\Vert F'(u)-\lambda G'(u) \bigr\Vert \leq \frac{8\varepsilon }{ \delta }, \quad\quad \Vert u-v \Vert \leq 2\delta .$$

## 3 Proof of Theorem 1.1

In this section, we shall study system (1.2). Suppose that $$\lambda _{1}, \lambda _{2}, \mu _{1}, \mu _{2}>0$$, $$\beta \in R$$, $$k \in (-\sqrt{\lambda _{1}\lambda _{2}},0)\cup (0,\sqrt{\lambda _{1} \lambda _{2}})$$. Let $$H:=H^{s}(R^{3})\times H^{s}(R^{3})$$. We define an inner product on H as follows:

\begin{aligned} \bigl((u_{1},v_{1}),(u_{2},v_{2}) \bigr)&= \int _{R^{3}}(-\triangle )^{s/2}u_{1}(- \triangle )^{s/2}u_{2}+\lambda _{1}u_{1}(x)u_{2}(x) \,dx \\ & \quad {} + \int _{R^{3}}(- \triangle )^{s/2}v_{1}(- \triangle )^{s/2}v_{2}+\lambda _{2}v_{1}(x)v _{2}(x)\,dx \\ & \quad {}+k \int _{R^{3}}u_{1}(x)v_{1}(x)\,dx+k \int _{R^{3}}u_{2}(x)v_{2}(x)\,dx, \end{aligned}

for $$(u_{1},v_{1}), (u_{2},v_{2})\in H$$. $$\Vert (u,v)\Vert =((u,v),(u,v))^{ \frac{1}{2}}$$ is the corresponding norm if $$\vert k\vert <\sqrt{\lambda _{1} \lambda _{2}}$$. This is equivalent to the standard product norm on the product space $$H^{s}(R^{3})\times H^{s}(R^{3})$$.

For $$(u,v)\in H$$, the energy functional associated with (1.2) is

\begin{aligned} I(u,v) &=\frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}u \bigr\vert ^{2}+ \lambda _{1}u^{2}(x)\,dx+ \frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}v \bigr\vert ^{2}+\lambda _{2}v^{2}(x)\,dx \\ & \quad {} +k \int _{R^{3}}u(x)v(x)\,dx- \frac{1}{4}\mu _{1} \int _{R^{3}}u^{4}(x)\,dx-\frac{1}{4}\mu _{2} \int _{R ^{3}}v^{4}(x)\,dx \\ & \quad {}-\frac{1}{2}\beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx \\ &= \frac{1}{2} \bigl\Vert (u,v) \bigr\Vert ^{2}- \frac{1}{4}\mu _{1} \int _{R^{3}}u^{4}(x)\,dx- \frac{1}{4}\mu _{2} \int _{R^{3}}v^{4}(x)\,dx-\frac{1}{2}\beta \int _{R^{3}}u ^{2}(x)v^{2}(x)\,dx. \end{aligned}

Then, under our assumptions, I is well defined on H and $$I\in C^{1}(H,R)$$.

Let us define the Nehari manifold

$$N:=\bigl\{ (u,v)\in H\setminus \bigl\{ (0,0)\bigr\} : F(u,v)=\bigl\langle I'(u,v),(u,v) \bigr\rangle =0\bigr\} .$$

Now we state some properties of N.

### Lemma 3.1

If the assumptions of Theorem 1.1 hold, then the following statements hold.

1. (1)

$$N\neq\emptyset$$;

2. (2)

N is a $$C^{1}$$ manifold;

3. (3)

there exists a positive constant $$\rho _{0}>0$$ such that $$\Vert (u,v)\Vert \geq \rho _{0}$$ for all $$(u,v)\in N$$;

4. (4)

the critical points of $$I\vert _{N}$$ are the critical points of I in H;

5. (5)

if $$\{(u_{n},v_{n})\}\subseteq H$$ is a P.S. sequence for $$I\vert _{N}$$, then $$\{(u_{n},v_{n})\}$$ is a P.S. sequence for I;

6. (6)

if $$\beta \in (-\sqrt{\mu _{1}\mu _{2}},+\infty )$$ then, for any $$(u,v)\in H\setminus \{(0,0)\}$$, there exists a unique $$t_{u,v}>0$$ such that $$(t_{u,v}u,t_{u,v}v)\in N$$.

### Proof

(1) The proof of (1) is simple, so we omit the details here.

(2) Since $$F(u,v)$$ is a $$C^{1}$$ functional, in order to prove that N is a $$C^{1}$$ manifold, it is sufficient to prove that $$F'(u,v) \neq0$$ for all $$(u,v)\in N$$. For $$(u,v)\in N$$

\begin{aligned} \bigl\langle F'(u,v),(u,v)\bigr\rangle &=2 \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}u \bigr\vert ^{2}+\lambda _{1}u^{2}\,dx+2 \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}v \bigr\vert ^{2}+\lambda _{2}v^{2}\,dx+4k \int _{R^{3}}uv \,dx \\ &\quad {} -4\mu _{1} \int _{R^{3}}u^{4}\,dx-4\mu _{2} \int _{R^{3}}v^{4}\,dx-8\beta \int _{R^{3}}u ^{2}v^{2}\,dx \\ &=-2 \bigl\Vert (u,v) \bigr\Vert ^{2}< 0. \end{aligned}

(3) Notice that if $$(u,v)\in N$$, then

$$\bigl\Vert (u,v) \bigr\Vert ^{2}=\mu _{1} \int _{R^{3}}u^{4}\,dx+\mu _{2} \int _{R^{3}}v^{4}\,dx-2 \beta \int _{R^{3}}u^{2}v^{2}\,dx.$$

Therefore, using the fact that $$(u,v)\in N$$, we obtain

\begin{aligned} I(u,v) &=\frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}u \bigr\vert ^{2}+ \lambda _{1}u^{2}(x)\,dx+ \frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}v \bigr\vert ^{2}+\lambda _{2}v^{2}(x)\,dx \\ &\quad {} +k \int _{R^{3}}u(x)v(x)\,dx- \frac{1}{4}\mu _{1} \int _{R^{3}}u^{4}(x)\,dx-\frac{1}{4}\mu _{2} \int _{R ^{3}}v^{4}(x)\,dx \\ &\quad {}-\frac{1}{2}\beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx \\ &= \frac{1}{2} \bigl\Vert (u,v) \bigr\Vert ^{2}- \frac{1}{4}\mu _{1} \int _{R^{3}}u^{4}(x)\,dx- \frac{1}{4}\mu _{2} \int _{R^{3}}v^{4}(x)\,dx-\frac{1}{2}\beta \int _{R^{3}}u ^{2}(x)v^{2}(x)\,dx \\ &=\frac{1}{2} \bigl\Vert (u,v) \bigr\Vert ^{2}- \frac{1}{4} \bigl\Vert (u,v) \bigr\Vert ^{2}= \frac{1}{4} \bigl\Vert (u,v) \bigr\Vert ^{2}. \end{aligned}

On the other hand, from $$(u,v)\in N$$ and the Sobolev embedded theorem we get

\begin{aligned} \mu _{1} \int _{R^{3}}u^{4}(x)\,dx+\mu _{2} \int _{R^{3}}v^{4}(x)\,dx+2\beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx&\leq C \bigl\vert (u,v) \bigr\vert _{L^{4}(R^{3})\times L^{4}(R ^{3})}^{4}\\&\leq C \bigl\Vert (u,v) \bigr\Vert ^{4}, \end{aligned}

and hence

$$\bigl\Vert (u,v) \bigr\Vert ^{2}\leq C \bigl\Vert (u,v) \bigr\Vert ^{4},$$

which implies that the conclusion (3) holds.

(4) Assume that $$(u,v)\in N$$ is a critical point of $$I\vert _{N}$$, then there exists $$\gamma \in R$$ such that

$$I'(u,v)=\gamma F'(u,v).$$

Multiplying the above equality by $$(u,v)$$, we have

$$\bigl\langle I'(u,v),(u,v)\bigr\rangle =\gamma \bigl\langle F'(u,v),(u,v)\bigr\rangle .$$

Now $$F(u,v)=\langle I'(u,v),(u,v)\rangle =0$$ and $$\langle F'(u,v),(u,v) \rangle =-2\Vert (u,v)\Vert ^{2}\leq -2\rho _{0}$$ for any $$(u,v)\in N$$ imply that $$\gamma =0$$, which means $$I'(u,v)=0$$.

(5) Let $$\{(u_{n},v_{n})\}\subset H$$ be a Palais–Smale sequence of $$I\vert _{N}$$, that is, $$\{I(u_{n},v_{n})\}$$ is bounded and $$I'\vert _{N}(u_{n},v _{n})\rightarrow 0$$. In the following, we claim that $$I'(u_{n},v_{n}) \rightarrow 0$$.

The proof of (3) shows $$\{(u_{n},v_{n})\}$$ is bounded in H. Hence after passing to a subsequence if necessary, we may assume that $$(u_{n},v_{n})\rightharpoonup (u,v)$$ in H. Since $$\{(u_{n},v_{n})\}\subset H$$ be a Palais–Smale sequence of $$I\vert _{N}$$, then there exists a sequence $$\{\gamma _{n}\} \subseteq R$$ such that

$$I\vert '_{N}(u_{n},v_{n})=I'(u_{n},v_{n})- \gamma _{n} F'(u_{n},v_{n}).$$

Multiplying the above equality by $$(u_{n},v_{n})$$, we have

$$o_{n}(1)=\bigl\langle I'(u_{n},v_{n}),(u_{n},v_{n}) \bigr\rangle -\gamma _{n} \bigl\langle F'(u_{n},v_{n}),(u_{n},v_{n}) \bigr\rangle =-\gamma _{n}\bigl\langle F'(u _{n},v_{n}),(u_{n},v_{n})\bigr\rangle .$$

It follows from $$\langle F'(u_{n},v_{n}),(u_{n},v_{n})\rangle \leq -2\rho _{0}$$ that $$\gamma _{n}\rightarrow 0$$ as $$n\rightarrow +\infty$$. Notice the fact that $$\{(u_{n},v_{n})\}$$ is bounded in N implies $$F'(u_{n},v_{n})$$ is bounded. Hence

$$I'(u_{n},v_{n})=\gamma _{n} F'(u_{n},v_{n})+o(1)=o(1), \quad \text{as } n \rightarrow \infty .$$

(6) For fixed $$(u,v)\in H\setminus \{(0,0)\}$$ and $$t>0$$, we consider the map $$h: t\mapsto I(tu,tv)$$ defined by

\begin{aligned} h(t): &=\frac{t^{2}}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}u \bigr\vert ^{2}+ \lambda _{1}u^{2}(x)\,dx+ \frac{t^{2}}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}v \bigr\vert ^{2}+\lambda _{2}v^{2}(x)\,dx \\ &\quad {} +kt^{2} \int _{R^{3}}u(x)v(x)\,dx-t ^{4} \int _{R^{3}}\frac{\mu _{1}}{4}u^{4}(x)+ \frac{\mu _{2}}{4}v^{4}(x)+ \frac{1}{2}\beta u^{2}(x)v^{2}(x)\,dx \\ &=\frac{1}{2}t^{2} \bigl\Vert (u,v) \bigr\Vert ^{2}-t ^{4} \int _{R^{3}}\frac{\mu _{1}}{4}u^{4}(x)+ \frac{\mu _{2}}{4}v^{4}(x)+ \frac{1}{2}\beta u^{2}(x)v^{2}(x)\,dx. \end{aligned}

Using the condition $$\beta \in (-\sqrt{\mu _{1}\mu _{2}},+\infty )$$, one can easily get $$h(t)\rightarrow -\infty$$ as $$t\rightarrow +\infty$$. Now we claim that $$h(t)>0$$ for $$t>0$$ small enough. Indeed, by the Sobolev embedding theorem, we have

$$h(t)\geq \frac{1}{2}t^{2} \bigl\Vert (u,v) \bigr\Vert ^{2}-Ct^{4} \bigl\Vert (u,v) \bigr\Vert ^{4},$$

which implies that $$h(t)>0$$ if $$t>0$$ is small enough. Hence there exists $$t_{u,v}>0$$ such that $$h(t)$$ has a positive maximum and $$h'(t_{u,v})=0$$. Notice that $$F(tu,tv)=th'(t)$$, so we have $$F(t_{u,v}u,t_{u,v}v)=0$$. Moreover, if $$F(t_{u,v}u,t_{u,v}v)=0$$, then we get

$$t_{u,v}^{2} \bigl\Vert (u,v) \bigr\Vert ^{2}=t_{u,v}^{4}\biggl( \int _{R^{3}}\mu _{1} u^{4}(x)+\mu _{2} v^{4}(x)+2\beta u^{2}(x)v^{2}(x) \,dx\biggr).$$

Hence, $$t_{u,v}=(\frac{\Vert (u,v)\Vert ^{2}}{\int _{R^{3}}\mu _{1} u^{4}(x)+\mu _{2} v^{4}(x)+2\beta u^{2}(x)v^{2}(x)\,dx})^{\frac{1}{2}}$$, which is the unique critical point of $$h(t)$$ corresponding to its maximum, that is, $$I(t_{u,v}u,t_{u,v}v)= \max_{t>0}I(tu,tv)$$. □

Now we give the proof of Theorem 1.1.

### Proof of Theorem 1.1

We divide the proof into three steps.

Step 1. Existence. We set $$c= \inf_{(u,v)\in N}I(u,v)$$. Then, by the proof of Lemma 3.1, we infer that $$c\geq \frac{1}{4}\rho _{0}>0$$. In view of Lemma 2.4, we can find a sequence $$\{(u_{n},v_{n})\}\subseteq N$$ such that $$\{(u_{n},v_{n})\}$$ is a $$(\mathit{P.S.})_{c}$$ sequence for $$I\vert _{N}$$. And consequently $$\{(u_{n},v _{n})\}$$ is a $$(\mathit{P.S.})_{c}$$ sequence for I by (5) of Lemma 3.1. Moreover, $$\{(u_{n},v_{n})\}$$ is a bounded sequence in H.

Suppose

$$\liminf_{n\rightarrow +\infty }\sup_{x\in R^{3}} \int _{B_{1}(x)}u_{n} ^{2}(y)\,dy=0 \quad \text{and} \quad \liminf_{n\rightarrow +\infty }\sup_{x\in R^{3}} \int _{B_{1}(x)}v_{n} ^{2}(y)\,dy=0,$$

then, by Lemma 2.3, we get

$$(u_{n},v_{n})\rightarrow (0,0) \quad \text{in } L^{4}\bigl(R^{3}\bigr)\times L^{4} \bigl(R^{3}\bigr).$$

Hence, we have

$$\bigl\Vert (u_{n},v_{n}) \bigr\Vert ^{2}=\mu _{1} \int _{R^{3}}u_{n}^{4}(x)\,dx+\mu _{2} \int _{R^{3}}v_{n}^{4}(x)\,dx+2\beta \int _{R^{3}}u_{n}^{2}(x)v_{n}^{2}(x) \,dx \rightarrow 0,$$

contrary to $$\Vert (u_{n},v_{n})\Vert >\rho _{0}$$. Thus, without loss of generality, there is a constant $$a>0$$ such that

$$\liminf_{n\rightarrow +\infty }\sup_{x\in R^{3}} \int _{B_{1}(x)}u_{n} ^{2}(y)\,dy=a.$$

Consequently, going to a subsequence if necessary, we can find a sequence $$\{x_{n}\}\subseteq R^{3}$$ that satisfies

$$\int _{B_{1}(x_{n})}u_{n}^{2}(y)\,dy\geq \frac{a}{2}.$$
(3.1)

Since

$$I\bigl(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n})\bigr)=\frac{1}{4} \bigl\Vert u_{n}(\cdot +x_{n}),v_{n}(\cdot +x _{n}) \bigr\Vert ^{2}=\frac{1}{4} \bigl\Vert (u_{n},v_{n}) \bigr\Vert ^{2}\rightarrow c,$$

we see that $$\{(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x _{n}))\}$$ is bounded. Hence, up to a subsequence, there exists $$(u,v)\in H$$ such that $$(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\rightharpoonup (u,v)$$ in H, $$(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\rightarrow (u,v)$$ in $$L^{2}_{\mathrm{loc}}(R^{3})\times L^{2}_{\mathrm{loc}}(R^{3})$$, $$u_{n}(\cdot +x_{n})\rightarrow u$$, $$v_{n}(\cdot+x_{n})\rightarrow v$$ for $$\textit{a.e. }x\in R^{3}$$. We pass to the limit in (3.1) and we get

$$\int _{B_{1}(0)}u^{2}(y)\,dy\geq \frac{a}{2},$$
(3.2)

which implies $$u\neq0$$. We use the invariance of N and I by translation to conclude that $$\{(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\} \in N$$ and $$\Vert I'(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\Vert =\Vert I'(u_{n},v_{n})\Vert$$, which shows that $$\{(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\}$$ is also a $$(\mathit{P.S.})_{c}$$ sequence of I. Consequently

$$I'(u,v)=0, \quad\quad F(u,v)=0.$$
(3.3)

It follows from $$u\neq0$$ that $$(u,v)\in N$$. Thus, by the weakly lower semi-continuity of $$\Vert \cdot \Vert$$ we obtain

\begin{aligned}[b] c &\leq I(u,v)=\frac{1}{4} \bigl\Vert (u,v) \bigr\Vert ^{2}\leq \frac{1}{4} \liminf _{n\rightarrow \infty } \bigl\Vert \bigl(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n})\bigr) \bigr\Vert ^{2} \\ &= \liminf_{n\rightarrow \infty }I\bigl(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n})\bigr)=c, \end{aligned}
(3.4)

which implies that $$I(u,v)=c$$.

Step 2. If $$k\in (-\sqrt{\lambda _{1}\lambda _{2}},0)$$, then $$u>0$$, $$v>0$$.

It is not difficult to see that

$$\begin{gathered} \int _{R^{3}}u(x)v(x)\,dx\leq \int _{R^{3}} \bigl\vert u(x) \bigr\vert \bigl\vert v(x) \bigr\vert \,dx, \\k \int _{R^{3}}u(x)v(x)\,dx\geq k \int _{R^{3}} \bigl\vert u(x) \bigr\vert \bigl\vert v(x) \bigr\vert \,dx.\end{gathered}$$
(3.5)

Thus, combining (3.5) and $$F(u,v)=0$$ shows

\begin{aligned}[b] &\bigl\Vert \bigl( \vert u \vert , \vert v \vert \bigr) \bigr\Vert ^{2} \\ &\quad = \int _{R^{3}} \int _{R^{3}} \frac{( \vert u(x) \vert - \vert u(y) \vert )^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy+ \int _{R^{3}} \int _{R^{3}} \frac{( \vert v(x) \vert - \vert v(y) \vert )^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy \\ &\quad \quad {} + \int _{R^{3}}\lambda _{1}u^{2}(x)+\lambda _{2}v^{2}(x)+2k \bigl\vert u(x) \bigr\vert \bigl\vert v(x) \bigr\vert \,dx \\ &\quad \leq \int _{R^{3}} \int _{R^{3}}\frac{(u(x)-u(y))^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy+ \int _{R^{3}} \int _{R^{3}}\frac{(v(x)-v(y))^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy \\ &\quad \quad {} + \int _{R^{3}} \lambda _{1}u^{2}(x)+\lambda _{2}v^{2}(x)+2ku(x)v(x)\,dx \\ &\quad = \bigl\Vert (u,v) \bigr\Vert ^{2}=\mu _{1} \int _{R^{3}}u^{4}(x)\,dx+\mu _{2} \int _{R^{3}}v^{4}(x)\,dx+2 \beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx. \end{aligned}
(3.6)

Consider

\begin{aligned} \varphi (t)&=I\bigl(t \vert u \vert ,t \vert v \vert \bigr)\\&= \frac{t^{2}}{2} \bigl\Vert \bigl( \vert u \vert , \vert v \vert \bigr) \bigr\Vert ^{2}- \frac{t^{4}}{4}\biggl(\mu _{1} \int _{R^{3}}u^{4}(x)\,dx+\mu _{2} \int _{R^{3}}v ^{4}(x)\,dx+2\beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx\biggr). \end{aligned}

Let $$t^{*}=\frac{\Vert (\vert u\vert ,\vert v\vert )\Vert }{\Vert (u,v)\Vert }\in (0,1]$$. It is easy to check that $$\varphi (t)$$ is strictly increasing in $$(0,t^{*})$$ and is strictly decreasing in $$(t^{*},+ \infty )$$, which shows that $$\varphi (t)$$ has a unique critical point $$t^{*}>0$$. So $$(t^{*}\vert u\vert ,t^{*}\vert v\vert )\in N$$. Then it follows from $$c\leq I(t^{*}\vert u\vert ,t^{*}\vert v\vert )=\frac{(t^{*})^{2}}{4}\Vert (\vert u\vert ,\vert v\vert ) \Vert ^{2}\leq \frac{1}{4}\Vert (u,v)\Vert ^{2}=c$$ that

$$\bigl\Vert \bigl( \vert u \vert , \vert v \vert \bigr) \bigr\Vert = \bigl\Vert (u,v) \bigr\Vert , \quad \bigl( \vert u \vert , \vert v \vert \bigr)\in N, \quad\quad I\bigl(\bigl( \vert u \vert , \vert v \vert \bigr)\bigr)=c.$$

Therefore, we may assume without loss of generality that $$u\geq 0$$, $$v \geq 0$$. By Lemma 3.1, we know that $$(u,v)$$ is a critical point of I and hence is a ground state solution.

Through a similar argument in [17], one can show that $$\vert u\vert _{L^{\infty }}<+\infty$$, $$\vert u\vert _{L^{\infty }}<+\infty$$. By using the strong maximum principle to each single equation in (1.2), we obtain

$$u>0 \quad \text{and} \quad v>0.$$
(3.7)

Hence, Step 2 is proved.

Step 3. If $$k\in (0,\sqrt{\lambda _{1}\lambda _{2}})$$, then $$u>0$$, $$v<0$$ or $$u<0$$, $$v>0$$.

It is not difficult to see that

$$k \int _{R^{3}}u(x)v(x)\,dx\geq -k \int _{R^{3}} \bigl\vert u(x) \bigr\vert \bigl\vert v(x) \bigr\vert \,dx.$$
(3.8)

Thus, combining (3.8) and $$F(u,v)=0$$ shows

\begin{aligned}[b] &\bigl\Vert \bigl( \vert u \vert ,- \vert v \vert \bigr) \bigr\Vert ^{2} \\ &\quad = \int _{R^{3}} \int _{R^{3}} \frac{( \vert u(x) \vert - \vert u(y) \vert )^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy+ \int _{R^{3}} \int _{R^{3}} \frac{(- \vert v(x) \vert + \vert v(y) \vert )^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy \\ &\quad \quad {} + \int _{R^{3}}\lambda _{1}u^{2}(x)+\lambda _{2}v^{2}(x)-2k \bigl\vert u(x) \bigr\vert \bigl\vert v(x) \bigr\vert \,dx \\ &\quad \leq \int _{R^{3}} \int _{R^{3}}\frac{(u(x)-u(y))^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy+ \int _{R^{3}} \int _{R^{3}}\frac{(v(x)-v(y))^{2}}{ \vert x-y \vert ^{3+2s}}\,dx\,dy \\ &\quad \quad {} + \int _{R^{3}} \lambda _{1}u^{2}(x)+\lambda _{2}v^{2}(x)+2ku(x)v(x)\,dx \\ &\quad = \bigl\Vert (u,v) \bigr\Vert ^{2}=\mu _{1} \int _{R^{3}}u^{4}(x)\,dx+\mu _{2} \int _{R^{3}}v^{4}(x)\,dx+2 \beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx. \end{aligned}
(3.9)

Consider

\begin{aligned} \psi (s)&=I\bigl(s \vert u \vert ,-s \vert v \vert \bigr)\\&= \frac{s^{2}}{2} \bigl\Vert \bigl( \vert u \vert ,- \vert v \vert \bigr) \bigr\Vert ^{2}- \frac{s^{4}}{4}\biggl(\mu _{1} \int _{R^{3}}u^{4}(x)\,dx+\mu _{2} \int _{R^{3}}v ^{4}(x)\,dx+2\beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx\biggr). \end{aligned}

Let $$s^{*}=\frac{\Vert (\vert u\vert ,-\vert v\vert )\Vert }{\Vert (u,v)\Vert }\in (0,1]$$. It is easy to check that $$\psi (s)$$ is strictly increasing in $$(0,s^{*})$$ and is strictly decreasing in $$(s^{*},+\infty )$$, which shows that $$\psi (s)$$ has a unique critical point $$s^{*}>0$$. So $$(s^{*}\vert u\vert ,-s ^{*}\vert v\vert )\in N$$. Then it follows from $$c\leq I(s^{*}\vert u\vert ,-s^{*}\vert v\vert )=\frac{(s ^{*})^{2}}{4}\Vert (\vert u\vert ,-\vert v\vert )\Vert ^{2}\leq \frac{1}{4}\Vert (u,v)\Vert ^{2}=c$$ that

$$\bigl\Vert \bigl( \vert u \vert ,- \vert v \vert \bigr) \bigr\Vert = \bigl\Vert (u,v) \bigr\Vert , \quad \bigl( \vert u \vert ,- \vert v \vert \bigr)\in N, \quad\quad I\bigl(\bigl( \vert u \vert ,- \vert v \vert \bigr)\bigr)=c.$$

Therefore, we may assume without loss of generality that $$u\geq 0$$, $$v \leq 0$$. By Lemma 3.1, we know that $$(u,v)$$ is a critical point of I and hence is a ground state solution. Similar to the proof of (3.7), we can prove that $$u>0$$, $$v<0$$.

The proof of Theorem 1.1 is completed. □

## 4 Proof of Theorem 1.2

Now we turn to the system (1.1). Let $$\lambda _{1}, \lambda _{2}, \mu _{1}, \mu _{2}>0$$, $$\beta \in (-\sqrt{\mu _{1}\mu _{2}},+\infty )$$, $$k\in (-\sqrt{\lambda _{1}\lambda _{2}},0)\cup (0,\sqrt{\lambda _{1}\lambda _{2}})$$ and $$(V)$$ holds. We only consider the case that $$V(x)\not \equiv \varLambda$$, otherwise Theorem 1.2 comes down to Theorem 1.1.

We shall search solutions to the system (1.1) as critical points for the functional

\begin{aligned} I_{V}(u,v) &=\frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}u \bigr\vert ^{2}+\bigl( \lambda _{1}+V(x)\bigr)u^{2}(x) \,dx \\ &\quad {}+\frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}v \bigr\vert ^{2}+\bigl(\lambda _{2}+V(x)\bigr)v^{2}(x) \,dx \\ &\quad {} +k \int _{R^{3}}u(x)v(x)\,dx- \frac{1}{4}\mu _{1} \int _{R^{3}}u^{4}(x)\,dx\\ &\quad {}-\frac{1}{4}\mu _{2} \int _{R ^{3}}v^{4}(x)\,dx-\frac{1}{2}\beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx, \end{aligned}

which is well defined on the Hilbert space $$H^{s}(R^{3})\times H^{s}(R ^{3})$$, equipped with the inner product as follows:

\begin{aligned} \bigl((u_{1},v_{1}),(u_{2},v_{2}) \bigr)_{V}&= \int _{R^{3}}(-\triangle )^{s/2}u _{1}(- \triangle )^{s/2}u_{2}+\bigl(\lambda _{1}+V(x) \bigr)u_{1}(x)u_{2}(x)\,dx \\ &\quad {} + \int _{R^{3}}(-\triangle )^{s/2}v_{1}(- \triangle )^{s/2}v_{2}+\bigl(\lambda _{2}+V(x) \bigr)v_{1}(x)v_{2}(x)\,dx \\ &\quad {}+k \int _{R^{3}}u_{1}(x)v_{1}(x)\,dx+k \int _{R^{3}}u_{2}(x)v_{2}(x)\,dx, \end{aligned}

for $$(u_{1},v_{1}), (u_{2},v_{2})\in H^{s}(R^{3})\times H^{s}(R^{3})$$. $$\Vert (u,v)\Vert _{V}=((u,v),(u,v))_{V}^{\frac{1}{2}}$$ is the corresponding norm if $$\vert k\vert <\sqrt{\lambda _{1}\lambda _{2}}$$, this is equivalent to the standard product norm on the product space $$H^{s}(R^{3})\times H^{s}(R^{3})$$.

The limit system of (1.1) is

$$\textstyle\begin{cases} (-\triangle )^{s}u+(\lambda _{1}+\varLambda )u+kv=\mu _{1}u^{3}+\beta uv ^{2}, \quad \text{in } R^{3},\\ (-\triangle )^{s}v+(\lambda _{2}+\varLambda )v+ku=\mu _{2}v ^{3}+\beta u^{2}v, \quad \text{in } R^{3},\\ u, v\in H^{s}(R^{3}). \end{cases}$$
(4.1)

The energy functional of the limit system (4.1) is given by

\begin{aligned} I_{\varLambda }(u,v) &=\frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}u \bigr\vert ^{2}+( \lambda _{1}+\varLambda )u^{2}(x) \,dx+\frac{1}{2} \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}v \bigr\vert ^{2}+(\lambda _{2}+\varLambda )v^{2}(x)\,dx \\ &\quad {} +k \int _{R^{3}}u(x)v(x)\,dx- \frac{1}{4}\mu _{1} \int _{R^{3}}u^{4}(x)\,dx-\frac{1}{4}\mu _{2} \int _{R ^{3}}v^{4}(x)\,dx \\ &\quad {}-\frac{1}{2}\beta \int _{R^{3}}u^{2}(x)v^{2}(x)\,dx. \end{aligned}

Let

\begin{aligned}& N_{V}:=\bigl\{ u\in H^{s}\bigl(R^{3}\bigr)\times H^{s}\bigl(R^{3}\bigr)\setminus \bigl\{ (0,0)\bigr\} : F _{V}(u,v)=\big\langle I'_{V}(u,v),(u,v)\big\rangle =0\bigr\} , \\& c_{V}= \inf_{(u,v)\in N_{V}}I_{V}(u,v), \end{aligned}

and define $$c_{\varLambda }$$ to be the constant which corresponds to the c in Sect. 2 when $$\lambda _{1}$$, $$\lambda _{2}$$ is replaced by $$\lambda _{1}+\varLambda$$, $$\lambda _{2}+\varLambda$$. The Nehari manifold $$N_{V}$$ shares the characteristic with N that has been defined in Sect. 3. In order to prove Theorem 1.2, we need the following lemma.

### Lemma 4.1

If the condition V holds, then $$c_{V}< c_{\varLambda }$$.

### Proof

Since $$\beta \in (-\sqrt{\mu _{1}\mu _{2}},+\infty )$$, we have

$$c_{V}= \inf_{\eta \in \varGamma }\sup _{t\in [0,1]}I_{V}\bigl(\eta (t)\bigr),$$
(4.2)

where $$\varGamma :=\{\eta \in C([0,1],H^{s}(R^{3})\times H^{s}(R^{3}): \eta (0)=(0,0), I_{V}(\eta (1))<0)\}$$. Let $$(\bar{u},\bar{v})$$ be the ground state solution to (4.1) which was given in Theorem 1.1. In view of

$$V(x)\leq \varLambda ,\quad\quad V(x)\neq \varLambda , \quad\quad \bar{u}\neq0, \quad\quad \bar{v} \neq0,$$

there is a positive constant $$t_{*}>0$$ such that

$$\max_{t>0}I_{V}(t\bar{u},t \bar{v})=I_{V}(t_{*}\bar{u},t_{*}\bar{v})< I _{\varLambda }(t_{*}\bar{u},t_{*}\bar{v})\leq \max _{t>0}I_{\varLambda }(t\bar{u},t\bar{v})=c_{\varLambda }.$$
(4.3)

Hence

$$c_{V}< c_{\varLambda }.$$

The proof of Lemma 4.1 is complete. □

Now we give the proof of Theorem 1.2.

### Proof of Theorem 1.2

Our arguments are similar to the ones developed in Theorem 1.4 of [1] but we give the details for the reader’s convenience.

The definition of $$c_{V}$$ shows that $$c_{V}>0$$. In view of Lemma 2.4, we can find a sequence $$\{(u_{n},v_{n})\}\subseteq N_{V}$$ such that $$\{(u_{n},v_{n})\}$$ is a $$(\mathit{P.S.})_{c_{V}}$$ sequence for $$I_{V}\vert _{N_{V}}$$. And consequently $$\{(u_{n},v_{n})\}$$ is a $$(\mathit{P.S.})_{c_{V}}$$ sequence for I by (5) of Lemma 3.1. Moreover, $$\{(u_{n},v_{n})\}$$ is a bounded sequence in $$H^{s}(R^{3})\times H^{s}(R^{3})$$. Thus, up to a subsequence, there exists $$(u,v)\in H^{s}(R^{3})\times H^{s}(R^{3})$$ such that $$(u_{n},v_{n})\rightharpoonup (u,v)$$ in $$H^{s}(R^{3})\times H^{s}(R^{3})$$. In the following, we claim that $$(u,v)\neq(0,0)$$.

Indeed, if $$(u,v)=(0,0)$$, then $$u_{n}\rightharpoonup 0$$, $$v_{n}\rightharpoonup 0$$ in $$H^{s}(R^{3})$$. From $$\lim_{\vert x\vert \rightarrow +\infty }V(x)=\varLambda$$ we see that, for any $$\varepsilon >0$$, there exists R large enough, such that

$$\int _{B_{R}^{c}(0)} \bigl\vert V(x)-\varLambda \bigr\vert u_{n}^{2}(x)\,dx< \frac{\varepsilon }{2}.$$

Since $$V(x)\in L^{\infty }$$ and $$(u_{n},v_{n})\rightharpoonup (0,0)$$ we obtain

$$\int _{B_{R}(0)} \bigl\vert V(x)-\varLambda \bigr\vert u_{n}^{2}(x)\,dx< \frac{\varepsilon }{2},$$

for large n, so that

$$\int _{R^{3}} \bigl\vert V(x)-\varLambda \bigr\vert u_{n}^{2}(x)\,dx=o_{n}(1), \quad n\rightarrow + \infty .$$
(4.4)

With the same computation we see that $$\forall \varphi \in H^{s}(R ^{3})$$

\begin{aligned}& \int _{R^{3}} \bigl\vert V(x)-\varLambda \bigr\vert v_{n}^{2}(x)\,dx=o_{n}(1),\quad n\rightarrow + \infty , \end{aligned}
(4.5)
\begin{aligned}& \int _{R^{3}} \bigl\vert V(x)-\varLambda \bigr\vert u_{n}(x)\varphi (x)\,dx=o_{n}(1),\quad n\rightarrow +\infty , \end{aligned}
(4.6)
\begin{aligned}& \int _{R^{3}} \bigl\vert V(x)-\varLambda \bigr\vert v_{n}(x)\varphi (x)\,dx=o_{n}(1),\quad n\rightarrow +\infty . \end{aligned}
(4.7)

Consequently,

\begin{aligned}& I_{\varLambda }(u_{n},v_{n})=I_{V}(u_{n},v_{n})+o_{n}(1)=c_{V}+o_{n}(1), \end{aligned}
(4.8)
\begin{aligned}& I'_{\varLambda }(u_{n},v_{n})=I'_{V}(u_{n},v_{n})+o_{n}(1)=o_{n}(1). \end{aligned}
(4.9)

Assume

$$\liminf_{n\rightarrow +\infty }\sup_{x\in R^{3}} \int _{B_{1}(x)}u_{n} ^{2}(y)\,dy=0 \quad \text{and} \quad \liminf_{n\rightarrow +\infty }\sup_{x\in R^{3}} \int _{B_{1}(x)}v_{n} ^{2}(y)\,dy=0,$$

then, by Lemma 2.3, we get

$$(u_{n},v_{n})\rightarrow (0,0) \quad \text{in } L^{4}\bigl(R^{3}\bigr)\times L^{4} \bigl(R^{3}\bigr).$$

Hence, we have

$$\bigl\Vert (u_{n},v_{n}) \bigr\Vert _{V}^{2}=\mu _{1} \int _{R^{3}}u_{n}^{4}(x)\,dx+\mu _{2} \int _{R^{3}}v_{n}^{4}(x)\,dx+2\beta \int _{R^{3}}u_{n}^{2}(x)v_{n}^{2}(x) \,dx \rightarrow 0,$$

which is contrary to the fact that $$\Vert (u_{n},v_{n})\Vert _{V}^{2}>C>0$$. Thus, without loss of generality, there is a constant such that $$\alpha _{V}>0$$

$$\liminf_{n\rightarrow +\infty }\sup_{x\in R^{3}} \int _{B_{1}(x)}u_{n} ^{2}(y)\,dy=\alpha _{V}.$$

Consequently, going if necessary to a subsequence, we can find a sequence $$\{x_{n}\}\subseteq R^{3}$$ that satisfies

$$\int _{B_{1}(x_{n})}u_{n}^{2}(y)\,dy\geq \frac{\alpha _{V}}{2}.$$
(4.10)

Since $$V(x)\leq \varLambda$$, we deduce from (4.4) and (4.5) that

\begin{aligned} &\bigl\Vert \bigl(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n})\bigr) \bigr\Vert _{V}^{2} \\ &\quad \leq \int _{R^{3}} \bigl\vert (- \triangle )^{s/2}u_{n} \bigr\vert ^{2}+(\lambda _{1}+\varLambda )u_{n}^{2}(x)\,dx \\ &\quad\quad {} + \int _{R^{3}} \bigl\vert (-\triangle )^{s/2}v_{n} \bigr\vert ^{2}+(\lambda _{2}+\varLambda )v _{n}^{2}(x)\,dx+2k \int _{R^{3}}u_{n}(x)v_{n}(x)\,dx \\ &\quad = \int _{R^{3}} \bigl\vert (- \triangle )^{s/2}u_{n} \bigr\vert ^{2}+\bigl(\lambda _{1}+V(x) \bigr)u_{n}^{2}(x)\,dx \\ &\quad\quad {} + \int _{R^{3}} \bigl\vert (-\triangle )^{s/2}v_{n} \bigr\vert ^{2}+\bigl(\lambda _{2}+V(x) \bigr)v_{n} ^{2}(x)\,dx+2k \int _{R^{3}}u_{n}(x)v_{n}(x)\,dx+o(1) \\ &\quad =4I_{V}(u_{n},v _{n})+o_{n}(1)=4c_{V}+o_{n}(1), \quad \text{as } n\rightarrow \infty , \end{aligned}

so that $$\{(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\}$$ is bounded. Thus, up to a subsequence, there exists $$(u_{*},v_{*})\in H^{s}(R^{3})\times H ^{s}(R^{3})$$ such that $$(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\rightharpoonup (u_{*},v_{*})$$ in $$H^{s}(R^{3})\times H^{s}(R^{3})$$, $$(u_{n}(\cdot +x_{n}),v _{n}(\cdot +x_{n}))\rightarrow (u_{*},v_{*})$$ in $$L^{2}_{\mathrm{loc}}(R^{3})\times L^{2}_{\mathrm{loc}}(R^{3})$$, $$u_{n}(\cdot +x_{n})\rightarrow u_{*}$$, $$v_{n}(\cdot +x_{n}) \rightarrow v_{*}$$ for $$\textit{a.e. }x\in R^{3}$$. We pass to the limit in (4.10) and we get

$$\int _{B_{1}(0)}u_{*}^{2}(y)\,dy\geq \frac{\alpha _{V}}{2},$$
(4.11)

which implies $$u_{*}\neq0$$. From (4.8) and (4.9), we derive that $$\{(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))\}$$ is a $$(\mathit{P.S.})_{c_{V}}$$ sequence for $$I_{\varLambda }$$. We also know that $$I'_{\varLambda }(u_{*},v _{*})=0$$. Therefore

\begin{aligned} I_{\varLambda }(u_{*},v_{*})={} &\frac{1}{4} \int _{R^{3}} \bigl\vert (-\triangle )^{ \frac{s}{2}}u_{*} \bigr\vert ^{2}+(\lambda _{1}+\varLambda )u_{*}^{2}(x)\,dx+ \frac{1}{4} \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}v_{*} \bigr\vert ^{2}+(\lambda _{2}+\varLambda )v_{*}^{2}(x)\,dx \\ & {} +\frac{1}{2}k \int _{R^{3}}u_{*}(x)v _{*}(x)\,dx. \end{aligned}

Let $$(\bar{u_{n}},\bar{v_{n}})=(u_{n}(\cdot +x_{n}),v_{n}(\cdot +x_{n}))$$. Now $$(\bar{u_{n}},\bar{v_{n}})\rightharpoonup (u_{*},v_{*})$$ implies that

\begin{aligned}& \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}u_{*} \bigr\vert ^{2}+(\lambda _{1}+ \varLambda )u_{*}^{2}(x)\,dx+ \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}v_{*} \bigr\vert ^{2}+( \lambda _{2}+\varLambda )v_{*}^{2}(x)\,dx \\& \quad \quad {}+2k \int _{R^{3}}u_{*}(x)v_{*}(x)\,dx \\& \quad \leq \liminf_{n\rightarrow \infty } \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}\bar{u _{n}} \bigr\vert ^{2}+(\lambda _{1}+\varLambda ){\bar{u_{n}}}^{2}\,dx+ \int _{R^{3}} \bigl\vert (- \triangle )^{\frac{s}{2}} \bar{v_{n}} \bigr\vert ^{2}+(\lambda _{2}+ \varLambda ){\bar{v _{n}}}^{2}\,dx \\& \quad \quad {}+2k \int _{R^{3}}\bar{u_{n}}\bar{v_{n}}\,dx \\& \quad = \liminf_{n\rightarrow \infty } \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}u _{n} \bigr\vert ^{2}+(\lambda _{1}+\varLambda )u_{n}^{2}\,dx+ \int _{R^{3}} \bigl\vert (-\triangle )^{\frac{s}{2}}v_{n} \bigr\vert ^{2}+(\lambda _{2}+\varLambda )v_{n}^{2}\,dx \\& \quad \quad {}+2k \int _{R^{3}}u_{n}v_{n}\,dx \\& \quad = \liminf_{n\rightarrow \infty } \bigl\Vert (u_{n},v_{n}) \bigr\Vert _{V}^{2} \end{aligned}

so that

$$c_{\varLambda }\leq I_{\varLambda }(u_{*},v_{*})\leq \liminf_{n\rightarrow \infty }\frac{1}{4} \bigl\Vert (u_{n},v_{n}) \bigr\Vert _{V}^{2}= \liminf_{n\rightarrow \infty }I_{V}(u_{n},v_{n})=c_{V},$$

this contradicts Lemma 4.1. Hence $$u\neq0$$ or $$v\neq0$$.

Then we can prove Theorem 1.2 similarly by using the same method as has been used in Theorem 1.1. □

## References

1. Li, K., Zhang, Z.: Existence of solutions for a Schrödinger system with linear and nonlinear couplings. J. Math. Phys. 57, 081504 (2016)

2. Tian, R., Zhang, Z.: Existence and bifurcation of solution for a double coupled sysytem of Schrödinger equations. Sci. China Math. 58, 1607–1620 (2015)

3. Zhang, Z., Wang, W.: Structure of positive solutions to a Schrödinger system. J. Fixed Point Theory Appl. 19, 877–887 (2017)

4. Lv, D., Peng, S.: On the positive vector sloutions for nonlinear fractional Laplacian systems with linear coupling. Discrete Contin. Dyn. Syst. 6, 3327–3352 (2017)

5. Guo, Q., He, X.: Least energy solutions for a weakly coupled fractional Schrödinger system. Nonlinear Anal. 132, 141–159 (2016)

6. Guo, Q., He, X.: Semiclassical states for weakly coupled fractional Schrödinger systems. J. Differ. Equ. 263, 1986–2023 (2017)

7. Di Nezza, E., Palatucci, G., Valdinoci, E.: Hitchhiker’s guide to the fractional Sobolev space. Bull. Sci. Math. 136, 521–573 (2012)

8. Servadei, R., Valdinoci, E.: The Brezis–Nirenberg result for the fractional Laplacian. Trans. Am. Math. Soc. 367, 67–102 (2015)

9. Chang, X., Wang, Z.-Q.: Ground state of scalar field equations involving a fractional Laplacian with general nonlinearity. Nonlinearity 26, 479–494 (2013)

10. Wang, Z., Zhou, H.: Radial sign-changing solution for fractional Schrödinger equation. Discrete Contin. Dyn. Syst. 36, 499–508 (2016)

11. Chen, W., Li, C., Li, Y.: A direct method of moving planes for the fractional Laplacian. Adv. Math. 308, 404–437 (2017)

12. Chen, W., Li, Y., Zhang, R.: A direct method of moving spheres on fractional order equations. J. Funct. Anal. 272, 4131–4157 (2017)

13. Secchi, S.: On fractional Schrödinger equations in $$R^{n}$$ without the Ambrosetti–Rabinowitz condition. Topol. Methods Nonlinear Anal. 47(1), 19–41 (2016)

14. Cotsiolis, G., Tavoularis, N.: Best constans for Sobolev inequalities for higher order fractional derivatives. J. Math. Anal. Appl. 295, 225–236 (2004)

15. Secchi, S.: Ground state solutions for nonlinear fractional Schrödinger equations in $$R^{n}$$. J. Math. Phys. 54, 031501 (2013)

16. Du, X., Mao, A.: Existence and multiplicity of nontrivial solutions for a class of semilinear fractional Schrödinger equations. J. Funct. Spaces 2017, Article ID 3793872 (2017). https://doi.org/10.1155/2017/3793872

17. Barrios, B., Colorado, E., Servadei, R., Soria, F.: A critical fractional equation with concave-convex power nonlinearities. Ann. Inst. Henri Poincaré, Anal. Non Linéaire 32, 875–900 (2015)

18. Xiang, M., Zhang, B., Radulescu, V.D.: Existence of solutions for perturbed fractional p-Laplacian equations. J. Differ. Equ. 260(2), 1392–1413 (2016)

19. Ambrosio, V.: Zero mass case for a fractional Berestycki–Lions-type problem. Adv. Nonlinear Anal. 7(3), 365–374 (2018)

20. Diaz, J.I., Gomez-Castro, D., Vazquez, J.L.: The fractional Schrödinger equation with general nonnegative potentials. The weighted space approach. Nonlinear Anal. 177(part A), 325–360 (2018)

21. Mingqi, X., Radulescu, V.D., Zhang, B.: Combined effects for fractional Schrödinger–Kirchhoff systems with critical nonlinearities. ESAIM Control Optim. Calc. Var. 24(3), 1249–1273 (2018)

22. Molica Bisci, G., Radulescu, V.D., Servadei, R.: Variational Methods for Nonlocal Fractional Problems. Encyclopedia of Mathematics and Its Applications, vol. 162. Cambridge University Press, Cambridge (2016)

23. Palatucci, G.: The Dirichlet problem for the p-fractional Laplace equation. Nonlinear Anal. 177, 699–732 (2018)

24. Lyons, J., Neugebauer, J.: Positive solutions of a singular fractional boundary value problem with a fractional boundary condition. Opusc. Math. 37(3), 421–434 (2017)

25. Jiang, J., Liu, L., Wu, Y.: Positive solutions to singular fractional differential system with coupled boundary conditions. Commun. Nonlinear Sci. Numer. Simul. 18, 3061–3074 (2013)

26. Jiang, J., Liu, L.: Existence of solutions for a sequential fractional differential system with coupled boundary conditions. Bound. Value Probl. 2016, 159 (2016)

27. Jiang, J., Liu, L., Wu, Y.: Multiple positive solutions of singular fractional differential system involving Stieltjes integral conditions. Electron. J. Qual. Theory Differ. Equ. 2012, Article ID 43 (2012)

28. Wang, Y., Jiang, J.: Existence and nonexistence of positive solutions for the fractional coupled system involving generalized p-Laplacian. Adv. Differ. Equ. 2017, Article ID 337 (2017)

29. Jiang, J., Liu, W., Wang, H.: Positive solutions to singular Dirichlet-type boundary value problems of nonlinear fractional differential equations. Adv. Differ. Equ. 2018, 169 (2018)

30. Zhang, K.: On sign-changing solution for some fractional differential equations. Bound. Value Probl. 2017, 59 (2017)

31. Wang, Y., Liu, L.: Positive solutions for a class of fractional 3-point boundary value problems at resonance. Adv. Differ. Equ. 2017, 7 (2017). https://doi.org/10.1186/s13662-016-0162-5

32. Hao, X., Wang, H., Liu, L., Cui, Y.: Positive solutions for a system of nonlinear fractional nonlocal boundary value problems with parameters and p-Laplacian operator. Bound. Value Probl. 2017, 182 (2017)

33. Zuo, M., Hao, X., Liu, L., Cui, Y.: Existence results for impulsive fractional integro-differential equation of mixed type with constant coefficient and antiperiodic boundary conditions. Bound. Value Probl. 2017, 161 (2017)

34. Liu, L.L., Zhang, X., Liu, L., Wu, Y.: Iterative positive solutions for singular nonlinear fractional differential equation with integral boundary conditions. Adv. Differ. Equ. 2016, 154 (2016). https://doi.org/10.1186/s13662-016-0876-5

35. Guan, Y., Zhao, Z., Lin, X.: On the existence of positive solutions and negative solutions of singular fractional differential equations via global bifurcation techniques. Bound. Value Probl. 2016, 141 (2016). https://doi.org/10.1186/s13661-016-0650-3

36. Zhang, X., Liu, L., Wu, Y., Wiwatanapataphee, B.: The spectral analysis for a singular fractional differential equation with a signed measure. Appl. Math. Comput. 257, 252–263 (2015)

37. Willem, M.: Minimax Theorems. Birkhäuser, Boston (1996)

Download references

### Acknowledgements

This paper was written during Xinsheng Du’s visit to Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, People’s Republic of China as a research fellow. The authors would like to express their sincere gratitude to Professor Zhitao Zhang for advising us to study this problem and also giving valuable comments. The authors would like to thank the referee for his/her very important comments, which improved the results and the quality of the paper.

Not applicable.

## Funding

Research supported by the National Natural Science Foundation of China (11471187, 11571197).

## Author information

Authors

### Contributions

XD and AM developed the idea for the study, performed the research and wrote the paper. All authors read and approved the final manuscript.

### Corresponding author

Correspondence to Xinsheng Du.

## Ethics declarations

### Competing interests

The authors declare that there is no conflict of interest regarding the publication of this paper.

## Additional information

### Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

## About this article

### Cite this article

Du, X., Mao, A. Existence of ground state solutions to a class of fractional Schrödinger system with linear and nonlinear couplings. Bound Value Probl 2019, 78 (2019). https://doi.org/10.1186/s13661-019-1185-1

Download citation

• Received:

• Accepted:

• Published:

• DOI: https://doi.org/10.1186/s13661-019-1185-1