Semiclassical states for non-cooperative singularly perturbed fractional Schrödinger systems

Abstract

We study the following non-cooperative type singularly perturbed systems involving the fractional Laplacian operator:

$$ \textstyle\begin{cases} \varepsilon ^{2s}(-\Delta )^{s} u+a(x)u=g(v), & \text{in } \mathbb{R}^{N}, \\ \varepsilon ^{2s}(-\Delta )^{s} v+a(x)v=f(u), & \text{in } \mathbb{R}^{N}, \end{cases} $$

where $s\in (0,1)$, $N>2s$, and $(-\Delta )^{s}$ is the s-Laplacian, $\varepsilon >0$ is a small parameter. f and g are power-type nonlinearities having superlinear and subcritical growth at infinity. The corresponding energy functional is strongly indefinite, which is different from the one of the single equation case and the one of a cooperative type. By considering some truncated problems and establishing some auxiliary results, the semiclassical solutions of the original system are obtained using “indefinite functional theorem”. The concentration phenomenon is also studied. It is shown that the semiclassical solutions can concentrate around the global minima of the potential.

1 Introduction and main results

In these last years, a great deal of work has been devoted to the study of the weak solutions for the following singularly perturbed fractional Schrödinger systems

$$ \textstyle\begin{cases} \varepsilon ^{2s}(-\Delta )^{s} u+a(x)u=f(u,v), & \text{in } \mathbb{R}^{N}, \\ \varepsilon ^{2s}(-\Delta )^{s} v+a(x)v=g(u,v), & \text{in } \mathbb{R}^{N}, \end{cases} $$

(1.1)

where $s\in (0,1)$ with $N>2s$, $\varepsilon >0$ is a small parameter, $a(x)\in C(\mathbb{R}^{N})$ is the external potential, and f, g satisfy appropriate conditions in order to use a variational method. To describe the transition from quantum to classical mechanics, we let $\varepsilon \rightarrow 0$, and thus the existence of solutions to (1.1) for small ε, which are called semiclassical states, has an important physical interest. For small value ε, the wave functions of (1.1) tend to concentrate as a material particle.

Recently, there has been tremendous interest in developing the fractional Laplacian problem in various fields, for instance, thin obstacle problems, optimization, population dynamics, geophysical fluid dynamics, mathematical finance, phases transitions, anomalous diffusion, crystal dislocation, ultra-relativistic limits of quantum mechanics, etc., see [1]. Different from the classical Laplace operator, the analytical methods for elliptic PDEs cannot be directly applied to (1.1) since the operator $(-\Delta )^{s}$ is nonlocal. In [2], Caffaralli and Silvestre gave a new formulation of the fractional Laplacian through Dirichlet-Neumann maps. This is extensively used in the recent literature since it allows to transform nonlocal problems into local ones, which enables the of use variational methods. For example, for the single nonlocal problems, this is, $u=v$, $f=g$ in (1.1), there have been many results on the existence and concentration, which were studied using the idea of the s-harmonic extension [3–9].

In the case of the standard Laplacian operator ($s=1$, local case), the existence of a solution for the Schrödinger systems has been studied, and relatively complete methods have been formed. However, for the fractional Schrödinger systems like (1.1), there are only some literature on the semiclassical states for nonlocal singularly perturbed problems; for example, see [7, 10–13].

In [7], Q. Guo and X.M. He considered the following nonlinear system of two weakly coupled Schrödinger equations

$$ \textstyle\begin{cases} \varepsilon ^{2s}(-\Delta )^{s} u+P_{1}(x)u=(\lvert u \rvert ^{2p}+b \lvert u \rvert ^{p-1}\lvert v\rvert ^{p+1})u, &\text{in } \mathbb{R}^{N}, \\ \varepsilon ^{2s}(-\Delta )^{s} v+P_{2}(x)v=(\lvert v \rvert ^{2p}+b \lvert v \rvert ^{p-1}\lvert u \rvert ^{p+1})v, & \text{in } \mathbb{R}^{N}, \end{cases} $$

and investigated the existence of nontrivial nonnegative solutions which concentrate around local minimal of the potentials.

Later, Vincenzo Ambrosio [10] applied penalization techniques, Nehari manifold arguments, and Ljusternik-Schnirelmann theory and investigated the existence, multiplicity, and concentration of positive solutions of the following nonlocal system of fractional Schrödinger equations

$$ \textstyle\begin{cases} \varepsilon ^{2s}(-\Delta )^{s} u+V(x)u=Q_{u}(u,v), & \text{in } \mathbb{R}^{N}, \\ \varepsilon ^{2s}(-\Delta )^{s} v+W(x)v=Q_{v}(u,v), & \text{in } \mathbb{R}^{N}, \end{cases} $$

where $V,W:\mathbb{R}^{N} \rightarrow \mathbb{R}$ are positive continuous potentials, Q is a homogeneous $C^{2}$-function with subcritical growth.

We note that Vincenzo Ambrosio [11] also dealt with the following nonlocal systems of fractional Schrödinger equations

$$ \textstyle\begin{cases} \varepsilon ^{2s}(-\Delta )^{s} u+V(x)u=Q_{u}(u,v)+\gamma H_{u}(u,v), & \text{in } \mathbb{R}^{N}, \\ \varepsilon ^{2s}(-\Delta )^{s} v+W(x)v=Q_{v}(u,v)+\gamma H_{v}(u,v), & \text{in } \mathbb{R}^{N}, \end{cases} $$

(1.2)

where $V,W:\mathbb{R}^{N} \rightarrow \mathbb{R}$ are continuous potentials, Q is a homogeneous $C^{2}$-function with subcritical growth, $\gamma \in (0,1)$ and $H(u,v)= (2/(\alpha + \beta ))\lvert u \rvert ^{\alpha}\lvert v \rvert ^{\beta}$ with $\alpha , \beta \geq 1$ such that $\alpha +\beta =2_{s}^{*}$. They investigated the subcritical case $(\gamma =0)$ and the critical case $(\gamma =1)$, and using the Ljusternik-Schnirelmann theory, they related the number of solutions with the topology of the set where the potentials V and W attain their minimum values.

We point out that Manassés de Souza in [12] also considered the existence and multiplicity of solutions of the following more general nonlocal system involving the fractional Laplacian

$$ \varepsilon ^{2s}(-\Delta )^{s} u_{i}+a_{i}(x)u_{i}=f_{i}(x,u_{1}, \ldots , u_{m}),\quad \text{in } \mathbb{R}^{N}, i=1, \ldots ,m, $$

where $a_{i}(x)$ are continuous and unbounded potentials that may change sign, and the nonlinearities $f_{i}(x,u_{1},\ldots , u_{m})$ are continuous functions that may be unbounded in x.

In line with the above works, it is worth mentioning that the nonlinearities are cooperative type in [7, 10–13] hence the energy functions corresponding to them can be proved to have mountain pass structure, and Nehari manifold arguments can be used. However, when the nonlinearities are non-cooperative type, the corresponding energy functional is strongly indefinite; that is, the quadratic part of the energy functional has no longer a positive sign, the problems become rather complicated mathematically. In [14], Manasses de Souza established a weighted Thudinger-Morse type inequality and, as the application of this result using the Galerkin methods and a linking theorem, proved the existence of weak solutions for the following elliptic system:

$$ \textstyle\begin{cases} (-\Delta )^{\frac{1}{2}} u+V(x)u=g(x,v), & \text{in } \mathbb{R}^{N}, \\ (-\Delta )^{\frac{1}{2}} v+V(x)v=f(x,u), & \text{in } \mathbb{R}^{N}. \end{cases} $$

However, another question arises: for the following more general non-cooperative type singularly perturbed fractional systems:

$$ \textstyle\begin{cases} \varepsilon ^{2s}(-\Delta )^{s} u+a(x)u=g(v), & \text{in } \mathbb{R}^{N}, \\ \varepsilon ^{2s}(-\Delta )^{s} v+a(x)v=f(u), & \text{in } \mathbb{R}^{N}, \end{cases} $$

(1.3)

whether the results [7, 10–13] on the existence of semiclassical states and concentration can be obtained? Answering this question constitutes the goal of this paper.

Since we are interested in positive solutions, we assume the continuous functions $a(x)$, f, g satisfy the following conditions:

($A_{0}$):

$0< a(0):=\min_{x\in \mathbb{R}^{N}}a(x)< \liminf_{\lvert x \rvert \rightarrow \infty}a(x)$;

($A_{1}$):

$f(0)=g(0)=f'(0)=g'(0)=0$, $f(t)=g(t)=0$ for $t\leqslant 0$;

($A_{2}$):

there exist real numbers $l_{1},l_{2}>0$ and $p,q>2$ such that $\frac{1}{p}+\frac{1}{q}>\frac{N-2s}{N}$ and

$$ \lim_{\lvert t \rvert \rightarrow \infty} \frac{f^{\prime }(t)}{\lvert t \rvert ^{p-2}}=l_{1}, \qquad \lim _{\lvert t \rvert \rightarrow \infty}\frac{g^{\prime }(t)}{\lvert t \rvert ^{q-2}}=l_{2}; $$

($A_{3}$):

there exists $\delta >0$ such that $0<(1+\delta )f(t)t\leqslant f^{\prime }(t)t^{2}$ for every $t\in \mathbb{R}$ and similarly for g;

($A_{4}$):

for every $\mu >0$, there exists $C_{\mu}>0$ such that

$$ \bigl\lvert f(u)v \bigr\rvert +\bigl\lvert g(v)u \bigr\rvert \leqslant \mu \bigl(u^{2}+v^{2}\bigr)+C_{\mu}\bigl(f(u)u+g(v)v\bigr),\quad u,v\in \mathbb{R}. $$

The main result of this paper is stated as follows:

Theorem 1.1

Suppose ($A_{0}$), ($A_{1}$)–($A_{4}$) are satisfied, $s\in (0,1)$, then for all small $\varepsilon >0$,

• (i) (Existence) the nonlocal system (1.2) admits a least energy solution $(\omega _{\varepsilon}, \xi _{\varepsilon})$;

• (ii) (Concentration) both functions $\omega _{\varepsilon}$ and $\xi _{\varepsilon}$ attain their maximum value at some unique and common point $z_{\varepsilon}\in \mathbb{R}^{N}$ such that

  $$ \lim_{\varepsilon \rightarrow 0}a(z_{\varepsilon})=a(0)=\min _{x\in \mathbb{R}^{N}}a(x); $$

• (iii) (Decay estimates) there exist constants $0< C_{1}< C_{2}$ and large $R>0$ such that

  $$ \frac{C_{1}\varepsilon ^{N+2s}}{\lvert x-z_{\varepsilon}\rvert ^{N+2s}} \leqslant \omega _{\varepsilon}(x), \qquad \xi _{\varepsilon}(x) \leqslant \frac{C_{1}\varepsilon ^{N+2s}}{\lvert x-z_{\varepsilon}\rvert ^{N+2s}} $$

  for all $\lvert x \rvert \geqslant R$.

A typical example of functions verifying the assumption ($A_{1}$)–($A_{4}$) is given by $f(t)=l_{1} \lvert t \rvert ^{p-2}t$, $g(t)=l_{2} \lvert t \rvert ^{q-2}t$ with $l_{1},l_{2}>0$ and $p,q>2$ such that $\frac{1}{p}+\frac{1}{q}>\frac{N-2s}{N}$.

Remark 1.2

The difficulties in treating system (1.3) originate in at least five facts:

• (i) Although we have a variational problem, the functional $I_{\varepsilon}$ associated with (1.3) is strongly indefinite, compared to the single equation case and cooperative type systems, the quadratic part of the energy functional has no longer a positive sign, and so we have to recourse to the “Indefinite Functional Theorem” introduced by Benci and Rabinowitiz in [15], which is an extension of both the mountain-pass theorem and the saddle point theorem.

• (ii) There is a lack of compactness due to the fact that we are working in $\mathbb{R}^{N}$, in order to attain (PS) condition, we need to consider some truncated problems in Sect. 4.

• (iii) No uniqueness and non-degenerate results seem to be known for the autonomous system of (1.2), and thus, the Lyapunov-Schmidt reduction method can not be used.

• (iv) We employ the ideas in [16] and [17] to prove Theorem 1.1; however, our systems are nonlocal, a delicate analysis is needed to overcome the lack of localization. The proof is different from that of the classical case $s=1$.

• (v) Under the natural assumption on p and q, that is $p,q>2$ such that $\frac{1}{p}+\frac{1}{q}>\frac{N-2s}{N}$, which is more general than assuming that $2< p$, $q<2^{*}_{s}$, the associated functional $I_{\varepsilon}$ may not to be well defined in the space $H^{s}(\mathbb{R}^{N})\times H^{s}(\mathbb{R}^{N})$, because it may happen that say $p<2^{*}_{s}=\frac{2N}{N-2s}<q$. However, as explained in Sect. 5, we only have to prove Theorem 1.1 in the case of $2< p=q<2^{*}_{s}$.

In fact, given $n\in \mathbb{N}$, we can define the truncated functions,

$$ g_{n}(t)= \textstyle\begin{cases} g(t), & t\leqslant n, \\ A_{n}t^{p-1}+B_{n} , &t> n, \end{cases} $$

where the coefficients are chosen so that $g_{n}$ is $C^{1}$. Thus, in view of $(A_{2})$, we see that $A_{n}=(\frac{l_{2}}{p-1}+o(1))\cdot n^{q-p}$, $B_{n}=(\frac{l_{2}(p-q)}{(p-1)(q-1)}+o(1))\cdot n^{q-1}$. We show in Sect. 5 that the solutions $(u_{\varepsilon _{n}},v_{\varepsilon _{n}})$ of the corresponding system obtained using Theorem 1.1 applied to the truncated problem are such that $\|u_{\varepsilon _{n}}\|_{\infty}$, $\|v_{\varepsilon _{n}}\|_{\infty} \leqslant C$ for some $C>0$ independent of n, and therefore they solve the original problem (1.2) if n is taken sufficiently large. Thus, in Sects. 2–4, we assume that $2< p=q<2^{*}_{s}$.

This paper is organized as follows: In Sect. 2, we review certain notations related to the fractional Laplacian and describe the appropriate functional setting, including the definition of the equivalent problems. In order to study the concentration phenomenon of semiclassical states for system (1.3), Sect. 3 is devoted to studying the autonomous systems of (1.3). We show that (PS) condition holds for the energy functional associated with (1.3) at energy levels in a suitable range in Sect. 4.1; we discuss some auxiliary problems involving appropriate truncated functions in the place of $a(x)$ in Sect. 4.2. The proof of Theorem 1.1 is given in Sect. 4.3. In Sect. 5, we will show that the solutions are bounded in $L^{\infty}(\mathbb{R}^{N})$; for this, some Liouville-type theorems need to be established. Therefore, during the proof on the existence of weak solutions for (1.3), we may assume that $2< p=q<2^{*}_{s}$.

Notations

Here, we list some notations that will be used throughout the paper.

• We denote by $\mathbb{R}^{N+1}_{+}$ the upper half-space $\{(x,y):x\in \mathbb{R}^{N},y>0\}$.

• The letter z represents a variable in the $\mathbb{R}^{N+1}_{+}$. Also, it is written as $z=(x,y)$ with $x\in \mathbb{R}^{N}$ and $y\in \mathbb{R}^{+}$.

• For $k\in \mathbb{N}$, we denote by $B_{k}(x_{0}, r)$ the ball $\{x\in \mathbb{R}^{k}:\lvert x-x_{0}\rvert < r\}$ for each $x_{0}\in \mathbb{R}^{k}$ and $r>0$. $B^{+}_{N+1}(x_{0}, r):=B_{N+1}(x_{0}, r)\cap \mathbb{R}^{N+1}_{+}$.

• $C>0$ is a generic constant that may vary from line to line.

• For a function $U\in X^{s}(\mathbb{R}_{+}^{N+1})$, we denote its trace on $\mathbb{R}^{N}\times \{y=0\}$ as $u=\operatorname{Tr}(U)$.

2 Preliminaries

In this section, we first introduce some definitions and notations. We consider the fractional Sobolev space

$$ H^{s}\bigl(\mathbb{R}^{N}\bigr)=\biggl\{ u\in L^{2}\bigl(\mathbb{R}^{N}\bigr): \Vert u \Vert _{H^{s}}:=\biggl( \int _{\mathbb{R}^{N}}\lvert \xi \rvert ^{2s}\bigl\lvert \mathscr{F}u ( \xi )\bigr\rvert \,d\xi \biggr)^{\frac{1}{2}}< +\infty \biggr\} , $$

where $\mathscr{F}u$ denotes the Fourier transform of u, and the fractional Laplacian $(-\Delta )^{s}:H^{s}(\mathbb{R}^{N})\rightarrow H^{-s}(\mathbb{R}^{N})$ is defined to be given $u\in H^{s}(\mathbb{R}^{N})$,

$$ \widehat{(-\Delta )^{s}u(\xi )}=\lvert \xi \rvert ^{2s} \widehat{u(\xi )} \quad \text{for any } \xi \in \mathbb{R}^{N}. $$

When u is assumed, in addition, sufficiently regular, we obtain the direct representation

$$ (-\Delta )^{s}u(x)=C_{N,s} \int _{{\mathbb{R}^{N}}} \frac{u(x)-u(y)}{\lvert x-y\rvert ^{N+2s}}\,dy, $$

for a suitable constant $C_{N,s}$ and the integral is understood in a principal value sense.

The dual space $H^{-s}(\mathbb{R}^{N})$ is defined in the standard way, as well as the inverse operator $(-\Delta )^{-s}$.

It is standard that (1.2) is equivalent to, by letting $u(x)=\omega (\varepsilon x)$, $v(x)=\xi (\varepsilon x)$,

$$ \textstyle\begin{cases} (-\Delta )^{s} u+a(\varepsilon x)u=g(v), & \text{in } \mathbb{R}^{N}, \\ (-\Delta )^{s} v+a(\varepsilon x)v=f(u), & \text{in } \mathbb{R}^{N}, \end{cases} $$

(2.1)

we now consider the problem (2.1). Since the above definition of the fractional Laplacian allows integrating by parts in the proper spaces, a natural definition of the energy solution to the problem (2.1) is the following.

Definition 2.1

We say that $(u,v)\in H\times H$ (Here $H:=\{u\in H^{s}(\mathbb{R}^{N}):\int _{\mathbb{R}^{N}}a(\varepsilon x)u^{2}\,dx<+ \infty \}$) are the weak solutions of (2.1) if the identity

$$ \begin{aligned} & \int _{\mathbb{R}^{N}}\bigl((-\Delta )^{\frac{s}{2}}u(-\Delta )^{ \frac{s}{2}}\psi +(-\Delta )^{\frac{s}{2}}\varphi (-\Delta )^{ \frac{s}{2}}v\bigr)\,dx \\ & \quad {}+ \int _{\mathbb{R}^{N}}a(\varepsilon x) (u\psi +\varphi v)\,dx= \int _{ \mathbb{R}^{N}}\bigl(f(u)\varphi +g(v)\psi \bigr)\,dx \end{aligned} $$

holds for every functions $\psi ,\varphi \in H$.

Associated to the problem (2.1), we consider the energy functionals

$$ I_{\varepsilon}(u,v)= \int _{\mathbb{R}^{N}}\bigl[(-\Delta )^{\frac{s}{2}}u(- \Delta )^{\frac{s}{2}}v+a(\varepsilon x)uv\bigr]\,dx- \int _{\mathbb{R}^{N}}F(u)\,dx- \int _{\mathbb{R}^{N}}G(v)\,dx, $$

where $F(t):=\int _{0}^{t} f(\xi )\,d\xi$, $G(t):=\int _{0}^{t} g(\xi )\,d\xi$. These functionals are well defined in $H\times H$ when $2< p=q<2^{*}_{s}$, and moreover, the critical points of $I_{\varepsilon}$ correspond to the weak solutions of (2.1).

We now include the main ingredients of a recently developed technique by Caffarelli and Silverstre [2]. Let u be a regular function in $\mathbb{R}^{N}$, we say that $U=E_{s}(u)$ is the s-harmonic extension of u to the upper half-space $\mathbb{R}_{+}^{N+1}$, if U is a solution to the problem

$$ \textstyle\begin{cases} \operatorname{div}(y^{1-2s}\nabla U)=0, & \text{in } \mathbb{R}_{+}^{N+1}, \\ U(x,0)=u, & \text{on } \mathbb{R}^{N}. \end{cases} $$

(2.2)

In [8] it is proved that

$$ \lim_{y\rightarrow 0^{+}}y^{1-2s} \frac{\partial U}{\partial y}(x,y)=-k_{s}^{-1}(- \Delta )^{s} u(x), $$

(2.3)

where $k_{s}=\frac{2^{1-2s}\Gamma (1-s)}{\Gamma (s)}$. Observe that $k_{s}=1$ for $s=\frac{1}{2}$ and $k_{s}\sim \frac{1}{2-2s}$ as $s\rightarrow 1^{-}$. Identity (2.3) allows the formulation of nonlocal problems involving the fractional powers of the Laplacian in $\mathbb{R}^{N}$ as local problems in divergence form in the half-space $\mathbb{R}_{+}^{N+1}$.

Remarking (2.2), we introduce the function space $X^{s}(\mathbb{R}_{+}^{N+1})$ that is defined as the completion of $C_{0}^{\infty}(\overline{\mathbb{R}_{+}^{N+1}})$ with respect to the norm

$$ \Vert U \Vert _{X^{s}}=\biggl(k_{s} \int _{\mathbb{R}_{+}^{N+1}}y^{1-2s}\lvert \nabla U\rvert ^{2}\,dz\biggr)^{\frac{1}{2}}. $$

Then it is a Hilbert space endowed with the inner product

$$ \langle U,V\rangle =k_{s} \int _{\mathbb{R}_{+}^{N+1}}y^{1-2s}\langle \nabla U,\nabla V\rangle \,dz, \quad \text{for } U,V\in X^{s}\bigl(\mathbb{R}_{+}^{N+1} \bigr). $$

With the constant $k_{s}$, we have the extension operator to be an isometry between $H^{s}$ $(\mathbb{R}^{N})$ and $X^{s}(\mathbb{R}_{+}^{N+1})$; that is

$$ \Vert U \Vert _{X^{s}}^{2}= \Vert u \Vert _{H^{s}}^{2}= \bigl\Vert (-\Delta )^{\frac{s}{2}}u \bigr\Vert _{2}^{2}. $$

On the other hand, for a function $U\in X^{s}(\mathbb{R}_{+}^{N+1})$, we will denote its trace on $\mathbb{R}^{N}\times \{y=0\}$ as $\operatorname{Tr}(U)$. This trace operator is also well defined, and it satisfies

$$ \bigl\Vert \operatorname{Tr}(U) \bigr\Vert _{H^{s}}\leqslant \Vert U \Vert _{X^{s}}. $$

For convenience, we will use the following notation:

$$ \begin{aligned} &L_{s}w: = -\operatorname{div} \bigl(y^{1-2s}\nabla w\bigr), \\ &\partial _{\nu}w: = -k_{s}\biggl(\lim _{y\rightarrow 0^{+}}y^{1-2s} \frac{\partial w}{\partial y}(x,y)\biggr), \quad \text{for } x\in \mathbb{R}^{N}. \end{aligned} $$

With the above extension, we can reformulate our problem (2.1) as

$$ \textstyle\begin{cases} L_{s}U=L_{s}V=0 & \text{in } \mathbb{R}_{+}^{N+1}, \\ \partial _{\nu}U=g(V)-a(\varepsilon x)U & \text{in } \mathbb{R}^{N} \times \{y=0\}, \\ \partial _{\nu}V=f(U)-a(\varepsilon x)V & \text{in } \mathbb{R}^{N} \times \{y=0\}, \\ U=u,\qquad V=v & \text{on } \mathbb{R}^{N}\times \{y=0\}. \end{cases} $$

(2.4)

The energy solutions to (2.4) are functions $(U,V)\in X\times X$ such that

$$ \begin{aligned} & k_{s} \int _{\mathbb{R}_{+}^{N+1}}y^{1-2s}\bigl(\langle \nabla U,\nabla \Psi \rangle +\langle \nabla \Phi ,\nabla V\rangle \bigr)\,dz \\ & \quad {}+ \int _{\mathbb{R}^{N}\times \{y=0\}}a(\varepsilon x) (U\Psi + \Phi V)\,dx= \int _{\mathbb{R}^{N}\times \{y=0\}}\bigl(f(U)\Phi +g(V)\Psi \bigr)\,dx, \end{aligned} $$

for any $\Phi ,\Psi \in X$, here $X:=\{U\in X^{s}(\mathbb{R}_{+}^{N+1}):\int _{\mathbb{R}^{N}\times \{y=0 \}}a(\varepsilon x)U^{2}\,dx<+\infty \}$. For any energy solutions $(U,V)\in X\times X$ to this problem, the functions $(u,v)=(U(x,0),V(x,0))$, defined in the sense of traces, belong to the space $H\times H$ and are the energy solutions to the problem (2.1). The converse is also true. Therefore, both formulations are equivalent.

The associated energy functionals $J_{\varepsilon}:X\times X\rightarrow \mathbb{R}^{1}$ to (2.4) are given by

$$ \begin{aligned} J_{\varepsilon}(U,V)={}&k_{s} \int _{\mathbb{R}_{+}^{N+1}} y^{1-2s} \langle \nabla U,\nabla V \rangle \,dz+ \int _{\mathbb{R}^{N}\times \{y=0 \}}a(\varepsilon x)UV\,dx \\ & {}- \int _{\mathbb{R}^{N}\times \{y=0\}}F(U)\,dx- \int _{\mathbb{R}^{N} \times \{y=0\}}G(V)\,dx. \end{aligned} $$

(2.5)

They are the $C^{2}$ functionals well defined over the Hilbert space $E:=X\times X$,

$$\begin{aligned}& \bigl\Vert (U,V) \bigr\Vert _{E}^{2}= \Vert U \Vert _{X}^{2}+ \Vert V \Vert _{X}^{2}, \\& \Vert U \Vert _{X}^{2}=k_{s} \int _{\mathbb{R}_{+}^{N+1}}y^{1-2s}\lvert \nabla U \rvert ^{2}\,dz+ \int _{\mathbb{R}^{N}\times \{y=0\}}a(\varepsilon x)U^{2}\,dx. \end{aligned}$$

Clearly, the critical points of $J_{\varepsilon}$ in E correspond to the ones of $I_{\varepsilon}$ in $H\times H$.

Remark 2.2

In the sequel, in view of the above equivalence, we will see both formulations of the problem (2.1), in $\mathbb{R}^{N}$ or in $\mathbb{R}_{+}^{N+1}$, whenever we may take some advantages. In particular, we will use the extension version when dealing with the fractional operator acting on products of functions since it is not clear how to calculate this action.

In Sect. 5, we will utilize the following Sobolev inequality on weighted spaces appeared in Theorem 1.3 of [18].

Proposition 2.3

Let Ω be an open bounded set in $\mathbb{R}_{+}^{N+1}$. Then there exists a constant $C=C(N,s,\Omega )>0$ such that

$$ \biggl( \int _{\Omega}y^{1-2s}\bigl\lvert U(x,y)\bigr\rvert ^{\frac{2(N+1)}{N}}\,dx\,dy\biggr)^{ \frac{N}{2(N+1)}}\leq C\biggl( \int _{\Omega}y^{1-2s}\bigl\lvert \nabla U(x,y) \bigr\rvert ^{2}\,dx\,dy\biggr)^{\frac{1}{2}} $$

holds for any function U whose support is contained in Ω whenever the right-hand side is well-defined.

It can be observed that the following orthogonal splitting holds $E=E^{-}\oplus E^{+}$, where $E^{\pm}:=\{(\Phi ,\pm \Phi ):\Phi \in X\}$ (Since for any $(U,V)\in E$, $(U,V)=(\frac{U+V}{2}, \frac{U+V}{2})+(\frac{U-V}{2}, \frac{V-U}{2})$). So that, denoting by Q the quadratic term of the energy functional $J_{\varepsilon}$, namely

$$ Q(U,V)=k_{s} \int _{\mathbb{R}_{+}^{N+1}} y^{1-2s}\langle \nabla U, \nabla V \rangle \,dz+ \int _{\mathbb{R}^{N}\times \{y=0\}}a(\varepsilon x)UV\,dx. $$

We have that Q is positive definite (respectively, negative definite) in $E^{+}$ (respectively, in $E^{-}$). Therefore, $J_{\varepsilon}$ are indefinite functionals; we refer to “Indefinite functional theorem” in [15] to obtain nontrivial critical points of $J_{\varepsilon}$.

What as follows, we recall that the definitions of the relative Morse index and solutions having finite index.

Let E be a real Hilbert space; for a closed subspace of $V\subset E$, we denote by $P_{V}$ the orthogonal projection onto V and by $V^{\perp}$ the orthogonal complement of V. Following [19] and [20], we say that the closed subspaces V, W of E are commensurable if $P_{V^{\perp}}P_{W}$ and $P_{W^{\perp}}P_{V}$ are compact operators.

If V and W are commensurable, the relative dimension of W with respect to V is defined as

$$ \dim_{V}W=\dim\bigl(W\cap V^{\perp}\bigr)-\dim \bigl(W^{\perp}\cap V\bigr). $$

Commensurability guarantees that both terms in the above formula are finite.

Definition 2.4

The relative Morse index of a critical point $(U,V)$ of a functional J with respect to the splitting $E=E^{+}\oplus E^{-}$ can be defined as the integer

$$ m(U,V)=\dim_{E^{-}}\bigl[ \text{negative eigenspace of } J^{\prime \prime }(U,V)\bigr]. $$

We will also borrow the definition of solutions having a finite index as defined in [21].

Definition 2.5

Let $(U,V)$ be a weak solution of (2.4), we say that $m(U,V)<+\infty$ if there exists $R_{0}>0$ with the property that for every $\phi \in C_{0}^{\infty}(\mathbb{R}^{N+1}_{+})$ such that $\phi =1$ in $B^{+}_{N+1}(0,2R_{0})\setminus B^{+}_{N+1}(0,R_{0})$ and $\operatorname{Supp}\phi \subset B^{+}_{N+1}(0,3.5R_{0})\setminus B^{+}_{N+1}(0,0.5R_{0})$, it holds that

$$ \begin{aligned} J^{\prime \prime }_{\varepsilon}(U,V) (\phi , \phi ) (\phi ,\phi )={}&2 \Vert \phi \Vert _{X}^{2}- \int _{\mathbb{R}^{N}\times \{y=0\}}f{'}(U)\phi ^{2}(x,0) \,dx \\ & {}- \int _{\mathbb{R}^{N}\times \{y=0\}}g{'}(V)\phi ^{2}(x,0) \,dx \geqslant 0. \end{aligned} $$

(2.6)

3 The autonomous system

In order to investigate the semiclassical states and their concentration phenomenon of the noncooperative type system (2.1), we firstly give some results on the autonomous system as follows:

$$ \textstyle\begin{cases} (-\Delta )^{s} u+\lambda u=g(v), & \text{in } \mathbb{R}^{N}, \\ (-\Delta )^{s} v+\lambda v=f(u), & \text{in } \mathbb{R}^{N}, \end{cases} $$

(3.1)

where $\lambda >0$ is any constant.

Theorem 3.1

Assume that f, g satisfy ($A_{1}$)–($A_{4}$), $s\in (0,1)$, $N>2s$, then the autonomous problem (3.1) has at least one positive solution $(u,v)\in H^{s}$ $(\mathbb{R}^{N})\times H^{s}(\mathbb{R}^{N})$, which, indeed, is a least energy solution.

In view of hypothesis $2< p=q<2^{*}_{s}$, we work with the space $E:=X^{s}(\mathbb{R}_{+}^{N+1})\times X^{s}(\mathbb{R}_{+}^{N+1})$. So, we consider the functional $J_{\lambda}:E\rightarrow \mathbb{R}^{1}$ defined by

$$ \begin{aligned} J_{\lambda}(U,V)={}&k_{s} \int _{\mathbb{R}_{+}^{N+1}} y^{1-2s} \langle \nabla U,\nabla V \rangle \,dx\,dy+\lambda \int _{\mathbb{R}^{N} \times \{0\}}UV\,dx \\ & {}- \int _{\mathbb{R}^{N}\times \{0\}}F(U)\,dx - \int _{\mathbb{R}^{N} \times \{0\}}G(V)\,dx, \end{aligned} $$

$J_{\lambda}$ is a $C^{2}$ functional and

$$ \begin{aligned} J_{\lambda}^{\prime }(U,V) (\Phi , \Psi )={}&k_{s} \int _{\mathbb{R}_{+}^{N+1}} y^{1-2s}\bigl[\langle \nabla U,\nabla \Psi \rangle +\langle \nabla \Phi , \nabla V\rangle \bigr] \,dx\,dy \\ & {}+\lambda \int _{\mathbb{R}^{N}\times \{0\}}[U\Psi +\Phi V]\,dx- \int _{ \mathbb{R}^{N}\times \{0\}}f(U)\Phi \,dx \\ & {}- \int _{\mathbb{R}^{N}\times \{0\}}g(V)\Psi \,dx, \end{aligned} $$

for any $\Psi ,\Phi \in X^{s}(\mathbb{R}_{+}^{N+1})$. So, the critical points of $J_{\lambda}$ satisfy the equations

$$ k_{s} \int _{\mathbb{R}_{+}^{N+1}} y^{1-2s}\langle \nabla U,\nabla \Psi \rangle \,dx\,dy+\lambda \int _{\mathbb{R}^{N}\times \{0\}}U\Psi \,dx - \int _{\mathbb{R}^{N}\times \{0\}} g(V)\Psi \,dx=0, $$

(3.2)

and

$$ k_{s} \int _{\mathbb{R}_{+}^{N+1}} y^{1-2s}\langle \nabla \Phi , \nabla V \rangle \,dx\,dy+\lambda \int _{\mathbb{R}^{N}\times \{0\}}\Phi V\,dx - \int _{\mathbb{R}^{N}\times \{0\}}f(U)\Phi \,dx=0, $$

(3.3)

for any $\Psi ,\Phi \in X^{s}(\mathbb{R}_{+}^{N+1})$. Equations (3.2)–(3.3) are the weak formulation of (3.1).

The following Lemma will play a significant role in the sequel, whose proof is similar to [22], Lemma 2.1, so we omit it.

Lemma 3.2

Let $(U_{n},V_{n})$ be a $(\mathit{PS})_{c}$ sequence for the functional $J_{\lambda}$, namely

$$\begin{aligned}& J_{\lambda}(U_{n},V_{n})\rightarrow c\in \mathbb{R}^{+}, \\& \mu _{n}:=\sup \bigl\{ \bigl\lvert J_{\lambda}^{\prime }(U_{n},V_{n}) (\Phi ,\Psi ) \bigr\rvert , \Phi ,\Psi \in X^{s}\bigl( \mathbb{R}_{+}^{N+1}\bigr), \Vert \Phi \Vert _{X^{s}}+ \Vert \Psi \Vert _{X^{s}}\leqslant 1\bigr\} \rightarrow 0. \end{aligned}$$

Then $(U_{n},V_{n})$ is bounded in E and

$$ \sup_{E^{-}\oplus \mathbb{R}^{+}(U_{n},V_{n})}J_{\lambda}=J_{\lambda}(U_{n},V_{n})+O \bigl( \mu _{n}^{2}\bigr). $$

Pproof of Theorem 3.1

By the assumptions ($A_{1}$)–($A_{3}$), it is easy to check that the energy function $J_{\lambda}$ possesses the linking structure; that is, $J_{\lambda}\leqslant 0$ in $E^{-}$, $J_{\lambda}\geqslant \rho >0$ in $E^{+}\cap \partial B^{+}_{N+1}(0,r)$, for some small $r>0$, $\rho >0$; moreover, if $r>0$ is sufficiently large and $e=(e_{1},e_{2})\in E$, $e_{1}>0$, $e_{2}>0$, then

$$ \sup_{(E^{-}\oplus \mathbb{R}^{+}e)\cap \partial B^{+}_{N+1}(0,r)}J_{ \lambda}\leqslant 0. $$

Then, according to “Indefinite functional theorem” [15], $J_{\lambda}$ has a $(\mathit{PS})_{c}$ sequence $\{(U_{n},V_{n})\}\subset E$, where $0<\rho \leqslant c\leqslant \sup_{E^{-}\oplus \mathbb{R}^{+}e}J_{ \lambda}$, using Lemma 3.2, $(U_{n},V_{n})$ are bounded in E and may assume $(U_{n},V_{n})\rightharpoonup (U,V)$ as $n\rightarrow \infty$, then clearly $J_{\lambda}^{\prime }(U,V)=0$. Next we need to show that there exists a non-trivial critical point. For this purpose, by concentration compact principle [17], it is possible to find a sequence $\{x_{n}\}\subset \mathbb{R}^{N}$ and some constants $R>0$ and $\beta >0$ such that

$$ \int _{B_{N}(x_{n},R)}u_{n}^{2}\,dx>\beta ,\qquad \int _{B_{N}(x_{n},R)}v_{n}^{2}\,dx> \beta ,\quad \text{for any } n\in \mathbb{N}. $$

Indeed, assuming the contrary, we have

$$ u_{n}(x)\rightarrow 0,\qquad v_{n}(x)\rightarrow 0\quad \text{in } L^{p}\bigl( \mathbb{R}^{N}\bigr)\ \bigl(2\leqslant p< 2^{*}_{s}\bigr). $$

But then, for large n and some constants $a>0$ and $C_{1},C_{2}>0$, we have

$$ \begin{aligned} 2c+o(1)&= 2J_{\lambda}(U_{n},V_{n})-J_{\lambda}^{\prime }(U_{n},V_{n}) (U_{n},V_{n}) \\ &= \int _{\mathbb{R}^{N}}\bigl[f(u_{n})u_{n}-2F(u_{n})+g(v_{n})v_{n}-2G(v_{n}) \bigr]\,dx \\ &\leqslant \int _{\mathbb{R}^{N}}C_{1}\bigl(\lvert u_{n} \rvert ^{2}+ \lvert v_{n}\rvert ^{2} \bigr)+C_{2}\bigl(\lvert u_{n}\rvert ^{p}+ \lvert v_{n} \rvert ^{p}\bigr)\,dx \end{aligned} $$

proving a contradiction, since $c>0$.

Now we define $\widetilde{U_{n}}(x,y)=U_{n}(x+x_{n},y)$, $\widetilde{V_{n}}(x,y)=V_{n}(x+x_{n},y)$, then $(\widetilde{U_{n}},\widetilde{V_{n}})\rightharpoonup (U_{0},V_{0}) \neq (0,0)$ is a non-trivial critical point of $J_{\lambda}$.

Let $c(\lambda )=\inf \{J_{\lambda}(U,V):(U,V)\neq (0,0),J_{\lambda}^{\prime }(U,V)=0 \}$, using the standard arguments, the infimum is actually a minimum, and it follows that $J_{\lambda}$ admits a ground state critical level $c(\lambda )$. Follow the proof of Lemma 3.1 in [23], we have that the map $\lambda \rightarrow c(\lambda )$ is continuous and increasing, and $\lim_{\lambda \rightarrow +\infty}c(\lambda )=+\infty$. This completes the proof of Theorem 3.1. □

4 The noncooperative singularly perturbed system

4.1 For the original problem (2.1)

Now we temporarily come back to our original problem (2.1). Note that using the similar arguments as Sect. 3, $J_{\varepsilon}$ also possess the linking structure, and thus there exist $(\mathit{PS})_{c_{\varepsilon}}$ sequences with

$$ 0< c_{\varepsilon}\leqslant \sup_{E^{-}\oplus \mathbb{R}^{+}e}J_{\varepsilon}, \quad e=(e_{1},e_{2})\in E, e_{1}>0,e_{2}>0. $$

(4.1)

On the other hand, according to the assumption ($A_{0}$), we may fix $\overline{a}\in \mathbb{R}$ such that

$$ 0< a(0)=\min_{x\in \mathbb{R}^{N}}a(x)< \overline{a}< \liminf _{\lvert x \rvert \rightarrow \infty}a(x). $$

We denote by $J_{0}$ and $c_{0}$ the energy functional defined in (2.5) with $a(0)$ in place of $a(\varepsilon x)$ and least energy, respectively. Let $(U_{0},V_{0})$ be a ground-state for $J_{0}$, it is easy to check that

$$ J_{\varepsilon}(U_{0},V_{0})=J_{0}(U_{0},V_{0})+o_{\varepsilon}(1)=c_{0}+o(1), $$

(4.2)

and

$$ J^{\prime }_{\varepsilon}(U_{0},V_{0}) ( \Phi ,\Psi )=J^{\prime }_{0}(U_{0},V_{0}) ( \Phi ,\Psi )+o(1), $$

uniformly for bounded Φ, $\Psi \in X$. Applying Theorem 3.1 and Lemma 3.2 to the functionals $J_{\varepsilon}$, we deduce that

$$ \sup_{E^{-}\oplus \mathbb{R}^{+}(U_{0},V_{0})}J_{\varepsilon}=J_{\varepsilon}(U_{0},V_{0})=c_{0}+o(1)< c( \overline{a})+o(1), $$

which complies with (4.1) to conclude that $0< c_{\varepsilon}< c(\overline{a})$.

4.2 Some auxiliary problems

To apply “Indefinite functional theorem” to get the positive solutions of (2.1), we only need to show that $(\mathit{PS})_{c_{\varepsilon}}$ condition holds for $0< c_{\varepsilon}< c(\overline{a})$. For this purpose, in this subsection, we will study some auxiliary problems and establish several auxiliary results employing the ideas from [16].

Consider the nonlocal system as follows:

$$ \textstyle\begin{cases} (-\Delta )^{s}u+b(x)u=g(v), & \text{in } \mathbb{R}^{N}, \\ (-\Delta )^{s}v+b(x)v=f(u), & \text{in } \mathbb{R}^{N}, \end{cases} $$

(4.3)

where $b(x)\in C(\mathbb{R}^{N})$, $b(x)\geqslant \overline{b}>0$ for any $x\in \mathbb{R}^{N}$ and $\lim_{\lvert x \rvert \rightarrow \infty}b(x)=b_{\infty}\in \mathbb{R}$.

The associated energy functional to the extension problem of (4.3) is defined by

$$ \begin{aligned} J_{b}(U,V)={}&k_{s} \int _{\mathbb{R}_{+}^{N+1}}y^{1-2s}\langle \nabla U, \nabla V\rangle \,dz+ \int _{\mathbb{R}^{N}\times \{y=0\}}b(x)UV\,dx \\ & {}- \int _{\mathbb{R}^{N}\times \{y=0\}}\bigl(F(U)+G(V)\bigr)\,dx. \end{aligned} $$

We denote by $J_{\infty}$ the corresponding functional with $b_{\infty}$ in place of $b(x)$. Of course, here we work in the space $E:=X\times X$, $X=\{U\in X^{s}(\mathbb{R}_{+}^{N+1}): \int _{ \mathbb{R}^{N}\times \{y=0\}}b(x)U^{2}\,dx<+\infty \}$.

Now we prove the following Lemma.

Lemma 4.1

Under the assumptions ($A_{1}$)–($A_{4}$), the $(\mathit{PS})$ condition holds for $J_{b}$ at critical level $0< c< c(b_{\infty})$. Moreover, $J_{b}$ admits a ground-state critical level $c_{b}$ and $c_{b}\geqslant c(\overline{b})$.

Proof

Let $(U_{n},V_{n})$ be such that $J_{b}(U_{n},V_{n})\rightarrow c\in (0,c(b_{\infty}))$ and $J^{\prime }_{b}(U_{n},V_{n})\rightarrow 0$ as $n\rightarrow \infty$. It follows from Lemma 3.2 that $(U_{n},V_{n})$ is bounded in E and assumes that $(U_{n},V_{n})\rightharpoonup (U,V)$ in E, clearly, $J_{b}^{\prime }(U,V)=0$. In particular,

$$ \begin{aligned} 2J_{b}(U,V)&=2J_{b}(U,V)-J^{\prime }_{b}(U,V) (U,V) \\ &= \int _{\mathbb{R}^{N}}\bigl(f(u)u-2F(u)+g(v)v-2G(v)\bigr)\,dx\geqslant 0. \end{aligned} $$

(4.4)

Putting $\overline{U}_{n}=U_{n}-U$, $\overline{V}_{n}=V_{n}-V$, we next show that $\overline{U}_{n}\rightarrow 0$, $\overline{V}_{n}\rightarrow 0$ in X.

Indeed, one has

$$ \begin{aligned} J_{b}(\overline{U}_{n}, \overline{V}_{n})={}&k_{s} \int _{\mathbb{R}_{+}^{N+1}}y^{1-2s} \bigl\langle \nabla (U_{n}-U),\nabla (V_{n}-V)\bigr\rangle \,dz \\ &{}+ \int _{\mathbb{R}^{N}\times \{y=0\}}b(x) (U_{n}-U) (V_{n}-V) \,dx \\ & {}- \int _{\mathbb{R}^{N}\times \{y=0\}}\bigl(F(U_{n}-U)+G(V_{n}-V) \bigr)\,dx \\ ={}&J_{b}(U_{n},V_{n})-J_{b}(U,V)+ \int _{\mathbb{R}^{N}\times \{y=0\}}\bigl(F(U_{n})-F(U)-F(U_{n}-U) \\ &{}+ G(V_{n})-G(V)-G(V_{n}-V)\bigr)\,dx, \end{aligned} $$

(4.5)

and

$$\begin{aligned}& J^{\prime }_{b}( \overline{U}_{n},\overline{V}_{n}) (\Phi ,\Psi ) \\& \quad = k_{s} \int _{\mathbb{R}_{+}^{N+1}}y^{1-2s}\bigl[\bigl\langle \nabla \Phi , \nabla (V_{n}-V)\bigr\rangle +\bigl\langle \nabla (U_{n}-U),\nabla \Psi \bigr\rangle \bigr] \,dz \\& \qquad {} + \int _{\mathbb{R}^{N}\times \{y=0\}}b(x)\bigl[\Phi (x,0) (V_{n}-V)+ \Psi (x,0) (U_{n}-U)\bigr]\,dx \\& \qquad {} - \int _{\mathbb{R}^{N}\times \{y=0\}}\bigl[f(U_{n}-U)\Phi (x,0)+g(V_{n}-V) \Psi (x,0)\bigr]\,dx \\& \quad = J^{\prime }_{b}(U_{n},V_{n}) ( \Phi ,\Psi )-J^{\prime }_{b}(U,V) (\Phi ,\Psi ) \\& \qquad {} + \int _{\mathbb{R}^{N}}\bigl[\bigl(f(u_{n}-u)-f(u_{n})+f(u) \bigr)\Phi (x,0) \\& \qquad {} +\bigl(g(v_{n}-v)-g(v_{n})+g(v)\bigr)\Psi (x,0) \bigr]\,dx, \end{aligned}$$

(4.6)

for any bounded functions $\Phi ,\Psi \in X$.

Now we compute the third term of (4.5) and (4.6), respectively,

$$\begin{aligned}& \biggl\lvert \int _{\mathbb{R}^{N}}\bigl(F(u_{n})-F(u)-F(u_{n}-u)+G(v_{n})-G(v)-G(v_{n}-v) \bigr)\,dx \biggr\rvert \\& \quad = \int _{\lvert x \rvert \leqslant R}+ \int _{\lvert x \rvert >R} \\& \quad = o_{n}(1)+C \int _{\lvert x \rvert >R}\bigl(\lvert u_{n} \rvert ^{2}+ \lvert u_{n} \rvert ^{p}+\lvert u \rvert ^{2}+\lvert u \rvert ^{p} \\& \qquad {} +\lvert v_{n}\rvert ^{2}+\lvert v_{n} \rvert ^{p}+\lvert v\rvert ^{2}+ \lvert v \rvert ^{p}\bigr)\,dx \\& \quad \leqslant o_{n}(1)+o_{R}(1). \end{aligned}$$

Similarly,

$$ \begin{aligned} & \int _{\mathbb{R}^{N}}\bigl[\bigl(f(u_{n}-u)-f(u_{n})+f(u) \bigr)\Phi (x,0) \\ & \qquad {}+\bigl(g(v_{n}-v)-g(v_{n})+g(v)\bigr)\Psi (x,0) \bigr]\,dx \\ &\quad \leqslant o_{n}(1)+o_{R}(1), \end{aligned} $$

uniformly for any bounded Φ and Ψ in X.

Combining these estimates with (4.4)–(4.6) gives

$$ J_{b}(\overline{U}_{n},\overline{V}_{n})=J_{b}(U_{n},V_{n})-J_{b}(U,V)+o(1) \leqslant c+o(1), $$

(4.7)

and

$$ \sup \bigl\{ \bigl\lvert J_{b}^{\prime }( \overline{U}_{n},\overline{V}_{n}) (\Phi , \Psi )\bigr\rvert , \Vert \Phi \Vert _{X}+ \Vert \Psi \Vert _{X}\leqslant 1\bigr\} =o(1). $$

(4.8)

Since $\overline{U}_{n}\rightharpoonup 0$, $\overline{V}_{n}\rightharpoonup 0$ in X and $b_{\infty}=\lim_{\lvert x \rvert \rightarrow \infty}b(x)$, a similar conclusion as (4.7) and (4.8) holds for $J_{\infty}(\overline{U}_{n},\overline{V}_{n})$ and $J^{\prime }_{\infty}(\overline{U}_{n},\overline{V}_{n})$; if $\liminf_{n\rightarrow \infty}$ $J_{\infty}(\overline{U}_{n},\overline{V}_{n})>0$, we deduce from Lemma 3.2 that

$$ c(b_{\infty})\leqslant \sup_{E^{-}\oplus \mathbb{R}^{+}(\overline{U}_{n}, \overline{V}_{n})}J_{\infty}=J_{\infty}(\overline{U}_{n},\overline{V}_{n}) \leqslant c+o(1), $$

which contradicts with the assumption $c< c(b_{\infty})$. Consequently,

$$ \liminf_{n\rightarrow \infty}J_{\infty}(\overline{U}_{n}, \overline{V}_{n})\leqslant 0. $$

This implies that

$$ \begin{aligned} &\liminf_{n\rightarrow \infty} \bigl[2J_{\infty}(\overline{U}_{n}, \overline{V}_{n})-J^{\prime }_{\infty}(\overline{U}_{n},\overline{V}_{n}) ( \overline{U}_{n},\overline{V}_{n})\bigr] \\ &\quad = \liminf_{n\rightarrow \infty} \int _{\mathbb{R}^{N}\times \{y=0\}}\bigl[f(\overline{U}_{n}) \overline{U}_{n}-2F(\overline{U}_{n})+g( \overline{V}_{n})\overline{V}_{n} \\ & \qquad {} -2G(\overline{V}_{n})\bigr]\,dx \\ &\quad \leqslant 0, \end{aligned} $$

and thus

$$ \liminf_{n\rightarrow \infty} \int _{\mathbb{R}^{N}\times \{y=0 \}}\bigl(F(\overline{U}_{n})+G( \overline{V}_{n})\bigr)\,dx\leqslant 0, $$

which turns to be

$$ \liminf_{n\rightarrow \infty}\bigl( \Vert \overline{U}_{n} \Vert _{X}+ \Vert \overline{V}_{n} \Vert _{X}\bigr)=0. $$

Therefore, $U_{n}\rightarrow U$, $V_{n}\rightarrow V$ as $n\rightarrow \infty$; that is the $(\mathit{PS})_{c}$ condition holds for $0< c< c(b_{\infty})$. Applying “Indefinite functional theorem”, we can derive that there exists a nontrivial critical point $(U,V)$ for the functional $J_{b}$.

We set

$$ c_{b}:=\inf \bigl\{ J_{b}(U,V),(U,V)\neq (0,0),J_{b}^{\prime }(U,V)=0\bigr\} . $$

By the standard arguments, the infimum is actually a minimum, and it follows that $J_{b}$ admits a ground-state critical level $c_{b}$.

Finally, we compare the least energy levels of $J_{b}$ with the ones of $J_{\bar{b}}$; this estimate is crucial for the proof of Theorem 1.1.

Assume that $c(\bar{b})>c_{b}$. For $t\in [0,1]$, let $b_{t}(x):=(1-t)b(x)+t\bar{b}$ and denote by $c'$, $c_{t}$ the corresponding ground-state level and linking level, respectively. Since $b_{t}(x)=b(x)+t(\bar{b}-b(x))\leqslant b(x)$, $c_{t}\leqslant c'\leqslant c$. Consequently, $c_{t}\leqslant c_{b}$. It follows from the assumption $c(\bar{b})>c_{b}$ and the fact $(1-t)b_{\infty}+t\bar{b}\geqslant \bar{b}$ and Lemma 3.2 that

$$ c_{t}\leqslant c_{b}< c(\bar{b})\leqslant c \bigl((1-t)b_{\infty}+t\bar{b}\bigr), $$

for every $t\in [0,1]$. This is a contradiction for $t=1$, since $c_{1}=c(\bar{b})$, which ends the proof. □

Next, utilizing Lemma 4.1, we prove an important auxiliary result that is directly applied to the proof of Theorem 1.1.

Lemma 4.2

Let $b(x)\geqslant \overline{b}>0$ and suppose $\{(U_{n},V_{n})\}$ is a $(\mathit{PS})_{c}$ sequence for $J_{b}$ with $\liminf_{n\rightarrow \infty}J_{b}(U_{n},V_{n})>0$, then $c(\overline{b})\leqslant J_{b}(U_{n},V_{n})+o(1)$.

Proof

By Lemma 4.1, we can choose M so large that

$$ \sup_{E^{-}\oplus \mathbb{R}^{+}e}J_{b}=J_{b}(U_{n},V_{n})+o(1)< c(M). $$

(4.9)

Define the truncated functions

$$ b_{n}(x)= \textstyle\begin{cases} b(x), & \lvert x\rvert \leqslant n, \\ M , &\lvert x\rvert > n, \end{cases} $$

then $b_{n}(x)\geqslant \overline{b}>0$ and $b_{n}(x)\rightarrow M$ as $\lvert x\rvert \rightarrow \infty$. Take n so large that

$$ J_{b_{n}}(U_{n},V_{n})=J_{b}(U_{n},V_{n})+o(1)=c+o(1), $$

(4.10)

and

$$ J^{\prime }_{b_{n}}(U_{n},V_{n}) (\Phi ,\Psi )=o(1), $$

(4.11)

uniformly for all bounded functions Φ and Ψ in X. From (4.9)–(4.11), we can employ Lemma 4.1 to get $J_{b_{n}}$ admitting a ground state critical level $c_{b_{n}}>0$ and

$$ c(\overline{b})\leqslant c_{b_{n}}\leqslant \sup_{E^{-}\oplus \mathbb{R}^{+}(U_{n},V_{n})}J_{b_{n}}=J_{b_{n}}(U_{n},V_{n})+o(1)=J_{b}(U_{n},V_{n})+o(1). $$

This completes the proof. □

4.3 The proof of Theorem 1.1

In this subsection, we prove Theorem 1.1 divided into three results. First of all, we give the existence result.

Theorem 1.1(i)

Assume that f, g satisfy ($A_{1}$)–($A_{4}$), then for all small $\varepsilon >0$, $J_{\varepsilon}$ have ground states $(U_{\varepsilon},V_{\varepsilon})\in E$ with critical values $0< c_{\varepsilon}< c(\overline{a})$ and $c_{\varepsilon}\rightarrow c_{0}$ as $\varepsilon \rightarrow 0$.

Proof

Recall that there exists a $(\mathit{PS})_{c_{\varepsilon}}$ sequence $\{(U_{n},V_{n})\}\subset E$ for $J_{\varepsilon}$, assume that $(U_{n},V_{n})\rightharpoonup (U,V)\in E$, and for sufficiently small $\varepsilon >0$, $0< c_{\varepsilon}< c(\overline{a})$. And there exists a constant $R_{0}>0$ such that for some fixed $\varepsilon >0$,

$$ a(\varepsilon x)\geqslant \overline{a} \quad \text{for all } \lvert x \rvert \geqslant R_{0}. $$

According to “Indefinite functional theorem” [15], we only need to show that for fixed $\varepsilon >0$, the $(\mathit{PS})_{c_{\varepsilon}}$ condition holds for $J_{\varepsilon}$ at critical levels $0< c_{\varepsilon}< c(\overline{a})$.

Introduce the following truncated function

$$ b(x)= \textstyle\begin{cases} a(\varepsilon x), & \lvert x\rvert \geqslant R_{0}, \\ \geqslant \overline{a} , & \lvert x\rvert < R_{0}, \end{cases} $$

then $b(x)\geqslant \overline {a}$ for all $x\in \mathbb{R}^{N}$. Remark that $\overline{U}_{n}=U_{n}-U$, $\overline{V}_{n}=V_{n}-V$ and $\overline{U}_{n}\rightharpoonup 0$, $\overline{V}_{n}\rightharpoonup 0$ in X.

Observe that

$$ \begin{aligned} \int _{\mathbb{R}^{N}\times \{y=0\}}b(x)\overline{U}_{n} \overline{V}_{n}\,dx&= \int _{\lvert x\rvert \geqslant R_{0}}+ \int _{ \lvert x\rvert < R_{0}} \\ &= \int _{\mathbb{R}^{N}\times \{y=0\}}a(\varepsilon x)\overline{U}_{n} \overline{V}_{n}\,dx+o_{n}(1). \end{aligned} $$

Using the similar arguments as Lemma 4.1, it is easy to check that

$$\begin{aligned}& J_{b}(\overline{U}_{n},\overline{V}_{n})= J_{\varepsilon}(\overline{U}_{n}, \overline{V}_{n})+o_{n}(1) \leqslant c_{\varepsilon}+o(1), \\& J^{\prime }_{b}(\overline{U}_{n}, \overline{V}_{n})=o(1). \end{aligned}$$

If $\liminf_{n\rightarrow \infty}J_{b}(\overline{U}_{n}, \overline{V}_{n})>0$, we can derive applying Lemma 4.2 that

$$ c(\overline{a})\leqslant J_{b}(\overline{U}_{n}, \overline{V}_{n}) \leqslant c_{\varepsilon}+o(1), $$

which is a contradiction. As a result, $\liminf_{n\rightarrow \infty}J_{b}(\overline{U}_{n}, \overline{V}_{n})\leqslant 0$. This gives $\overline{U}_{n}\rightarrow 0$, $\overline{V}_{n}\rightarrow 0$ in X.

Next, we show $c_{\varepsilon}\rightarrow c_{0}$ as $\varepsilon \rightarrow 0$.

By (4.2), we obtain that $\limsup_{\varepsilon \rightarrow 0}c_{\varepsilon}\leqslant c_{0}$. On the other hand, let $(U_{\varepsilon},V_{\varepsilon})$ be the ground states for $J_{\varepsilon}$, then $(U_{\varepsilon}, V_{\varepsilon})$ are bounded in E. In particular, we have

$$ J_{0}(U_{\varepsilon},V_{\varepsilon})=J_{\varepsilon}(U_{\varepsilon},V_{\varepsilon})+ \int _{\mathbb{R}^{N}\times \{y=0\}}\bigl(a(0)-a( \varepsilon x)\bigr)U_{\varepsilon}V_{\varepsilon}\,dx=c_{\varepsilon}+o(1), $$

and for any $\Phi ,\Psi \in X$,

$$ \begin{aligned} \bigl\lvert J_{0}^{\prime }(U_{\varepsilon},V_{\varepsilon}) (\Phi ,\Psi ) \bigr\rvert &= \int _{\mathbb{R}^{N}\times \{y=0\}}\bigl\lvert a(0)-a( \varepsilon x)\bigr\rvert \lvert U_{\varepsilon}\Phi +V_{\varepsilon}\Psi \rvert \,dx \\ &\leqslant o(1) \bigl( \Vert \Phi \Vert _{X}+ \Vert \Psi \Vert _{X}\bigr). \end{aligned} $$

By Lemma 3.2, we conclude that

$$ c_{0}\leqslant \sup_{E^{-}\oplus \mathbb{R}^{+}(U_{\varepsilon},V_{\varepsilon})}J_{0}=J_{0}(U_{\varepsilon},V_{\varepsilon})=c_{\varepsilon}+o(1). $$

Hence, $c_{0}\leqslant \liminf_{\varepsilon \rightarrow 0}c_{\varepsilon}$. This completes the proof. □

From now, we consider the positive functions $U_{\varepsilon}>0$, $V_{\varepsilon}>0$ given by Theorem 1.1(i), which satisfy $u_{\varepsilon}=U_{\varepsilon}(x,0)\in H$, $v_{\varepsilon}=V_{\varepsilon}(x,0)\in H$ and

$$ \textstyle\begin{cases} (-\Delta )^{s}u_{\varepsilon}+a(\varepsilon x)u_{\varepsilon}=g(v_{\varepsilon}), & \text{in } \mathbb{R}^{N}, \\ (-\Delta )^{s}v_{\varepsilon}+a(\varepsilon x)v_{\varepsilon}=f(u_{\varepsilon}), & \text{in } \mathbb{R}^{N}. \end{cases} $$

Next, we study the concentration behavior of this family of solution $(u_{\varepsilon},v_{\varepsilon})$ as $\varepsilon \rightarrow 0$.

Theorem 1.1(ii)

Suppose that $(u_{\varepsilon},v_{\varepsilon})$ is the ground states of problem (2.1) for all ε sufficiently small, then $(u_{\varepsilon},v_{\varepsilon})$ attain their maximum value at some unique and common points $x_{\varepsilon}\in \mathbb{R}^{N}$ such that

$$ \lim_{\varepsilon \rightarrow 0}a(\varepsilon x_{\varepsilon})=a(0)= \min _{x\in \mathbb{R}^{N}}a(x). $$

Proof

Since $u_{\varepsilon}>0$, $v_{\varepsilon}>0$, for each small $\varepsilon >0$, there exist $x_{\varepsilon}, y_{\varepsilon}\in \mathbb{R}^{N}$ such that, respectively,

$$ u_{\varepsilon}(\varepsilon x_{\varepsilon})=\max_{x\in \mathbb{R}^{N}}u_{\varepsilon}, \qquad v_{\varepsilon}(\varepsilon y_{\varepsilon})=\max _{y\in \mathbb{R}^{N}}v_{\varepsilon}. $$

We split the proof into several steps.

Step 1. $\{\varepsilon x_{\varepsilon}\}$ and $\{\varepsilon y_{\varepsilon}\}$ are bounded.

Suppose by contradiction that there exists a subsequence $\{\varepsilon _{j}x_{\varepsilon _{j}}\}$ such that $\lvert \varepsilon _{j}x_{\varepsilon _{j}}\rvert \rightarrow + \infty$ as $\varepsilon _{j}\rightarrow 0$. Define the functions $\widetilde{u}_{j}(x)=u_{\varepsilon _{j}}(x+x_{\varepsilon _{j}})$ and $\widetilde{v}_{j}(x)=v_{\varepsilon _{j}}(x+x_{\varepsilon _{j}})$. Observe that these functions satisfy the following system:

$$ \textstyle\begin{cases} (-\Delta )^{s}\widetilde{u}_{j}+a_{j}(x)\widetilde{u}_{j}=g( \widetilde{v}_{j}), \\ (-\Delta )^{s}\widetilde{v}_{j}+a_{j}(x)\widetilde{v}_{j}=f( \widetilde{u}_{j}), \end{cases} $$

here $a_{j}(x)=a(\varepsilon _{j}x+\varepsilon _{j}x_{\varepsilon _{j}})$. Denote by $I_{j}$ the corresponding energy functional,

$$ I_{j}(\widetilde{u}_{j},\widetilde{v}_{j}) = \int _{\mathbb{R}^{N}}(- \Delta )^{s}\widetilde{u}_{j} \widetilde{v}_{j}\,dx + \int _{\mathbb{R}^{N}}a_{j}(x) \widetilde{u}_{j} \widetilde{v}_{j}\,dx - \int _{\mathbb{R}^{N}}\bigl(F( \widetilde{u}_{j})+G( \widetilde{v}_{j})\bigr)\,dx. $$

(4.12)

From Lemma 3.2, the families $\{(\widetilde{u}_{j},\widetilde{v}_{j})\}$ are bounded in E, and let $(\widetilde{u}_{j},\widetilde{v}_{j})\rightharpoonup (u,v)\neq (0,0) \in E$. Recall from Theorem 1.1(i) that

$$ 0< \liminf_{j\rightarrow \infty}I_{j}(\widetilde{u}_{j}, \widetilde{v}_{j}) \leqslant \limsup_{j\rightarrow \infty}I_{j}( \widetilde{u}_{j},\widetilde{v}_{j})< c(\overline{a}). $$

(4.13)

Let $b_{j}(x):=\max \{\overline{a},a_{j}(x)\}$ and denote by $\overline{I}_{j}$ the corresponding energy functional defined as in (4.12) with $a_{j}(x)$ replaced by $b_{j}(x)$. By the assumption $\lvert \varepsilon _{j}x_{\varepsilon _{j}}\rvert \rightarrow + \infty$, it holds that $(\widetilde{u}_{j},\widetilde{v}_{j})$ is a $(\mathit{PS})$ sequence for $\overline{I}_{j}$ and

$$ \overline{I}_{j}(\widetilde{u}_{j}, \widetilde{v}_{j})=I_{j}( \widetilde{u}_{j}, \widetilde{v}_{j})+o_{\varepsilon _{j}}(1). $$

Since $b_{j}(x)\geqslant \overline{a}$ for every $x\in \mathbb{R}^{N}$, it follows then from Lemma 4.2 that

$$ c(\overline{a})\leqslant \overline{I}_{j}(\widetilde{u}_{j}, \widetilde{v}_{j})+o(1)=I_{j}(\widetilde{u}_{j} ,\widetilde{v}_{j})+o(1), $$

which contradicts with (4.13). Hence $\{\varepsilon x_{\varepsilon}\}$ is bounded, the same is $\{\varepsilon y_{\varepsilon}\}$.

Furthermore, we have that there is a subsequence $\{\widetilde{u}_{j_{n}}\}$ and $\{\widetilde{v}_{j_{n}}\}$ such that for any $\eta >0$, there exists $r_{\eta}>0$ satisfying

$$ \limsup_{n\rightarrow \infty} \int _{B_{N}(0, n)\setminus B_{N}(0,r)}\bigl( \widetilde{u}^{q}_{j_{n}}+ \widetilde{v}^{q}_{j_{n}}\bigr)\,dx\leqslant \eta $$

for all $r\geqslant r_{\eta}$ (see an argument to [24, Lemma 5.7]). Here $q\in [2, 2^{*}_{s})$, which implies together with the assumptions $(A_{1})(A_{2})$ that for any $\eta >0$, there exists $r_{\eta}>0$,

$$ \limsup_{n\rightarrow \infty} \int _{B_{N}(0, n)\setminus B_{N}(0,r)}\bigl(f( \widetilde{u}_{j_{n}}) \widetilde{u}_{j_{n}}+g(\widetilde{v}_{j_{n}}) \widetilde{v}_{j_{n}}\bigr)\,dx\leqslant \eta , $$

(4.14)

for all $r\geqslant r_{\eta}$.

According to the above arguments, assume that $\varepsilon x_{\varepsilon}\rightarrow x_{0}$, $\varepsilon y_{\varepsilon}\rightarrow y_{0}$ as $\varepsilon \rightarrow 0$; now we can conclude from $a(\varepsilon x+\varepsilon x_{\varepsilon})\rightarrow a(x_{0})$, $a( \varepsilon x+\varepsilon y_{\varepsilon})\rightarrow a(y_{0})$ pointwise and (4.14) that $c_{\varepsilon}\rightarrow c(a(x_{0}))$ and $c_{\varepsilon}\rightarrow c(a(y_{0}))$, which combine with Lemma 4.3 that $a(x_{0})=a(y_{0})=a(0)=\min_{x\in \mathbb{R}^{N}}a(x)$. In conclusion, any maximum points $\varepsilon x_{\varepsilon}$, $\varepsilon y_{\varepsilon}$ of $u_{\varepsilon}$, $v_{\varepsilon}$, respectively, have the concentration point as $\varepsilon \rightarrow 0$.

Step 2. $x_{\varepsilon}=y_{\varepsilon}$ as $\varepsilon \rightarrow 0$ and the maximum point of $u_{\varepsilon}$ (and of $v_{\varepsilon}$ as well) is unique if $\varepsilon >0$ is sufficiently small.

Let us prove that there exists $C>0$ such that, for every subsequence $\{\varepsilon _{j}\}$ and every large j,

$$ \lvert x_{\varepsilon _{j}}-y_{\varepsilon _{j}} \rvert \leqslant C. $$

(4.15)

For this purpose, let $\overline{u}_{j}=u_{\varepsilon _{j}}(x+y_{\varepsilon _{j}})$, $\overline{v}_{j}=v_{\varepsilon _{j}}(x+y_{\varepsilon _{j}})$, we can follow Step 1 to assume $(\overline{u}_{j},\overline{v}_{j})\rightharpoonup (\overline{u}, \overline{v})\neq (0,0)$; it follows from (4.14) that there is a subsequence $\{\overline{u}_{j_{n}}\}$ and $\{\overline{v}_{j_{n}}\}$ such that, for any $\eta >0$ there exists $R>0$ such that, for every large n,

$$ \int _{{B_{N}(0,R)}^{c}}\bigl(f(\overline{u}_{j_{n}}) \overline{u}_{j_{n}}+g( \overline{v}_{j_{n}}) \overline{v}_{j_{n}}\bigr)\,dx\leqslant \eta . $$

Denoting $w_{j_{n}}:=y_{\varepsilon _{j_{n}}}-x_{\varepsilon _{j_{n}}}$, the above inequality reads as

$$ \int _{{B_{N}(w_{j_{n}},R)}^{c}}\bigl(f(\widetilde{u}_{j_{n}}) \widetilde{u}_{j_{n}}+g(\widetilde{v}_{j_{n}}) \widetilde{v}_{j_{n}}\bigr)\,dx \leqslant \eta . $$

So, if $\lvert w_{j_{n}}\rvert \rightarrow +\infty$, we conclude from (4.14) that for every $\eta >0$, for every large n,

$$ \int _{\mathbb{R}^{N}}\bigl(f(\widetilde{u}_{j_{n}}) \widetilde{u}_{j_{n}}+g( \widetilde{v}_{j_{n}}) \widetilde{v}_{j_{n}}\bigr)\,dx\leqslant 2\eta , $$

which conclude that $\int _{\mathbb{R}^{N}}(f(u)u+g(v)v)\,dx=0$, and thus $u=v=0$. This is a contradiction.

Now, according to (4.15), let $w_{0}\in \mathbb{R}^{N}$ be such that $y_{\varepsilon _{j}}-x_{\varepsilon _{j}}\rightarrow w_{0}( \varepsilon _{j}\rightarrow 0)$. Since $\overline{u}_{j}(x)=\widetilde{u}_{j}(x+y_{\varepsilon _{j}}-x_{ \varepsilon _{j}})\rightharpoonup u(x+w_{0})$, combining with the fact that both $\widetilde{u}_{j}(x)$ and $\overline{u}_{j}(x)$ have the same limit function u (and similarly for $\widetilde{v}_{j}(x)$ and $\overline{v}_{j}(x)$), we can derive $u(x)=u(x+w_{0})$ (and similarly for v). u has 0 and $w_{0}$ as maximum points, so that $w_{0}=0$. This establishes $y_{\varepsilon}=x_{\varepsilon}$ as $\varepsilon \rightarrow 0$.

Similarly, if $z_{\varepsilon}\in \mathbb{R}^{N}$ is also a maximum point of $u_{\varepsilon}$, then the preceding arguments yield that $z_{\varepsilon}=x_{\varepsilon}(=y_{\varepsilon})$ for small values of ε.

Step 3. $u_{\varepsilon}, v_{\varepsilon}(x)\rightarrow 0$ as $\lvert x \rvert \rightarrow \infty$ uniformly for all small ε.

Indeed, assume by contradiction that there exist $\delta >0$ and $\xi _{\varepsilon}\in \mathbb{R}^{N}$ with $\lvert \xi _{\varepsilon}\rvert \rightarrow \infty$ such that

$$ \delta \leqslant \bigl\lvert u_{\varepsilon}(\xi _{\varepsilon})\bigr\rvert \leqslant C \int _{B_{N}(\xi _{\varepsilon},1)}\bigl\lvert u_{\varepsilon}(y) \bigr\rvert \,dy. $$

Assume that $u_{\varepsilon}\rightharpoonup u$ in H, we obtain, as $\varepsilon \rightarrow 0$,

$$ \begin{aligned} \delta &\leqslant C\biggl( \int _{B_{N}(\xi _{\varepsilon},1)}\bigl\lvert u_{\varepsilon}(y)\bigr\rvert ^{2}\,dy\biggr)^{\frac{1}{2}} \\ &\leqslant C\biggl( \int _{B_{N}(\xi _{\varepsilon},1)}\bigl\lvert u_{\varepsilon}(y)-u(y) \bigr\rvert ^{2}\,dy\biggr)^{\frac{1}{2}} +C\biggl( \int _{B_{N}(\xi _{\varepsilon},1)} \bigl\lvert u(y)\bigr\rvert ^{2}\,dy \biggr)^{\frac{1}{2}}\rightarrow 0, \end{aligned} $$

which is a contradiction completing the proof. □

Define

$$ \omega _{\varepsilon}(x)=u_{\varepsilon}\biggl(\frac{x}{\varepsilon} \biggr),\qquad \xi _{\varepsilon}(x)=v_{\varepsilon}\biggl( \frac{x}{\varepsilon}\biggr) \quad \text{and}\quad z_{\varepsilon}=\varepsilon x_{\varepsilon}. $$

Then $(\omega _{\varepsilon},\xi _{\varepsilon})$ is a solution to (1.2) for all small $\varepsilon >0$. Since $z_{\varepsilon}$ is a maximum point of $\omega _{\varepsilon}$ and $\xi _{\varepsilon}$, we have

$$ \lim_{\varepsilon \rightarrow 0}a(z_{\varepsilon})=a(0). $$

Now, we study the decay behavior of this family of solution $(\omega _{\varepsilon},\xi _{\varepsilon})$.

Theorem 1.1(iii)

There exist $0< C_{1}\leqslant C_{2}$ and $R>1$ such that for all small $\varepsilon >0$,

$$ \frac{C_{1}\varepsilon ^{N+2s}}{\lvert x-z_{\varepsilon}\rvert ^{N+2s}} \leqslant \omega _{\varepsilon}(x), \qquad \xi _{\varepsilon}(x)\leqslant \frac{C_{2}\varepsilon ^{N+2s}}{\lvert x-z_{\varepsilon}\rvert ^{N+2s}} $$

for all $\lvert x \rvert \geqslant R$.

Proof

Firstly, we use the following Claims, according to [25].

(i) There is a continuous function $w_{1}$ in $\mathbb{R}^{N}$ satisfying

$$ (-\Delta )^{s}w_{1}(x)+\mu w_{1}(x)=0,\quad \text{if } \lvert x \rvert >1, $$

and

$$ w_{1}(x)\geqslant \frac{C_{1}}{\lvert x \rvert ^{N+2s}}, $$

for an appropriate $C_{1}>0$, where $\mu :=\sup a(\varepsilon x)$;

(ii) There is a continuous function $w_{2}$ in $\mathbb{R}^{N}$ satisfying

$$ (-\Delta )^{s}w_{2}(x)+\tau w_{2}(x)=0,\quad \text{if } \lvert x \rvert >1. $$

and

$$ w_{2}(x)\leqslant \frac{C_{2}}{\lvert x \rvert ^{N+2s}}, $$

for an appropriate $C_{2}>0$, where $0<\tau <\frac{1}{2}\inf a(\varepsilon x)$.

By $u_{\varepsilon}(x),v_{\varepsilon}(x)\rightarrow 0$ as $\lvert x \rvert \rightarrow \infty$ uniformly for all small ε and the condition ($A_{0}$), ($A_{1}$), we conclude that there is a large $R_{1}>0$ such that

$$ (-\Delta )^{s}(u_{\varepsilon}+v_{\varepsilon})+ \frac{a(\varepsilon x)}{2}(u_{\varepsilon}+v_{\varepsilon})=g(v_{\varepsilon})+f(u_{\varepsilon})-\frac{a(\varepsilon x)}{2}(u_{\varepsilon}+v_{\varepsilon})\leqslant 0 \quad \text{in } {B_{R_{1}}}^{c}. $$

Moreover, by the continuity of solution $(u_{\varepsilon},v_{\varepsilon})$ and $w_{2}$, for all small $\varepsilon >0$, there exists $C>0$ such that

$$ u_{\varepsilon}(x)+v_{\varepsilon}(x)-Cw_{2}(x)\leqslant 0 \quad \text{in } \overline{B_{R_{1}}}. $$

Therefore,

$$ (-\Delta )^{s}(u_{\varepsilon}+v_{\varepsilon}-Cw_{2})+\tau (u_{\varepsilon}+v_{\varepsilon}-Cw_{2})\leqslant 0\quad \text{in } {B_{R_{1}}}^{c}. $$

Using comparison arguments, we get

$$ u_{\varepsilon}+v_{\varepsilon}\leq Cw_{2}\leq \frac{C_{2}}{\lvert x \rvert ^{N+2s}} \quad \text{for } \lvert x \rvert \geq R_{1}. $$

Since $(u_{\varepsilon},v_{\varepsilon})$ is a positive solution, then

$$ u_{\varepsilon}(x)\leqslant \frac{C}{\lvert x \rvert ^{N+2s}},\qquad v_{\varepsilon}(x) \leqslant \frac{C}{\lvert x \rvert ^{N+2s}},\quad \text{for } \lvert x \rvert \geq R_{1}. $$

Hence, by rescaling, it follows that there exists $C_{2}>0$ such that

$$ \omega _{\varepsilon}(x),\xi _{\varepsilon}(x)\leqslant \frac{C_{2}\varepsilon ^{N+2s}}{\lvert x-z_{\varepsilon}\rvert ^{N+2s}}. $$

On the other hand, by the continuity of $(u_{\varepsilon},v_{\varepsilon})$ and $w_{1}$, there exist constants $C_{2}, C_{3}>0$ such that, respectively,

$$\begin{aligned}& u_{\varepsilon}(x)-C_{2}w_{1}(x)\geqslant 0\quad \text{in } \overline{B_{1}}, \\& v_{\varepsilon}(x)-C_{3}w_{1}(x)\geqslant 0\quad \text{in } \overline{B_{1}}, \end{aligned}$$

which imply that

$$\begin{aligned}& (-\Delta )^{s}(u_{\varepsilon}-C_{2}w_{1})+ \mu (u_{\varepsilon}-C_{2}w_{1}) \geqslant 0\quad \text{in } {\overline{B_{1}}}^{c}, \\& (-\Delta )^{s}(v_{\varepsilon}-C_{3}w_{1})+ \mu (v_{\varepsilon}-C_{3}w_{1}) \geqslant 0\quad \text{in } {\overline{B_{1}}}^{c}. \end{aligned}$$

By the similar comparison arguments, we conclude the second inequality, which ends the proof of Theorem 1.1. □

5 The case $p\neq q$

In Sect. 3 and Sect. 4, we have proved Theorem 1.1(i)(ii)(iii) except that we have worked with a truncated problem as explained in Remark 1.2. The full statement of Theorem 1.1(i)(ii)(iii) will be established once we prove uniform bounds in $L^{\infty}$ of the solutions constructed so far. So, in this section, let us suppose that $p,q>2$ are such that $\frac{1}{p}+\frac{1}{q}>\frac{N-2s}{N}$ with say, $2< p<2^{*}_{s}$ and $p< q$. We only show that the weak solutions to the modified problem of (3.1) are bounded uniformly in $L^{\infty}$. In the same way, the weak solutions to the modified problem of (2.1) are ones of the original system (2.1) for large value of n.

Given $n\in \mathbb{N}$, we can define the truncated functions,

$$ g_{n}(t)= \textstyle\begin{cases} g(t), & t\leqslant n, \\ A_{n}t^{p-1}+B_{n} , &t> n, \end{cases} $$

where the coefficients are chosen in such a way that $g_{n}$ is $C^{1}$. Thus, in view of $(A_{2})$, we see that $A_{n}=(\frac{l_{2}}{p-1}+o(1))\cdot n^{q-p}$, $B_{n}=(\frac{l_{2}(p-q)}{(p-1)(q-1)}+o(1))\cdot n^{q-1}$. The energy functionals associated to the modified problem of (3.1) are given by

$$ \begin{aligned} J_{n}(U,V)&=k_{s} \int _{\mathbb{R}_{+}^{N+1}} y^{1-2s}\langle \nabla U,\nabla V\rangle \,dx\,dy+\lambda \int _{\mathbb{R}^{N}\times \{0 \}}UV\,dx \\ & \quad {}- \int _{\mathbb{R}^{N}\times \{0\}}F(U)\,dx- \int _{\mathbb{R}^{N} \times \{0\}}G_{n}(V)\,dx, \end{aligned} $$

where $G_{n}$ is the primitive of $g_{n}$. They are $C^{2}$ functionals defined over the Hilbert space E. The critical points of $J_{n}$ correspond to weak solutions of the modified problem

$$ \textstyle\begin{cases} (-\bigtriangleup )^{s}u+\lambda u=g_{n}(v), & \text{in } \mathbb{R}^{N}, \\ (-\bigtriangleup )^{s}v+\lambda v=f(u), & \text{in } \mathbb{R}^{N}. \end{cases} $$

(5.1)

For a fixed n, thanks to the Sect. 3, there are positive solutions $(u_{n},v_{n})$ of the modified problem (5.1) satisfying the conclusion of Theorem 3.1.

Remark 5.1

According to the Sect. 3, we find the solutions $(u_{n},v_{n})$ to the modified problem (5.1), having relative Morse index ⩽1.

Now, we state the main result of this section.

Theorem 5.2

Assume that ($A_{1}$)–($A_{4}$), for any given $n\in \mathbb{N}$; let $(U_{n},V_{n})$ be solutions to the problem (5.1). If there exists $k\in \mathbb{N}$ such that $m(u_{n},v_{n})\leqslant k$ for every n, then there exists $M>0$ such that

$$ \Vert u_{n} \Vert _{\infty}+ \Vert v_{n} \Vert _{\infty}\leqslant M,\quad \forall n. $$

In particular, $u_{n}$ and $v_{n}$ are solutions to the problem (3.1) for large values of n.

The proof of Theorem 5.2 is based on the following simple fact, whose proof is the same as Lemma 1.2 in [26].

Lemma 5.3

Assume that ($A_{1}$)–($A_{4}$), and let $(U_{n},V_{n})$ be any solutions of the problem (5.1). If there exist $\lambda >0$ and $k+1$ functions $\Psi _{1},\Psi _{2},\ldots ,\Psi _{k+1}\in X$ having disjoint supports, such that

$$ J_{n}^{\prime \prime }(U_{n},V_{n}) (\Psi _{i},\lambda \Psi _{i}) (\Psi _{i}, \lambda \Psi _{i})< 0,\quad \forall i=1,\ldots ,k+1, $$

then $m(U_{n},V_{n})\geqslant k+1$.

Next, we will prove a Liouville-type theorem, which is crucial for the proof of Theorem 5.2.

Proposition 5.4

Let $f_{\infty},g_{\infty}\in C^{1}(\mathbb{R})$ and $(u,v)$ satisfy

$$ \textstyle\begin{cases} (-\Delta )^{s}u=g_{\infty}(v), & \textit{in } \mathbb{R}^{N}, \\ (-\Delta )^{s}v=f_{\infty}(u), & \textit{in } \mathbb{R}^{N}, \end{cases} $$

(5.2)

and $m(u,v)<+\infty$ in the sense of Definitions 2.4and 2.5. Let $f_{\infty}(t)=c\lvert t\rvert ^{p-1}t$ with $c>0$ and $2< p<2^{*}_{s}$.

• (i) If $g_{\infty}=0$, then $u=0$;

• (ii) If $g_{\infty}$ satisfies the following conditions, for $p\leqslant q, \frac{1}{p}+\frac{1}{q}>\frac{N-2s}{N}$ and some $C_{1}, C_{2}>0$,

  • (a) $C_{1}\lvert t\rvert ^{q}\leqslant g_{\infty}(t)t\leqslant C_{2} \lvert t\rvert ^{q}$;

  • (b) $g_{\infty}(t)t\leqslant qG_{\infty}(t)$;

  • (c) $(p-1)g_{\infty}(t)t\leqslant g'_{\infty}(t)t^{2}$;

  then $u=0=v$.

Proof

(i) It is obvious.

(ii) We may assume that $c=1$, suppose $(-\Delta )^{s}u=g_{\infty}(v)$, $(-\Delta )^{s}v=\lvert u \rvert ^{p-1}u$, with $g_{\infty}$ satisfying the conditions (a)–(c). The associated energy functionals $J_{\infty}:X^{s}(\mathbb{R}_{+}^{N+1})\times X^{s}(\mathbb{R}_{+}^{N+1}) \rightarrow \mathbb{R}^{1}$ to the extension problem of (5.2) are given by

$$ J_{\infty}(U,V)=k_{s} \int _{\mathbb{R}_{+}^{N+1}} y^{1-2s}\langle \nabla U,\nabla V\rangle \,dx\,dy -\frac{1}{p} \int _{\mathbb{R}^{N}} \lvert u \rvert ^{p}\,dx- \int _{\mathbb{R}^{N}}G_{\infty}(v)\,dx. $$

Fix any smooth function $\Psi \in C_{0}^{\infty}(\mathbb{R}^{N+1}_{+})$ such that $\Psi =0$ in $B^{+}_{N+1}(0,0.5R_{0})$, $\Psi =1$ in $B^{+}_{N+1}(0,2R_{0})\setminus B^{+}_{N+1}(0,R_{0})$ and $\operatorname{supp}\Psi \subset B^{+}_{N+1}(0,3.5R_{0})\setminus B^{+}_{N+1}(0,0.5R_{0})$. For any large R, let $\Phi \in C_{0}^{\infty}(\mathbb{R}^{N+1}_{+},[0,1])$ supported on $B^{+}_{N+1}(0,2R)\subset \mathbb{R}^{N+1}_{+}$ satisfying $\Phi =1$ on $B^{+}_{N+1}(0,R)$ and $\lvert \nabla \Phi \rvert ^{2}\leqslant \|\Phi \|$.

In view of $m(u,v)<+\infty$, replace ϕ with $U\Phi ^{m}\Psi$ in (2.6), then the assumption (2.6) reads as

$$ \begin{aligned} &(p-1) \int _{\mathbb{R}^{N}\times \{0\}}U^{p}\Phi ^{2m}\Psi ^{2}\,dx+ \int _{\mathbb{R}^{N}\times \{0\}}g'_{\infty}(V)U^{2}\Phi ^{2m}\Psi ^{2}\,dx \\ &\quad \leqslant Ck_{s} \int _{\mathbb{R}^{N+1}_{+}}y^{1-2s}\bigl(m^{2}U^{2} \Phi ^{2(m-1)}\Psi ^{2}\lvert \nabla \Phi \rvert ^{2}+U^{2}\Phi ^{2m} \lvert \nabla \Psi \rvert ^{2} \\ & \qquad {}+\Phi ^{2m}\Psi ^{2}\lvert \nabla U \rvert ^{2}\bigr)\,dx\,dy+ \int _{ \mathbb{R}^{N}\times \{0\}}g_{\infty}(V)U\Phi ^{2m}\Psi ^{2}\,dx, \end{aligned} $$

which implies that

$$ \begin{aligned} &(p-1) \int _{\mathbb{R}^{N}\times \{0\}}U^{p}\Phi ^{2m}\Psi ^{2}\,dx \\ &\quad \leqslant C(R_{0})k_{s} \int _{\operatorname{supp}\Psi}y^{1-2s}\bigl(U^{2}\Phi ^{2(m-1)} \lvert \nabla \Phi \rvert ^{2}+U^{2} \Phi ^{2m}+\lvert \nabla U \rvert ^{2}\Phi ^{2m}\bigr)\,dx\,dy \\ &\qquad {}+C(R_{0}) \int _{\operatorname{supp}\Psi (x,0)\times \{0\}}g_{\infty}(V)U\Phi ^{2m}\,dx. \end{aligned} $$

(5.3)

Now, we estimate the right terms of the above inequality. It follows from Proposition 2.3 and Hölder inequality with m large that

$$ k_{s} \int _{\operatorname{supp}\Psi}y^{1-2s}\bigl(U^{2}\Phi ^{2(m-1)}\lvert \nabla \Phi \rvert ^{2}+U^{2}\Phi ^{2m}+\lvert \nabla U \rvert ^{2}\Phi ^{2m} \bigr)\,dx\,dy< C. $$

(5.4)

On the other hand, in view of (5.2) and the similar arguments as (5.4), we arrive at

$$ \begin{aligned} & \int _{\operatorname{supp}\Psi (x,0)\times \{0\}}g_{\infty}(V)U\Phi ^{2m}\,dx \\ &\quad =k_{s} \int _{\operatorname{supp}\Psi}y^{1-2s}\bigl\langle \nabla U,\nabla \bigl(U\Phi ^{2m}\bigr) \bigr\rangle \,dx\,dy \\ &\quad \leqslant k_{s} \int _{\operatorname{supp}\Psi}y^{1-2s}\bigl(\Phi ^{2m} \lvert \nabla U \rvert ^{2}+m^{2}\Phi ^{4m-2}U^{2}+ \lvert \nabla U \rvert ^{2} \lvert \nabla \Phi \rvert ^{2} \bigr)\,dx\,dy< C. \end{aligned} $$

(5.5)

We conclude by combing (5.3)–(5.5), which together lead to $u\in L^{p}(\mathbb{R}^{N})$. Thanks to $\int _{\mathbb{R}^{N}}(-\Delta )^{s}uv\,dx=\int _{\mathbb{R}^{N}}g_{\infty}(v)v\,dx=\int _{\mathbb{R}^{N}}u^{p}\,dx<+\infty$ and the condition (a), $v\in L^{q}(\mathbb{R}^{N})$ also.

Making use of the well-known Pohozăve–Rellich type identity,

$$ \int _{\mathbb{R}^{N}}(-\Delta )^{s}uv\,dx= \frac{N}{N-2s} \int _{ \mathbb{R}^{N}}\bigl(F_{\infty}(u)+G_{\infty}(v) \bigr)\,dx. $$

By the condition (c), we deduce that

$$ \begin{aligned} \frac{N-2s}{N} \int _{\mathbb{R}^{N}}u^{p}\,dx&=\frac{1}{p} \int _{ \mathbb{R}^{N}}u^{p}\,dx+ \int _{\mathbb{R}^{N}}G_{\infty}(v)\,dx \\ &\geqslant \frac{1}{p} \int _{\mathbb{R}^{N}}u^{p}\,dx+\frac{1}{q} \int _{ \mathbb{R}^{N}}g_{\infty}(v)v\,dx=\biggl( \frac{1}{p}+\frac{1}{q}\biggr) \int _{ \mathbb{R}^{N}}u^{p}\,dx. \end{aligned} $$

Since $\frac{1}{p}+\frac{1}{q}>\frac{N-2s}{N}$, this implies that $u=v=0$ and concludes the proof of Proposition 5.4. □

Once Proposition 5.4 is settled, we may use the classical blow-up argument to give the proof of Theorem 5.2 that is similar to [27], we omit it.