Normalized solutions for a coupled fractional Schrödinger system in low dimensions

Abstract

We consider the following coupled fractional Schrödinger system:

$$ \textstyle\begin{cases} (-\Delta )^{s}u+\lambda _{1}u=\mu _{1} \vert u \vert ^{2p-2}u+ \beta \vert v \vert ^{p} \vert u \vert ^{p-2}u, \\ (-\Delta )^{s}v+\lambda _{2}v=\mu _{2} \vert v \vert ^{2p-2}v+\beta \vert u \vert ^{p} \vert v \vert ^{p-2}v \end{cases}\displaystyle \quad \text{in } {\mathbb{R}^{N}}, $$

with $0< s<1$, $2s< N\le 4s$ and $1+\frac{2s}{N}< p<\frac{N}{N-2s}$, under the following constraint:

$$ \int _{\mathbb{R}^{N}} \vert u \vert ^{2}\,dx=a_{1}^{2} \quad \text{and}\quad \int _{ \mathbb{R}^{N}} \vert v \vert ^{2}\,dx=a_{2}^{2}. $$

Assuming that the parameters $\mu _{1}$, $\mu _{2}$, $a_{1}$, $a_{2}$ are fixed quantities, we prove the existence of normalized solution for different ranges of the coupling parameter $\beta >0$.

1 Introduction

In this paper, we consider the following fractional Schrödinger system with $1+\frac{2s}{N}< p<\frac{N}{N-2s} \text{and} 2s<N\le 4s$:

$$ \textstyle\begin{cases} (-\Delta )^{s}u+\lambda _{1}u=\mu _{1} \vert u \vert ^{2p-2}u+\beta \vert v \vert ^{p} \vert u \vert ^{p-2}u, \\ (-\Delta )^{s}v+\lambda _{2}v=\mu _{2} \vert v \vert ^{2p-2}v+\beta \vert u \vert ^{p} \vert v \vert ^{p-2}v \end{cases}\displaystyle \quad \text{in } {\mathbb{R}^{N}}, $$

(1)

under the constraint

$$ \int _{\mathbb{R}^{N}} \vert u \vert ^{2}\,dx=a_{1}^{2} \quad \text{and}\quad \int _{ \mathbb{R}^{N}} \vert v \vert ^{2}\,dx=a_{2}^{2}. $$

(2)

The parameters $\mu _{1}>0$, $\mu _{2}>0$ and $\beta >0$.

More precisely, we analyze the existence of solutions $(\lambda _{1},\lambda _{2},u,v)\in \mathbb{R}^{2}\times H^{s}( \mathbb{R}^{N})\times H^{s}(\mathbb{R}^{N})$ to the system (1) satisfying the additional condition (2) for different ranges of the coupling parameter $\beta >0$.

Condition (2) is called the normalization condition, which imposes a normalization on the $L^{2}$-masses of u and v. The solutions to the system (1) under the constraint (2) are usually referred as normalized solutions. In order to obtain the solution to the system (1) satisfying the normalization condition (2), one need to consider the critical point with the $H_{a}$ (see (4)). Then $\lambda _{1}$ and $\lambda _{2}$ appear as Lagrange multipliers with respect to the mass constraint, which cannot be determined a priori, but are part of the unknown.

The normalized solutions of nonlinear Schrödinger equations and systems have gradually attracted the attention of a large number of researchers in recent years, both for the pure mathematical research and in view of its very important applications in many physical problems; see for more details [1–4].

On the one hand, for the nonlinear Schrödinger equation with $s=1$, in [5], the author studied existence and properties of ground states for the nonlinear Schrödinger equation with combined power nonlinearities p, q which satisfy $2< q\leq 2+\frac{4}{N}\leq p$, $p\neq q$. In [6], the author studied existence and properties of ground states for the nonlinear Schrödinger equation with combined power nonlinearities q, 2^∗.

On the other hand, for the nonlinear Schrödinger system with $s=1$, Thomas et al. [3] recently proved the existence of positive solutions for the system with any arbitrary number of components in three-dimensional space. In [7], the authors considered the existence of multiple positive solutions to the nonlinear Schrödinger systems set on $H^{1}(\mathbb{R}^{N})\times H^{1}(\mathbb{R}^{N})$. In [8], the authors proved the existence of solutions $(\lambda _{1},\lambda _{2},u,v)\in \mathbb{R}^{2}\times H^{1}( \mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$ to systems of coupled Schrödinger equations.

The fractional Schrödinger equation is introduced by Laskin [9, 10] through expanding the Feynman path integral from Brownian-like to Lévy-like mechanical paths. The path integral over the Lévy-like quantum-mechanical paths allows one to develop the generalization of the quantum mechanics. The existence of normalized solution for fractional Schrödinger system is an interesting problem.

The fractional Laplacian $(-\Delta )^{s}$ with $s\in (0,1)$ of a function $f:\mathbb{R}^{N}\rightarrow \mathbb{R}$ is expressed by the formula

$$ (-\Delta )^{s}f(x)=C_{N,s}\,\mathrm{P.V.} \int _{\mathbb{R}^{N}} \frac{f(x)-f(z)}{ \vert x-z \vert ^{N+2s}}\,dz, $$

where $\mathrm{P.V.}$ stands for the Cauchy principal value, and $C_{N,s}$ is a normalization constant.

It can also be defined as a pseudo-differential operator

$$ \mathscr{F}\bigl((-\Delta )^{s}f\bigr) (\xi )= \vert \xi \vert ^{2s}\mathscr{F}(f) (\xi )= \vert \xi \vert ^{2s} \hat{f}(\xi ), $$

where $\mathscr{F}$ is the Fourier transform. For more details about the fractional Laplacian we refer to [11–15] and the references therein. The nature function space associated with $(-\Delta )^{s}$ in N dimension is

$$ H^{s}\bigl(\mathbb{R}^{N}\bigr):= \biggl\{ u \Bigm| \int _{\mathbb{R}^{2N}} \frac{ \vert u(x)-u(z) \vert ^{2}}{ \vert x-z \vert ^{N+2s}}\,dx\,dz< +\infty \text{ and } \int _{ \mathbb{R}^{N}} \bigl\vert u(x) \bigr\vert ^{2}\,dx< +\infty \biggr\} , $$

equipped with the norm

$$ \Vert u \Vert _{H^{s}(\mathbb{R}^{N})}= \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}}u \bigr\vert ^{2}\,dx+ \int _{\mathbb{R}^{N}} \vert u \vert ^{2}\,dx \biggr)^{\frac{1}{2}}, $$

where, by the Fourier transform,

$$\begin{aligned} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}}u \bigr\vert ^{2}\,dx&= \int _{ \mathbb{R}^{N}} \vert \xi \vert ^{2s}\hat{u}(\xi )^{2}d\xi = \int _{\mathbb{R}^{N}}(- \Delta )^{s} u\cdot u\,dx \\ &=C_{N,s} \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{(u(x)-u(z))u(x)}{ \vert x-z \vert ^{N+2s}}\,dz\,dx \\ &=\frac{C_{N,s}}{2} \int _{\mathbb{R}^{2N}} \frac{ \vert u(x)-u(z) \vert ^{2}}{ \vert x-z \vert ^{N+2s}}\,dz\,dx. \end{aligned}$$

The energy functional associated with (1) is

$$ \begin{aligned} E(u,v) ={}&\frac{1}{2} \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u \bigr\vert ^{2}+ \bigl\vert (- \Delta )^{\frac{s}{2}}v \bigr\vert ^{2}\bigr)\,dx \\ &{}-\frac{1}{2p} \int _{\mathbb{R}^{N}}\bigl( \mu _{1} \vert u \vert ^{2p}+2\beta \vert u \vert ^{p} \vert v \vert ^{p}+\mu _{2} \vert v \vert ^{2p}\bigr)\,dx \end{aligned} $$

(3)

on the constraint $H_{a_{1}}\times {H_{a_{2}}}$. For $a\in \mathbb{R}^{+}$, we define

$$ H_{a}:=\biggl\{ u\in H^{s}\bigl( \mathbb{R}^{N}\bigr): \int _{\mathbb{R}^{N}} \vert u \vert ^{2}\,dx=a^{2} \biggr\} . $$

(4)

We prove the existence of normalized solution for different ranges of the coupling parameter $\beta >0$. Our first main theorem, which is the generalization of the corresponding result ($s=1$, $N=3$, $p=2$) given in [3], it stated as follows.

Theorem 1.1

Assume $0< s<1$, $2s< N\le 4s$ and $1+\frac{2s}{N}< p<\frac{N}{N-2s}$. Let $a_{1}$, $a_{2}$, $\mu _{1}$ and $\mu _{2}>0$ be fixed, and let $\beta _{1}>0$ be defined by

$$ \begin{aligned} &\max \biggl\{ \frac{1}{a_{1}^{\frac{4ps-2(p-1)N}{(p-1)N-2s}}\mu _{1}^{\frac{2s}{(p-1)N-2s}}}, \frac{1}{a_{2}^{\frac{4ps-2(p-1)N}{(p-1)N-2s}}\mu _{2}^{\frac{2s}{(p-1)N-2s}}} \biggr\} \\ &\quad = \frac{1}{a_{1}^{\frac{4ps-2(p-1)N}{(p-1)N-2s}}(\mu _{1}+\beta _{1})^{\frac{2s}{(p-1)N-2s}}}+ \frac{1}{a_{2}^{\frac{4ps-2(p-1)N}{(p-1)N-2s}}(\mu _{2}+\beta _{1})^{\frac{2s}{(p-1)N-2s}}}. \end{aligned} $$

(5)

If $0<\beta <\beta _{1}$, then (1) has a solution $(\tilde{\lambda }_{1},\tilde{\lambda }_{2},\tilde{u},\tilde{v})$ with $(\tilde{u},\tilde{v})$ on the constraint $H_{a_{1}}\times {H_{a_{2}}}$, such that $\tilde{\lambda }_{1},\tilde{\lambda }_{2}>0$ and ũ and ṽ are both positive and radial.

For the next result, we introduce a Pohozaev-type constraint as follows:

$$ {F}:=\bigl\{ (u,v)\in {H_{a_{1}}}\times {H_{a_{2}}}:G(u,v)=0\bigr\} , $$

(6)

where

$$ \begin{aligned} G(u,v) ={}& \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u \bigr\vert ^{2}+ \bigl\vert (- \Delta )^{\frac{s}{2}}v \bigr\vert ^{2}\bigr)\,dx \\ &{}-\frac{(p-1)N}{2ps} \int _{\mathbb{R}^{N}}\bigl( \mu _{1} \vert u \vert ^{2p}+2\beta \vert u \vert ^{p} \vert v \vert ^{p}+\mu _{2} \vert v \vert ^{2p} \bigr)\,dx. \end{aligned} $$

(7)

We define a Rayleigh-type quotient as

$$ \mathcal{R}(u,v):= \frac{R_{0} (\int _{\mathbb{R}^{N}}( \vert (-\Delta )^{\frac{s}{2}} u \vert ^{2}+ \vert (-\Delta )^{\frac{s}{2}}v \vert ^{2})\,dx )^{\frac{(p-1)N}{(p-1)N-2s}}}{ (\int _{\mathbb{R}^{N}}(\mu _{1} \vert u \vert ^{2p}+2\beta \vert u \vert ^{p} \vert v \vert ^{p}+\mu _{2} \vert v \vert ^{2p})\,dx )^{\frac{2s}{(p-1)N-2s}}}, $$

(8)

where

$$ R_{0}=\frac{(p-1)N-{2s}}{2(p-1)N} \biggl( \frac{2ps}{(p-1)N} \biggr)^{\frac{2s}{(p-1)N-2s}}. $$

(9)

Theorem 1.2

Assume $0< s<1$, $2s< N\le 4s$ and $1+\frac{2s}{N}< p<\frac{N}{N-2s}$. Let $a_{1}$, $a_{2}$, $\mu _{1}$ and $\mu _{2}>0$ be fixed, and let $\beta _{2}>0$ be defined by

$$ \begin{aligned} &\min \biggl\{ \frac{1}{a_{1}^{\frac{4ps-2(p-1)N}{(p-1)N-2s}}\mu _{1}^{\frac{2s}{(p-1)N-2s}}}, \frac{1}{a_{2}^{\frac{4ps-2(p-1)N}{(p-1)N-2s}}\mu _{2}^{\frac{2s}{(p-1)N-2s}}} \biggr\} \\ &\quad = \frac{(a_{1}^{2}+a_{2}^{2})^{\frac{(p-1)N}{(p-1)N-2s}}}{(\mu _{1}a_{1}^{2p}+2\beta _{2}a_{1}^{p}a_{2}^{p}+\mu _{2}a_{2}^{2p})^{\frac{2s}{(p-1)N-2s}}}. \end{aligned} $$

(10)

If $\beta >\beta _{2}$, then (1) has a solution $(\bar{\lambda }_{1},\bar{\lambda }_{2},\bar{u},\bar{v})$ with $(\bar{u},\bar{v})$ on the constraint $H_{a_{1}}\times {H_{a_{2}}}$, such that $\bar{\lambda }_{1},\bar{\lambda }_{2}>0$ and ū and v̄ are both positive and radial. Moreover, $(\bar{\lambda }_{1},\bar{\lambda }_{2},\bar{u},\bar{v})$ is a solution in the sense that

$$ E(\bar{u},\bar{v}) =\inf \bigl\{ E(u,v):(u,v)\in F\bigr\} =\inf _{(u,v)\in {H_{a_{1}}} \times {H_{a_{2}}}}\mathcal{R}(u,v) $$

holds.

Remark 1.1

In the system (1) with prescribed $L^{2}$ constraint, the problem appears to be more complicated as the Lagrange multipliers $\lambda _{i}$ are also need to be determined simultaneously. The exponent $2p\in (2+\frac{4s}{N},\frac{2N}{N-2s})$ brings another difficulty as it is $L^{2}$-supercritical and $E(u,v)$ is unbounded from below on the $L^{2}$ constraint. To overcome these difficulties, the idea introduced by Jeanjean in [3, 16] can be adopted to our system: A minimax argument can be applied to E, allowing one to construct a Palais–Smale sequence on the constraint satisfying the Pohozaev identity in limit sense. This leads to the boundedness of the Palais–Smale sequence. Some a priori estimates on $\lambda _{i}$ and a Liouville-type result for the fractional Laplacian (Lemma 2.7) ensure $H^{s}$-convergence of the Palais–Smale sequence.

We do not know if the results are still true in high dimensions. Since $u\in H^{s}(\mathbb{R}^{N})$, when the Liouville-type result is applied, we require that $2\le \frac{N}{N-2s}$ to get our results. It should be interesting to consider the problem in high dimension, even in the Laplacian case.

Remark 1.2

The quantities $\beta _{1}$ given in (5) and $\beta _{2}$ given in (10) are complicated, however, when $N=3$, $s=1$, $p=2$, the condition (5) becomes

$$ \max \biggl\{ \frac{1}{a_{1}^{2}\mu _{1}^{2}}, \frac{1}{a_{2}^{2}\mu _{2}^{2}} \biggr\} = \frac{1}{a_{1}^{2}(\mu _{1}+\beta _{1})^{2}}+ \frac{1}{a_{2}^{2}(\mu _{2}+\beta _{1})^{2}}, $$

and (10) becomes

$$ \frac{(a_{1}^{2}+a_{2}^{2})^{3}}{(\mu _{1}a_{1}^{4}+2\beta a_{1}^{2}a_{2}^{2}+\mu _{2}a_{2}^{4})^{2}}= \min \biggl\{ \frac{1}{a_{1}^{2}\mu _{1}^{2}}, \frac{1}{a_{2}^{2}\mu _{2}^{2}} \biggr\} , $$

which are the conditions given in [3].

The paper is organized as follows. In Sect. 2, we introduce some important lemmas. In Sect. 3, we prove Theorem 1.1 and in Sect. 4, the proof of Theorem 1.2 is given.

2 Preliminaries

In this section, we will show some facts about the fractional NLS equation, which are used later. First, we need the fractional Gagliardo–Nirenberg–Sobolev inequality, which can be found in [12, 13]. For the reader’s convenience, we give the proof here.

Lemma 2.1

(The fractional Gagliardo–Nirenberg–Sobolev inequality)

$$ \int _{\mathbb{R}^{N}} \vert u \vert ^{\alpha +2}\,dx \leqslant C_{\mathrm{opt}} \biggl( \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}}u \bigr\vert ^{2}\,dx \biggr)^{\frac{N\alpha }{4s}} \biggl( \int _{\mathbb{R}^{N}} \vert u \vert ^{2}\,dx \biggr)^{ \frac{\alpha (2s-N)}{4s}+1}. $$

(11)

Here $N>2s$, $0<\alpha <\frac{4s}{N-2s}$ and $C_{\mathrm{opt}}>0$ denotes the sharp constant (depending on α, N and s).

Proof

We consider the “Weinstein functional” given by

$$ J(u)= \frac{ (\int _{\mathbb{R}^{N}} \vert (-\Delta )^{\frac{s}{2}}u \vert ^{2}\,dx )^{\frac{N\alpha }{4s}} (\int _{\mathbb{R}^{N}} \vert u \vert ^{2}\,dx )^{\frac{\alpha (2s-N)}{4s}+1}}{\int _{\mathbb{R}^{N}} \vert u \vert ^{\alpha +2}\,dx} $$

defined for $u\in H^{s}(\mathbb{R}^{N})$ with $u\not \equiv 0$. Set $u^{\lambda ,\mu }=\mu u(\lambda x)$, then we can obtain

$$ J\bigl(u^{\lambda ,\mu }\bigr)=J(u),\qquad \bigl\Vert u^{\lambda ,\mu } \bigr\Vert _{2}^{2}=\lambda ^{-N} \mu ^{2} \Vert u \Vert ^{2}_{2},\qquad \bigl\Vert (-\Delta )^{\frac{s}{2}} u^{\lambda , \mu } \bigr\Vert _{2}^{2}= \lambda ^{2s-N}\mu ^{2} \bigl\Vert (-\Delta )^{\frac{s}{2}} u \bigr\Vert ^{2}_{2}. $$

Since $J(u)> 0$, there exists a minimizing sequence $\{u_{n}\}^{\infty }_{n=1}\subset H^{s}(\mathbb{R}^{N})\cap L^{ \alpha +2}(\mathbb{R}^{N})$. Therefore, it follows that $0\le \eta =\inf_{u\in H^{s}(\mathbb{R}^{N})\setminus \{0\}} J(u)= \lim_{n\rightarrow \infty }J(u_{n})<\infty$. Since $\int _{\mathbb{R}^{N}}|(-\Delta )^{\frac{s}{2}} |u||^{2}\,dx\le \int _{ \mathbb{R}^{N}}|(-\Delta )^{\frac{s}{2}}u|^{2}\,dx$, we may assume that $u_{n}\ge 0$. By Schwarz symmetrization, we may also assume that $u_{n}=u_{n}(|x|)$. If we take $\lambda _{n}= ( \frac{\|u_{n}\|^{2}_{2}}{\|(-\Delta )^{\frac{s}{2}}u_{n}\|_{2}^{2}} )^{\frac{1}{2s}}$, $\mu _{n}= \frac{(\|u\|_{2})^{\frac{2s}{N}-1}}{(\|(-\Delta )^{\frac{s}{2}}u_{n}\|_{2})^{\frac{2s}{N}}}$, then we can obtain a sequence $v_{n}=u^{\lambda _{n},\mu _{n}}$ satisfying

$$\begin{aligned}& 0\leq v_{n}=v_{n}\bigl( \vert x \vert \bigr)\in H^{s}\bigl(\mathbb{R}^{N}\bigr),\qquad \Vert v_{n} \Vert _{2}^{2}=1, \\& \bigl\Vert (- \Delta )v_{n} \bigr\Vert _{2}^{2}=1,\qquad J(v_{n})\downarrow \eta \quad \text{as } n\rightarrow \infty . \end{aligned}$$

Since $\{v_{n}\}$ is bounded in $H^{s}(\mathbb{R}^{N})$, there exists a v such that $v_{n}\rightharpoonup v$ weakly in $H^{s}(\mathbb{R}^{N})$. Because of the radial symmetry of $v_{n}$, by Sobolev embedding, we can obtain $v_{n}\rightarrow v$ strongly in $L^{\alpha +2}(\mathbb{R}^{N})$. By weak convergence, we have $\|v\|^{2}_{2}\leq 1$ and $\|(-\Delta )^{\frac{s}{2}}v\|^{2}_{2}\leq 1$. Hence

$$ \eta \leq J(v)\leq \frac{1}{\int _{\mathbb{R}^{N}} \vert v \vert ^{\alpha +2}}= \lim_{n\rightarrow \infty }J(v_{n})= \eta . $$

Therefore $\|v\|^{2}_{2}=\|(-\Delta )^{\frac{s}{2}}v\|^{2}_{2}=1$, hence $v_{n}\rightarrow v$ strongly in $H^{s}(\mathbb{R}^{N})$, this also proves that $\eta >0$.

Since v minimizes the functional J, it follows that v satisfies the Euler–Lagrange equation

$$ \frac{d}{d\varepsilon }{\bigg|}_{\varepsilon =0}J(v+\varepsilon \varphi )=0 \quad \text{for all }\varphi \in C^{\infty }_{0}\bigl( \mathbb{R}^{N}\bigr). $$

By applying $\|v\|^{2}_{2}=\|(-\Delta )^{\frac{s}{2}}v\|^{2}_{2}=1$, then we can derive that

$$\begin{aligned} &\frac{N\alpha }{2s} \int _{\mathbb{R}^{N}}(-\Delta )^{\frac{s}{2}}v(- \Delta )^{\frac{s}{2}}\varphi \,dx+\biggl(\frac{\alpha (2s-N)}{2s}+2\biggr) \int _{ \mathbb{R}^{N}}v\varphi \,dx \\ &\quad {}-\eta (\alpha +2) \int _{\mathbb{R}^{N}}\bigl( \vert v \vert ^{ \alpha +1}\varphi \bigr)\,dx=0, \end{aligned}$$

for all $\varphi \in C^{\infty }_{0}(\mathbb{R}^{N})$. Therefore v satisfies

$$ \frac{N\alpha }{4s}(-\Delta )^{s}v+\biggl(\frac{\alpha (2s-N)}{4s}+1 \biggr)v=\eta \biggl( \frac{\alpha }{2}+1\biggr) \vert v \vert ^{\alpha +1}. $$

Taking $\hat{v}=[\eta (\frac{\alpha }{2}+1)]^{-\frac{1}{\alpha }}v$, it satisfies

$$ \frac{N\alpha }{4s}(-\Delta )^{s}\hat{v}+\biggl( \frac{\alpha (2s-N)}{4s}+1\biggr) \hat{v}= \vert \hat{v} \vert ^{\alpha +1}. $$

Let $C^{-1}_{\mathrm{opt}}=\inf_{u\neq 0}J(u)$. Then the inequality is established. □

It is well known that, when $N>2s$,

$$ H^{s}\bigl(\mathbb{R}^{N}\bigr) \hookrightarrow L^{p}\bigl(\mathbb{R}^{N}\bigr),\quad \text{for all } 2\leqslant p\leqslant \frac{2N}{N-2s}. $$

(12)

Consider the general fractional Laplacian equation

$$ (-\Delta )^{s} u=f(u) \quad \text{in } \mathbb{R}^{N} $$

(13)

with $f\in C^{2}(\mathbb{R})$. Assume that $u\in {H}^{s}(\mathbb{R}^{N})\cap L^{\infty }(\mathbb{R}^{N})$ is a solution to (13), the Pohozaev identity for (13) is proved in [17].

Theorem 2.2

([17])

Let $u\in {H}^{s}(\mathbb{R}^{N})\cap L^{\infty }(\mathbb{R}^{N})$ be a solution to (13) and $F(u)\in L^{1}(\mathbb{R}^{N})$. Then

$$ (N-2s) \int _{\mathbb{R}^{N}} uf(u)\,dx=2N \int _{\mathbb{R}^{N}}F(u)\,dx, $$

(14)

where $F(u)=\int ^{u}_{0}f(t)\,dt$.

Let us consider the scalar problem

$$ \textstyle\begin{cases} (-\Delta )^{s} w+w= \vert w \vert ^{2p-2}w \quad \text{in } { \mathbb{R}^{N}}, \\ w>0 \quad \text{in } {\mathbb{R}^{N}}, \\ w(0)=\max w\quad \text{and}\quad w\in H^{s}\bigl(\mathbb{R}^{N} \bigr). \end{cases} $$

(15)

It is shown in [13] that there is a unique positive radial solution $w_{0}\in H^{s}(\mathbb{R}^{N})\cap L^{\infty }(\mathbb{R}^{N})$ to (15) for $1< p<\frac{N}{N-2s}$ and $N>2s$; see Proposition 3.1 in [13].

We set

$$ C_{0}:= \int _{\mathbb{R}^{N}} \vert w_{0} \vert ^{2}\,dx\quad \text{and}\quad C_{1}:= \int _{\mathbb{R}^{N}} \vert w_{0} \vert ^{2p}\,dx. $$

(16)

By the Pohozaev identity for (15), we can get

$$ \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}}w_{0} \bigr\vert ^{2}\,dx= \frac{(p-1)N}{2ps} \int _{\mathbb{R}^{N}} \vert w_{0} \vert ^{2p}\,dx= \frac{(p-1)NC_{1}}{2ps}. $$

(17)

Remark 2.1

For the constant $C_{\mathrm{opt}}$ in Gagliardo–Nirenberg–Sobolev inequality (11) with $\alpha =2p-2$, it can be evaluated by $w_{0}$

$$\begin{aligned} \frac{1}{C_{\mathrm{opt}}} &=\inf_{u\in H^{s}\setminus \{0\}} \frac{ (\int _{\mathbb{R}^{N}} \vert (-\Delta )^{\frac{s}{2}}u \vert ^{2}\,dx )^{\frac{(p-1)N}{2s}} (\int _{\mathbb{R}^{N}} \vert u \vert ^{2}\,dx )^{\frac{2ps-(p-1)N}{2s}}}{\int _{\mathbb{R}^{N}} \vert u \vert ^{2p}\,dx} \\ &= \frac{ (\int _{\mathbb{R}^{N}} \vert (-\Delta )^{\frac{s}{2}}w_{0} \vert ^{2}\,dx )^{\frac{(p-1)N}{2s}}C_{0}^{\frac{2ps-(p-1)N}{2s}}}{C_{1}} \\ &= \frac{ (\frac{(p-1)N}{2ps}C_{1} )^{\frac{(p-1)N}{2s}}C_{0}^{\frac{2ps-(p-1)N}{2s}}}{C_{1}} \\ &= \frac{C_{0}^{\frac{2ps-(p-1)N}{2s}}C_{1}^{\frac{(p-1)N-2s}{2s}}}{(\frac{2ps}{(p-1)N})^{\frac{(p-1)N}{2s}}}, \end{aligned}$$

which implies that

$$ C_{\mathrm{opt}}= \frac{(\frac{2ps}{(p-1)N})^{\frac{(p-1)N}{2s}}}{C_{0}^{\frac{2ps-(p-1)N}{2s}}C_{1}^{\frac{(p-1)N-2s}{2s}}}. $$

(18)

For $a,\mu >0$ fixed, we search for $(\lambda ,w)\in \mathbb{R}\times H^{s}(\mathbb{R}^{N})$, with $\lambda >0$ in $\mathbb{R}$, solving

$$ \textstyle\begin{cases} (-\Delta )^{s} w+\lambda w=\mu \vert w \vert ^{2p-2}w \quad \text{in } { \mathbb{R}^{N}}, \\ w>0 \quad \text{in } {\mathbb{R}^{N}}, \\ w(0)=\max w \quad \text{and}\quad \int _{\mathbb{R}^{N}} \vert w \vert ^{2}\,dx=a^{2}. \end{cases} $$

(19)

Solutions to (19) can be found as the critical points of $I_{\mu }:H^{s}(\mathbb{R}^{N})\rightarrow \mathbb{R}$, defined by

$$ I_{\mu }(w)= \int _{\mathbb{R}^{N}} \biggl(\frac{1}{2} \bigl\vert (-\Delta )^{\frac{s}{2}}w \bigr\vert ^{2}- \frac{\mu }{2p} \vert w \vert ^{2p} \biggr)\,dx, $$

(20)

constrained on the $L^{2}$-sphere $H_{a}:=\{u\in H^{s}(\mathbb{R}^{N}):\int _{\mathbb{R}^{N}}|u|^{2}=a^{2} \}$, and λ appears as the Lagrange multiplier. It is well known that it can be obtained from $w_{0}$ by scaling.

Lemma 2.3

Equation (19) has a unique positive solution $(\lambda _{a,\mu },w_{a,\mu })$ defined by

$$ \lambda _{a,\mu }:= \biggl[\frac{1}{\mu }\biggl( \frac{C_{0}}{a^{2}}\biggr)^{p-1} \biggr]^{\frac{2s}{(p-1)N-2s}},\qquad w_{a,\mu }:= \biggl( \frac{C_{0}^{2s}}{\mu ^{N} a^{4s}} \biggr)^{\frac{1}{2(p-1)N-4s}}w_{0} \bigl(\lambda _{a,\mu }^{\frac{1}{2s}}x \bigr). $$

Furthermore, $w_{a,\mu }$ satisfies

$$\begin{aligned}& \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} w_{a,\mu } \bigr\vert ^{2}\,dx= \frac{(p-1)N}{2ps} \frac{C_{1}C_{0}^{\frac{2ps-(p-1)N}{(p-1)N-2s}}}{\mu ^{\frac{2s}{(p-1)N-2s}}a^{\frac{4ps-2(p-1)N}{(p-1)N-2s}}}, \end{aligned}$$

(21)

$$\begin{aligned}& \int _{\mathbb{R}^{N}} \vert w_{a,\mu } \vert ^{2p}\,dx= \frac{C_{1}C_{0}^{\frac{2ps-(p-1)N}{(p-1)N-2s}}}{\mu ^{\frac{(p-1)N}{(p-1)N-2s}}a^{\frac{4ps-2(p-1)N}{(p-1)N-2s}}}, \end{aligned}$$

(22)

$$\begin{aligned}& I_{\mu }(w_{a,\mu })=\frac{(p-1)N-2s}{4ps} \frac{C_{1}C_{0}^{\frac{2ps-(p-1)N}{(p-1)N-2s}}}{\mu ^{\frac{2s}{(p-1)N-2s}}a^{\frac{4ps-2(p-1)N}{(p-1)N-2s}}}. \end{aligned}$$

(23)

Proof

We can directly check that $w_{a,\mu }$ satisfies the equation (19) with $\lambda =\lambda _{a,\mu }$ and $w_{a,\mu }$ is the unique positive radial solution of (19) by [13]. By direct calculation,

$$ \int _{\mathbb{R}^{N}} \vert w_{a,\mu } \vert ^{2p}\,dx= \frac{C_{0}^{\frac{2ps-(p-1)N}{(p-1)N-2s}}}{\mu ^{\frac{(p-1)N}{(p-1)N-2s}}a^{\frac{4ps-2(p-1)N}{(p-1)N-2s}}} \int _{\mathbb{R}^{N}} \vert w_{0} \vert ^{2p}\,dx, $$

we get (22). We have

$$ \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} w_{a,\mu } \bigr\vert ^{2}\,dx= \frac{C_{0}^{\frac{2ps-(p-1)N}{(p-1)N-2s}}}{\mu ^{\frac{2s}{(p-1)N-2s}}a^{\frac{4ps-2(p-1)N}{(p-1)N-2s}}} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} w_{0} \bigr\vert ^{2}\,dx $$

and combined with (17), we get (21). Combining (21) and (22), we obtain (23). □

Let us introduce the set

$$ \mathcal{P}(a,\mu ):= \biggl\{ w\in H_{a}: \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}}w \bigr\vert ^{2}\,dx=\frac{(p-1)N\mu }{2ps} \int _{\mathbb{R}^{N}} \vert w \vert ^{2p}\,dx \biggr\} . $$

(24)

When $1+\frac{2s}{N}< p<\frac{N}{N-2s}$, we have the following lemma.

Lemma 2.4

Assume that $1+\frac{2s}{N}< p<\frac{N}{N-2s}$, if w is a solution of (19), then $w\in \mathcal{P}(a,\mu )$. In addition the positive solution of (19) minimizes $I_{\mu }$ on $\mathcal{P}(a,\mu )$.

Proof

Let $(w,\lambda )\in H_{a}\times \mathbb{R}$ be a solution of (19). By the Pohozaev identity (14),

$$ (N-2s) \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}}w \bigr\vert ^{2}\,dx =2N \biggl(-\frac{\lambda }{2} \int _{\mathbb{R}^{N}} \vert w \vert ^{2}\,dx+ \frac{\mu }{2p} \int _{\mathbb{R}^{N}} \vert w \vert ^{2p}\,dx \biggr), $$

and combined with

$$ \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}}w \bigr\vert ^{2}\,dx+\lambda \int _{ \mathbb{R}^{N}} \vert w \vert ^{2}\,dx=\mu \int _{\mathbb{R}^{N}} \vert w \vert ^{2p}\,dx, $$

we get

$$ \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}}w \bigr\vert ^{2}\,dx= \frac{(p-1)N\mu }{2ps} \int _{\mathbb{R}^{N}} \vert w \vert ^{2p}\,dx, $$

(25)

thus, $w\in \mathcal{P}(a,\mu )$.

In the following, we prove that the positive solution $w_{a,\mu }$ of (19) minimizes $I_{\mu }$ on $\mathcal{P}(a,\mu )$. For any $u\in \mathcal{P}(a,\mu )$, by the Gagliardo–Nirenberg–Sobolev inequality (11) and the fact that $\|u\|_{L^{2}}=a$, we have

$$ \int _{\mathbb{R}^{N}} \vert u \vert ^{2p}\,dx\leqslant C_{\mathrm{opt}}a^{ \frac{2ps-(p-1)N}{s}} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{ \frac{s}{2}}u \bigr\vert ^{2}\,dx \biggr)^{\frac{(p-1)N}{2s}}. $$

(26)

Together with (24), we obtain

$$ \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}}u \bigr\vert ^{2}\,dx \biggr)^{\frac{(p-1)N-2s}{2s}}\geq \frac{2ps}{(p-1)N\mu C_{\mathrm{opt}}a^{\frac{2ps-(p-1)N}{s}}}. $$

Therefore, for any $u\in \mathcal{P}(a,\mu )$,

$$ \begin{aligned} I_{\mu }(u)&=\frac{(p-1)N-2s}{2(p-1)N}\int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{ \frac{s}{2}}u \bigr\vert ^{2}\,dx \\ &\geq \frac{(p-1)N-2s}{2(p-1)N} \biggl( \frac{2ps}{(p-1)N\mu C_{\mathrm{opt}}a^{\frac{2ps-(p-1)N}{s}}} \biggr)^{ \frac{2s}{(p-1)N-2s}}. \end{aligned} $$

(27)

It is clear that equality in (27) is obtained by $w_{a,\mu }$ due to the Pohozaev identity (14) and the fact that $C_{\mathrm{opt}}$ is achieved by $w_{a,\mu }$ (see [13]). Therefore

$$ I_{\mu }(w_{a,\mu })=\inf_{u\in \mathcal{P}(a,\mu )} I_{\mu }(u). $$

 □

Lemma 2.5

For $1+\frac{2s}{N}< p<\frac{N}{N-2s}$, let $u\in H_{a}$ be arbitrary but fixed. Define $(l\star u)(x):=e^{\frac{Nsl}{2}}u(e^{sl}x)$, then we have

• (i) $\|(-\Delta )^{\frac{s}{2}} (l\star u)\|_{L^{2}}\rightarrow 0$ and $I_{\mu }(l\star u)\rightarrow 0$ as $l\rightarrow -\infty$,

• (ii) $\|(-\Delta )^{\frac{s}{2}} (l\star u)\|_{L^{2}}\rightarrow +\infty$ and $I_{\mu }(l\star u)\rightarrow -\infty$ as $l\rightarrow +\infty$,

• (iii) $f_{u}(l)=I_{\mu }(l\star u)$ reaches its unique maximum value at $l(u)\in \mathbb{R}$ with $l(u)\star u\in \mathcal{P}(a,\mu )$.

Proof

By direct calculation, we have

$$ \Vert l\star u \Vert _{L^{2}}=a\quad \text{and} \quad \bigl\Vert (- \Delta )^{\frac{s}{2}} (l \star u) \bigr\Vert _{L^{2}}=e^{s^{2}l} \bigl\Vert (-\Delta )^{\frac{s}{2}} u \bigr\Vert _{L^{2}}, $$

thus, $\|(-\Delta )^{\frac{s}{2}} (l\star u)\|_{L^{2}}\rightarrow 0$ as $l\rightarrow -\infty$, and $\|(-\Delta )^{\frac{s}{2}} (l\star u)\|_{L^{2}}\rightarrow +\infty$ as $l\rightarrow +\infty$.

Now we compute $f_{u}(l)$,

$$ \begin{aligned} f_{u}(l)&=I_{\mu }(l\star u)=\int _{\mathbb{R}^{N}}\frac{1}{2} \bigl\vert (-\Delta )^{\frac{s}{2}}(l\star u) \bigr\vert ^{2}-\frac{\mu }{2p} \vert l\star u \vert ^{2p}\,dx \\ &=\frac{e^{2s^{2}l}}{2} \bigl\Vert (-\Delta )^{\frac{s}{2}}u \bigr\Vert ^{2}_{L^{2}}- \frac{e^{(p-1)Nsl}}{2p}\mu \Vert u \Vert ^{2p}_{L^{2p}}, \end{aligned} $$

(28)

thus, $I_{\mu }(l\star u)\rightarrow 0$ as $l\rightarrow -\infty$. Due to $p>1+\frac{2s}{N}$, we have $I_{\mu }(l\star u)\rightarrow -\infty$ as $l\rightarrow +\infty$. (i), (ii) are proved. To show the third claim, by (28), we have

$$ \begin{aligned} f'_{u}(l)&=s^{2}e^{2s^{2}l} \bigl\Vert (-\Delta )^{\frac{s}{2}}u \bigr\Vert ^{2}_{L^{2}}- \frac{(p-1)Ns\mu }{2p}{e^{(p-1)Nsl}} \Vert u \Vert _{L^{2p}}^{2p} \\ &=s^{2} \bigl\Vert (-\Delta )^{\frac{s}{2}}(l\star u) \bigr\Vert ^{2}_{L^{2}}- \frac{(p-1)Ns\mu }{2p} \Vert l\star u \Vert _{L^{2p}}^{2p}. \end{aligned} $$

(29)

Therefore $f'_{u}(l)=0$ is equivalent to

$$ e^{s[(p-1)N-2s]l}= \frac{ \Vert (-\Delta )^{\frac{s}{2}}u \Vert ^{2}_{L^{2}}}{\frac{(p-1)N\mu }{2ps} \Vert u \Vert _{L^{2p}}^{2p}}. $$

(30)

So there exists a unique $l_{0}\in \mathbb{R}$ such that $f'_{u}(l)|_{l=l_{0}}=0$ and $l_{0}\star u\in \mathcal{P}(a,\mu )$. Furthermore, we have

$$\begin{aligned} f''_{u}(l)|_{l=l_{0}}&= \biggl(2s^{4}e^{2s^{2}l} \bigl\Vert (-\Delta )^{\frac{s}{2}}u \bigr\Vert ^{2}_{L^{2}}- \frac{{(p-1)}^{2}N^{2}s^{2}\mu }{2p}{e^{(p-1)Nsl}} \Vert u \Vert _{L^{2p}}^{2p} \biggr)\bigg|_{l=l_{0}} \\ &= \bigl(2s-(p-1)N \bigr)\frac{e^{(p-1)Nsl_{0}}}{2p}s^{2}(p-1)N\mu \Vert u \Vert _{L^{2p}}^{2p} \\ &< 0. \end{aligned}$$

Note that

$$ f'_{u}(l)= \textstyle\begin{cases} >0 & \text{if } l< l_{0}, \\ =0 & \text{if } l=l_{0}, \\ < 0 & \text{if } l>l_{0}. \end{cases} $$

This implies that $f_{u}(l)$ gets its unique maximum value at $l_{0}(u)$. If $u\in \mathcal{P}(a,\mu )$, then, by (30), $l_{0}=0$. □

When $\mu _{0}=({C_{0}}/{a^{2}})^{p-1}$ in (19), by Lemma 2.3, $\lambda _{a,\mu _{0}}=1$, i.e., $w_{a,\mu _{0}}$ is the unique positive solution of the following equation:

$$ \textstyle\begin{cases} (-\Delta )^{s}{w}+w=\mu _{0} \vert w \vert ^{2p-2}w \quad \text{in } \mathbb{R}^{N}, \\ w(0)=\max {w},\quad \text{and}\quad \int _{\mathbb{R}^{N}}{ \vert w \vert ^{2}}\,dx=a^{2}, \end{cases} $$

(31)

and hence is a minimizer of $I_{\mu _{0}}$ on $\mathcal{P}(a,\mu _{0})$. Our next result shows that this level can also be characterized as the infimum of a Rayleigh-type quotient.

Lemma 2.6

$$ \inf_{u\in \mathcal{P}(a,\mu _{0})}I_{\mu _{0}}(u)=\inf _{u\in H_{a}} \mathcal{R}(u), $$

(32)

where

$$ \mathcal{R}(u):= \frac{R_{0}(\int _{\mathbb{R}^{N}} \vert (-\Delta )^{\frac{s}{2}} u \vert ^{2}\,dx)^{\frac{(p-1)N}{(p-1)N-2s}}}{(\mu _{0}\int _{\mathbb{R}^{N}} \vert u \vert ^{2p}\,dx)^{\frac{2s}{(p-1)N-2s}}}, $$

and $R_{0}$ is defined in (9).

Proof

If $u\in \mathcal{P}(a,\mu _{0})$, then

$$ \frac{2ps\int _{\mathbb{R}^{N}} \vert (-\Delta )^{\frac{s}{2}}u \vert ^{2}\,dx}{(p-1)N\mu _{0}\int _{\mathbb{R}^{N}} \vert u \vert ^{2p}\,dx}=1 \quad \text{and}\quad I_{\mu _{0}}(u)= \biggl(\frac{1}{2}-\frac{s}{(p-1)N}\biggr) \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}}u \bigr\vert ^{2}\,dx. $$

(33)

Therefore,

$$ \begin{aligned} I_{\mu _{0}}(u)&=\biggl(\frac{1}{2}-\frac{s}{(p-1)N}\biggr)\int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}}u \bigr\vert ^{2}\,dx \biggl( \frac{2ps\int _{\mathbb{R}^{N}} \vert (-\Delta )^{\frac{s}{2}}u \vert ^{2}\,dx}{(p-1)N\mu _{0}\int _{\mathbb{R}^{N}} \vert u \vert ^{2p}\,dx} \biggr)^{\frac{2s}{(p-1)N-2s}}, \\ &=\mathcal{R}(u), \end{aligned} $$

which proves that

$$ \inf_{u\in \mathcal{P}(a,\mu _{0})}I_{\mu _{0}}(u)\geqslant \inf _{u \in H_{a}}\mathcal{R}(u). $$

On the other hand, for all $l\in \mathbb{R}$ and $u\in H_{a}$, direct calculation shows that

$$ \mathcal{R}(u)=\mathcal{R}(l\star u). $$

By Lemma 2.5, we know that, for $u\in H_{a}$ arbitrary but fixed, there exists a unique $l_{0}(u)\in \mathbb{R}$ such that $l_{0}(u)\star u\in \mathcal{P}(a,\mu _{0})$, and $I_{\mu _{0}}(l\star u)$ reaches its unique maximum at $l_{0}(u)\star u$. Hence, for every $u\in H_{a}$, we have

$$ \mathcal{R}(u)=\mathcal{R}\bigl(l_{0}(u)\star u \bigr)=I_{\mu _{0}}\bigl(l_{0}(u) \star u\bigr)\geqslant \inf _{v\in \mathcal{P}(a,\mu _{0})}I_{\mu _{0}}(v), $$

which proves that

$$ \inf_{u\in \mathcal{P}(a,\mu _{0})}I_{\mu _{0}}(u)\leqslant \inf _{u \in H_{a}}\mathcal{R}(u). $$

 □

Next, we give a Liouville-type result for fractional Laplacian. A similar Liouville-type result for Laplacian can be found in [18].

Lemma 2.7

Let $u\in {H^{s}(\mathbb{R}^{N})}$ with $N>2s$,

• (i) If u satisfies

  $$ \textstyle\begin{cases} (-\Delta )^{s}{u}\geq 0\quad \textit{in } {\mathbb{R}^{N}}, \\ u\in {L^{q}\bigl({\mathbb{R}^{N}}\bigr)},\quad q\in {(0, \frac{N}{N-2s}]}, \\ u\geq 0, \end{cases} $$

  then $u\equiv 0$.

• (ii) If u satisfies

  $$ \textstyle\begin{cases} (-\Delta )^{s}{u}\geq {u^{q}}\quad \textit{in } {\mathbb{R}^{N}}, \\ u\geq 0, \quad \textit{and}\quad q\in (1,\frac{N}{N-2s}], \end{cases} $$

  then $u\equiv 0$.

Proof

We prove (i) by contradiction. If $u\not \equiv 0$, by the maximum principle, we have $u>0$ in ${\mathbb{R}^{N}}$. Let $v(x)=\frac{1}{|x|^{N-2s}}u(\frac{x}{|x|^{2}})$, then $v(x)>0$ in $\mathbb{R}^{N}\setminus \{0\}$, and $v(x)$ satisfies

$$ (-\Delta )^{s}{v(x)}=\frac{1}{ \vert x \vert ^{N+2s}}(-\Delta )^{s}{u\biggl( \frac{x}{ \vert x \vert ^{2}}\biggr)}\quad \text{in } { \mathbb{R}^{N}}\setminus \{0\}, $$

so $(-\Delta )^{s}{v}\geq 0$ in the distribution sense. Since $u\in H^{s}(\mathbb{R}^{N})\subset {L_{2s}}(\mathbb{R}^{N})$, where

$$ L_{2s}\bigl(\mathbb{R}^{N}\bigr)=\biggl\{ w(x):{ \mathbb{R}^{N}}\rightarrow {\mathbb{R}} \Bigm| \int _{\mathbb{R}^{N}}\frac{ \vert w(x) \vert }{1+{ \vert x \vert ^{N+2s}}}\,dx< +\infty \biggr\} , $$

we can see that $v\in L_{2s}(\mathbb{R}^{N})$. By Theorem 1 in [19], there exists a constant $C>0$ such that

$$ \inf_{|x|< \frac{1}{2}}v(x)\geq {C}. $$

Therefore, we obtain

$$ u(x)\geq \frac{C}{ \vert x \vert ^{N-2s}},\quad \vert x \vert >2. $$

For $q\in (0,\frac{N}{N-2s}]$, we can compute

$$ \int _{\mathbb{R}^{N}}u^{q}\,dx\geq C \int _{ \vert x \vert >2} \frac{1}{ \vert x \vert ^{(N-2s)q}}\,dx\geq C \int _{ \vert x \vert >2}\frac{1}{ \vert x \vert ^{N}}\,dx=+ \infty , $$

which is a contradiction to $u\in L^{q}({\mathbb{R}^{N}})$. So $u\equiv 0$.

To prove (ii), let φ be the first eigenfunction of

$$ \textstyle\begin{cases} (-\Delta )^{s}{\varphi }={ \lambda _{1}} {\varphi }\quad \text{in } {B_{1}(0)}, \\ {\varphi }\equiv 0 \quad \text{in } {B^{c}_{1}(0)}, \end{cases} $$

where $B_{1}(0)$ is the unit ball in ${\mathbb{R}^{N}}$, $\varphi >0$ in $B_{1}(0)$ and $\lambda _{1}>0$ is the first eigenvalue of $(-\Delta )^{s}$ in $B_{1}(0)$. For any $R>0$ but fixed, let $\varphi _{R}(x)=\varphi (\frac{x}{R})$, then

$$ \textstyle\begin{cases} (-\Delta )^{s}{\varphi _{R}}=R^{-2s}\lambda _{1}{\varphi _{R}}\quad \text{in } B_{R}(0), \\ {\varphi _{R}}\equiv 0\quad \text{in } B^{c}_{R}(0). \end{cases} $$

We can compute

$$\begin{aligned} \begin{aligned} \int _{B_{R}(0)}u^{q}{\varphi _{R}}\,dx&= \int _{\mathbb{R}^{N}}u^{q}{ \varphi _{R}}\,dx \leq \int _{\mathbb{R}^{N}}(-\Delta )^{s}u{\varphi _{R}}\,dx \\ &= \int _{\mathbb{R}^{N}}(-\Delta )^{s}{\varphi _{R}}u\,dx= \int _{B_{R}(0)}uR^{-2s} \lambda _{1}{ \varphi _{R}}\,dx+ \int _{B^{c}_{R}(0)}(-\Delta )^{s}{ \varphi _{R}}u\,dx \\ &\leq \int _{B_{R}(0)}uR^{-2s}\lambda _{1}{ \varphi _{R}}\,dx\leq {R^{-2s} \lambda _{1}} \biggl( \int _{B_{R}(0)}u^{q}{\varphi _{R}}\,dx \biggr)^{ \frac{1}{q}} \biggl( \int _{B_{R}(0)}{\varphi _{R}}\,dx \biggr)^{1- \frac{1}{q}}, \end{aligned} \end{aligned}$$

in the above, we have use the fact that $(-\Delta )^{s}\varphi _{R}<0$ in $B_{R}^{c}(0)$. Therefore

$$ \int _{B_{R}(0)}u^{q}{\varphi _{R}}\,dx \leq {C}R^{-\frac{2sq}{q-1}} \int _{ \mathbb{R}^{N}}\varphi _{R}\,dx\le CR^{\frac{N(q-1)-2sq}{q-1}}. $$

(34)

When $q\in (1,\frac{N}{N-2s})$, we have

$$ \min_{B_{\frac{1}{2}}(0)}\varphi \cdot \int _{B_{\frac{R}{2}}(0)}u^{q}\,dx \leq \int _{\mathbb{R}^{N}}u^{q}{\varphi _{R}}\,dx \leq {C}R^{ \frac{N(q-1)-2sq}{q-1}}\rightarrow 0,\quad \text{as } R\rightarrow \infty . $$

So we have $u\equiv 0$.

When $q=\frac{N}{N-2s}$, we have

$$ \min_{B_{\frac{1}{2}}(0)}\varphi \cdot \int _{B_{\frac{R}{2}}(0)}u^{q}\,dx \leq \int _{\mathbb{R}^{N}}u^{q}{\varphi _{R}}\,dx \leq {C} \quad \text{for all $R>0$}, $$

with C independent of R by (34), so $u\in L^{q}(\mathbb{R}^{N})$. By (i), we obtain $u\equiv 0$. □

3 Proof of Theorem 1.1

In this section, we give the proof of Theorem 1.1. We work in a radial setting. That is, we find the critical point of the functional E constrained on $H^{\mathrm{rad}}_{a_{1}}\times {H^{\mathrm{rad}}_{a_{2}}}$, where for any $a>0$, we define

$$ H^{\mathrm{rad}}_{a}:=H_{a}\cap H_{r}^{s} \bigl(\mathbb{R}^{N}\bigr), $$

and $H^{s}_{r}(\mathbb{R}^{N})$ is the subset of $H^{s}(\mathbb{R}^{N})$ containing all the functions which are radial with respect to the origin. We know that $H^{s}_{r}(\mathbb{R}^{N})\hookrightarrow L^{p}(\mathbb{R}^{N})$ is compact when $2< p<\frac{2N}{N-2s}$. Due to the Palais principle of symmetric criticality, the critical points of E constrained on $H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}$ are true critical points of E constrained in the full product $H_{a_{1}}\times H_{a_{2}}$.

For $a_{1}$, $a_{2}$, $\mu _{1}$ and $\mu _{2}>0$, let $\beta _{1}>0$ be defined by (5).

Lemma 3.1

For $0<\beta <\beta _{1}$,

$$ \inf \bigl\{ E(u_{1},u_{2}):(u_{1},u_{2}) \in \mathcal{P}(a_{1},\mu _{1}+ \beta )\times \mathcal{P}(a_{2},\mu _{2}+\beta )\bigr\} >\max \bigl\{ I_{\mu _{1}}(w_{a_{1}, \mu _{1}}),I_{\mu _{2}}(w_{a_{2},\mu _{2}}) \bigr\} , $$

where $I_{\mu _{i}}(w_{a_{i},\mu _{1}})$, $i=1,2$ is defined by (23).

Proof

For $(u_{1},u_{2})\in \mathcal{P}(a_{1},\mu _{1}+\beta )\times \mathcal{P}(a_{2},\mu _{2}+\beta )\}$, we have

$$\begin{aligned} E(u_{1},u_{2})&= \int _{\mathbb{R}^{N}} \biggl(\frac{1}{2} \bigl\vert (-\Delta )^{\frac{s}{2}}u_{1} \bigr\vert ^{2}- \frac{\mu _{1}}{2p} \vert u_{1} \vert ^{2p} \biggr)\,dx \\ &\quad {}+ \int _{\mathbb{R}^{N}} \biggl(\frac{1}{2} \bigl\vert (-\Delta )^{\frac{s}{2}}u_{2} \bigr\vert ^{2}- \frac{\mu _{2}}{2p} \vert u_{2} \vert ^{2p} \biggr)\,dx-\frac{\beta }{p} \int _{ \mathbb{R}^{N}} \vert u_{1} \vert ^{p} \vert u_{2} \vert ^{p}\,dx \\ &\geqslant I_{\mu _{1}}(u_{1})+I_{\mu _{2}}(u_{2})- \frac{\beta }{2p} \int _{\mathbb{R}^{N}} \vert u_{1} \vert ^{2p}\,dx-\frac{\beta }{2p} \int _{ \mathbb{R}^{N}} \vert u_{2} \vert ^{2p}\,dx \\ &=I_{\mu _{1}+\beta }(u_{1})+I_{\mu _{2}+\beta }(u_{2}) \\ &\geqslant \inf_{u\in \mathcal{P}(a_{1},\mu _{1}+\beta )}I_{\mu _{1}+ \beta }(u)+\inf _{v\in \mathcal{P}(a_{1},\mu _{1}+\beta )}I_{\mu _{2}+ \beta }(v) \\ &=I_{\mu _{1}+\beta }(w_{a_{1},\mu _{1}+\beta })+I_{\mu _{2}+\beta }(w_{a_{2}, \mu _{2}+\beta }), \end{aligned}$$

by Lemma 2.4. From (23) and (5), it is easy to get, when $0<\beta <\beta _{1}$,

$$\begin{aligned} &\max \bigl\{ I_{\mu _{1}}(w_{a_{1},\mu _{1}}),I_{\mu _{2}}(w_{a_{2},\mu _{2}}) \bigr\} \\ &\quad =\max \biggl\{ \frac{(p-1)N-2s}{4ps} \frac{C_{1}C_{0}^{\frac{2ps-(p-1)N}{(p-1)N-2s}}}{{\mu _{1}}^{\frac{2s}{(p-1)N-2s}}a_{1}^{\frac{4ps-2(p-1)N}{(p-1)N-2s}}} , \frac{(p-1)N-2s}{4ps} \frac{C_{1}C_{0}^{\frac{2ps-(p-1)N}{(p-1)N-2s}}}{{\mu _{2}} ^{\frac{2s}{(p-1)N-2s}}a_{2}^{\frac{4ps-2(p-1)N}{(p-1)N-2s}}} \biggr\} \\ &\quad < I_{\mu _{1}+\beta }(w_{a_{1},\mu _{1}+\beta })+I_{\mu _{2}+\beta }(w_{a_{2}, \mu _{2}+\beta }). \end{aligned}$$

Therefore,

$$\begin{aligned} &\inf \bigl\{ E(u_{1},u_{2}):(u_{1},u_{2}) \in \mathcal{P}(a_{1},\mu _{1}+ \beta )\times \mathcal{P}(a_{2},\mu _{2}+\beta )\bigr\} \\ &\quad >\max \bigl\{ I_{\mu _{1}}(w_{a_{1},\mu _{1}}),I_{\mu _{2}}(w_{a_{2},\mu _{2}}) \bigr\} . \end{aligned}$$

 □

Now we fix $0<\beta <\beta _{1}$ and choose $\varepsilon >0$ such that

$$\begin{aligned} \begin{aligned} &\inf \bigl\{ E(u_{1},u_{2}):(u_{1},u_{2}) \in \mathcal{P}(a_{1},\mu _{1}+ \beta )\times \mathcal{P}(a_{2},\mu _{2}+\beta )\bigr\} \\ &\quad >\max \bigl\{ I_{\mu _{1}}(w_{a_{1},\mu _{1}}),I_{\mu _{2}}(w_{a_{2},\mu _{2}}) \bigr\} +\varepsilon . \end{aligned} \end{aligned}$$

(35)

Denote

$$ w_{1}:=w_{a_{1},\mu _{1}+\beta }\quad \text{and}\quad w_{2}:=w_{a_{2}, \mu _{2}+\beta }, $$

(36)

and, for $i=1,2$,

$$ \varphi _{i}(l):=I_{\mu _{i}}(l\star w_{i})\quad \text{and}\quad \tilde{\varphi }_{i}(l):= \frac{\partial }{\partial l}I_{\mu _{i}+\beta }(l \star w_{i}). $$

(37)

Lemma 3.2

For $i=1,2$, there exist $\rho _{i}<0$ and $R_{i}>0$, depending on ε and β, such that

• (i) $0<\varphi _{i}(\rho _{i})<\varepsilon$ and $\varphi _{i}(R_{i})\leqslant 0$;

• (ii) $\tilde{\varphi }_{i}(l)>0$ for any $l<0$, $\tilde{\varphi }_{i}(0)=0$ and $\tilde{\varphi }_{i}(l)<0$ for any $l>0$. In particular, $\tilde{\varphi }_{i}(\rho _{i})>0$ and $\tilde{\varphi }_{i}(R_{i})<0$.

Proof

By Lemma 2.4 and Lemma 2.5, we have

$$\begin{aligned} \varphi _{i}(l)&= \int _{\mathbb{R}^{N}} \biggl(\frac{1}{2} \bigl\vert (-\Delta )^{\frac{s}{2}}(l\star w_{i}) \bigr\vert ^{2}- \frac{\mu _{i}}{2p} \vert l\star w_{i} \vert ^{2p} \biggr)\,dx \\ &=\frac{e^{2s^{2}l}}{2} \bigl\Vert (-\Delta )^{\frac{s}{2}}w_{i} \bigr\Vert ^{2}_{L^{2}}- \frac{e^{(p-1)Nsl}}{2p}\mu _{i} \Vert w_{i} \Vert ^{2p}_{L^{2p}} \\ &= \biggl(\frac{(p-1)N(\mu _{i}+\beta )}{2ps}\frac{e^{2s^{2}l}}{2}- \frac{e^{(p-1)Nsl}}{2p}\mu _{i} \biggr) \Vert w_{i} \Vert ^{2p}_{L^{2p}}, \end{aligned}$$

thus, $\varphi _{i}(l)\rightarrow 0^{+}$ as $l\rightarrow -\infty$, and $\varphi _{i}(l)\rightarrow -\infty$ as $l\rightarrow +\infty$. Therefore, there exist $\rho _{i}<0$ and $R_{i}>0$, such that $0<\varphi _{i}(\rho _{i})<\varepsilon$ and $\varphi _{i}(R_{i})\leqslant 0$.

$$\begin{aligned} \tilde{\varphi }_{i}(l)&=s^{2}e^{2s^{2}l} \bigl\Vert (-\Delta )^{\frac{s}{2}}w_{i} \bigr\Vert ^{2}_{L^{2}}-\frac{e^{(p-1)Nsl}(p-1)N}{2p}s(\mu _{i}+ \beta ) \int _{ \mathbb{R}^{N}} \vert w_{i} \vert ^{2p}\,dx \\ &= \biggl(\frac{(p-1)N(\mu _{i}+\beta )}{2ps}s^{2}e^{2s^{2}l}- \frac{e^{(p-1)Nsl}(p-1)N}{2p}s(\mu _{i}+\beta ) \biggr) \int _{ \mathbb{R}^{N}} \vert w_{i} \vert ^{2p}\,dx \\ &=\frac{(p-1)N(\mu _{i}+\beta )}{2p}se^{(p-1)Nsl} \bigl(e^{(2s-(p-1)N)sl}-1 \bigr) \int _{\mathbb{R}^{N}} \vert w_{i} \vert ^{2p}\,dx, \end{aligned}$$

then

$$\begin{aligned} \tilde{\varphi }_{i}(l)= \textstyle\begin{cases} >0 & \text{if } l< 0, \\ =0 & \text{if } l=0, \\ < 0 & \text{if } l>0, \end{cases}\displaystyle \end{aligned}$$

(38)

which implies that (ii) holds. □

Let $Q:=[\rho _{1},R_{1}]\times [\rho _{2},R_{2}]$, and let

$$ \gamma _{0}(t_{1},t_{2}):=(t_{1} \star w_{1},t_{2}\star w_{2})\in H^{\mathrm{rad}}_{a_{1}} \times H^{\mathrm{rad}}_{a_{2}},\quad \forall (t_{1},t_{2})\in Q. $$

We introduce the minimax class

$$ \Gamma :=\bigl\{ \gamma \in C\bigl(Q, H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}\bigr): \gamma =\gamma _{0} \text{ on } \partial Q\bigr\} . $$

Lemma 3.3

We have

$$ \sup_{\partial Q}E(\gamma _{0})\leqslant \max \bigl\{ I_{\mu _{1}}(w_{a_{1}, \mu _{1}}),I_{\mu _{2}}(w_{a_{2},\mu _{2}}) \bigr\} +\varepsilon . $$

Proof

For every $(u_{1},u_{2})\in H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}$, we have

$$ E(u_{1},u_{2})=I_{\mu _{1}}(u_{1})+I_{\mu _{2}}(u_{2})- \frac{\beta }{p} \int _{\mathbb{R}^{N}} \vert u_{1} \vert ^{p} \vert u_{2} \vert ^{p}\,dx \leq I_{ \mu _{1}}(u_{1})+I_{\mu _{2}}(u_{2}). $$

Then, from Lemma 3.2,

$$\begin{aligned} \begin{aligned} E(t_{1}\star w_{1},\rho _{2}\star w_{2}) &\leqslant I_{\mu _{1}}(t_{1} \star w_{1})+I_{\mu _{2}}(\rho _{2}\star w_{2}) \\ &\leqslant I_{\mu _{1}}(t_{1}\star w_{1})+ \varepsilon \\ &\leqslant \sup_{l\in \mathbb{R}}I_{\mu _{1}}(l\star w_{1})+ \varepsilon . \end{aligned} \end{aligned}$$

By Lemma 2.3, we have

$$ w_{a_{i},\mu _{i}}=\bar{l}_{i}\star w_{i}, \quad \text{for } e^{\bar{l}_{i}}:=\biggl( \frac{\mu _{i}+\beta }{\mu _{i}} \biggr)^{\frac{1}{s[(p-1)N-2s]}}. $$

Then, due to $l_{1}\star (l_{2}\star w)=(l_{1}+l_{2})\star w$ for every $l_{1},l_{2}\in \mathbb{R}$ and $w\in H^{s}(\mathbb{R})$, we have

$$ \sup_{l\in \mathbb{R}}I_{\mu _{1}}(l\star w_{1})= \sup_{l\in \mathbb{R}}I_{\mu _{1}}(l\star w_{a_{1},\mu _{1}}). $$

As a consequence of Lemma 2.5,

$$ \sup_{l\in \mathbb{R}}I_{\mu _{1}}(l\star w_{a_{1},\mu _{1}})=I_{ \mu _{1}}(w_{a_{1},\mu _{1}}). $$

Therefore, we have

$$ E(t_{1}\star w_{1},\rho _{2}\star w_{2})\leqslant I_{\mu _{1}}(w_{a_{1}, \mu _{1}})+\varepsilon , \quad \forall t_{1}\in [\rho _{1},R_{1}]. $$

Similarly, we have

$$\begin{aligned}& E(\rho _{1}\star w_{1},t_{2}\star w_{2})\leqslant I_{\mu _{2}}(w_{a_{2}, \mu _{2}})+\varepsilon ,\quad \forall t_{2}\in [\rho _{2},R_{2}], \\& \begin{aligned} E(t_{1}\star w_{1},R_{2}\star w_{2}) &\leqslant I_{\mu _{1}}(t_{1} \star w_{1})+I_{\mu _{2}}(R_{2}\star w_{2}) \\ &\leqslant \sup_{l\in \mathbb{R}} I_{\mu _{1}}(l\star w_{1})=I_{\mu _{1}}(w_{a_{1}, \mu _{1}}), \quad \forall t_{1}\in [\rho _{1},R_{1}], \end{aligned} \end{aligned}$$

and

$$ E(R_{1}\star w_{1},t_{2}\star w_{2})\leqslant I_{\mu _{2}}(w_{a_{2}, \mu _{2}}), \quad \forall t_{2}\in [\rho _{2},R_{2}]. $$

Hence, the conclusion of Lemma 3.3 holds. □

Lemma 3.4

For every $\gamma \in \Gamma$, there exists $(t_{1,\gamma },t_{2,\gamma })\in Q$ such that $\gamma (t_{1,\gamma },t_{2,\gamma })\in \mathcal{P}(a_{1},\mu _{1}+ \beta )\times \mathcal{P}(a_{2},\mu _{2}+\beta )$.

Proof

For $\gamma \in \Gamma$, we use the notation $\gamma (t_{1},t_{2})=(\gamma _{1}(t_{1},t_{2}),\gamma _{2}(t_{1},t_{2})) \in H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}$. Considering the map $F_{\gamma }:Q\rightarrow \mathbb{R}^{2}$ defined by

$$ F_{\gamma }(t_{1},t_{2}):= \biggl( \frac{\partial }{\partial l}I_{\mu _{1}+ \beta }\bigl(l\star \gamma _{1}(t_{1},t_{2}) \bigr)\big|_{l=0}, \frac{\partial }{\partial l}I_{\mu _{2}+\beta }\bigl(l\star \gamma _{2}(t_{1},t_{2})\bigr)\big|_{l=0} \biggr). $$

From

$$\begin{aligned} &\frac{\partial }{\partial l}I_{\mu _{i}+\beta }\bigl(l\star \gamma _{i}(t_{1},t_{2})\bigr)\big|_{l=0} \\ &\quad =\frac{\partial }{\partial l} \biggl(\frac{e^{2s^{2}l}}{2} \bigl\Vert (-\Delta )^{\frac{s}{2}}\gamma _{i}(t_{1},t_{2}) \bigr\Vert ^{2}_{L^{2}}- \frac{e^{(p-1)Nsl}}{2p}(\mu _{i}+\beta ) \bigl\Vert \gamma _{i}(t_{1},t_{2}) \bigr\Vert ^{2p}_{L^{2p}} \biggr)\bigg|_{l=0} \\ &\quad =s^{2} \bigl\Vert (-\Delta )^{\frac{s}{2}}\gamma _{i}(t_{1},t_{2}) \bigr\Vert ^{2}_{L^{2}}- \frac{(p-1)Ns}{2p}(\mu _{i}+ \beta ) \bigl\Vert \gamma _{i}(t_{1},t_{2}) \bigr\Vert ^{2p}_{L^{2p}}, \end{aligned}$$

we deduce that

$$ F_{\gamma }(t_{1},t_{2})=(0,0) \quad \text{if and only if} \quad \gamma (t_{1},t_{2}) \in \mathcal{P}(a_{1},\mu _{1}+\beta )\times \mathcal{P}(a_{2},\mu _{2}+ \beta ). $$

Now, we will show that $F_{\gamma }(t_{1},t_{2})=(0,0)$ has a solution in Q for every $\gamma \in \Gamma$. Since

$$\begin{aligned} F_{\gamma _{0}}(t_{1},t_{2})={}& \biggl(s^{2}e^{2s^{2}t_{1}} \bigl\Vert (-\Delta )^{\frac{s}{2}} w_{1} \bigr\Vert ^{2}_{L^{2}}- \frac{(p-1)Ns}{2p}e^{(p-1)Nst_{1}}( \mu _{1}+\beta ) \Vert w_{1} \Vert ^{2p}_{L^{2p}}, \\ & s^{2}e^{2s^{2}t_{2}} \bigl\Vert (-\Delta )^{\frac{s}{2}}w_{2} \bigr\Vert ^{2}_{L^{2}}- \frac{(p-1)Ns}{2p}e^{(p-1)Nst_{2}}( \mu _{2}+\beta ) \Vert w_{2} \Vert ^{2p}_{L^{2p}} \biggr) \\ ={}&\bigl(\tilde{\varphi }_{1}(t_{1}),\tilde{\varphi }_{2}(t_{2})\bigr). \end{aligned}$$

By Lemma 3.2, we get $(0,0)\notin F_{\gamma _{0}}(\partial Q)$, and $(0,0)$ is the only solution to $F_{\gamma _{0}}(t_{1},t_{2})=(0,0)$ in Q. It is easy to compute

$$ \deg \bigl(F_{\gamma _{0}},Q,(0,0)\bigr)=\operatorname{sgn}\bigl(\tilde{ \varphi }'_{1}(0)\cdot \tilde{\varphi }'_{2}(0)\bigr)=1. $$

Now, for any $\gamma \in \Gamma$, since $F_{\gamma }(\partial ^{+}Q)=F_{\gamma _{0}}(\partial ^{+}Q)$, therefore, $(0,0)\notin F_{\gamma }(\partial Q)$, we get

$$ \deg \bigl(F_{\gamma },Q,(0,0)\bigr)=\deg \bigl(F_{\gamma _{0}},Q,(0,0) \bigr)=1. $$

Hence, there exists a $(t_{1,\gamma },t_{2,\gamma })\in Q$ such that $F_{\gamma }(t_{1,\gamma },t_{2,\gamma })=(0,0)$. □

Lemma 3.5

There exists a bounded Palais–Smale sequence $(u_{n},v_{n})$ for E on ${H}^{\mathrm{rad}}_{a_{1}}\times {H}^{\mathrm{rad}}_{a_{2}}$ at the level

$$ c:=\inf_{\gamma \in \Gamma } \max_{(t_{1},t_{2})\in Q} E\bigl(\gamma (t_{1},t_{2})\bigr)> \max \bigl\{ I_{\mu _{1}}(w_{a_{1},\mu _{1}}), I_{\mu _{2}}(w_{a_{2},\mu _{2}}) \bigr\} , $$

(39)

satisfying the additional condition

$$ G(u_{n},v_{n})=o(1), $$

(40)

where $o(1)\rightarrow 0$ as $n\rightarrow \infty$. Furthermore, there exists $\bar{C}>0$ such that

$$ \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u_{n} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}v_{n} \bigr\vert ^{2}\bigr)\,dx\geqslant \bar{C} \quad \textit{for all } n, $$

and $u_{n}^{-},v_{n}^{-}\rightarrow 0$ a.e. in $\mathbb{R}^{N}$ as $n\rightarrow \infty$.

Proof

The idea comes from [3]. Equation (39) is simply from Lemma 3.4. We consider the augmented functional $\tilde{E}:\mathbb{R}\times H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}} \rightarrow \mathbb{R}$ defined by $\tilde{E}(l,u_{1},u_{2}):=E(l\star u_{1},l\star u_{2})$. Let

$$\begin{aligned}& \tilde{\gamma }(t_{1},t_{2}):=\bigl(l(t_{1},t_{2}), \gamma _{1}(t_{1},t_{2}), \gamma _{2}(t_{1},t_{2})\bigr), \\& \tilde{\gamma }_{0}(t_{1},t_{2}):=\bigl(0, \gamma _{0}(t_{1},t_{2}) \bigr)=(0,t_{1} \star w_{1},t_{2}\star w_{2}), \\& \tilde{\Gamma }:=\bigl\{ \tilde{\gamma }\in C\bigl(Q,\mathbb{R}\times H^{\mathrm{rad}}_{a_{1}} \times H^{\mathrm{rad}}_{a_{2}}: \tilde{\gamma }=\tilde{\gamma }_{0} \text{ on } \partial Q \bigr)\bigr\} , \end{aligned}$$

and

$$ \tilde{c}:=\inf_{\tilde{\gamma }\in \tilde{\Gamma }} \max_{(t_{1},t_{2}) \in Q} \tilde{E}\bigl(\tilde{\gamma }(t_{1},t_{2})\bigr). $$

Since, for any $\gamma (t_{1},t_{2})=(\gamma _{1}(t_{1},t_{2}),\gamma _{2}(t_{1},t_{2})) \in \Gamma$, $(0,\gamma _{1}(t_{1},t_{2}),\gamma _{2}(t_{1},t_{2}))\in \tilde{\Gamma }$, we have $\tilde{c}\leqslant c$. On the other hand, for any $\tilde{\gamma }\in \tilde{\Gamma }$ and $(t_{1},t_{2})\in Q$, we have

$$ \tilde{E}\bigl(\tilde{\gamma }(t_{1},t_{2})\bigr)=E \bigl(l(t_{1},t_{2})\star \gamma _{1}(t_{1},t_{2}),l(t_{1},t_{2}) \star \gamma _{2}(t_{1},t_{2})\bigr), $$

and $(l(\cdot )\star \gamma _{1}(\cdot ),l(\cdot )\star \gamma _{2}( \cdot ))\in \Gamma$ due to $\tilde{\gamma }=\tilde{\gamma }_{0}$ on ∂Q, so $c\leqslant \tilde{c}$. Hence, $c=\tilde{c}$.

Now take a sequence of $\{\tilde{\gamma }_{n}\}\subset \tilde{\Gamma }$ such that

$$ \lim_{n\to +\infty }\max_{(t_{1},t_{2})\in Q} \tilde{E}\bigl( \tilde{\gamma }_{n}(t_{1},t_{2})\bigr)= \tilde{c}=c. $$

We may also assume that $\tilde{\gamma }_{n}=(l_{n},\gamma _{1,n},\gamma _{2,n})$ satisfies the following two additional properties: for all $(t_{1},t_{2})\in Q$:

• $l_{n}(t_{1},t_{2})\equiv 0$,

• $\gamma _{1,n}(t_{1},t_{2})\ge 0$, $\gamma _{2,n}(t_{1},t_{2})\ge 0$, a.e. in $\mathbb{R}^{N}$.

The first property comes from the fact that

$$\begin{aligned} \tilde{E}\bigl(\tilde{\gamma }(t_{1},t_{2})\bigr) &=E \bigl(l(t_{1},t_{2})\star \gamma _{1}(t_{1},t_{2}),l(t_{1},t_{2}) \star \gamma _{2}(t_{1},t_{2})\bigr) \\ &=\tilde{E}\bigl(0,l(t_{1},t_{2})\star \gamma _{1}(t_{1},t_{2}),l(t_{1},t_{2}) \star \gamma _{2}(t_{1},t_{2})\bigr), \end{aligned}$$

and the second one is the consequence of $\tilde{E}(l,|u|,|v|)\le \tilde{E}(l,u,v)$ and the definition of c̃.

Applying Theorem 3.2 in [20], there exists a Palais–Smale sequence $(l_{n},u_{n},v_{n})$ for Ẽ on $\mathbb{R}\times H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}$ at level c̃, such that

• $\lim_{n\to +\infty }\tilde{E}(l_{n},u_{n},v_{n})=\tilde{c}=c$,

• $\lim_{n\to +\infty }|l_{n}|+\operatorname{dist}((u_{n},v_{n}), \tilde{\gamma }_{n}(Q))=0$,

• For all $u,v\in H_{r}^{s}(\mathbb{R}^{N})$ with $\int _{\mathbb{R}^{N}}u_{n}u\,dx=0$, $\int _{\mathbb{R}^{N}}v_{n}v\,dx=0$ and $\forall l\in \mathbb{R}$,

  $$ \bigl\langle \tilde{E}'(l_{n},u_{n},v_{n}),(l,u,v) \bigr\rangle =o(1) \bigl( \vert l \vert + \Vert u \Vert _{H^{s}}+ \Vert v \Vert _{H^{s}}\bigr). $$

Take $(l,u,v)=(1,0,0)$, direct calculations gives

$$\begin{aligned} \begin{aligned} &\bigl\langle \tilde{E}'(l_{n},u_{n},v_{n}),(1,0,0) \bigr\rangle \\ &\quad =s^{2}e^{2s^{2}l_{n}} \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u_{n} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}v_{n} \bigr\vert ^{2}\bigr)\,dx \\ & \qquad {}-\frac{e^{(p-1)Nsl_{n}}(p-1)Ns}{2p} \int _{\mathbb{R}^{N}}\bigl(\mu _{1} \vert u_{n} \vert ^{2p}+2 \beta \vert u_{n} \vert ^{p} \vert v_{n} \vert ^{p}+\mu _{2} \vert v_{n} \vert ^{2p}\bigr)\,dx. \end{aligned} \end{aligned}$$

(41)

From the above, we can get

$$\begin{aligned} &se^{2s^{2}l_{n}}\biggl(\frac{(p-1)N}{2}-s\biggr) \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u_{n} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}v_{n} \bigr\vert ^{2}\bigr)\,dx \\ &\quad =(p-1)Ns\tilde{E}(l_{n},u_{n},v_{n})- \bigl\langle \tilde{E}'(l_{n},u_{n},v_{n}),(1,0,0) \bigr\rangle \\ &\quad \to (p-1)Nsc, \quad \text{as } n\to +\infty . \end{aligned}$$

Since $l_{n}\to 0$ and $p>1+\frac{2s}{N}$, we see that there exist $\bar{C}>0$ and $C>0$, such that

$$ \bar{C}\le \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u_{n} \bigr\vert ^{2}+ \bigl\vert (- \Delta )^{\frac{s}{2}}v_{n} \bigr\vert ^{2}\bigr)\,dx\le C, $$

(42)

therefore $(u_{n},v_{n})$ is bounded in $H^{s}_{r}(\mathbb{R}^{N})\times H^{s}_{r}(\mathbb{R}^{N})$. Using $l_{n}\to 0$ and (41) again, we conclude that $(u_{n},v_{n})$ satisfies (40). Now take $(l,u,v)=(0,u,v)$ for any $(u,v)\in H^{s}_{r}(\mathbb{R}^{N})\times H^{s}_{r}(\mathbb{R}^{N})$ with $\int _{\mathbb{R}^{N}}u_{n}u\,dx=0$, $\int _{\mathbb{R}^{N}}v_{n}v\,dx=0$, due to the boundedness of $(u_{n},v_{n})$ and $l_{n}\to 0$, it is easy to see that

$$\begin{aligned} \bigl\langle E'(u_{n},v_{n}),(u,v)\bigr\rangle &=\bigl\langle \tilde{E}'(l_{n},u_{n},v_{n}),(0,u,v) \bigr\rangle +O\bigl( \vert l_{n} \vert \bigr) \bigl( \Vert u \Vert _{H^{s}}+ \Vert v \Vert _{H^{s}}\bigr) \\ &=o(1) \bigl( \Vert u \Vert _{H^{s}}+ \Vert v \Vert _{H^{s}}\bigr). \end{aligned}$$

Therefore, $(u_{n},v_{n})$ is a bounded Palais–Smale sequence for E on $H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}$ at level c with additional condition (40). Finally, $u_{n}^{-},v_{n}^{-}\rightarrow 0$ a.e. in $\mathbb{R}^{N}$ as $n\rightarrow \infty$ is a simple consequence of $\gamma _{1,n}(t_{1},t_{2})\ge 0$, $\gamma _{2,n}(t_{1},t_{2})\ge 0$ and $\lim_{n\to +\infty } \operatorname{dist}((u_{n},v_{n}),\tilde{\gamma }_{n}(Q))=0$. □

From Lemma 3.5, there exist nonnegative functions ũ, ṽ in $H_{r}^{s}(\mathbb{R}^{N})$, such that, up to a subsequence,

$$\begin{aligned} &(u_{n},v_{n})\rightharpoonup (\tilde{u},\tilde{v}), \quad \text{weakly in } H^{s}\bigl(\mathbb{R}^{N}\bigr)\times H^{s}\bigl(\mathbb{R}^{N}\bigr), \\ &(u_{n},v_{n})\rightarrow (\tilde{u},\tilde{v}), \quad \text{strongly in } L^{2p}\bigl(\mathbb{R}^{N}\bigr)\times L^{2p}\bigl(\mathbb{R}^{N}\bigr), \\ &(u_{n},v_{n})\rightarrow (\tilde{u},\tilde{v}), \quad \mbox{a.e. in } \mathbb{R}^{N}. \end{aligned}$$

As a consequence $E'|_{H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}}(u_{n},v_{n})\rightarrow 0$, there exist two sequences of real number $\{\lambda _{1,n}\}$ and $\{\lambda _{2,n}\}$ such that

$$\begin{aligned} \begin{aligned} & \int _{\mathbb{R}^{N}} \bigl((-\Delta )^{\frac{s}{2}}u_{n}(- \Delta )^{ \frac{s}{2}}g+(-\Delta )^{\frac{s}{2}}v_{n}(-\Delta )^{\frac{s}{2}}h- \mu _{1} \vert u_{n} \vert ^{2p-2}u_{n}g-\mu _{2} \vert v_{n} \vert ^{2p-2}v_{n}h \bigr)\,dx \\ &\qquad {}- \int _{\mathbb{R}^{n}}\bigl(\beta \vert u_{n} \vert ^{p-2} \vert v_{n} \vert ^{p}u_{n}g+ \vert u_{n} \vert ^{p} \vert v_{n} \vert ^{p-2}v_{n}h\bigr)\,dx+ \int _{\mathbb{R}^{N}}(\lambda _{1,n}u_{n}g+ \lambda _{2,n}v_{n}h)\,dx \\ &\quad =o(1) \bigl( \Vert g \Vert _{H^{s}}+ \Vert h \Vert _{H^{s}}\bigr), \end{aligned} \end{aligned}$$

(43)

for every $g, h\in H^{s}({\mathbb{R}^{N}})$ with $o(1)\rightarrow 0$, $\text{as} n\rightarrow \infty$.

Lemma 3.6

Both $\{\lambda _{1,n}\}$ and $\{\lambda _{2,n}\}$ are bounded sequences and at least one of them is converging, up to a sequence, to a positive value.

Proof

By using $(u_{n},0)$ and $(0,v_{n})$ as test functions in (43), we get

$$\begin{aligned} & \int _{\mathbb{R}^{N}} \bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u_{n} \bigr\vert ^{2}-\mu _{1} \vert u_{n} \vert ^{2p}- \beta \vert u_{n} \vert ^{p} \vert v_{n} \vert ^{p} \bigr)\,dx+ \lambda _{1,n}a_{1}^{2}=o(1), \\ & \int _{\mathbb{R}^{N}} \bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}v_{n} \bigr\vert ^{2}-\mu _{2} \vert v_{n} \vert ^{2p}- \beta \vert u_{n} \vert ^{p} \vert v_{n} \vert ^{p} \bigr)\,dx+ \lambda _{2,n}a_{2}^{2}=o(1), \end{aligned}$$

with $o(1)\rightarrow 0$, $\text{as} n\rightarrow \infty$. Hence the boundedness of $\{\lambda _{i,n}\}$ follows from the boundedness of $u_{n}$, $v_{n}$ in $H^{s}({\mathbb{R}^{N}})$ and in $L^{2p}({\mathbb{R}^{N}})$. Furthermore, since $(u_{n},v_{n})$ satisfies (40),

$$\begin{aligned} &{\lambda _{1,n}} {a_{1}^{2}}+{\lambda _{2,n}} {a_{2}^{2}} \\ &\quad =- \int _{ \mathbb{R}^{N}} \bigl({ \bigl\vert (-\Delta )^{\frac{s}{2}}u_{n} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{ \frac{s}{2}}v_{n} \bigr\vert ^{2}}-\mu _{1}{ \vert u_{n} \vert ^{2p}}-2\beta { \vert u_{n} \vert ^{p}} { \vert v_{n} \vert ^{p}}- \mu _{2}{ \vert v_{n} \vert ^{2p}} \bigr)\,dx+o(1) \\ &\quad =\biggl(\frac{2ps}{(p-1)N}-1\biggr) \int _{\mathbb{R}^{N}} \bigl( \bigl\vert (-\Delta )^{ \frac{s}{2}}u_{n} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}v_{n} \bigr\vert ^{2} \bigr)\,dx+o(1), \end{aligned}$$

therefore by (42),

$$ \biggl(\frac{ps}{(p-1)N}-\frac{1}{2}\biggr)\bar{C}\le {\lambda _{1,n}} {a_{1}^{2}}+{ \lambda _{2,n}} {a_{2}^{2}}\le 2\biggl( \frac{2ps}{(p-1)N}-1\biggr)C, $$

for $1+\frac{2s}{N}< p<\frac{N}{N-2s}$ and every n sufficiently large. Therefore, at least one sequence of $\{\lambda _{i,n}\}$ is positive and bounded away from 0. This shows that at least one sequence of $\{\lambda _{i,n}\}$ is converging, up to a sequence, to a positive value. □

Next, we consider converging subsequence $\lambda _{1,n}\rightarrow \tilde{\lambda }_{1}\in {\mathbb{R}}$ and $\lambda _{2,n}\rightarrow \tilde{\lambda }_{2}\in {\mathbb{R}}$, as $n\rightarrow \infty$. The sign of $\tilde{\lambda }_{i}$ plays an important role for the strong convergence of $u_{n}$, $v_{n}$ in $H^{s}(\mathbb{R}^{N})$.

Lemma 3.7

If $\tilde{\lambda }_{1}>0$ (resp. $\tilde{\lambda }_{2}>0$), then $u_{n}\rightarrow \tilde{u}$ (resp. $v_{n}\rightarrow \tilde{v}$) strongly in $H^{s}(\mathbb{R}^{N})$.

Proof

Let us suppose that $\tilde{\lambda }_{1}>0$. By the weak convergence of $u_{n}$ in $H^{s}(\mathbb{R}^{N})$ and the strong convergence in $L^{2p}(\mathbb{R}^{N})$, it is easy to get from (43)

$$\begin{aligned} \begin{aligned} o(1)&=\bigl\langle E'({u_{n}},{v_{n}})-E'( \tilde{u},\tilde{v}),({u_{n}}- \tilde{u},0)\bigr\rangle +{\tilde{ \lambda }_{1}} \int _{\mathbb{R}^{N}}({u_{n}}- \tilde{u})^{2}\,dx \\ &= \int _{\mathbb{R}^{N}} \bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}({u_{n}}- \tilde{u}) \bigr\vert ^{2}+\tilde{\lambda }_{1}({u_{n}}- \tilde{u})^{2} \bigr)\,dx+o(1), \end{aligned} \end{aligned}$$

with $o(1)\rightarrow 0$ and $n\rightarrow \infty$. Since $\tilde{\lambda }_{1}>0$, this is equivalent to the strong convergence of $u_{n}$ in $H^{s}(\mathbb{R}^{N})$. The proof in the case $\tilde{\lambda }_{2}>0$ is similar. □

Having arrived at the end of this section, we give the proof of Theorem 1.1.

Proof of Theorem 1.1

By the convergence of $\{\lambda _{1,n}\}$ and $\{\lambda _{2,n}\}$, and the weak convergence $(u_{n},v_{n})\rightharpoonup (\tilde{u},\tilde{v})$, we see that $(\tilde{\lambda }_{1},\tilde{\lambda }_{2},\tilde{u},\tilde{v})$ is a solution of (1) with at least one $\tilde{\lambda }_{i}$ positive. We will show that both $\tilde{\lambda }_{1}$, $\tilde{\lambda }_{2}$ are positive, hence by Lemma 3.7, $\tilde{u}\in H_{a_{1}}$, $\tilde{v}\in H_{a_{2}}$ and the proof is complete.

We prove by contradiction. Without loss of generality, by Lemma 3.7, we may assume that $\tilde{\lambda }_{1}>0$ and $\tilde{\lambda }_{2}\le 0$. Since $(\tilde{\lambda }_{1},\tilde{\lambda }_{2},\tilde{u},\tilde{v})$ is a solution of (1) and $\tilde{u},\tilde{v}\geq 0$, we have

$$ (-\Delta )^{s} \tilde{v}=-\tilde{\lambda }_{2} \tilde{v} +\mu _{2} { \tilde{v}}^{2p-1}+\beta {\tilde{u}}^{p}{ \tilde{v}}^{p-1}\geq 0\quad \text{in } {\mathbb{R}^{N}}, $$

and since $2s< N\le 4s$, i.e., $2\le \frac{N}{N-2s}$, from Lemma 2.7(i), we can deduce that $\tilde{v}\equiv 0$. In particular, this implies that ũ solves

$$ \textstyle\begin{cases} (-\Delta )^{s}{\tilde{u}}+{ \tilde{\lambda }_{1}} {\tilde{u}} -{\mu _{1}} { \tilde{u}}^{2p-1}=0 \quad \text{in } {\mathbb{R}^{N}}, \\ \int _{\mathbb{R}^{N}} \tilde{u}^{2}\,dx=a_{1}^{2},\quad \text{and}\quad \tilde{u}>0 \quad \text{in } {\mathbb{R}^{N}}, \end{cases} $$

(44)

so that $\tilde{u}=w_{a_{1},\mu _{1}}\in \mathcal{P}(a_{1},\mu _{1})$. However, due to strong convergence of $u_{n}$, $v_{n}$ in $L^{2p}(\mathbb{R}^{N})$, we obtain due to (40),

$$\begin{aligned} c&=\lim_{n\rightarrow \infty }E(u_{n},v_{n})=\lim _{n\rightarrow \infty }{\frac{(p-1)N-2s}{4ps}} \int _{\mathbb{R}^{N}}\bigl(\mu _{1} \vert u_{n} \vert ^{2p}+2 \beta \vert u_{n} \vert ^{p} \vert v_{n} \vert ^{p}+\mu _{2} \vert v_{n} \vert ^{2p}\bigr)\,dx \\ &=\frac{(p-1)N-2s}{4ps} \int _{\mathbb{R}^{N}}\mu _{1} \vert {w_{a_{1},\mu _{1}}} \vert ^{2p}\,dx=I_{ \mu _{1}}(w_{a_{1},\mu _{1}}). \end{aligned}$$

This is a contradiction with Lemma 3.5. Therefore, both $\tilde{\lambda }_{1}$, $\tilde{\lambda }_{2}$ are positive. □

4 Proof of Theorem 1.2

In this section, we prove Theorem 1.2. The proof is divided into two parts. Firstly, we show the existence of a positive solution $(\bar{u},\bar{v})$, and secondly we characterize it as a ground state. The proof of the theorem is based on a mountain pass argument. For $(u,v)\in H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}$, we consider the function

$$ \begin{aligned} E\bigl(l\star (u,v)\bigr)={}&\frac{e^{2s^{2}l}}{2} \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (- \Delta )^{\frac{s}{2}}u \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}v \bigr\vert ^{2}\bigr)\,dx \\ &{}- \frac{e^{(p-1)Nsl}}{2p} \int _{\mathbb{R}^{N}}\bigl(\mu _{1} \vert u \vert ^{2p}+2 \beta \vert u \vert ^{p} \vert v \vert ^{p}+\mu _{2} \bigl\vert {v^{2p}} \bigr\vert \bigr)\,dx, \end{aligned} $$

where $l\star (u,v)=(l\star {u},l\star {v})$. If $(u,v)\in {H^{\mathrm{rad}}_{a_{1}}}\times {H^{\mathrm{rad}}_{a_{2}}}$, then $l\star (u,v)\in {H^{\mathrm{rad}}_{a_{1}}}\times {H^{\mathrm{rad}}_{a_{2}}}$ for any $l\in \mathbb{R}$. Similar to Lemma 2.5, we have the following lemma.

Lemma 4.1

Let $(u,v)\in {H^{\mathrm{rad}}_{a_{1}}\times {H^{\mathrm{rad}}_{a_{2}}}}$. Then

$$\begin{aligned} &\lim_{l\rightarrow {-\infty }}{ \int _{\mathbb{R}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}{l \star {u}} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}{l\star {v}} \bigr\vert ^{2}\bigr)\,dx=0}, \end{aligned}$$

(45)

$$\begin{aligned} &\lim_{l\rightarrow {+\infty }}{ \int _{\mathbb{R}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}{l \star {u}} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}{l\star {v}} \bigr\vert ^{2}\bigr)\,dx={+ \infty }}, \end{aligned}$$

(46)

$$\begin{aligned} &\lim_{l\rightarrow {-\infty }}{E\bigl(l\star (u,v)\bigr)=0^{+}}\quad \textit{and}\quad \lim_{l \rightarrow {+\infty }}{E\bigl(l\star (u,v)\bigr)={-\infty }}. \end{aligned}$$

(47)

The next lemma enlightens the mountain pass structure of the problem.

Lemma 4.2

There exists $K>0$ sufficiently small such that

$$ \sup_{A}E< \inf_{B}E \quad \textit{and}\quad E(u,v)>0 \quad \textit{on } A, $$

(48)

where

$$\begin{aligned} &A=\bigl\{ (u,v)\in {H^{\mathrm{rad}}_{a_{1}}}\times {H^{\mathrm{rad}}_{a_{2}}}, \bigl\Vert (-\Delta )^{ \frac{s}{2}}u \bigr\Vert ^{2}_{L^{2}}+ \bigl\Vert (-\Delta )^{\frac{s}{2}}v \bigr\Vert ^{2}_{L^{2}} \leq {K}\bigr\} , \end{aligned}$$

(49)

$$\begin{aligned} &B=\bigl\{ (u,v)\in {H^{\mathrm{rad}}_{a_{1}}}\times {H^{\mathrm{rad}}_{a_{2}}}, \bigl\Vert (-\Delta )^{ \frac{s}{2}}u \bigr\Vert ^{2}_{L^{2}}+ \bigl\Vert (-\Delta )^{\frac{s}{2}}v \bigr\Vert ^{2}_{L^{2}}=2K \bigr\} . \end{aligned}$$

(50)

Proof

By the Gagliardo–Nirenberg–Sobolev inequality (11),

$$\begin{aligned} & \int _{\mathbb{R}^{N}}\bigl(\mu _{1} \vert u \vert ^{2p}+2\beta \vert u \vert ^{p} \vert v \vert ^{p}+\mu _{2} \vert v \vert ^{2p}\bigr)\,dx \\ &\quad \leq C \int _{\mathbb{R}^{N}}\bigl( \vert u \vert ^{2p}+ \vert v \vert ^{2p}\bigr)\,dx \\ &\quad \leq C \biggl( \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u \bigr\vert ^{2}+ \bigl\vert (- \Delta )^{\frac{s}{2}}v \bigr\vert ^{2}\bigr)\,dx \biggr)^{\frac{(p-1)N}{2s}}, \end{aligned}$$

for every $(u,v)\in {H^{\mathrm{rad}}_{a_{1}}}\times {H^{\mathrm{rad}}_{a_{2}}}$, where $C>0$ depends on $\mu _{1},\mu _{2},\beta ,a_{1},a_{2}>0$, but not on the choice of $(u,v)$. Now if $(u_{1},v_{1})\in B$ and $(u_{2},v_{2})\in A$ (with K to be determined), we have

$$\begin{aligned} &E(u_{1},v_{1})-E(u_{2},v_{2}) \\ &\quad \geq {\frac{1}{2}} \biggl\{ \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{ \frac{s}{2}}{u_{1}} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}{v_{1}} \bigr\vert ^{2}\bigr)\,dx- \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}{u_{2}} \bigr\vert ^{2}+ \bigl\vert (- \Delta )^{\frac{s}{2}}{v_{2}} \bigr\vert ^{2}\bigr)\,dx \biggr\} \\ &\qquad {} -\frac{1}{2p} \int _{\mathbb{R}^{N}} \bigl(\mu _{1} \vert u_{1} \vert ^{2p}+2 \beta { \vert u_{1} \vert ^{p} \vert v_{1} \vert ^{p}}+\mu _{2} \vert v_{1} \vert ^{2p} \bigr)\,dx \geq \frac{K}{2}-\frac{C}{2p}(2K)^{\frac{(p-1)N}{2s}}\ge \frac{K}{4}, \end{aligned}$$

provided $K>0$ is sufficiently small. Furthermore if necessary, we can make K smaller, then

$$\begin{aligned} \begin{aligned} E(u_{2},v_{2})&\geq \frac{1}{2} \biggl( \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (- \Delta )^{\frac{s}{2}}u_{2} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}v_{2} \bigr\vert ^{2}\bigr)\,dx \biggr) \\ &\quad {}-\frac{C}{2p} \biggl( \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u_{2} \bigr\vert ^{2}+ \bigl\vert (- \Delta )^{\frac{s}{2}}v_{2} \bigr\vert ^{2}\bigr)\,dx \biggr)^{\frac{(p-1)N}{2s}}>0, \end{aligned} \end{aligned}$$

for every $(u_{2},v_{2})\in A$. □

For the next part, we shall introduce a suitable minimax class. Define

$$ D:=\bigl\{ (u,v)\in {H^{\mathrm{rad}}_{a_{1}}}\times {H^{\mathrm{rad}}_{a_{2}}}: \bigl\Vert (-\Delta )^{ \frac{s}{2}}u \bigr\Vert ^{2}_{L^{2}}+ \bigl\Vert (-\Delta )^{\frac{s}{2}}v \bigr\Vert ^{2}_{L^{2}} \geq 3K \text{ and } E(u,v)\leq 0\bigr\} . $$

(51)

Recall from Lemma 2.3 that $w_{a,\mu }$ is the unique positive radial solution of (19). By Lemma 4.1, there exist $l_{1}<0$ and $l_{2}>0$ such that

$$\begin{aligned} &l_{1}\star (w_{a_{1},{C_{0}}/{a_{1}^{2}}},w_{a_{2},{C_{0}}/{a_{2}^{2}}})=:( \bar{u}_{1},\bar{v}_{1})\in A, \\ &l_{2}\star (w_{a_{1},{C_{0}}/{a_{1}^{2}}},w_{a_{2},{C_{0}}/{a_{2}^{2}}})=:( \bar{u}_{2},\bar{v}_{2})\in D. \end{aligned}$$

At last, we define

$$ \bar{\Gamma }:=\bigl\{ \bar{\gamma }\in {C\bigl([0,1],{H^{\mathrm{rad}}_{a_{1}}} \times {H^{\mathrm{rad}}_{a_{2}}}\bigr)}: \bar{\gamma }(0)=( \bar{u}_{1},\bar{v}_{1}),\bar{\gamma }(1)=( \bar{u}_{2}, \bar{v}_{2})\bigr\} . $$

Similarly to the proof of Lemma 3.5, we derive

Lemma 4.3

There exists a bounded Palais–Smale sequence $(u_{n},v_{n})$ for E on $H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}$ at the level

$$ d:=\inf_{\bar{\gamma }\in \bar{\Gamma }}\max_{t\in [0,1]}E\bigl(\bar{ \gamma }(t)\bigr), $$

satisfying the additional condition

$$ G(u_{n},v_{n})=o(1), $$

with $o(1)\rightarrow 0$ as $n\rightarrow \infty$. Furthermore, $u^{-}_{n},v^{-}_{n}\rightarrow 0$ a.e. in $\mathbb{R}^{N}$ as $n\rightarrow \infty$.

Lemma 4.4

Let $\beta _{2}$ be defined in (10), if $\beta >\beta _{2}$, then

$$ \sup_{l\in \mathbb{R}}E \bigl(l\star (w_{a_{1},({C_{0}}/{a^{2}_{1}})^{p-1}},w_{a_{2},({C_{0}}/{a^{2}_{2}})^{p-1}}) \bigr)< \min \bigl\{ I_{\mu _{1}}(w_{a_{1},\mu _{1}}),I_{\mu _{2}}(w_{a_{2}, \mu _{2}}) \bigr\} . $$

(52)

Proof

By Lemma 2.3, direct computation gives

$$\begin{aligned}& \int _{\mathbb{R}^{N}} (l\star w_{a_{1},({C_{0}}/{a^{2}_{1}})^{p-1}} )^{p} (l \star w_{a_{2},({C_{0}}/{a^{2}_{2}})^{p-1}} )^{p}\,dx \\& \quad = \int _{\mathbb{R}^{N}}e^{(p-1)Nsl} \biggl( \frac{{a_{1}}}{C_{0}^{\frac{1}{2}}}{w_{0}(x)} \biggr)^{p} \biggl( \frac{{a_{2}}}{C_{0}^{\frac{1}{2}}}{w_{0}(x)} \biggr)^{p}\,dx \\& \quad =\frac{a_{1}^{p}a_{2}^{p}}{C_{0}^{p}}e^{(p-1)Nsl} \int _{\mathbb{R}^{N}}w_{0}^{2p}\,dx= \frac{a_{1}^{p}a_{2}^{p}C_{1}}{C_{0}^{p}}e^{(p-1)Nsl}, \\& \begin{aligned} E \bigl(l\star (w_{a_{1},({C_{0}}/{a^{2}_{1}})^{p-1}},w_{a_{2},({C_{0}}/{a^{2}_{2}})^{p-1}}) \bigr)={}& \frac{(p-1)Ne^{2s^{2}l}}{4ps}\biggl( \frac{a_{1}^{2}C_{1}+a_{2}^{2}C_{1}}{C_{0}}\biggr) \\ &{}-\frac{e^{(p-1)Nsl}}{2p}\biggl( \frac{\mu _{1}C_{1}a_{1}^{2p}+2\beta C_{1} a_{1}^{p}a_{2}^{p}+\mu _{2}C_{1}a_{2}^{2p}}{C_{0}^{p}}\biggr). \end{aligned} \end{aligned}$$

Therefore, it is easy to get

$$\begin{aligned} &\max_{l\in \mathbb{R}}E \bigl(l\star (w_{a_{1},({C_{0}}/{a^{2}})^{p-1}},w_{a_{2},({C_{0}}/{a^{2}})^{p-1}}) \bigr) \\ &\quad =\frac{(p-1)N-2s}{4ps} \frac{C_{1}C_{0}^{\frac{2ps-(p-1)N}{(p-1)N-2s}}(a_{1}^{2}+a_{2}^{2})^{\frac{(p-1)N}{(p-1)N-2s}}}{(\mu _{1}a_{1}^{2p}+2\beta a_{1}^{p}a_{2}^{p}+\mu _{2}a_{2}^{2p})^{\frac{2s}{(p-1)N-2s}}}. \end{aligned}$$

Due to (10) and (23), if $\beta >\beta _{2}$, (52) is satisfied. □

Existence of a positive solution at level d

We will prove the existence of positive solution at level d by contradiction. By Lemma 4.3, up to a subsequence, we may assume that

$$\begin{aligned} &(u_{n},v_{n})\rightharpoonup (\bar{u},\bar{v}), \quad \text{weakly in } H^{s}\bigl(\mathbb{R}^{N}\bigr)\times H^{s}\bigl(\mathbb{R}^{N}\bigr), \\ &(u_{n},v_{n})\rightarrow (\bar{u},\bar{v}), \quad \text{strongly in } L^{2p}\bigl(\mathbb{R}^{N}\bigr)\times L^{2p}\bigl(\mathbb{R}^{N}\bigr), \\ &(u_{n},v_{n})\rightarrow (\bar{u},\bar{v}), \quad \mbox{a.e. in } \mathbb{R}^{N}. \end{aligned}$$

Then it can be easily derived that $(\bar{u},\bar{v})$ is a solution of (1) for some constants $\bar{\lambda }_{1},\bar{\lambda }_{2}\in \mathbb{R}$. Moreover, Lemma 3.6 and Lemma 3.7 are applicable. We may assume that $\bar{\lambda }_{1}>0$ and $u_{n}\to \bar{u}$ strongly in $H^{s}(\mathbb{R}^{N})$. If $\bar{\lambda }_{2}\le 0$, we can derive that $\bar{v}\equiv 0$ and $\bar{u}=w_{a_{1},\mu _{1}}$ as in the proof of Theorem 1.1. By $G(u_{n},v_{n})\to 0$ and strong convergence in $L^{2p}(\mathbb{R}^{N})$, $d=I_{\mu _{1}}(w_{a_{1},\mu _{1}})$. We can consider the path

$$ \bar{\gamma }(t):=\bigl((1-t)l_{1}+tl_{2}\bigr)\star (w_{a_{1},\mu _{1}},w_{a_{2}, \mu _{2}}). $$

Obviously, $\bar{\gamma }\in \bar{\Gamma }$. Then, by Lemma 4.4,

$$ d\leq \sup_{t\in [0,1]}E\bigl(\bar{\gamma }(t)\bigr)\leq \sup _{l\in \mathbb{R}}E\bigl(l \star (w_{a_{1},\mu _{1}},w_{a_{2},\mu _{2}}) \bigr)< I_{\mu _{1}}(w_{a_{1}, \mu _{1}}), $$

which is a contradiction. Therefore, $\bar{\lambda }_{2}>0$ and $v_{n}\to \bar{v}$ strongly in $H^{s}(\mathbb{R}^{N})$. This shows that $(\bar{\lambda }_{1},\bar{\lambda }_{2},\bar{u},\bar{v})$ is a solution of (1) with $\bar{\lambda }_{1},\bar{\lambda }_{2}>0$ and $(\bar{u},\bar{v})\in H_{a_{1}}\times H_{a_{2}}$.

Obviously, we can see that $G(\bar{u},\bar{v})=0$, i.e., $(\bar{u},\bar{v})\in F$.

Variational characterization of $(\bar{u},\bar{v})$

In the following, we will prove that

$$ E(\bar{u},\bar{v}) =\inf \bigl\{ E(u,v);(u,v)\in F\bigr\} =\inf _{(u,v)\in H_{a_{1}} \times H_{a_{2}}}\mathcal{R}(u,v), $$

where F and $\mathcal{R}$ are defined in (6) and (8). Recall the definition of A in (49) and D in (51), let us define

$$\begin{aligned} &A^{+}:=\bigl\{ (u,v)\in A, u, v\geq 0 \text{ a.e. in } \mathbb{R}^{N}\bigr\} , \\ &D^{+}:=\bigl\{ (u,v)\in D, u, v\geq 0 \text{ a.e. in } \mathbb{R}^{N}\bigr\} . \end{aligned}$$

For any $(u_{1},v_{1})\in A^{+}$ and $(u_{2},v_{2})\in D^{+}$, let

$$ \bar{\Gamma }(u_{1},v_{1},u_{2},v_{2})=: \bigl\{ \bar{\gamma }\in C\bigl([0,1],{H^{\mathrm{rad}}_{a_{1}}} \times {H^{\mathrm{rad}}_{a_{2}}}\bigr):\bar{\gamma }(0)=(u_{1},v_{1}), \bar{\gamma }(1)=(u_{2},v_{2}) \bigr\} . $$

Lemma 4.5

The sets $A^{+}$ and $D^{+}$ are connected by arcs, so that

$$ d=\inf_{\bar{\gamma }\in \bar{\Gamma }(u_{1},v_{1},u_{2},v_{2})}\max_{t \in [0,1]}E \bigl(\bar{\gamma }(t)\bigr), $$

(53)

for every $(u_{1},v_{1})\in A^{+}$ and $(u_{2},v_{2})\in D^{+}$.

Proof

Equality (53) follows easily once we show that $A^{+}$ and $D^{+}$ are connected by arcs ( as $l*(\bar{u},\bar{v})$ is a path from $A^{+}$ to $D^{+}$). Let $(u_{1},v_{1}),(u_{2},v_{2})\in {H^{\mathrm{rad}}_{a_{1}}}\times {H^{\mathrm{rad}}_{a_{2}}}$ be nonnegative functions such that

$$ \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}{u_{1}} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}{v_{1}} \bigr\vert ^{2}\bigr)\,dx= \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u_{2} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}v_{2} \bigr\vert ^{2}\bigr)\,dx=\alpha ^{2}, $$

(54)

for some $\alpha >0$. For $l\in \mathbb{R}$ and $\theta \in [0,\frac{\pi }{2}]$,

$$ h(l,\theta )=\bigl(\cos \theta (l\star u_{1}) (x)+\sin \theta (l \star u_{2}) (x), \cos \theta (l\star v_{1}) (x)+\sin \theta (l\star v_{2}) (x)\bigr). $$

Set $h=(h_{1},h_{2})$, we have $h_{1}(l,\theta ), h_{2}(l,\theta )\geq 0$ a.e. in $\mathbb{R}^{N}$. It is not difficult to check that

$$\begin{aligned} & \int _{\mathbb{R}^{N}}{h^{2}_{1}(l,\theta )}\,dx=a^{2}_{1}+\sin (2 \theta ) \int _{\mathbb{R}^{N}}{u_{1}} {u_{2}}\,dx, \\ & \int _{ \mathbb{R}^{N}}{h^{2}_{2}(l,\theta )}\,dx=a^{2}_{2}+\sin (2\theta ) \int _{\mathbb{R}^{N}}{v_{1}} {v_{2}}\,dx, \\ & \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}{h_{1}(l, \theta )} \bigr\vert ^{2}+ \bigl\vert (- \Delta )^{\frac{s}{2}}{h_{2}(l,\theta )} \bigr\vert ^{2} \bigr)\,dx \\ &\quad =e^{2s^{2}l} \biggl(\alpha ^{2}+\sin (2\theta ) \int _{\mathbb{R}^{N}}\bigl((- \Delta )^{\frac{s}{2}}{u_{1}}(- \Delta )^{\frac{s}{2}}{u_{2}}+(-\Delta )^{\frac{s}{2}}{v_{1}}(- \Delta )^{\frac{s}{2}}{v_{2}}\bigr)\,dx \biggr), \end{aligned}$$

for all $(l,\theta )\in \mathbb{R}\times [0,\frac{\pi }{2}]$. We can deduce that

$$ a_{1}^{2}\leq \int _{\mathbb{R}^{N}}h^{2}_{1}(l,\theta )\,dx\leq 2a_{1}^{2}\quad \text{and}\quad a_{2}^{2} \leq \int _{\mathbb{R}^{N}}h^{2}_{2}(l, \theta )\,dx\leq 2a_{2}^{2}. $$

Therefore

$$ \alpha ^{2}+\sin (2\theta ) \int _{\mathbb{R}^{N}}\bigl((-\Delta )^{\frac{s}{2}}{u_{1}}(- \Delta )^{\frac{s}{2}}{u_{2}}+(-\Delta )^{\frac{s}{2}}{v_{1}}(- \Delta )^{\frac{s}{2}}{v_{2}}\bigr)\,dx>0, $$

and it is continuous in $\theta \in [0,\frac{\pi }{2}]$, so there is a constant $C>0$ independent of l, θ, such that

$$ C\alpha ^{2}e^{2s^{2}l}\leq \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}h_{1}(l, \theta ) \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}h_{2}(l, \theta ) \bigr\vert ^{2}\bigr)\,dx\leq 2\alpha ^{2}e^{2s^{2}l}. $$

Thus we can define the function

$$ \hat{h}(l,\theta ) (x)=\biggl(a_{1} \frac{h_{1}(l,\theta )}{ \Vert h_{1}(l,\theta ) \Vert _{L^{2}}},a_{2} \frac{h_{2}(l,\theta )}{ \Vert h_{2}(l,\theta ) \Vert _{L^{2}}}\biggr), $$

for $(l,\theta )\in \mathbb{R}\times [0,\frac{\pi }{2}]$.

Notice that $\hat{h}(l,\theta )\in {H^{\mathrm{rad}}_{a_{1}}}\times {H^{\mathrm{rad}}_{a_{2}}}$ for every $(l,\theta )$, we see that

$$ C c_{0}\alpha ^{2}e^{2s^{2}l} \leq \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}} \hat{h_{1}}(l,\theta ) \bigr\vert ^{2}+ \bigl\vert (- \Delta )^{\frac{s}{2}} \hat{h_{2}}(l,\theta ) \bigr\vert ^{2}\bigr)\,dx\leq 2 c_{1}\alpha ^{2}e^{2s^{2}l}, $$

(55)

with $c_{0}= \frac{\min \{a^{2}_{1},a^{2}_{2}\}}{\max \{a^{2}_{1},a^{2}_{2}\}}$ and $c_{1}= \frac{\max \{a^{2}_{1},a^{2}_{2}\}}{\min \{a^{2}_{1},a^{2}_{2}\}}$.

For $u,v\ge 0$ and $t\in [0,\frac{\pi }{2}]$,

$$\begin{aligned} \bigl\Vert \cos t\cdot u(x)+\sin t\cdot v(x) \bigr\Vert _{L^{2p}}& \ge \max \bigl(\cos t \Vert u \Vert _{L^{2p}}, \sin t \Vert v \Vert _{L^{2p}}\bigr) \\ &\ge \frac{ \Vert u \Vert _{L^{2p}} \Vert v \Vert _{L^{2p}}}{\sqrt{ \Vert u \Vert ^{2}_{L^{2p}}+ \Vert v \Vert ^{2}_{L^{2p}}}}. \end{aligned}$$

Therefore, we have, for some constant $C>0$ independent of l, θ,

$$ \int _{\mathbb{R}^{N}}\hat{h}^{2p}_{1}(l,\theta )\,dx\geq {Ce^{(p-1)Nsl}} \quad \text{and}\quad \int _{\mathbb{R}^{N}}\hat{h}^{2p}_{2}(l,\theta )\,dx\geq Ce^{(p-1)Nsl}, $$

(56)

for all $(l,\theta )\in \mathbb{R}\times [0,\frac{\pi }{2}]$. Let $(u_{1},v_{1}),(u_{2},v_{2})\in A^{+}$, and let ĥ be as previously. From (55), we can deduce there exists $l_{0}>0$ such that

$$ \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}} \hat{h}_{1}(-l_{0}, \theta ) \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}\hat{h}_{2}(-l_{0}, \theta ) \bigr\vert ^{2}\bigr)\,dx \leq {K}, $$

for all $\theta \in [0,\frac{\pi }{2}]$, where K is defined in Lemma 4.2. For the choice of $l_{0}$, let

$$ \sigma _{1}(r):= \textstyle\begin{cases} -r\star {(u_{1},v_{1})}=\hat{h}(-r,0), &{0\leq r\leq {l_{0}}}, \\ \hat{h}(-{l_{0}},r-{l_{0}}),& {{l_{0}}< r \leq {{l_{0}}+\frac{\pi }{2}}}, \\ (r-2{l_{0}}-\frac{\pi }{2})\star {(u_{2},v_{2})}=\hat{h}({r-2{l_{0}}- \frac{\pi }{2}},\frac{\pi }{2}),& {{l_{0}}+ \frac{\pi }{2}< r\leq {2{l_{0}}+ \frac{\pi }{2}}}. \end{cases} $$

It is not difficult to check that $\sigma _{1}$ is a continuous path connecting $(u_{1},v_{1})$ and $(u_{2},v_{2})$ and lying in $A^{+}$. For the case that condition (54) is not satisfied, suppose for instance

$$ \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u_{1} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}v_{1} \bigr\vert ^{2}\bigr)\,dx> \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u_{2} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}v_{2} \bigr\vert ^{2}\bigr)\,dx. $$

Then, by Lemma 4.1, there exists $l_{1}<0$ such that

$$ \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}(l_{1} \star u_{1}) \bigr\vert ^{2}+ \bigl\vert (- \Delta )^{\frac{s}{2}}(l_{1}\star v_{1}) \bigr\vert ^{2}\bigr)\,dx= \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (- \Delta )^{\frac{s}{2}}u_{2} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}v_{2} \bigr\vert ^{2}\bigr)\,dx. $$

Therefore, to connect $(u_{1},v_{1})$ and $(u_{2},v_{2})$ by a path in $A^{+}$, we can at first connect $(u_{1},v_{1})$ with $l_{1}\star (u_{1},v_{1})$ along arc $l*(u_{1},v_{1})$, then connect $l_{1}\star (u_{1},v_{1})$ with $(u_{2},v_{2})$. This shows that $A^{+}$ is path connected. In a similar way, we can prove that $D^{+}$ is also path connected. □

From the previous notation,

$$ F: =\bigl\{ (u,v)\in H_{a_{1}}\times H_{a_{2}}:G(u,v)=0\bigr\} , $$

we define its radial subset and positive radial subset

$$\begin{aligned}& F_{\mathrm{rad}}: =\bigl\{ (u,v)\in H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}:G(u,v)=0\bigr\} , \\& F^{+}: =\bigl\{ (u,v)\in F: u\ge 0, v\ge 0\bigr\} , \\& F^{+}_{\mathrm{rad}}: =\bigl\{ (u,v)\in F_{\mathrm{rad}}: u\ge 0, v\ge 0\bigr\} , \end{aligned}$$

where

$$\begin{aligned} G(u,v)={}& \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u \bigr\vert ^{2}+ \bigl\vert (- \Delta )^{\frac{s}{2}}v \bigr\vert ^{2}\bigr)\,dx \\ &{}-\frac{(p-1)N}{2ps} \int _{\mathbb{R}^{N}}\bigl( \mu _{1} \vert u \vert ^{2p}+2\beta \vert u \vert ^{p} \vert v \vert ^{p}+\mu _{2} \vert v \vert ^{2p} \bigr)\,dx. \end{aligned}$$

For $(u,v)\in H_{a_{1}}\times H_{a_{2}}$, let us set

$$ \Psi _{(u,v)}(l)=E\bigl(l\star {(u,v)}\bigr), $$

where $l\star {(u,v)}=(l\star u,l\star v)$ for short. Similar to the proof of Lemma 2.5, we have the following lemma.

Lemma 4.6

For every $(u,v)\in H_{a_{1}}\times H_{a_{2}}$, there exists a unique $l_{(u,v)}\in {\mathbb{R}}$ such that $l_{(u,v)}\star {(u,v)}\in F$. Moreover, $l_{(u,v)}$ is the unique critical point of $\Psi _{(u,v)}$, which is a strict maximum.

Lemma 4.7

We have $\inf_{F}E=\inf_{F^{+}}E=\inf_{F_{\mathrm{rad}}^{+}}E$.

Proof

We prove the lemma by contradiction. Suppose there exists $(u,v)\in F$ such that

$$ 0< E(u,v)< \inf_{F^{+}}E. $$

(57)

For any $u\in {H^{s}(\mathbb{R}^{N})}$, since $\|(-\Delta )^{\frac{s}{2}}|u|\|_{L^{2}}\le \|(-\Delta )^{\frac{s}{2}} u\|_{L^{2}}$, we get $E(|u|,|v|)\le E(u,v)$ and $G(|u|,|v|)\le G(u,v)=0$. Thus, there exists $l_{0}\leq 0$ such that $G(l_{0}\star (|u|,|v|))=0$. We obtain

$$\begin{aligned} E\bigl(l_{0}\star \bigl( \vert u \vert , \vert v \vert \bigr)\bigr)&=\biggl(\frac{1}{2}-\frac{s}{(p-1)N} \biggr)e^{2s^{2}l_{0}} \biggl\{ \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}} \vert u \vert \bigr\vert ^{2}+ \bigl\vert (- \Delta )^{\frac{s}{2}} \vert v \vert \bigr\vert ^{2}\bigr)\,dx \biggr\} \\ &\leq \biggl(\frac{1}{2}-\frac{s}{(p-1)N}\biggr)e^{2s^{2}l_{0}} \biggl\{ \int _{ \mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}v \bigr\vert ^{2}\bigr)\,dx \biggr\} \\ &=e^{2s^{2}l_{0}}E(u,v). \end{aligned}$$

Therefore

$$ 0< E(u,v)< \inf_{F^{+}}E\leq E\bigl(l_{0}\star \bigl( \vert u \vert , \vert v \vert \bigr)\bigr)\leq e^{2s^{2}l_{0}}E(u,v), $$

which contradicts $l_{0}\leq 0$. Thus $\inf_{F}E=\inf_{F^{+}}E$.

Next, if there exists $(u,v)\in F^{+}$ such that

$$ 0< E(u,v)< \inf_{F_{\mathrm{rad}}^{+}}E. $$

For $u\in {H^{s}(\mathbb{R}^{N})}$, let $u^{\ast }$ denotes its Schwarz spherical rearrangement. According to the property of Schwarz symmetrization, we have $E(u^{\ast },v^{\ast })\leq {E(u,v)}$ and $G(u^{\ast },v^{\ast })\leq {G(u,v)}=0$. Thus there exists $l_{0}\leq 0$ such that $G(l_{0}\star (u^{\ast },v^{\ast }))=0$. Similarly, we get

$$ 0< E(u,v)< \inf_{F_{\mathrm{rad}}^{+}}E\leq E\bigl(l_{0}\star \bigl(u^{\ast },v^{\ast }\bigr)\bigr)\leq e^{2s^{2}l_{0}}E(u,v), $$

which contradicts $l_{0}\leq 0$. Thus $\inf_{F^{+}}E=\inf_{F_{\mathrm{rad}}^{+}}E$. □

Proof of Theorem 1.2

We have showed that $(\bar{\lambda }_{1},\bar{\lambda }_{2},\bar{u},\bar{v})$ is a solution of (1). Since $(\bar{u},\bar{v})\in F^{+}_{\mathrm{rad}}$, we just need to show that

$$ E(\bar{u},\bar{v})=d\leq \inf_{F^{+}_{\mathrm{rad}}}E. $$

Then $E(\bar{u},\bar{v})=\inf_{F}E$ follows from Lemma 4.7. Choose any $(u,v)\in F^{+}_{\mathrm{rad}}$. Let us consider the function $\Psi _{(u,v)}(l)=E(l*(u,v))$. By Lemma 4.1 there exists $l_{0}\gg 1$ such that $(-l_{0})\star (u,v)\in A^{+}$ and $l_{0}\star (u,v)\in D^{+}$. Therefore, the continuous path

$$ \bar{\gamma }(t)=\bigl((2t-1)l_{0}\bigr)\star (u,v),\quad t\in [0,1], $$

connects $A^{+}$ with $D^{+}$, and by Lemma 4.5 and Lemma 4.6, we can deduce that

$$ d\leq \max_{t\in [0,1]}E\bigl(\bar{\gamma }(t)\bigr)=E(u,v). $$

Since this holds for all the elementary quantities in $F^{+}_{\mathrm{rad}}$, we have

$$ d\leq \inf_{F^{+}_{\mathrm{rad}}}E. $$

Finally, it remains to show that

$$ \inf_{F}E=\inf_{H_{a_{1}}\times H_{a_{2}}} \mathcal{R}. $$

(58)

The proof of (58) is similar to the case for the single equation; see Lemma 2.6. □