In this section, we give the proof of Theorem 1.1. We work in a radial setting. That is, we find the critical point of the functional E constrained on \(H^{\mathrm{rad}}_{a_{1}}\times {H^{\mathrm{rad}}_{a_{2}}}\), where for any \(a>0\), we define
$$ H^{\mathrm{rad}}_{a}:=H_{a}\cap H_{r}^{s} \bigl(\mathbb{R}^{N}\bigr), $$
and \(H^{s}_{r}(\mathbb{R}^{N})\) is the subset of \(H^{s}(\mathbb{R}^{N})\) containing all the functions which are radial with respect to the origin. We know that \(H^{s}_{r}(\mathbb{R}^{N})\hookrightarrow L^{p}(\mathbb{R}^{N})\) is compact when \(2< p<\frac{2N}{N-2s}\). Due to the Palais principle of symmetric criticality, the critical points of E constrained on \(H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}\) are true critical points of E constrained in the full product \(H_{a_{1}}\times H_{a_{2}}\).
For \(a_{1}\), \(a_{2}\), \(\mu _{1}\) and \(\mu _{2}>0\), let \(\beta _{1}>0\) be defined by (5).
Lemma 3.1
For \(0<\beta <\beta _{1}\),
$$ \inf \bigl\{ E(u_{1},u_{2}):(u_{1},u_{2}) \in \mathcal{P}(a_{1},\mu _{1}+ \beta )\times \mathcal{P}(a_{2},\mu _{2}+\beta )\bigr\} >\max \bigl\{ I_{\mu _{1}}(w_{a_{1}, \mu _{1}}),I_{\mu _{2}}(w_{a_{2},\mu _{2}}) \bigr\} , $$
where \(I_{\mu _{i}}(w_{a_{i},\mu _{1}})\), \(i=1,2\) is defined by (23).
Proof
For \((u_{1},u_{2})\in \mathcal{P}(a_{1},\mu _{1}+\beta )\times \mathcal{P}(a_{2},\mu _{2}+\beta )\}\), we have
$$\begin{aligned} E(u_{1},u_{2})&= \int _{\mathbb{R}^{N}} \biggl(\frac{1}{2} \bigl\vert (-\Delta )^{\frac{s}{2}}u_{1} \bigr\vert ^{2}- \frac{\mu _{1}}{2p} \vert u_{1} \vert ^{2p} \biggr)\,dx \\ &\quad {}+ \int _{\mathbb{R}^{N}} \biggl(\frac{1}{2} \bigl\vert (-\Delta )^{\frac{s}{2}}u_{2} \bigr\vert ^{2}- \frac{\mu _{2}}{2p} \vert u_{2} \vert ^{2p} \biggr)\,dx-\frac{\beta }{p} \int _{ \mathbb{R}^{N}} \vert u_{1} \vert ^{p} \vert u_{2} \vert ^{p}\,dx \\ &\geqslant I_{\mu _{1}}(u_{1})+I_{\mu _{2}}(u_{2})- \frac{\beta }{2p} \int _{\mathbb{R}^{N}} \vert u_{1} \vert ^{2p}\,dx-\frac{\beta }{2p} \int _{ \mathbb{R}^{N}} \vert u_{2} \vert ^{2p}\,dx \\ &=I_{\mu _{1}+\beta }(u_{1})+I_{\mu _{2}+\beta }(u_{2}) \\ &\geqslant \inf_{u\in \mathcal{P}(a_{1},\mu _{1}+\beta )}I_{\mu _{1}+ \beta }(u)+\inf _{v\in \mathcal{P}(a_{1},\mu _{1}+\beta )}I_{\mu _{2}+ \beta }(v) \\ &=I_{\mu _{1}+\beta }(w_{a_{1},\mu _{1}+\beta })+I_{\mu _{2}+\beta }(w_{a_{2}, \mu _{2}+\beta }), \end{aligned}$$
by Lemma 2.4. From (23) and (5), it is easy to get, when \(0<\beta <\beta _{1}\),
$$\begin{aligned} &\max \bigl\{ I_{\mu _{1}}(w_{a_{1},\mu _{1}}),I_{\mu _{2}}(w_{a_{2},\mu _{2}}) \bigr\} \\ &\quad =\max \biggl\{ \frac{(p-1)N-2s}{4ps} \frac{C_{1}C_{0}^{\frac{2ps-(p-1)N}{(p-1)N-2s}}}{{\mu _{1}}^{\frac{2s}{(p-1)N-2s}}a_{1}^{\frac{4ps-2(p-1)N}{(p-1)N-2s}}} , \frac{(p-1)N-2s}{4ps} \frac{C_{1}C_{0}^{\frac{2ps-(p-1)N}{(p-1)N-2s}}}{{\mu _{2}} ^{\frac{2s}{(p-1)N-2s}}a_{2}^{\frac{4ps-2(p-1)N}{(p-1)N-2s}}} \biggr\} \\ &\quad < I_{\mu _{1}+\beta }(w_{a_{1},\mu _{1}+\beta })+I_{\mu _{2}+\beta }(w_{a_{2}, \mu _{2}+\beta }). \end{aligned}$$
Therefore,
$$\begin{aligned} &\inf \bigl\{ E(u_{1},u_{2}):(u_{1},u_{2}) \in \mathcal{P}(a_{1},\mu _{1}+ \beta )\times \mathcal{P}(a_{2},\mu _{2}+\beta )\bigr\} \\ &\quad >\max \bigl\{ I_{\mu _{1}}(w_{a_{1},\mu _{1}}),I_{\mu _{2}}(w_{a_{2},\mu _{2}}) \bigr\} . \end{aligned}$$
□
Now we fix \(0<\beta <\beta _{1}\) and choose \(\varepsilon >0\) such that
$$\begin{aligned} \begin{aligned} &\inf \bigl\{ E(u_{1},u_{2}):(u_{1},u_{2}) \in \mathcal{P}(a_{1},\mu _{1}+ \beta )\times \mathcal{P}(a_{2},\mu _{2}+\beta )\bigr\} \\ &\quad >\max \bigl\{ I_{\mu _{1}}(w_{a_{1},\mu _{1}}),I_{\mu _{2}}(w_{a_{2},\mu _{2}}) \bigr\} +\varepsilon . \end{aligned} \end{aligned}$$
(35)
Denote
$$ w_{1}:=w_{a_{1},\mu _{1}+\beta }\quad \text{and}\quad w_{2}:=w_{a_{2}, \mu _{2}+\beta }, $$
(36)
and, for \(i=1,2\),
$$ \varphi _{i}(l):=I_{\mu _{i}}(l\star w_{i})\quad \text{and}\quad \tilde{\varphi }_{i}(l):= \frac{\partial }{\partial l}I_{\mu _{i}+\beta }(l \star w_{i}). $$
(37)
Lemma 3.2
For \(i=1,2\), there exist \(\rho _{i}<0\) and \(R_{i}>0\), depending on ε and β, such that
-
(i)
\(0<\varphi _{i}(\rho _{i})<\varepsilon \) and \(\varphi _{i}(R_{i})\leqslant 0\);
-
(ii)
\(\tilde{\varphi }_{i}(l)>0\) for any \(l<0\), \(\tilde{\varphi }_{i}(0)=0\) and \(\tilde{\varphi }_{i}(l)<0\) for any \(l>0\). In particular, \(\tilde{\varphi }_{i}(\rho _{i})>0\) and \(\tilde{\varphi }_{i}(R_{i})<0\).
Proof
By Lemma 2.4 and Lemma 2.5, we have
$$\begin{aligned} \varphi _{i}(l)&= \int _{\mathbb{R}^{N}} \biggl(\frac{1}{2} \bigl\vert (-\Delta )^{\frac{s}{2}}(l\star w_{i}) \bigr\vert ^{2}- \frac{\mu _{i}}{2p} \vert l\star w_{i} \vert ^{2p} \biggr)\,dx \\ &=\frac{e^{2s^{2}l}}{2} \bigl\Vert (-\Delta )^{\frac{s}{2}}w_{i} \bigr\Vert ^{2}_{L^{2}}- \frac{e^{(p-1)Nsl}}{2p}\mu _{i} \Vert w_{i} \Vert ^{2p}_{L^{2p}} \\ &= \biggl(\frac{(p-1)N(\mu _{i}+\beta )}{2ps}\frac{e^{2s^{2}l}}{2}- \frac{e^{(p-1)Nsl}}{2p}\mu _{i} \biggr) \Vert w_{i} \Vert ^{2p}_{L^{2p}}, \end{aligned}$$
thus, \(\varphi _{i}(l)\rightarrow 0^{+}\) as \(l\rightarrow -\infty \), and \(\varphi _{i}(l)\rightarrow -\infty \) as \(l\rightarrow +\infty \). Therefore, there exist \(\rho _{i}<0\) and \(R_{i}>0\), such that \(0<\varphi _{i}(\rho _{i})<\varepsilon \) and \(\varphi _{i}(R_{i})\leqslant 0\).
$$\begin{aligned} \tilde{\varphi }_{i}(l)&=s^{2}e^{2s^{2}l} \bigl\Vert (-\Delta )^{\frac{s}{2}}w_{i} \bigr\Vert ^{2}_{L^{2}}-\frac{e^{(p-1)Nsl}(p-1)N}{2p}s(\mu _{i}+ \beta ) \int _{ \mathbb{R}^{N}} \vert w_{i} \vert ^{2p}\,dx \\ &= \biggl(\frac{(p-1)N(\mu _{i}+\beta )}{2ps}s^{2}e^{2s^{2}l}- \frac{e^{(p-1)Nsl}(p-1)N}{2p}s(\mu _{i}+\beta ) \biggr) \int _{ \mathbb{R}^{N}} \vert w_{i} \vert ^{2p}\,dx \\ &=\frac{(p-1)N(\mu _{i}+\beta )}{2p}se^{(p-1)Nsl} \bigl(e^{(2s-(p-1)N)sl}-1 \bigr) \int _{\mathbb{R}^{N}} \vert w_{i} \vert ^{2p}\,dx, \end{aligned}$$
then
$$\begin{aligned} \tilde{\varphi }_{i}(l)= \textstyle\begin{cases} >0 & \text{if } l< 0, \\ =0 & \text{if } l=0, \\ < 0 & \text{if } l>0, \end{cases}\displaystyle \end{aligned}$$
(38)
which implies that (ii) holds. □
Let \(Q:=[\rho _{1},R_{1}]\times [\rho _{2},R_{2}]\), and let
$$ \gamma _{0}(t_{1},t_{2}):=(t_{1} \star w_{1},t_{2}\star w_{2})\in H^{\mathrm{rad}}_{a_{1}} \times H^{\mathrm{rad}}_{a_{2}},\quad \forall (t_{1},t_{2})\in Q. $$
We introduce the minimax class
$$ \Gamma :=\bigl\{ \gamma \in C\bigl(Q, H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}\bigr): \gamma =\gamma _{0} \text{ on } \partial Q\bigr\} . $$
Lemma 3.3
We have
$$ \sup_{\partial Q}E(\gamma _{0})\leqslant \max \bigl\{ I_{\mu _{1}}(w_{a_{1}, \mu _{1}}),I_{\mu _{2}}(w_{a_{2},\mu _{2}}) \bigr\} +\varepsilon . $$
Proof
For every \((u_{1},u_{2})\in H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}\), we have
$$ E(u_{1},u_{2})=I_{\mu _{1}}(u_{1})+I_{\mu _{2}}(u_{2})- \frac{\beta }{p} \int _{\mathbb{R}^{N}} \vert u_{1} \vert ^{p} \vert u_{2} \vert ^{p}\,dx \leq I_{ \mu _{1}}(u_{1})+I_{\mu _{2}}(u_{2}). $$
Then, from Lemma 3.2,
$$\begin{aligned} \begin{aligned} E(t_{1}\star w_{1},\rho _{2}\star w_{2}) &\leqslant I_{\mu _{1}}(t_{1} \star w_{1})+I_{\mu _{2}}(\rho _{2}\star w_{2}) \\ &\leqslant I_{\mu _{1}}(t_{1}\star w_{1})+ \varepsilon \\ &\leqslant \sup_{l\in \mathbb{R}}I_{\mu _{1}}(l\star w_{1})+ \varepsilon . \end{aligned} \end{aligned}$$
By Lemma 2.3, we have
$$ w_{a_{i},\mu _{i}}=\bar{l}_{i}\star w_{i}, \quad \text{for } e^{\bar{l}_{i}}:=\biggl( \frac{\mu _{i}+\beta }{\mu _{i}} \biggr)^{\frac{1}{s[(p-1)N-2s]}}. $$
Then, due to \(l_{1}\star (l_{2}\star w)=(l_{1}+l_{2})\star w\) for every \(l_{1},l_{2}\in \mathbb{R}\) and \(w\in H^{s}(\mathbb{R})\), we have
$$ \sup_{l\in \mathbb{R}}I_{\mu _{1}}(l\star w_{1})= \sup_{l\in \mathbb{R}}I_{\mu _{1}}(l\star w_{a_{1},\mu _{1}}). $$
As a consequence of Lemma 2.5,
$$ \sup_{l\in \mathbb{R}}I_{\mu _{1}}(l\star w_{a_{1},\mu _{1}})=I_{ \mu _{1}}(w_{a_{1},\mu _{1}}). $$
Therefore, we have
$$ E(t_{1}\star w_{1},\rho _{2}\star w_{2})\leqslant I_{\mu _{1}}(w_{a_{1}, \mu _{1}})+\varepsilon , \quad \forall t_{1}\in [\rho _{1},R_{1}]. $$
Similarly, we have
$$\begin{aligned}& E(\rho _{1}\star w_{1},t_{2}\star w_{2})\leqslant I_{\mu _{2}}(w_{a_{2}, \mu _{2}})+\varepsilon ,\quad \forall t_{2}\in [\rho _{2},R_{2}], \\& \begin{aligned} E(t_{1}\star w_{1},R_{2}\star w_{2}) &\leqslant I_{\mu _{1}}(t_{1} \star w_{1})+I_{\mu _{2}}(R_{2}\star w_{2}) \\ &\leqslant \sup_{l\in \mathbb{R}} I_{\mu _{1}}(l\star w_{1})=I_{\mu _{1}}(w_{a_{1}, \mu _{1}}), \quad \forall t_{1}\in [\rho _{1},R_{1}], \end{aligned} \end{aligned}$$
and
$$ E(R_{1}\star w_{1},t_{2}\star w_{2})\leqslant I_{\mu _{2}}(w_{a_{2}, \mu _{2}}), \quad \forall t_{2}\in [\rho _{2},R_{2}]. $$
Hence, the conclusion of Lemma 3.3 holds. □
Lemma 3.4
For every \(\gamma \in \Gamma \), there exists \((t_{1,\gamma },t_{2,\gamma })\in Q\) such that \(\gamma (t_{1,\gamma },t_{2,\gamma })\in \mathcal{P}(a_{1},\mu _{1}+ \beta )\times \mathcal{P}(a_{2},\mu _{2}+\beta )\).
Proof
For \(\gamma \in \Gamma \), we use the notation \(\gamma (t_{1},t_{2})=(\gamma _{1}(t_{1},t_{2}),\gamma _{2}(t_{1},t_{2})) \in H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}\). Considering the map \(F_{\gamma }:Q\rightarrow \mathbb{R}^{2}\) defined by
$$ F_{\gamma }(t_{1},t_{2}):= \biggl( \frac{\partial }{\partial l}I_{\mu _{1}+ \beta }\bigl(l\star \gamma _{1}(t_{1},t_{2}) \bigr)\big|_{l=0}, \frac{\partial }{\partial l}I_{\mu _{2}+\beta }\bigl(l\star \gamma _{2}(t_{1},t_{2})\bigr)\big|_{l=0} \biggr). $$
From
$$\begin{aligned} &\frac{\partial }{\partial l}I_{\mu _{i}+\beta }\bigl(l\star \gamma _{i}(t_{1},t_{2})\bigr)\big|_{l=0} \\ &\quad =\frac{\partial }{\partial l} \biggl(\frac{e^{2s^{2}l}}{2} \bigl\Vert (-\Delta )^{\frac{s}{2}}\gamma _{i}(t_{1},t_{2}) \bigr\Vert ^{2}_{L^{2}}- \frac{e^{(p-1)Nsl}}{2p}(\mu _{i}+\beta ) \bigl\Vert \gamma _{i}(t_{1},t_{2}) \bigr\Vert ^{2p}_{L^{2p}} \biggr)\bigg|_{l=0} \\ &\quad =s^{2} \bigl\Vert (-\Delta )^{\frac{s}{2}}\gamma _{i}(t_{1},t_{2}) \bigr\Vert ^{2}_{L^{2}}- \frac{(p-1)Ns}{2p}(\mu _{i}+ \beta ) \bigl\Vert \gamma _{i}(t_{1},t_{2}) \bigr\Vert ^{2p}_{L^{2p}}, \end{aligned}$$
we deduce that
$$ F_{\gamma }(t_{1},t_{2})=(0,0) \quad \text{if and only if} \quad \gamma (t_{1},t_{2}) \in \mathcal{P}(a_{1},\mu _{1}+\beta )\times \mathcal{P}(a_{2},\mu _{2}+ \beta ). $$
Now, we will show that \(F_{\gamma }(t_{1},t_{2})=(0,0)\) has a solution in Q for every \(\gamma \in \Gamma \). Since
$$\begin{aligned} F_{\gamma _{0}}(t_{1},t_{2})={}& \biggl(s^{2}e^{2s^{2}t_{1}} \bigl\Vert (-\Delta )^{\frac{s}{2}} w_{1} \bigr\Vert ^{2}_{L^{2}}- \frac{(p-1)Ns}{2p}e^{(p-1)Nst_{1}}( \mu _{1}+\beta ) \Vert w_{1} \Vert ^{2p}_{L^{2p}}, \\ & s^{2}e^{2s^{2}t_{2}} \bigl\Vert (-\Delta )^{\frac{s}{2}}w_{2} \bigr\Vert ^{2}_{L^{2}}- \frac{(p-1)Ns}{2p}e^{(p-1)Nst_{2}}( \mu _{2}+\beta ) \Vert w_{2} \Vert ^{2p}_{L^{2p}} \biggr) \\ ={}&\bigl(\tilde{\varphi }_{1}(t_{1}),\tilde{\varphi }_{2}(t_{2})\bigr). \end{aligned}$$
By Lemma 3.2, we get \((0,0)\notin F_{\gamma _{0}}(\partial Q)\), and \((0,0)\) is the only solution to \(F_{\gamma _{0}}(t_{1},t_{2})=(0,0)\) in Q. It is easy to compute
$$ \deg \bigl(F_{\gamma _{0}},Q,(0,0)\bigr)=\operatorname{sgn}\bigl(\tilde{ \varphi }'_{1}(0)\cdot \tilde{\varphi }'_{2}(0)\bigr)=1. $$
Now, for any \(\gamma \in \Gamma \), since \(F_{\gamma }(\partial ^{+}Q)=F_{\gamma _{0}}(\partial ^{+}Q)\), therefore, \((0,0)\notin F_{\gamma }(\partial Q)\), we get
$$ \deg \bigl(F_{\gamma },Q,(0,0)\bigr)=\deg \bigl(F_{\gamma _{0}},Q,(0,0) \bigr)=1. $$
Hence, there exists a \((t_{1,\gamma },t_{2,\gamma })\in Q\) such that \(F_{\gamma }(t_{1,\gamma },t_{2,\gamma })=(0,0)\). □
Lemma 3.5
There exists a bounded Palais–Smale sequence \((u_{n},v_{n})\) for E on \({H}^{\mathrm{rad}}_{a_{1}}\times {H}^{\mathrm{rad}}_{a_{2}}\) at the level
$$ c:=\inf_{\gamma \in \Gamma } \max_{(t_{1},t_{2})\in Q} E\bigl(\gamma (t_{1},t_{2})\bigr)> \max \bigl\{ I_{\mu _{1}}(w_{a_{1},\mu _{1}}), I_{\mu _{2}}(w_{a_{2},\mu _{2}}) \bigr\} , $$
(39)
satisfying the additional condition
$$ G(u_{n},v_{n})=o(1), $$
(40)
where \(o(1)\rightarrow 0\) as \(n\rightarrow \infty \). Furthermore, there exists \(\bar{C}>0\) such that
$$ \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u_{n} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}v_{n} \bigr\vert ^{2}\bigr)\,dx\geqslant \bar{C} \quad \textit{for all } n, $$
and \(u_{n}^{-},v_{n}^{-}\rightarrow 0\) a.e. in \(\mathbb{R}^{N}\) as \(n\rightarrow \infty \).
Proof
The idea comes from [3]. Equation (39) is simply from Lemma 3.4. We consider the augmented functional \(\tilde{E}:\mathbb{R}\times H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}} \rightarrow \mathbb{R}\) defined by \(\tilde{E}(l,u_{1},u_{2}):=E(l\star u_{1},l\star u_{2})\). Let
$$\begin{aligned}& \tilde{\gamma }(t_{1},t_{2}):=\bigl(l(t_{1},t_{2}), \gamma _{1}(t_{1},t_{2}), \gamma _{2}(t_{1},t_{2})\bigr), \\& \tilde{\gamma }_{0}(t_{1},t_{2}):=\bigl(0, \gamma _{0}(t_{1},t_{2}) \bigr)=(0,t_{1} \star w_{1},t_{2}\star w_{2}), \\& \tilde{\Gamma }:=\bigl\{ \tilde{\gamma }\in C\bigl(Q,\mathbb{R}\times H^{\mathrm{rad}}_{a_{1}} \times H^{\mathrm{rad}}_{a_{2}}: \tilde{\gamma }=\tilde{\gamma }_{0} \text{ on } \partial Q \bigr)\bigr\} , \end{aligned}$$
and
$$ \tilde{c}:=\inf_{\tilde{\gamma }\in \tilde{\Gamma }} \max_{(t_{1},t_{2}) \in Q} \tilde{E}\bigl(\tilde{\gamma }(t_{1},t_{2})\bigr). $$
Since, for any \(\gamma (t_{1},t_{2})=(\gamma _{1}(t_{1},t_{2}),\gamma _{2}(t_{1},t_{2})) \in \Gamma \), \((0,\gamma _{1}(t_{1},t_{2}),\gamma _{2}(t_{1},t_{2}))\in \tilde{\Gamma }\), we have \(\tilde{c}\leqslant c\). On the other hand, for any \(\tilde{\gamma }\in \tilde{\Gamma }\) and \((t_{1},t_{2})\in Q\), we have
$$ \tilde{E}\bigl(\tilde{\gamma }(t_{1},t_{2})\bigr)=E \bigl(l(t_{1},t_{2})\star \gamma _{1}(t_{1},t_{2}),l(t_{1},t_{2}) \star \gamma _{2}(t_{1},t_{2})\bigr), $$
and \((l(\cdot )\star \gamma _{1}(\cdot ),l(\cdot )\star \gamma _{2}( \cdot ))\in \Gamma \) due to \(\tilde{\gamma }=\tilde{\gamma }_{0}\) on ∂Q, so \(c\leqslant \tilde{c}\). Hence, \(c=\tilde{c}\).
Now take a sequence of \(\{\tilde{\gamma }_{n}\}\subset \tilde{\Gamma }\) such that
$$ \lim_{n\to +\infty }\max_{(t_{1},t_{2})\in Q} \tilde{E}\bigl( \tilde{\gamma }_{n}(t_{1},t_{2})\bigr)= \tilde{c}=c. $$
We may also assume that \(\tilde{\gamma }_{n}=(l_{n},\gamma _{1,n},\gamma _{2,n})\) satisfies the following two additional properties: for all \((t_{1},t_{2})\in Q\):
-
\(l_{n}(t_{1},t_{2})\equiv 0\),
-
\(\gamma _{1,n}(t_{1},t_{2})\ge 0\), \(\gamma _{2,n}(t_{1},t_{2})\ge 0\), a.e. in \(\mathbb{R}^{N}\).
The first property comes from the fact that
$$\begin{aligned} \tilde{E}\bigl(\tilde{\gamma }(t_{1},t_{2})\bigr) &=E \bigl(l(t_{1},t_{2})\star \gamma _{1}(t_{1},t_{2}),l(t_{1},t_{2}) \star \gamma _{2}(t_{1},t_{2})\bigr) \\ &=\tilde{E}\bigl(0,l(t_{1},t_{2})\star \gamma _{1}(t_{1},t_{2}),l(t_{1},t_{2}) \star \gamma _{2}(t_{1},t_{2})\bigr), \end{aligned}$$
and the second one is the consequence of \(\tilde{E}(l,|u|,|v|)\le \tilde{E}(l,u,v)\) and the definition of c̃.
Applying Theorem 3.2 in [20], there exists a Palais–Smale sequence \((l_{n},u_{n},v_{n})\) for Ẽ on \(\mathbb{R}\times H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}\) at level c̃, such that
-
\(\lim_{n\to +\infty }\tilde{E}(l_{n},u_{n},v_{n})=\tilde{c}=c\),
-
\(\lim_{n\to +\infty }|l_{n}|+\operatorname{dist}((u_{n},v_{n}), \tilde{\gamma }_{n}(Q))=0\),
-
For all \(u,v\in H_{r}^{s}(\mathbb{R}^{N})\) with \(\int _{\mathbb{R}^{N}}u_{n}u\,dx=0\), \(\int _{\mathbb{R}^{N}}v_{n}v\,dx=0\) and \(\forall l\in \mathbb{R}\),
$$ \bigl\langle \tilde{E}'(l_{n},u_{n},v_{n}),(l,u,v) \bigr\rangle =o(1) \bigl( \vert l \vert + \Vert u \Vert _{H^{s}}+ \Vert v \Vert _{H^{s}}\bigr). $$
Take \((l,u,v)=(1,0,0)\), direct calculations gives
$$\begin{aligned} \begin{aligned} &\bigl\langle \tilde{E}'(l_{n},u_{n},v_{n}),(1,0,0) \bigr\rangle \\ &\quad =s^{2}e^{2s^{2}l_{n}} \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u_{n} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}v_{n} \bigr\vert ^{2}\bigr)\,dx \\ & \qquad {}-\frac{e^{(p-1)Nsl_{n}}(p-1)Ns}{2p} \int _{\mathbb{R}^{N}}\bigl(\mu _{1} \vert u_{n} \vert ^{2p}+2 \beta \vert u_{n} \vert ^{p} \vert v_{n} \vert ^{p}+\mu _{2} \vert v_{n} \vert ^{2p}\bigr)\,dx. \end{aligned} \end{aligned}$$
(41)
From the above, we can get
$$\begin{aligned} &se^{2s^{2}l_{n}}\biggl(\frac{(p-1)N}{2}-s\biggr) \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u_{n} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}v_{n} \bigr\vert ^{2}\bigr)\,dx \\ &\quad =(p-1)Ns\tilde{E}(l_{n},u_{n},v_{n})- \bigl\langle \tilde{E}'(l_{n},u_{n},v_{n}),(1,0,0) \bigr\rangle \\ &\quad \to (p-1)Nsc, \quad \text{as } n\to +\infty . \end{aligned}$$
Since \(l_{n}\to 0\) and \(p>1+\frac{2s}{N}\), we see that there exist \(\bar{C}>0\) and \(C>0\), such that
$$ \bar{C}\le \int _{\mathbb{R}^{N}}\bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u_{n} \bigr\vert ^{2}+ \bigl\vert (- \Delta )^{\frac{s}{2}}v_{n} \bigr\vert ^{2}\bigr)\,dx\le C, $$
(42)
therefore \((u_{n},v_{n})\) is bounded in \(H^{s}_{r}(\mathbb{R}^{N})\times H^{s}_{r}(\mathbb{R}^{N})\). Using \(l_{n}\to 0\) and (41) again, we conclude that \((u_{n},v_{n})\) satisfies (40). Now take \((l,u,v)=(0,u,v)\) for any \((u,v)\in H^{s}_{r}(\mathbb{R}^{N})\times H^{s}_{r}(\mathbb{R}^{N})\) with \(\int _{\mathbb{R}^{N}}u_{n}u\,dx=0\), \(\int _{\mathbb{R}^{N}}v_{n}v\,dx=0\), due to the boundedness of \((u_{n},v_{n})\) and \(l_{n}\to 0\), it is easy to see that
$$\begin{aligned} \bigl\langle E'(u_{n},v_{n}),(u,v)\bigr\rangle &=\bigl\langle \tilde{E}'(l_{n},u_{n},v_{n}),(0,u,v) \bigr\rangle +O\bigl( \vert l_{n} \vert \bigr) \bigl( \Vert u \Vert _{H^{s}}+ \Vert v \Vert _{H^{s}}\bigr) \\ &=o(1) \bigl( \Vert u \Vert _{H^{s}}+ \Vert v \Vert _{H^{s}}\bigr). \end{aligned}$$
Therefore, \((u_{n},v_{n})\) is a bounded Palais–Smale sequence for E on \(H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}\) at level c with additional condition (40). Finally, \(u_{n}^{-},v_{n}^{-}\rightarrow 0\) a.e. in \(\mathbb{R}^{N}\) as \(n\rightarrow \infty \) is a simple consequence of \(\gamma _{1,n}(t_{1},t_{2})\ge 0\), \(\gamma _{2,n}(t_{1},t_{2})\ge 0\) and \(\lim_{n\to +\infty } \operatorname{dist}((u_{n},v_{n}),\tilde{\gamma }_{n}(Q))=0\). □
From Lemma 3.5, there exist nonnegative functions ũ, ṽ in \(H_{r}^{s}(\mathbb{R}^{N})\), such that, up to a subsequence,
$$\begin{aligned} &(u_{n},v_{n})\rightharpoonup (\tilde{u},\tilde{v}), \quad \text{weakly in } H^{s}\bigl(\mathbb{R}^{N}\bigr)\times H^{s}\bigl(\mathbb{R}^{N}\bigr), \\ &(u_{n},v_{n})\rightarrow (\tilde{u},\tilde{v}), \quad \text{strongly in } L^{2p}\bigl(\mathbb{R}^{N}\bigr)\times L^{2p}\bigl(\mathbb{R}^{N}\bigr), \\ &(u_{n},v_{n})\rightarrow (\tilde{u},\tilde{v}), \quad \mbox{a.e. in } \mathbb{R}^{N}. \end{aligned}$$
As a consequence \(E'|_{H^{\mathrm{rad}}_{a_{1}}\times H^{\mathrm{rad}}_{a_{2}}}(u_{n},v_{n})\rightarrow 0\), there exist two sequences of real number \(\{\lambda _{1,n}\}\) and \(\{\lambda _{2,n}\}\) such that
$$\begin{aligned} \begin{aligned} & \int _{\mathbb{R}^{N}} \bigl((-\Delta )^{\frac{s}{2}}u_{n}(- \Delta )^{ \frac{s}{2}}g+(-\Delta )^{\frac{s}{2}}v_{n}(-\Delta )^{\frac{s}{2}}h- \mu _{1} \vert u_{n} \vert ^{2p-2}u_{n}g-\mu _{2} \vert v_{n} \vert ^{2p-2}v_{n}h \bigr)\,dx \\ &\qquad {}- \int _{\mathbb{R}^{n}}\bigl(\beta \vert u_{n} \vert ^{p-2} \vert v_{n} \vert ^{p}u_{n}g+ \vert u_{n} \vert ^{p} \vert v_{n} \vert ^{p-2}v_{n}h\bigr)\,dx+ \int _{\mathbb{R}^{N}}(\lambda _{1,n}u_{n}g+ \lambda _{2,n}v_{n}h)\,dx \\ &\quad =o(1) \bigl( \Vert g \Vert _{H^{s}}+ \Vert h \Vert _{H^{s}}\bigr), \end{aligned} \end{aligned}$$
(43)
for every \(g, h\in H^{s}({\mathbb{R}^{N}})\) with \(o(1)\rightarrow 0\), \(\text{as} n\rightarrow \infty \).
Lemma 3.6
Both \(\{\lambda _{1,n}\}\) and \(\{\lambda _{2,n}\}\) are bounded sequences and at least one of them is converging, up to a sequence, to a positive value.
Proof
By using \((u_{n},0)\) and \((0,v_{n})\) as test functions in (43), we get
$$\begin{aligned} & \int _{\mathbb{R}^{N}} \bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}u_{n} \bigr\vert ^{2}-\mu _{1} \vert u_{n} \vert ^{2p}- \beta \vert u_{n} \vert ^{p} \vert v_{n} \vert ^{p} \bigr)\,dx+ \lambda _{1,n}a_{1}^{2}=o(1), \\ & \int _{\mathbb{R}^{N}} \bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}v_{n} \bigr\vert ^{2}-\mu _{2} \vert v_{n} \vert ^{2p}- \beta \vert u_{n} \vert ^{p} \vert v_{n} \vert ^{p} \bigr)\,dx+ \lambda _{2,n}a_{2}^{2}=o(1), \end{aligned}$$
with \(o(1)\rightarrow 0\), \(\text{as} n\rightarrow \infty \). Hence the boundedness of \(\{\lambda _{i,n}\}\) follows from the boundedness of \(u_{n}\), \(v_{n}\) in \(H^{s}({\mathbb{R}^{N}})\) and in \(L^{2p}({\mathbb{R}^{N}})\). Furthermore, since \((u_{n},v_{n})\) satisfies (40),
$$\begin{aligned} &{\lambda _{1,n}} {a_{1}^{2}}+{\lambda _{2,n}} {a_{2}^{2}} \\ &\quad =- \int _{ \mathbb{R}^{N}} \bigl({ \bigl\vert (-\Delta )^{\frac{s}{2}}u_{n} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{ \frac{s}{2}}v_{n} \bigr\vert ^{2}}-\mu _{1}{ \vert u_{n} \vert ^{2p}}-2\beta { \vert u_{n} \vert ^{p}} { \vert v_{n} \vert ^{p}}- \mu _{2}{ \vert v_{n} \vert ^{2p}} \bigr)\,dx+o(1) \\ &\quad =\biggl(\frac{2ps}{(p-1)N}-1\biggr) \int _{\mathbb{R}^{N}} \bigl( \bigl\vert (-\Delta )^{ \frac{s}{2}}u_{n} \bigr\vert ^{2}+ \bigl\vert (-\Delta )^{\frac{s}{2}}v_{n} \bigr\vert ^{2} \bigr)\,dx+o(1), \end{aligned}$$
therefore by (42),
$$ \biggl(\frac{ps}{(p-1)N}-\frac{1}{2}\biggr)\bar{C}\le {\lambda _{1,n}} {a_{1}^{2}}+{ \lambda _{2,n}} {a_{2}^{2}}\le 2\biggl( \frac{2ps}{(p-1)N}-1\biggr)C, $$
for \(1+\frac{2s}{N}< p<\frac{N}{N-2s}\) and every n sufficiently large. Therefore, at least one sequence of \(\{\lambda _{i,n}\}\) is positive and bounded away from 0. This shows that at least one sequence of \(\{\lambda _{i,n}\}\) is converging, up to a sequence, to a positive value. □
Next, we consider converging subsequence \(\lambda _{1,n}\rightarrow \tilde{\lambda }_{1}\in {\mathbb{R}}\) and \(\lambda _{2,n}\rightarrow \tilde{\lambda }_{2}\in {\mathbb{R}}\), as \(n\rightarrow \infty \). The sign of \(\tilde{\lambda }_{i}\) plays an important role for the strong convergence of \(u_{n}\), \(v_{n}\) in \(H^{s}(\mathbb{R}^{N})\).
Lemma 3.7
If \(\tilde{\lambda }_{1}>0\) (resp. \(\tilde{\lambda }_{2}>0\)), then \(u_{n}\rightarrow \tilde{u}\) (resp. \(v_{n}\rightarrow \tilde{v}\)) strongly in \(H^{s}(\mathbb{R}^{N})\).
Proof
Let us suppose that \(\tilde{\lambda }_{1}>0\). By the weak convergence of \(u_{n}\) in \(H^{s}(\mathbb{R}^{N})\) and the strong convergence in \(L^{2p}(\mathbb{R}^{N})\), it is easy to get from (43)
$$\begin{aligned} \begin{aligned} o(1)&=\bigl\langle E'({u_{n}},{v_{n}})-E'( \tilde{u},\tilde{v}),({u_{n}}- \tilde{u},0)\bigr\rangle +{\tilde{ \lambda }_{1}} \int _{\mathbb{R}^{N}}({u_{n}}- \tilde{u})^{2}\,dx \\ &= \int _{\mathbb{R}^{N}} \bigl( \bigl\vert (-\Delta )^{\frac{s}{2}}({u_{n}}- \tilde{u}) \bigr\vert ^{2}+\tilde{\lambda }_{1}({u_{n}}- \tilde{u})^{2} \bigr)\,dx+o(1), \end{aligned} \end{aligned}$$
with \(o(1)\rightarrow 0\) and \(n\rightarrow \infty \). Since \(\tilde{\lambda }_{1}>0\), this is equivalent to the strong convergence of \(u_{n}\) in \(H^{s}(\mathbb{R}^{N})\). The proof in the case \(\tilde{\lambda }_{2}>0\) is similar. □
Having arrived at the end of this section, we give the proof of Theorem 1.1.
Proof of Theorem 1.1
By the convergence of \(\{\lambda _{1,n}\}\) and \(\{\lambda _{2,n}\}\), and the weak convergence \((u_{n},v_{n})\rightharpoonup (\tilde{u},\tilde{v})\), we see that \((\tilde{\lambda }_{1},\tilde{\lambda }_{2},\tilde{u},\tilde{v})\) is a solution of (1) with at least one \(\tilde{\lambda }_{i}\) positive. We will show that both \(\tilde{\lambda }_{1}\), \(\tilde{\lambda }_{2}\) are positive, hence by Lemma 3.7, \(\tilde{u}\in H_{a_{1}}\), \(\tilde{v}\in H_{a_{2}}\) and the proof is complete.
We prove by contradiction. Without loss of generality, by Lemma 3.7, we may assume that \(\tilde{\lambda }_{1}>0\) and \(\tilde{\lambda }_{2}\le 0\). Since \((\tilde{\lambda }_{1},\tilde{\lambda }_{2},\tilde{u},\tilde{v})\) is a solution of (1) and \(\tilde{u},\tilde{v}\geq 0\), we have
$$ (-\Delta )^{s} \tilde{v}=-\tilde{\lambda }_{2} \tilde{v} +\mu _{2} { \tilde{v}}^{2p-1}+\beta {\tilde{u}}^{p}{ \tilde{v}}^{p-1}\geq 0\quad \text{in } {\mathbb{R}^{N}}, $$
and since \(2s< N\le 4s\), i.e., \(2\le \frac{N}{N-2s}\), from Lemma 2.7(i), we can deduce that \(\tilde{v}\equiv 0\). In particular, this implies that ũ solves
$$ \textstyle\begin{cases} (-\Delta )^{s}{\tilde{u}}+{ \tilde{\lambda }_{1}} {\tilde{u}} -{\mu _{1}} { \tilde{u}}^{2p-1}=0 \quad \text{in } {\mathbb{R}^{N}}, \\ \int _{\mathbb{R}^{N}} \tilde{u}^{2}\,dx=a_{1}^{2},\quad \text{and}\quad \tilde{u}>0 \quad \text{in } {\mathbb{R}^{N}}, \end{cases} $$
(44)
so that \(\tilde{u}=w_{a_{1},\mu _{1}}\in \mathcal{P}(a_{1},\mu _{1})\). However, due to strong convergence of \(u_{n}\), \(v_{n}\) in \(L^{2p}(\mathbb{R}^{N})\), we obtain due to (40),
$$\begin{aligned} c&=\lim_{n\rightarrow \infty }E(u_{n},v_{n})=\lim _{n\rightarrow \infty }{\frac{(p-1)N-2s}{4ps}} \int _{\mathbb{R}^{N}}\bigl(\mu _{1} \vert u_{n} \vert ^{2p}+2 \beta \vert u_{n} \vert ^{p} \vert v_{n} \vert ^{p}+\mu _{2} \vert v_{n} \vert ^{2p}\bigr)\,dx \\ &=\frac{(p-1)N-2s}{4ps} \int _{\mathbb{R}^{N}}\mu _{1} \vert {w_{a_{1},\mu _{1}}} \vert ^{2p}\,dx=I_{ \mu _{1}}(w_{a_{1},\mu _{1}}). \end{aligned}$$
This is a contradiction with Lemma 3.5. Therefore, both \(\tilde{\lambda }_{1}\), \(\tilde{\lambda }_{2}\) are positive. □