In this section, we define
$$ B=\inf_{\eta \in \Gamma }\max_{t\in [0,1]} I\bigl(\eta (t) \bigr), $$
where
$$ \Gamma =\bigl\{ \eta \in {\mathcal{C}}\bigl([0,1],H\bigr)| \eta (0)=(0,0), I \bigl( \eta (1)\bigr)< 0\bigr\} . $$
Set
$$ \mathcal{N}=\bigl\{ (u,v)\in H\setminus \bigl\{ (0,0)\bigr\} | \bigl\langle I'(u,v), (u,v) \bigr\rangle =0\bigr\} . $$
By simple calculation and analysis, we see that for any \((u,v)\ne (0,0)\), there exists \(t_{0}>0\) such that \(t_{0}(u,v)\in \mathcal{N}\) and \(I(t_{0}u,t_{0}v)=\max_{t\ge 0}I(tu,tv)\). Then, as in the proof of [23, Theorem 4.2], we know that
$$ B=\inf_{(u,v)\in H\setminus {(0,0)}}\max_{t\ge 0} I(tu,tv)=\inf _{ \mathcal{N}}I(u,v). $$
Moreover, since \({\mathcal{M}}\subset {\mathcal{N}}\), we have \(B\le \mathcal{B}\). We will show that B is attained by some positive solution \((u,v)\) of system (1.1). To begin with, we give an estimate of the upper bound of B, which is important in recovering the compactness of the Palais–Smale sequence.
Lemma 4.1
Assume that \((C1)\) and \((C2)\) hold. If \(\beta >0\), then
$$ B< \min \biggl\{ B_{1},B_{2},\frac{\alpha }{2(N+\alpha )}S_{0}^{ \frac{N+\alpha }{\alpha }} \biggr\} . $$
Proof
We first show that
$$ B< \frac{\alpha }{2(N+\alpha )}S_{0}^{\frac{N+\alpha }{\alpha }}. $$
(4.1)
Recall \((s_{m}U_{*}, U_{*})\) defined in Theorem 1.5, and let \(t>0\) be the constant such that \(t(s_{m}U_{*}, U_{*})\in \mathcal{N}\). Then, by Theorem 1.5 and direct calculation, we see that
$$\begin{aligned} B&\le I\bigl(t(s_{m}U_{*}, U_{*})\bigr) \\ & = \frac{1}{2}t^{2} \int \bigl(1+s_{m}^{2}\bigr) \vert \nabla U_{*} \vert ^{2}+\bigl(\lambda _{1}(x)s_{m}^{2}+ \lambda _{2}(x)\bigr)U_{*}^{2} \\ &\quad {}- \frac{N}{2(N+\alpha )}t^{ \frac{2(N+\alpha )}{N}} \bigl(\mu _{2}+\mu _{1}s_{m}^{ \frac{2(N+\alpha )}{N}}+2\beta s_{m}^{\frac{N+\alpha }{N}} \bigr) \int \bigl(I_{\alpha }* \vert U_{*} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert U_{*} \vert ^{ \frac{N+\alpha }{N}} \\ & = \frac{1}{2}t^{2} \int \bigl(1+s_{m}^{2}\bigr) U_{*}^{2} \\ &\quad {}- \frac{N}{2(N+\alpha )}t^{ \frac{2(N+\alpha )}{N}} \bigl(\mu _{2}+\mu _{1}s_{m}^{ \frac{2(N+\alpha )}{N}}+2\beta s_{m}^{\frac{N+\alpha }{N}} \bigr) \int \bigl(I_{\alpha }* \vert U_{*} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert U_{*} \vert ^{ \frac{N+\alpha }{N}} \\ & \quad{} + \frac{1}{2}t^{2} \int s_{m}^{2} \vert \nabla U_{*} \vert ^{2}+s_{m}^{2}\bigl(\lambda _{1}(x)-1\bigr)U_{*}^{2}+ \vert \nabla U_{*} \vert ^{2}+\bigl( \lambda _{2}(x)-1 \bigr)U_{*}^{2} \\ & \le \frac{\alpha }{2(N+\alpha )}\bigl(g(s_{m})S_{1} \bigr)^{ \frac{N+\alpha }{\alpha }}\\ &\quad {}+\frac{1}{2}t^{2} \int \bigl(1+s_{m}^{2}\bigr) \vert \nabla U_{*} \vert ^{2}+s_{m}^{2}\bigl( \lambda _{1}(x)-1\bigr)U_{*}^{2}+\bigl(\lambda _{2}(x)-1\bigr)U_{*}^{2} \\ &= \frac{\alpha }{2(N+\alpha )}S_{0}^{ \frac{N+\alpha }{\alpha }}+\frac{1}{2}t^{2} \int \bigl(1+s_{m}^{2}\bigr) \vert \nabla U_{*} \vert ^{2}+s_{m}^{2}\bigl( \lambda _{1}(x)-1\bigr)U_{*}^{2}+\bigl(\lambda _{2}(x)-1\bigr)U_{*}^{2}. \end{aligned}$$
Denote \(\phi _{i}(u)=\frac{1}{2}\int |\nabla u|^{2}+(\lambda _{i}(x)-1)u^{2}\), \(i=1,2 \). To get (4.1), it suffices to show
$$ \phi _{i}(U_{*})< 0, \quad i=1,2, $$
(4.2)
for some \(b\in {\mathbb{R}}^{N}\). By the fact that
$$ \int \frac{ \vert x \vert ^{2}}{(1+ \vert x \vert ^{2})^{N+2}}=\frac{N-2}{4(N+1)} \int \frac{1}{x^{2}(1+x^{2})^{N}}, $$
we obtain
$$ \int \vert \nabla U_{*} \vert ^{2}= \frac{N^{2}(N-2)}{4(N+1)} \int \frac{ \vert U_{*} \vert ^{2}}{ \vert x \vert ^{2}}. $$
After a transformation \(x=b+a y\), we have
$$ a^{2}\phi _{i}(U_{*})= \int \biggl(\frac{N^{2}(N-2)}{4(N+1) \vert y \vert ^{2}}-a^{2}\bigl(1- \lambda _{i}(b+a y)\bigr) \biggr)\frac{C^{2}}{(1+ \vert y \vert ^{2})^{N}}\,dy. $$
Then from \((C2)\) we see that (4.2) holds for \(b=0\), and (4.1) follows.
Next, we show \(B< B_{i}\), \(i=1,2\). Let \(w_{i}\) be a positive solution of (1.9) for \(i=1, 2\) and \(t(\tau )>0\) such that \((\sqrt{t(\tau )}w_{1},\sqrt{t(\tau )}\tau w_{1})\in \mathcal{N}\). Then
$$ t(\tau )^{\frac{\alpha }{N}}= \frac{ \int \vert \nabla w_{1} \vert ^{2}+\lambda _{1}(x)w_{1}^{2}+\tau ^{2}( \vert \nabla w_{1} \vert ^{2}+\lambda _{2}(x)w_{1}^{2})}{ (\mu _{1}+2\beta \tau ^{\frac{N+\alpha }{N}}+\mu _{2}\tau ^{\frac{2(N+\alpha )}{N}})\int (I_{\alpha }* \vert w_{1} \vert ^{\frac{N+\alpha }{N}}) \vert w_{1} \vert ^{\frac{N+\alpha }{N}}}. $$
By simple calculation, we get
$$ \lim_{\tau \to 0^{+}} \frac{t'(\tau )}{ \vert \tau \vert ^{\frac{\alpha }{N}-1}\tau }=- \frac{2(N+\alpha )}{\alpha \mu _{1}}\beta . $$
It follows that
$$ t(\tau )=1-\frac{2N}{\alpha \mu _{1}}\beta \tau ^{\frac{N+\alpha }{N}}\bigl(1+o(1)\bigr), \quad \text{as } \tau \to 0, $$
and
$$ t(\tau )^{\frac{N+\alpha }{N}}=1-\frac{2(N+\alpha )}{\alpha \mu _{1}} \beta \tau ^{\frac{N+\alpha }{N}} \bigl(1+o(1)\bigr),\quad \text{as } \tau \to 0. $$
Therefore,
$$ \begin{aligned} B&\le I\bigl(\sqrt{t(\tau )}w_{1},\sqrt{t( \tau )}\tau w_{1}\bigr) \\ & = \frac{\alpha }{2(N+\alpha )}t(\tau )^{ \frac{N+\alpha }{N}}\bigl(\mu _{1}+2\beta \tau ^{\frac{N+\alpha }{N}}+\mu _{2} \tau ^{\frac{2(N+\alpha )}{N}}\bigr) \int \bigl(I_{\alpha }* \vert w_{1} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert w_{1} \vert ^{\frac{N+\alpha }{N}} \\ & = \frac{\alpha }{2(N+\alpha )} \int \mu _{1}\bigl(I_{ \alpha }* \vert w_{1} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert w_{1} \vert ^{\frac{N+\alpha }{N}}\\ &\quad {}- \frac{N}{N+\alpha }\beta \tau ^{\frac{N+\alpha }{N}} \int \bigl(I_{\alpha }* \vert w_{1} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert w_{1} \vert ^{\frac{N+\alpha }{N}} +o\bigl( \tau ^{ \frac{N+\alpha }{N}}\bigr) \\ & < B_{1}\quad \text{for } \tau >0 \text{ small enough.} \end{aligned} $$
Similarly, we have \(B< B_{2}\). □
Next, we prove a Brezis–Lieb type lemma.
Lemma 4.2
Let \(\{(u_{n},v_{n})\}\) be a bounded sequence in H, and \((u_{n},v_{n})\to (u,v)\) a.e on \({\mathbb{R}}^{N}\) as \(n\to \infty \). Then
$$ \int \bigl(I_{\alpha }* \vert u_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n} \vert ^{ \frac{N+\alpha }{N}}- \int \bigl(I_{\alpha }* \vert u_{n}-u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n}-v \vert ^{ \frac{N+\alpha }{N}}\to \int \bigl(I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v \vert ^{ \frac{N+\alpha }{N}} $$
as \(n\to \infty \).
Proof
From the Brezis–Lieb lemma [23], we know that
$$\begin{aligned}& \vert u_{n} \vert ^{\frac{N+\alpha }{N}}- \vert u_{n}-u \vert ^{\frac{N+\alpha }{N}}\to \vert u \vert ^{ \frac{N+\alpha }{N}},\quad \text{in } L^{\frac{2N}{N+\alpha }}\bigl({ \mathbb{R}}^{N}\bigr), \\& \vert v_{n} \vert ^{\frac{N+\alpha }{N}}- \vert v_{n}-v \vert ^{\frac{N+\alpha }{N}}\to \vert v \vert ^{ \frac{N+\alpha }{N}},\quad \text{in } L^{\frac{2N}{N+\alpha }}\bigl({ \mathbb{R}}^{N}\bigr), \end{aligned}$$
as \(n\to \infty \). Then, according to the Hardy–Littlewood–Sobolev inequality, we have
$$\begin{aligned}& I_{\alpha }* \bigl( \vert u_{n} \vert ^{\frac{N+\alpha }{N}}- \vert u_{n}-u \vert ^{ \frac{N+\alpha }{N}} \bigr)\to I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}\quad \text{in } L^{\frac{2N}{N-\alpha }}\bigl({ \mathbb{R}}^{N}\bigr), \\& I_{\alpha }* \bigl( \vert v_{n} \vert ^{\frac{N+\alpha }{N}}- \vert v_{n}-v \vert ^{ \frac{N+\alpha }{N}} \bigr)\to I_{\alpha }* \vert v \vert ^{\frac{N+\alpha }{N}}\quad \text{in } L^{\frac{2N}{N-\alpha }}\bigl({ \mathbb{R}}^{N}\bigr), \end{aligned}$$
as \(n\to \infty \). Observing that
$$ \begin{gathered} \int \bigl(I_{\alpha }* \vert u_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n} \vert ^{ \frac{N+\alpha }{N}}- \int \bigl(I_{\alpha }* \vert u_{n}-u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n}-v \vert ^{ \frac{N+\alpha }{N}} \\ \quad = \int I_{\alpha }*\bigl( \vert u_{n} \vert ^{\frac{N+\alpha }{N}}- \vert u_{n}-u \vert ^{ \frac{N+\alpha }{N}}\bigr) \bigl( \vert v_{n} \vert ^{\frac{N+\alpha }{N}}- \vert v_{n}-v \vert ^{ \frac{N+\alpha }{N}}\bigr) \\ \qquad{} + \int I_{\alpha }*\bigl( \vert v_{n} \vert ^{\frac{N+\alpha }{N}}- \vert v_{n}-v \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert u_{n}-u \vert ^{\frac{N+\alpha }{N}} \\ \qquad{} + \int I_{\alpha }*\bigl( \vert u_{n} \vert ^{\frac{N+\alpha }{N}}- \vert u_{n}-u \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert v_{n}-v \vert ^{\frac{N+\alpha }{N}},\end{gathered} $$
(4.3)
and
$$ \vert u_{n}-u \vert ^{\frac{N+\alpha }{N}}\rightharpoonup 0,\qquad \vert v_{n}-v \vert ^{ \frac{N+\alpha }{N}}\rightharpoonup 0 \quad \text{in } L^{ \frac{2N}{N+\alpha }}\bigl({\mathbb{R}}^{N}\bigr), $$
we see that the conclusion holds. □
Proof of Theorem 1.4
According to the mountain pass theorem [23], we obtain that there is \(\{(u_{n},v_{n})\}\subset \mathcal{N}\) satisfying
$$ I(u_{n},v_{n})\to B, I'(u_{n},v_{n}) \to 0\quad \text{in } H^{-1}. $$
It follows that
$$ \begin{aligned} B+o(1)&\ge I(u_{n},v_{n})- \frac{N}{2(N+\alpha )} \bigl\langle I'(u_{n},v_{n}),(u_{n},v_{n}) \bigr\rangle \\ & = \frac{\alpha }{2(N+\alpha )} \int \vert \nabla u_{n} \vert ^{2}+\lambda _{1}(x)u_{n}^{2}+ \vert \nabla v_{n} \vert ^{2}+ \lambda _{2}(x)v_{n}^{2} \end{aligned} $$
for n large enough, which combined with assumption \((C1)\) implies that \(\{(u_{n},v_{n})\}\) is bounded in H. Then we may assume that
$$ \begin{gathered} (u_{n},v_{n})\rightharpoonup (u,v)\quad \text{in } H, \\ (u_{n},v_{n})\to (u,v)\quad \text{in } L_{loc}^{2}\bigl({ \mathbb{R}}^{N}\bigr)\times L_{loc}^{2}\bigl({\mathbb{R}}^{N}\bigr), \\ (u_{n},v_{n})\to (u,v)\quad \text{a.e on } { \mathbb{R}}^{N}.\end{gathered} $$
Since \(|u_{n}|^{\frac{N+\alpha }{N}}\) and \(|v_{n}|^{\frac{N+\alpha }{N}}\) are bounded in \(L^{\frac{2N}{N+\alpha }}({\mathbb{R}}^{N})\), we have
$$ \vert u_{n} \vert ^{\frac{N+\alpha }{N}}\rightharpoonup \vert u \vert ^{\frac{N+\alpha }{N}}, \qquad \vert v_{n} \vert ^{\frac{N+\alpha }{N}} \rightharpoonup \vert v \vert ^{ \frac{N+\alpha }{N}} \quad \text{in } L^{\frac{2N}{N+\alpha }} \bigl({ \mathbb{R}}^{N}\bigr). $$
Using the Hardy–Littlewood–Sobolev inequality, we obtain
$$ I_{\alpha }* \vert u_{n} \vert ^{\frac{N+\alpha }{N}}\rightharpoonup I_{\alpha }* \vert u \vert ^{ \frac{N+\alpha }{N}},\qquad I_{\alpha }* \vert v_{n} \vert ^{\frac{N+\alpha }{N}} \rightharpoonup I_{\alpha }* \vert v \vert ^{\frac{N+\alpha }{N}} \quad \text{in } L^{\frac{2N}{N-\alpha }}\bigl({ \mathbb{R}}^{N}\bigr). $$
Observing that
$$ \vert u_{n} \vert ^{\frac{\alpha }{N}-1}u_{n}\to \vert u \vert ^{\frac{\alpha }{N}-1}u,\qquad \vert v_{n} \vert ^{ \frac{\alpha }{N}-1}v_{n}\to \vert v \vert ^{\frac{\alpha }{N}-1}v \quad \text{in } L_{loc}^{\frac{2N}{\alpha }}\bigl({\mathbb{R}}^{N}\bigr), $$
we have, for any \(\phi \in C_{0}^{\infty }({\mathbb{R}}^{N})\),
$$ \begin{gathered} \int \bigl(I_{\alpha }* \vert u_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert u_{n} \vert ^{ \frac{\alpha }{N}-1}u_{n} \phi \to \int \bigl(I_{\alpha }* \vert u \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert u \vert ^{\frac{\alpha }{N}-1}u\phi , \\ \int \bigl(I_{\alpha }* \vert v_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n} \vert ^{ \frac{\alpha }{N}-1}v_{n} \phi \to \int \bigl(I_{\alpha }* \vert v \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert v \vert ^{\frac{\alpha }{N}-1}v\phi , \\ \int \bigl(I_{\alpha }* \vert u_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n} \vert ^{ \frac{\alpha }{N}-1}v_{n} \phi \to \int \bigl(I_{\alpha }* \vert u \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert v \vert ^{\frac{\alpha }{N}-1}v\phi , \\ \int \bigl(I_{\alpha }* \vert v_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert u_{n} \vert ^{ \frac{\alpha }{N}-1}u_{n} \phi \to \int \bigl(I_{\alpha }* \vert v \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert u \vert ^{\frac{\alpha }{N}-1}u\phi ,\end{gathered} $$
(4.4)
as \(n\to \infty \). Taking account of \(I'(u_{n},v_{n})\to 0\), (4.4), and the fact that \(C_{0}^{\infty }({\mathbb{R}}^{N})\) is dense in \(H^{1}({\mathbb{R}}^{N})\), we have \(I'(u,v)=0\). Denote \(z_{n}=u_{n}-u\), \(\omega _{n}=v_{n}-v\), then \((z_{n},\omega _{n})\rightharpoonup (0,0)\) in H, \((z_{n},\omega _{n})\to (0,0)\) in \(L_{loc}^{2}({\mathbb{R}}^{N})\times L_{loc}^{2}({\mathbb{R}}^{N})\), and \((z_{n},\omega _{n})\to (0,0)\) a.e on \({\mathbb{R}}^{N}\). By \((C1)\), there exists \(R>0\) sufficiently large such that
$$ \begin{aligned} \int \lambda _{1}(x)z_{n}^{2}+\lambda _{2}(x)\omega _{n}^{2}&= \int _{{ \mathbb{R}}^{N}\setminus B(0,R)}z_{n}^{2}+\omega _{n}^{2}+ \int _{B(0,R)} \lambda _{1}(x)z_{n}^{2}+ \lambda _{2}(x)\omega _{n}^{2}+o(1)\\ &= \int z_{n}^{2}+ \omega _{n}^{2}+o(1). \end{aligned} $$
(4.5)
Denote
$$ \begin{aligned} J(u,v)&= \frac{1}{2} \int \vert \nabla u \vert ^{2}+u^{2}+ \vert \nabla v \vert ^{2}+v^{2} \\ &\quad{} - \frac{N}{2(N+\alpha )} \int \bigl(\mu _{1}\bigl(I_{ \alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert u \vert ^{\frac{N+\alpha }{N}}+\mu _{2} \bigl(I_{ \alpha }* \vert v \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v \vert ^{\frac{N+\alpha }{N}}\\ &\quad {}+2\beta \bigl(I_{ \alpha }* \vert u \vert ^{\frac{N+\alpha }{N}} \bigr) \vert v \vert ^{\frac{N+\alpha }{N}}\bigr).\end{aligned} $$
Combining (4.5) with Lemma 4.2, we have, for n large enough,
$$ \bigl\langle J'(z_{n},w_{n}),(z_{n}, \omega _{n})\bigr\rangle =\bigl\langle I'(u_{n},v_{n}),(u_{n},v_{n}) \bigr\rangle -\bigl\langle I'(u,v),(u,v)\bigr\rangle =o(1) $$
(4.6)
and
$$ B+o(1)=I(u_{n},v_{n})=I(u,v)+J(z_{n}, \omega _{n})+o(1). $$
(4.7)
Set
$$ C_{n}= \int \vert \nabla z_{n} \vert ^{2}+z_{n}^{2}, \qquad D_{n}= \int \vert \nabla \omega _{n} \vert ^{2}+\omega _{n}^{2}. $$
Then it follows
$$ B=I(u,v)+\frac{\alpha }{2(N+\alpha )}(C_{n}+D_{n})+o(1). $$
(4.8)
We will show that \(u\not \equiv 0\), \(v\not \equiv 0\) by excluding the following three cases:
(i) \((u,v)\equiv (0,0)\). By (4.8), we know that
Denote
$$ E_{n}= \int \mu _{1}\bigl(I_{\alpha }* \vert z_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert z_{n} \vert ^{ \frac{N+\alpha }{N}}, \qquad F_{n}= \int \mu _{2}\bigl(I_{\alpha }* \vert w_{n} \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert w_{n} \vert ^{\frac{N+\alpha }{N}}. $$
If \(E_{n}\to 0\), then \(\int (I_{\alpha }*|w_{n}|^{\frac{N+\alpha }{N}})|z_{n}|^{ \frac{N+\alpha }{N}}\to 0\). So we have
$$\begin{aligned} \int \vert \nabla z_{n} \vert ^{2}+z_{n}^{2}+ \vert \nabla w_{n} \vert ^{2}+w_{n}^{2}&= \int \mu _{1}\bigl(I_{\alpha }* \vert w_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert w_{n} \vert ^{ \frac{N+\alpha }{N}}+o(1). \\ &\le \mu _{1} S_{1}^{-\frac{N+\alpha }{N}} \biggl( \int \vert w_{n} \vert ^{2} \biggr)^{\frac{N+\alpha }{N}}. \\ &\le \mu _{1} S_{1}^{-\frac{N+\alpha }{N}} \biggl( \int \vert \nabla z_{n} \vert ^{2}+z_{n}^{2}+ \vert \nabla w_{n} \vert ^{2}+w_{n}^{2} \biggr)^{ \frac{N+\alpha }{N}}, \end{aligned}$$
which implies
$$ \int \vert \nabla z_{n} \vert ^{2}+z_{n}^{2}+ \vert \nabla w_{n} \vert ^{2}+w_{n}^{2} \ge \mu _{1}^{-\frac{N}{\alpha }} S_{1}^{\frac{N+\alpha }{\alpha }}. $$
Then, by (4.8) and (1.11), we obtain
$$ B=I(u,v)+\frac{\alpha }{2(N+\alpha )}(C_{n}+D_{n})+o(1)\ge \frac{\alpha }{2(N+\alpha )}\mu _{1}^{-\frac{N}{\alpha }}S_{1}^{ \frac{N+\alpha }{\alpha }}>B_{1}, $$
which contradicts Lemma 4.1. Similarly, \(F_{n}\to 0\) also leads to a contradiction. Thus, \(E_{n}\ge \delta \) and \(F_{n}\ge \delta \) for some \(\delta >0\) and n large enough. Then there exists \(t_{n}>0\) such that
$$ \bigl\langle J'(t_{n}z_{n},t_{n} \omega _{n}),(t_{n}z_{n},t_{n}\omega _{n}) \bigr\rangle =0 $$
and
$$ \begin{gathered} J(t_{n}z_{n},t_{n} \omega _{n}) \\ \quad = \max_{s_{n}\ge 0}J(s_{n}z_{n},s_{n} \omega _{n}) \\ \quad \ge \max_{s_{n}\ge 0}\frac{1}{2}s_{n}^{2} \int \vert z_{n} \vert ^{2}+ \omega _{n}^{2} \\ \qquad {} -\frac{N s_{n}^{\frac{2(N+\alpha )}{N}}}{2(N+\alpha )} \int \bigl(\mu _{1}\bigl(I_{\alpha }* \vert z_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert z_{n} \vert ^{ \frac{N+\alpha }{N}}+\mu _{2}\bigl(I_{\alpha }* \vert \omega _{n} \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert \omega _{n} \vert ^{\frac{N+\alpha }{N}}\\ \qquad {}+2 \beta \bigl(I_{ \alpha }* \vert z_{n} \vert ^{\frac{N+\alpha }{N}} \bigr) \vert \omega _{n} \vert ^{ \frac{N+\alpha }{N}}\bigr) \\ \quad \ge \frac{\alpha }{2(N+\alpha )}S_{0}^{ \frac{N+\alpha }{\alpha }}, \end{gathered} $$
(4.9)
where the last inequality follows by Theorem 1.5. Moreover, by (4.6), we have \(t_{n}\to 1\). Then we have
$$ B=I(u,v)+J(z_{n},\omega _{n})=J(t_{n}z_{n},t_{n} \omega _{n})\ge \frac{\alpha }{2(N+\alpha )}S_{0}^{\frac{N+\alpha }{\alpha }}, $$
which also contradicts Lemma 4.1.
(ii) \(u\equiv 0\), \(v\not \equiv 0\). In this case, it is clear that v is a solution of (1.9) for \(i=2\). Then, by (4.7), we have \(B\ge I(0,v)\ge B_{2}\), which contradicts Lemma 4.1.
(iii) \(v\equiv 0\), \(u\not \equiv 0\). By similar arguments as in case (ii), we see that \(B\ge B_{1}\), which also contradicts Lemma 4.1.
Thus, we have proved that \(u\not \equiv 0\), \(v\not \equiv 0\), and \(I'(u,v)=0\). Then \(I(u,v)\ge B\), which combining with (4.7), (4.8) indicates \(I(u,v)=B\). Hence, \((u,v)\) is a ground state of system (1.1). Moreover, since \(I(|u|,|v|)=B\) and \((|u|,|v|)\in \mathcal{N}\), we know that \((|u|,|v|)\) is also a ground state of (1.1). By the strong maximum principle, we have \(|u|>0\), \(|v|>0\). Thus, system (1.1) has a positive ground state \((|u|,|v|)\). □
Remark 4.3
Let \((u,v)\) be a solution obtained in Theorem 1.4. Then it is obvious that \((u,v)\in \mathcal{M}\). Moreover, since \(\mathcal{M}\in \mathcal{N}\), we have
$$ B=I(u,v)\le \mathcal{B}\le I(u,v), $$
which implies that \(B=\mathcal{B}\).