Existence of normalized solutions for Schrödinger systems with linear and nonlinear couplings

Abstract

In this paper we study the nonlinear Bose–Einstein condensates Schrödinger system

$$ \textstyle\begin{cases} -\Delta u_{1}-\lambda _{1} u_{1}=\mu _{1} u_{1}^{3}+\beta u_{1}u_{2}^{2}+ \kappa (x) u_{2}\quad\text{in }\mathbb{R}^{3}, \\ -\Delta u_{2}-\lambda _{2} u_{2}=\mu _{2} u_{2}^{3}+\beta u_{1}^{2}u_{2}+ \kappa (x) u_{1}\quad\text{in }\mathbb{R}^{3}, \\ \int _{\mathbb{R}^{3}} u_{1}^{2}=a_{1}^{2},\qquad \int _{\mathbb{R}^{3}} u_{2}^{2}=a_{2}^{2}, \end{cases} $$

where $a_{1}$, $a_{2}$, $\mu _{1}$, $\mu _{2}$, $\kappa =\kappa (x)>0$, $\beta <0$, and $\lambda _{1}$, $\lambda _{2}$ are Lagrangian multipliers. We use the Ekeland variational principle and the minimax method on manifold to prove that this system has a solution that is radially symmetric and positive.

1 Introduction

In this paper we study the stationary nonlinear Bose–Einstein condensates Schrödinger system

$$ \textstyle\begin{cases} -\Delta u_{1}-\lambda _{1} u_{1}=\mu _{1} u_{1}^{3}+\beta u_{1}u_{2}^{2}+ \kappa (x) u_{2} \quad\text{in }\mathbb{R}^{3}, \\ -\Delta u_{2}-\lambda _{2} u_{2}=\mu _{2} u_{2}^{3}+\beta u_{1}^{2}u_{2}+ \kappa (x) u_{1} \quad\text{in }\mathbb{R}^{3}, \\ \int _{\mathbb{R}^{3}} u_{1}^{2}=a_{1}^{2},\qquad \int _{\mathbb{R}^{3}} u_{2}^{2}=a_{2}^{2}, \end{cases} $$

(1.1)

where $a_{1}$, $a_{2}$, $\mu _{1}$, $\mu _{2}$, $\kappa (x)>0$, $\beta <0$, and $\lambda _{1}$, $\lambda _{2}$ are Lagrangian multipliers that will be determined. If there exists $(\lambda _{1},\lambda _{2},u_{1},u_{2})\in \mathbb{R}^{2}\times H^{1}( \mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$ that satisfies (1.1), then we call $(\lambda _{1},\lambda _{2},u_{1},u_{2})$ a normalized solution of (1.1). This problem possesses several physical motivations such as nonlinear optics and Bose–Einstein condensation.

When $\mu _{1}$, $\mu _{2}$, $a_{1}$, $a_{2}>0$ and $\kappa (x)=0$, problem (1.1) has been considered by many mathematicians in recent years. In [3] Bartsch and Jeanjean studied the case $\beta >0$; they proved that there exists $\beta _{1}>0$ depending on $a_{i}$ and $\mu _{i}$, $i=1,2$, such that if $0<\beta <\beta _{1}$ then (1.1) has a solution $(\lambda _{1},\lambda _{2},\bar{u}_{1},\bar{u}_{2})$, where $\lambda _{1}$, $\lambda _{2}<0$ and $\bar{u}_{1}$ and $\bar{u}_{2}$ are both positive and radially symmetric, and there exists $\beta _{2}>0$ depending on $a_{i}$ and $\mu _{i}$ such that if $\beta >\beta _{2}$ then (1.1) has a solution $(\lambda _{1},\lambda _{2},\bar{u}_{1},\bar{u}_{2})$, where $\lambda _{1}$, $\lambda _{2}<0$ and $\bar{u}_{1}$ and $\bar{u}_{2}$ are both positive and radially symmetric. In [6–8] Bartsch and Soave studied the case $\beta <0$, and they proved the existence of positive solutions of (1.1). Moreover, if $\mu _{1}=\mu _{2}$, $a_{1}=a_{2}$ then (1.1) has infinitely many positive solutions. We refer the interested reader to [4, 5, 12, 13, 20] and the references therein for more results of this case.

When $\mu _{1}$, $\mu _{2}$, $a_{1}$, $a_{2}>0$ and $\kappa (x)\neq 0$, to the best of our knowledge, the only result available is presented in [25]. We proved the existence of solutions for (1.1) when $\beta >0$, the interaction is attractive, and $\kappa (x)$ is a radially symmetric function, which means $\kappa (x)=\kappa (|x|)$, by using the Ekeland variational principle and minimax theory on manifold.

It is necessary to point out that Schrödinger systems with fixed $\lambda _{i}$ have been widely studied in the last twenty years, the existence and behavior of solutions are well understood. For autonomous systems, we refer the interested reader to [1, 2, 9, 15, 17–19, 24, 27, 28] and the references therein. For nonautonomous systems, we refer the interested reader to [21, 26], which studied the ground state solutions of Schrödinger systems with potentials, but for system (1.1) it is far from being well understood. Furthermore, normalized solutions for the single equation were studied in [14, 16, 22, 23] and the references therein.

Now, let us focus on the repulsive case of system (1.1), which means $a_{1}$, $a_{2}$, $\mu _{1}$, $\mu _{2}$, $\kappa (x)>0$, and $\beta <0$. We also use the variational method to prove the existence of solutions for system (1.1), but different from the attractive case because $\beta <0$, the Liouville type theorem in [25] is no longer applicable, we need to establish a new Liouville type theorem for elliptic systems to make sure that the weak limit of P.S. sequence is nontrivial. We will prove the existence of solutions for (1.1), which will be found as critical points of the energy functional J on manifold $\mathcal{S}$, where

$$ J(u_{1},u_{2}):=\frac{1}{2} \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}- \frac{1}{4} \int _{\mathbb{R}^{3}}\mu _{1}u_{1}^{4}+ \mu _{2} u_{2}^{4}+2\beta u_{1}^{2}u_{2}^{2}- \int _{\mathbb{R}^{3}} \kappa (x)u_{1}u_{2} $$

and

$$ \mathcal{S}:=S_{a_{1}}\times S_{a_{2}}, S_{a}:= \biggl\{ u\in H^{1}_{rad}\bigl( \mathbb{R}^{3}\bigr): \int _{\mathbb{R}^{3}}u^{2}=a^{2} \biggr\} , $$

space $H^{1}_{rad}(\mathbb{R}^{3})$ denotes the space of radially symmetric functions in $H^{1}(\mathbb{R}^{3})$. We have the following results.

First, for the autonomous case ($\kappa (x)=\kappa$ is a constant). Because $\lambda _{1}$, $\lambda _{2}$ are unknown, the traditional Nehari manifold method is not available, so we need to combine the Nehari identity and the Pohožeav identity to get a new constraint for system (1.1):

$$ \mathcal{P}:= \biggl\{ (u_{1},u_{2})\in \mathcal{S}: \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}= \frac{3}{4} \int _{\mathbb{R}^{3}} \mu _{1} u_{1}^{4}+\mu _{2} u_{2}^{4}+2\beta u_{1}^{2}u_{2}^{2} \biggr\} . $$

From [6] we know that $\mathcal{P}$ is a $C^{2}$ submanifold of $\mathcal{S}$. First, we show that J is bounded from below and away from 0 on $\mathcal{P}$. Next, we use the Ekeland variational principle to find a P.S. sequence for J on $\mathcal{S}$ at level $c:=\inf_{\mathcal{P}}J(u_{1},u_{2})$ and prove that the P.S. sequence is bounded, then it has a weak limit in $H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$. Finally, we prove that the weak limit is also a strong limit by establishing a new Liouville type theorem for elliptic systems. Then we have the following existence theorem.

Theorem 1.1

Assume $a_{1}$, $a_{2}$, $\mu _{1}$, $\mu _{2}>0$, $\beta <0$, and $\kappa >0$ with the additional condition

$$ \kappa < \frac{8}{27C_{a_{1},a_{2}}^{2}a_{1}a_{2}}, $$

(1.2)

where $C_{a_{1},a_{2}}=\max \{\mu _{1} a_{1}G^{4},\mu _{2}a_{2}G^{4}\}$ and G is the best constant for the Gagliardo–Nirenberg inequality in Lemma 3.1. Then system (1.1) has a solution $(\bar{\lambda}_{1},\bar{\lambda}_{2},\bar{u}_{1},\bar{u}_{2})$ such that $\bar{\lambda}_{1}$, $\bar{\lambda}_{2}<0$, $\bar{u}_{1}$, $\bar{u}_{2}>0$. Moreover, $\bar{u}_{1}$ and $\bar{u}_{2}$ are radially symmetric.

Next, for the nonautonomous case ($\kappa (x)$ is not a constant). When $\kappa (x)$ is not a constant, the Pohožeav identity of (1.1) is very complicated, so we need to find the critical point of J on manifold $\mathcal{S}$ directly. The functional J on $\mathcal{S}$ is unbounded from below, so we try to construct a mountain pass structure of J on the manifold $\mathcal{S}$ and by the minimax theory on the Finsler manifold, which was introduced in [10], and to obtain the critical point of J on $\mathcal{S}$. We have the following theorem.

Theorem 1.2

Assume $a_{1}$, $a_{2}$, $\mu _{1}$, $\mu _{2}>0$, $-\sqrt{\mu _{1}\mu _{2}}<\beta <0$, and $\kappa (x)=\kappa (|x|)$ is positive and away from 0, $\kappa (x)\in L^{\infty}(\mathbb{R}^{3})$, $\frac{2}{3}\nabla \kappa (x)\cdot x+\kappa (x)\geq 0$, $\nabla \kappa (x)\cdot x$ is bounded and

$$ \bigl\vert \kappa (x) \bigr\vert _{\infty}< \frac{5}{18C_{a_{1},a_{2}}^{2}a_{1}a_{2}}, $$

where $C_{a_{1},a_{2}}=\max \{\mu _{1} a_{1}G^{4},\mu _{2}a_{2}G^{4}\}$ and G is the best constant for the Gagliardo–Nirenberg inequality in Lemma 3.1. Then system (1.1) has a solution $(\bar{\lambda}_{1},\bar{\lambda}_{2},\bar{u}_{1},\bar{u}_{2})$ such that $\bar{\lambda}_{1}$, $\bar{\lambda}_{2}<0$, $\bar{u}_{1}$, $\bar{u}_{2}>0$. Moreover, $\bar{u}_{1}$ and $\bar{u}_{2}$ are radially symmetric.

Finally, we need to point out that the proofs of Theorem 1.1 and Theorem 1.2 are different from [6] and [25]. To deal with the repulsive case and the linear coupling terms of system (1.1), we need to establish a new Liouville type theorem for elliptic systems (Lemma 3.8) by asymptotic estimates to prove that the weak limit of P.S. sequence is also a strong limit.

The paper is organized as follows. In Sect. 2 we give some notations and preliminaries. Sections 3 and 4 are devoted to the proofs of Theorems 1.1 and 1.2.

2 Notations

Throughout the paper we always work in the space $\mathbb{R}^{3}$, and we use the notation $|u|_{p}$ to denote the $L^{p}$-norm. Set $H^{1}(\mathbb{R}^{3})$ to be the usual Sobolev space, and its norm is denoted by

$$ \Vert u \Vert := \Vert u \Vert _{H^{1}}:=\bigl( \vert \nabla u \vert _{2}^{2}+ \vert u \vert _{2}^{2} \bigr)^{1/2}. $$

To use the compact embedding in whole space, we denote the radially symmetric space as follows:

$$ H_{r}^{1}:=H_{rad}^{1}\bigl(\mathbb{R}^{3}\bigr):=\bigl\{ u\in H^{1}\bigl(\mathbb{R}^{3} \bigr): u(x)=u\bigl( \vert x \vert \bigr)\bigr\} , $$

and we set

$$ \mathcal{S}:=S_{1}\times S_{2}, S_{i}:=S_{a_{i}}:= \bigl\{ u\in H_{r}^{1}: \vert u \vert _{2}=a_{i} \bigr\} , \quad i=1,2, $$

where $|u|_{p}:=|u|_{p,\mathbb{R}^{3}}:=(\int _{\mathbb{R}^{3}}|u|^{p})^{{1}/{p}}$, $p>1$. From standard variational arguments and the Palais principle of symmetric criticality, we know that critical points of the following functional on $\mathcal{S}$ are weak solutions of (1.1):

$$ J(u_{1},u_{2}):=\frac{1}{2} \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}- \frac{1}{4} \int _{\mathbb{R}^{3}}\mu _{1}u_{1}^{4}+ \mu _{2} u_{2}^{4}+2\beta u_{1}^{2}u_{2}^{2}- \int _{\mathbb{R}^{3}} \kappa (x)u_{1}u_{2}. $$

We will use the following fiber mapping, which was introduced in [14] originally. For $s\in \mathbb{R}$ and $u\in H^{1}(\mathbb{R}^{3})$, we define

$$ (s\star u) (x):=e^{\frac{3s}{2}}u\bigl(e^{s}x\bigr). $$

From the definition, we can easily check that $|s\star u|_{2}=|u|_{2}$ and $|\nabla (s\star u)|_{2}=e^{s}|\nabla u|_{2}$; as a consequence, take $s\in \mathbb{R}$, $(u_{1},u_{2})\in \mathcal{S}$, we have $s\star (u_{1},u_{2}):=(s\star u_{1}, s\star u_{2})\in \mathcal{S}$.

To deal with the autonomous case, we give the following notations. First define a function $\Phi _{(u_{1},u_{2})}(s)$ as follows:

$$ \Phi _{(u_{1},u_{2})}(s):=J\bigl(s\star (u_{1}, u_{2})\bigr), $$

when κ is a constant by changing variables, we have

$$ \Phi _{(u_{1},u_{2})}(s)=\frac{e^{2}s}{2} \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}- \frac{e^{3s}}{4} \int _{ \mathbb{R}^{3}}\mu _{1}u_{1}^{4}+\mu _{2} u_{2}^{4}+2\beta u_{1}^{2}u_{2}^{2}- \int _{\mathbb{R}^{3}}\kappa u_{1}u_{2}. $$

Next we introduce a subset of $\mathcal {S}$:

$$ \mathcal {T}:= \biggl\{ (u_{1},u_{2})\in \mathcal{S}: \int _{\mathbb{R}^{3}} \mu _{1} u_{1}^{4}+\mu _{2} u_{2}^{4}+2\beta u_{1}^{2}u_{2}^{2}>0 \biggr\} . $$

Clearly, $\mathcal{T}=\mathcal{S}$, when $-\sqrt{u_{1}u_{2}}<\beta <+\infty$, $\mathcal{T}$ is a proper subset of $\mathcal{S}$ while $\beta \leq -\sqrt{u_{1}u_{2}}$. Moreover, when $(u_{1},u_{2})\in \mathcal{T}$, the function $\Phi _{(u_{1},u_{2})}(s)$ has a unique strict maximum point, which is defined by

$$ s_{(u_{1},u_{2})}=\ln \biggl( \frac{4\int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}}{3\int _{\mathbb{R}^{3}}\mu _{1}u_{1}^{4}+\mu _{2} u_{2}^{4}+2\beta u_{1}^{2}u_{2}^{2}}\biggr). $$

(2.1)

It is clear that for any $(u_{1},u_{2})\in \mathcal{T}$, we have $s_{(u_{1},u_{2})}\star (u_{1},u_{2})\in \mathcal{P}$.

3 Autonomous systems

In this section, we prove Theorem 1.1.

We work on the space $\mathcal{H}:=H^{1}_{rad}(\mathbb{R}^{3})\times H^{1}_{rad}( \mathbb{R}^{3})$, the corresponding energy functional of (1.1) on $\mathcal{S}$ is

$$ J(u_{1},u_{2}):=\frac{1}{2} \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}- \frac{1}{4} \int _{\mathbb{R}^{3}}\mu _{1}u_{1}^{4}+ \mu _{2} u_{2}^{4}+2\beta u_{1}^{2}u_{2}^{2}- \int _{\mathbb{R}^{3}} \kappa u_{1}u_{2}. $$

We try to find the critical point of J on $\mathcal{S}$.

Lemma 3.1

(Gagliardo–Nirenberg inequality)

For any $u\in H^{1}(\mathbb{R}^{3})$, we have

$$ \int _{\mathbb{R}^{3}} u^{4}\leq G \biggl( \int _{\mathbb{R}^{3}}u^{2} \biggr)^{\frac{1}{2}} \biggl( \int _{\mathbb{R}^{3}} \vert \nabla u \vert ^{2} \biggr)^{\frac{3}{2}}, $$

where G is a universal constant.

Lemma 3.2

J is coercive on $\mathcal {P}$ and there exists $\delta >0$ such that

$$ \inf_{(u_{1},u_{2})\in \mathcal{P}}J(u_{1},u_{2})>\delta , $$

when $\kappa <\frac{8}{27C_{a_{1},a_{2}}a_{1}a_{2}}$, where $C_{a_{1},a_{2}}=\max \{\mu _{1} a_{1}G^{4},\mu _{2}a_{2}G^{4}\}$.

Proof

From Lemma 3.1 we know for all $u\in H^{1}(\mathbb{R}^{3})$:

$$ \int _{\mathbb{R}^{3}} u^{4}\leq G \biggl( \int _{\mathbb{R}^{3}}u^{2} \biggr)^{\frac{1}{2}} \biggl( \int _{\mathbb{R}^{3}} \vert \nabla u \vert ^{2} \biggr)^{\frac{3}{2}}. $$

If $(u_{1},u_{2})\in \mathcal{P}$, then $(u_{1},u_{2})\in \mathcal {T}$, we have

$$ \begin{aligned} 0&< \biggl( \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2} \biggr)^{\frac{2}{3}} \\ &= \biggl(\frac{3}{4} \int _{ \mathbb{R}^{3}}\mu _{1}u_{1}^{4}+\mu _{2} u_{2}^{4}+2\beta u_{1}^{2}u_{2}^{2} \biggr)^{\frac{2}{3}} \\ &\leq \biggl(\frac{3}{4} \int _{\mathbb{R}^{3}}\mu _{1}u_{1}^{4}+\mu _{2} u_{2}^{4} \biggr)^{\frac{2}{3}}\leq \biggl( \frac{3}{4}C_{a_{1},a_{2}} \biggr)^{\frac{2}{3}} \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}. \end{aligned} $$

Moreover, when $(u_{1},u_{2})\in \mathcal {P}$, we have

$$ J(u_{1},u_{2})=\frac{1}{6} \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}- \int _{\mathbb{R}^{3}}\kappa u_{1}u_{2}\geq \frac{8}{27C_{a_{1},a_{2}}^{2}}-\kappa a_{1}a_{2}. $$

Then, if $\kappa <\frac{8}{27C_{a_{1},a_{2}}^{2}a_{1}a_{2}}$, where $C_{a_{1},a_{2}}=\max \{\mu _{1} a_{1}G^{4},\mu _{2}a_{2}G^{4}\}$, then there exists $\delta >0$ such that

$$ \inf_{(u_{1},u_{2})\in \mathcal{P}}J(u_{1},u_{2})>\delta , $$

the coerciveness of J on $\mathcal {P}$ is obvious, which finishes the proof. □

From Lemma 3.2 we know

$$ c:=\inf_{(u_{1},u_{2})\in \mathcal{P}}J(u_{1},u_{2})>0, $$

and J is coercive. These properties inspire us to prove that c is the critical value of J on manifold $\mathcal {S}$.

First we define the functional $E: \mathcal{T}\rightarrow \mathbb{R}$ by

$$ E(u_{1},u_{2}):=J\bigl(s \star (u_{1},u_{2}) \bigr). $$

From the definition of $s_{(u_{1},u_{2})}$, we have $s_{(u_{1},u_{2})}\star (u_{1},u_{2})\in \mathcal{P}$. Together with (2.1) it is easy to check that

$$ \begin{aligned} E(u_{1},u_{2})&= \frac{1}{6} \int _{\mathbb{R}^{3}} \vert \nabla s_{(u_{1},u_{2})}\star u_{1} \vert ^{2}+ \vert \nabla s_{(u_{1},u_{2})} \star u_{2} \vert ^{2}- \int _{\mathbb{R}^{3}}\kappa u_{1} u_{2} \\ &=\frac{e^{2s_{(u_{1},u_{2})}}}{6} \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}- \int _{\mathbb{R}^{3}}\kappa u_{1} u_{2} \\ &= \frac{8\int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}}{27\int _{\mathbb{R}^{3}}\mu _{1}u_{1}^{4}+\mu _{2} u_{2}^{4}+2\beta u_{1}^{2}u_{2}^{2}}- \int _{\mathbb{R}^{3}}\kappa u_{1} u_{2}. \end{aligned} $$

(3.1)

Lemma 3.3

We have

$$ c=\inf_{(u_{1},u_{2})\in \mathcal{T}}E(u_{1},u_{2}). $$

Proof

For every $(u_{1},u_{2})\in \mathcal{P}$, we have $s_{(u_{1},u_{2})}=0$. Moreover,

$$ J(u_{1},u_{2})=E(u_{1},u_{2})\geq \inf _{(u_{1},u_{2})\in \mathcal{T}}E(u_{1},u_{2}) \quad \Rightarrow\quad c\geq \inf _{(u_{1},u_{2})\in \mathcal{T}}E(u_{1},u_{2}). $$

On the other hand, for every $(u_{1},u_{2})\in \mathcal{T}$, we have

$$ E(u_{1},u_{2})=J\bigl(s_{u_{1},u_{2}}\star (u_{1},u_{2})\bigr)\geq c \quad \Rightarrow\quad \inf _{(u_{1},u_{2})\in \mathcal{T}}E(u_{1},u_{2})\geq c. $$

Combining two inequations above, we finish the proof. □

The following lemma shows us the relations of derivative between J and E.

Lemma 3.4

The functional $E\in C^{1}(\mathcal{T},\mathbb{R})$, and

$$ \bigl(dE(u_{1},u_{2}),(\phi _{1}, \phi _{2})\bigr)=\bigl(dJ\bigl(s_{(u_{1},u_{2})} \star (u_{1},u_{2})\bigr),s_{(u_{1},u_{2})}\star ({\phi _{1}},\phi _{2})\bigr), $$

(3.2)

where $(u_{1},u_{2})\in \mathcal{T}$, $(\phi _{1},\phi _{2})\in T_{(u_{1},u_{2})}\mathcal {S}$, and $T_{(u_{1},u_{2})}\mathcal {S}$ is the tangent space of $\mathcal{S}$ in $\mathcal{H}$ at point $(u_{1},u_{2})$.

Proof

From (3.1) we know that $E\in C^{1}(\mathcal{T},\mathbb{R})$ is obvious, and take $(u_{1},u_{2})\in \mathcal{T}$, $(\phi _{1},\phi _{2})\in T_{(u_{1},u_{2})}\mathcal {S}$, we have

$$\begin{aligned}& \bigl(dE(u_{1},u_{2}),( \phi _{1},\phi _{2})\bigr) \\& \quad = \biggl( \frac{4\int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}}{3\int _{\mathbb{R}^{3}}\mu _{1}u_{1}^{4}+\mu _{2} u_{2}^{4}+2\beta u_{1}^{2}u_{2}^{2}} \biggr)^{2} \int _{\mathbb{R}^{3}}\nabla u_{1}\cdot \nabla \phi _{1}+ \nabla u_{2}\cdot \nabla \phi _{2} \\& \qquad {}- \biggl( \frac{4\int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}}{3\int _{\mathbb{R}^{3}}\mu _{1}u_{1}^{4}+\mu _{2} u_{2}^{4}+2\beta u_{1}^{2}u_{2}^{2}} \biggr)^{3} \int _{\mathbb{R}^{3}}\mu _{1}u_{1}^{3}+\mu _{2}u_{2}^{3}+ \beta u_{1}^{2}u_{2} \phi _{2}+\beta u_{1}u_{2}^{2}\phi _{1} \\& \qquad {}- \int _{\mathbb{R}^{3}}\kappa u_{1}\phi _{2}- \int _{\mathbb{R}^{3}} \kappa u_{2}\phi _{1} \\& \quad = e^{2s_{(u_{1},u_{2})}} \int _{\mathbb{R}^{3}}\nabla u_{1}\cdot \nabla \phi _{1}+\nabla u_{2}\cdot \nabla \phi _{2} \\& \qquad {}-e^{3s_{(u_{1},u_{2})}} \int _{\mathbb{R}^{3}}\mu _{1}u_{1}^{3}+\mu _{2}u_{2}^{3}+ \beta u_{1}^{2}u_{2} \phi _{2}+\beta u_{1}u_{2}^{2}\phi _{1}- \int _{ \mathbb{R}^{3}}\kappa u_{1}\phi _{2}- \int _{\mathbb{R}^{3}}\kappa u_{2} \phi _{1} \\& \quad = \int _{\mathbb{R}^{3}}\nabla (s_{(u_{1},u_{2})}\star u_{1})\cdot \nabla (s_{(u_{1},u_{2})}\star \phi _{1})+\nabla (s_{(u_{1},u_{2})} \star u_{2})\cdot \nabla (s_{(u_{1},u_{2})}\star \phi _{1}) \\& \qquad {}- \int _{\mathbb{R}^{3}}\mu _{1}(s_{(u_{1},u_{2})}\star u_{1})^{3}(s_{(u_{1},u_{2})} \star \phi _{1})+\mu _{2}(s_{(u_{1},u_{2})}\star u_{2})^{3}(s_{(u_{1},u_{2})} \star \phi _{2}) \\& \qquad {}- \int _{\mathbb{R}^{3}}\beta (s_{(u_{1},u_{2})}\star u_{1})^{2}(s_{(u_{1},u_{2})} \star u_{2}) (s_{(u_{1},u_{2})}\star \phi _{2}) \\& \qquad {}- \int _{\mathbb{R}^{3}}\beta (s_{(u_{1},u_{2})}\star u_{2})^{2}(s_{(u_{1},u_{2})} \star u_{1}) (s_{(u_{1},u_{2})}\star \phi _{1}) \\& \qquad {}- \int _{\mathbb{R}^{3}}\kappa (s_{(u_{1},u_{2})}\star u_{1}) (s_{(u_{1},u_{2})} \star \phi _{2})- \int _{\mathbb{R}^{3}}\kappa (s_{(u_{1},u_{2})} \star u_{2}) (s_{(u_{1},u_{2})}\star \phi _{1}) \\& \quad = \bigl(dJ\bigl(s_{(u_{1},u_{2})}\star (u_{1},u_{2}) \bigr),s_{(u_{1},u_{2})}\star ( \phi _{1},\phi _{2})\bigr), \end{aligned}$$

which finishes the proof. □

Next, from the Ekeland variational principle, we can find a P.S. sequence for J on $\mathcal{S}$ at level c.

Proposition 3.1

There exist two sequences $\{(\tilde{u}_{1,n},\tilde{u}_{2,n})\}$ and $\{(u_{1,n},u_{2,n})\}:=\{s_{n}\star (\tilde{u}_{1,n}\tilde{u}_{2,n})\}$ satisfying the following properties, where $s_{n}:=s_{(\tilde{u}_{1,n},\tilde{u}_{2,n})}$:

(a) $\{(\tilde{u}_{1,n},\tilde{u}_{2,n})\}$ is a P.S. sequence of E on manifold $\mathcal{S}$ at c;

(b) $s_{n}\rightarrow 0$ as $n\rightarrow +\infty$, and $\{(u_{1,n},u_{2,n})\}\in \mathcal{P}$ for every n;

(c) $\{(u_{1,n},u_{2,n})\}$ is a P.S. sequence of J on $\mathcal{S}$ at c.

Moreover, we can assume $u_{1,n}^{-}, u_{2,n}^{-}\rightarrow 0$ in $\mathcal{H}$.

Proof

First we can choose $(v_{1,n},v_{2,n})\in \mathcal{T}$ such that $E(v_{1,n},v_{2,n})\rightarrow c$. We take $(\tilde{ \tilde{u}}_{1,n},\tilde{ \tilde{u}}_{2,n}):=s_{(v_{1,n},v_{2,n})} \star (v_{1,n},v_{2,n})\in \mathcal{P}$. From the definition of E, we know that $E((\tilde{ \tilde{u}}_{1,n},\tilde{ \tilde{u}}_{2,n}))\rightarrow c$. From the Ekeland variational principle, there exists $(\tilde{u}_{1,n},\tilde{u}_{2,n})$ such that

$$\begin{aligned}& E(\tilde{u}_{1,n},\tilde{u}_{2,n})\rightarrow c,\\& d |_{\mathcal{S}}E(\tilde{u}_{1,n},\tilde{u}_{2,n})\rightarrow 0, \end{aligned}$$

and $\|\tilde{\tilde{u}}_{i,n}-\tilde{u}_{i,n}\|$, $i=1,2$. From the fact that $(\tilde{ \tilde{u}}_{1,n},\tilde{ \tilde{u}}_{2,n})\in \mathcal{P}$, we have $s_{n}:=s_{(\tilde{u}_{1,n},\tilde{u}_{2,n})}\rightarrow 0$, we define $(u_{1,n},u_{2,n}):=s_{n}\star (\tilde{u}_{1,n},\tilde{u}_{2,n})$. From Lemma 3.4 we have

$$ \bigl(d J(u_{1,n},u_{2,n}),(\phi _{1},\phi _{2})\bigr)=(dE\bigl(-s_{n}\star (u_{1,n},u_{2,n}) \bigr),\bigl(-s_{n} \star (\phi _{1},\phi _{2}) \bigr), $$

where $(\phi _{1},\phi _{2})\in T_{\mathcal {S}}(u_{1,n},u_{2,n})$. From the fact that $s_{n}\rightarrow 0$, there exists $C>0$ such that $0\leq s_{n}\leq C$. Moreover, there exist $C_{1}>0$, $C_{2}>0$ such that

$$\begin{aligned}& C_{1}< \frac{ \Vert -s_{n}\star (\phi _{1},\phi _{2}) \Vert }{ \Vert (\phi _{1},\phi _{2}) \Vert }< C_{2}, \\& \bigl\Vert d|_{\mathcal{S}}J(u_{1,n},u_{2,n}) \bigr\Vert _{*} =\sup_{ \substack{ \Vert (\phi _{1},\phi _{2}) \Vert =1\\\phi _{1},\phi _{2}\in T_{\mathcal {S}}(u_{1},u_{2})}} \bigl\vert \bigl(dJ(u_{1,n},u_{2,n}),(\phi _{1},\phi _{2})\bigr) \bigr\vert \\& \hphantom{ \bigl\Vert d|_{\mathcal{S}}J(u_{1,n},u_{2,n}) \bigr\Vert _{*}} =\sup_{ \substack{ \Vert (\phi _{1},\phi _{2}) \Vert =1\\\phi _{1},\phi _{2}\in T_{\mathcal {S}}(u_{1},u_{2})}} \bigl\vert \bigl(dE\bigl(-s_{n}\star (u_{1,n},u_{2,n})\bigr),\bigl(-s_{n}\star (\phi _{1},\phi _{2})\bigr)\bigr) \bigr\vert \\& \hphantom{ \bigl\Vert d|_{\mathcal{S}}J(u_{1,n},u_{2,n}) \bigr\Vert _{*}}=\sup_{ \substack{ \Vert (\phi _{1},\phi _{2}) \Vert =1\\\phi _{1},\phi _{2}\in T_{\mathcal {S}}(u_{1},u_{2})}} \bigl\vert \bigl(dE( \tilde{u}_{1,n}, \tilde{u}_{2,n}),\bigl(-s_{n}\star (\phi _{1},\phi _{2})\bigr)\bigr) \bigr\vert \\& \hphantom{ \bigl\Vert d|_{\mathcal{S}}J(u_{1,n},u_{2,n}) \bigr\Vert _{*}}= \bigl\Vert d|E(\tilde{u}_{1,n},\tilde{u}_{2,n}) \bigr\Vert _{*} \frac{ \Vert -s_{n}\star (\phi _{1},\phi _{2}) \Vert }{ \Vert (\phi _{1},\phi _{2}) \Vert } \\& \hphantom{ \bigl\Vert d|_{\mathcal{S}}J(u_{1,n},u_{2,n}) \bigr\Vert _{*}} \rightarrow 0. \end{aligned}$$

From the continuity of J, we have $J(u_{1,n},u_{2,n})\rightarrow c$. Moreover, because

$$ J\bigl( \vert u_{1} \vert , \vert u_{2} \vert \bigr) \leq J(u_{1},u_{2}), $$

we can choose $v_{1,n}, v_{2,n}\geq 0$, then we have $u_{1,n}^{-}, u_{2,n}^{-}\rightarrow 0$ in $\mathcal{H}$, which finishes the proof. □

Then we need to show that the P.S. sequence $\{(u_{1,n},u_{2,n})\}$, which is mentioned in Proposition 3.1, is bounded in $\mathcal {H}$.

Lemma 3.5

Sequence $\{(u_{1,n},u_{2,n})\}$ is bounded in $\mathcal{H}$.

Proof

First we have

$$ \begin{aligned} J(u_{1,n},u_{2,n})=& \frac{1}{2} \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}+ \vert \nabla u_{2,n} \vert ^{2}- \frac{1}{4} \int _{\mathbb{R}^{3}} \mu _{1}u_{1,n}^{4}+\mu _{2}u_{2,n}^{4}+2\beta u_{1,n}^{2}u_{2,n}^{2} \\ &{}- \int _{\mathbb{R}^{3}}\kappa u_{1,n}u_{2,n}\rightarrow c>0. \end{aligned} $$

(3.3)

From the fact that $\{(u_{1,n},u_{2,n})\}\in \mathcal{P}$, we have

$$ \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}+ \vert \nabla u_{2,n} \vert ^{2}= \frac{3}{4} \int _{\mathbb{R}^{3}}\mu _{1}u_{1,n}^{4}+\mu _{2}u_{2,n}^{4}+2 \beta u_{1,n}^{2}u_{2,n}^{2}. $$

(3.4)

Combining (3.3) and (3.4), we obtain

$$ \frac{1}{6} \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}+ \vert \nabla u_{2,n} \vert ^{2}- \int _{\mathbb{R}^{3}}\kappa u_{1,n}u_{2,n}\rightarrow c. $$

From the Schwarz inequation we have that $\{(\nabla u_{1,n},\nabla u_{2,n})\}$ is bounded in $L^{2}\times L^{2}$ together with $(u_{1,n},u_{2,n})\in \mathcal{S}$, then $\{(u_{1,n},u_{2,n})\}$ is bounded in $\mathcal{H}$. □

Then we have

$$ (u_{1,n},u_{2,n})\rightharpoonup (\bar{u}_{1},\bar{u}_{2})\in \mathcal{H}; $$

(3.5)

by the standard arguments of compact embedding, we have $u_{i,0}\geq 0$, $i=1,2$.

From the above discussion we have

$$ dJ|_{\mathcal{S}}(u_{1,n},u_{2,n})=dJ(u_{1,n},u_{2,n})- \lambda _{1,n}(u_{1,n},0)- \lambda _{2,n}(0,u_{2,n}) \rightarrow 0 $$

(3.6)

in $\mathcal{H}^{*}$, where

$$\begin{aligned}& \begin{aligned} \lambda _{1,n}&=\frac{1}{ \vert u_{1,n} \vert _{2}^{2}} \bigl(dJ(u_{1,n},u_{2,n}),(u_{1,n},0)\bigr) \\ &=\frac{1}{a_{1}^{2}}\biggl( \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}- \mu _{1} \int _{\mathbb{R}^{3}} u_{1,n}^{4}-\beta \int _{\mathbb{R}^{3}} u_{1,n}^{2}u_{2,n}^{2}- \kappa \int _{\mathbb{R}^{3}} u_{1,n}u_{2,n}\biggr), \end{aligned} \\& \begin{aligned} \lambda _{2,n}&=\frac{1}{ \vert u_{2,n} \vert _{2}^{2}} \bigl(dJ(u_{1,n},u_{2,n}),(0,u_{2,n})\bigr) \\ &=\frac{1}{a_{2}^{2}}\biggl( \int _{\mathbb{R}^{3}} \vert \nabla u_{2,n} \vert ^{2}- \mu _{2} \int _{\mathbb{R}^{3}} u_{2,n}^{4}-\beta \int _{\mathbb{R}^{3}} u_{1,n}^{2}u_{2,n}^{2}- \kappa \int _{\mathbb{R}^{3}} u_{1,n}u_{2,n}\biggr). \end{aligned} \end{aligned}$$

It is easy to check that $\{\lambda _{1,n}\}$ and $\{\lambda _{2,n}\}$ are bounded sequences, and we may assume $\lambda _{1,n}\rightarrow \bar{\lambda}_{1}$, $\lambda _{2,n}\rightarrow \bar{\lambda}_{2}$ up to the subsequence. Then, by weak convergence, we have

$$ dJ(\bar{u}_{1},\bar{u}_{2})-\bar{\lambda}_{1}(\bar{u}_{1},0)- \bar{\lambda}_{2}(0,\bar{u}_{2})=0 \quad \text{in }\mathcal{H}^{*}. $$

Moreover, $(\bar{\lambda}_{1},\bar{\lambda}_{2},\bar{u}_{1},\bar{u}_{2})$ is a solution of system

$$ \textstyle\begin{cases} -\Delta u_{1}-\lambda _{1} u_{1}=\mu _{1} u_{1}^{3}+\beta u_{1}u_{2}^{2}+ \kappa u_{2} \quad\text{in }\mathbb{R}^{3}, \\ -\Delta u_{2}-\lambda _{2} u_{2}=\mu _{2} u_{2}^{3}+\beta u_{1}^{2}u_{2}+ \kappa u_{1} \quad\text{in }\mathbb{R}^{3}. \end{cases} $$

(3.7)

If $(u_{1,n},u_{2,n})\rightarrow (\bar{u}_{1},\bar{u}_{2})\in \mathcal{H}$ strongly, then $(\bar{\lambda}_{1},\bar{\lambda}_{2},\bar{u}_{1},\bar{u}_{2})$ is a solution of system

$$ \textstyle\begin{cases} -\Delta u_{1}-\lambda _{1} u_{1}=\mu _{1} u_{1}^{3}+\beta u_{1}u_{2}^{2}+ \kappa u_{2} \quad\text{in }\mathbb{R}^{3}, \\ -\Delta u_{2}-\lambda _{2} u_{2}=\mu _{2} u_{2}^{3}+\beta u_{1}^{2}u_{2}+ \kappa u_{1} \quad\text{in }\mathbb{R}^{3}, \\ \int _{\mathbb{R}^{3}} u_{1}^{2}=a_{1}^{2},\qquad \int _{\mathbb{R}^{3}} u_{2}^{2}=a_{2}^{2}. \end{cases} $$

(3.8)

The following lemma gives a sufficient condition of strong convergence for the sequence $\{(u_{1,n},u_{2,n})\}$.

Lemma 3.6

If $\bar{\lambda}_{i}<0$, $i=1,2$, then we have strong convergence $u_{i,n}\rightarrow \bar{u}_{i}$ in $H_{r}^{1}$, $i=1,2$.

Proof

When $\bar{\lambda}_{1}<0$, we compute

$$ \begin{aligned} o(1)&=\bigl(dJ(u_{1,n},u_{2,n})-dJ( \bar{u}_{1},\bar{u}_{2}),(u_{1,n}- \bar{u}_{1},0)\bigr)-\bar{\lambda}_{1} \int _{\mathbb{R}^{3}}(u_{1,n}-\bar{u}_{1})^{2} \\ &= \int _{\mathbb{R}^{3}}(\nabla u_{1,n}-\nabla \bar{u}_{1})^{2}- \bar{\lambda}_{1} \int _{\mathbb{R}^{3}}(u_{1,n}-\bar{u}_{1})^{2}+o(1). \end{aligned} $$

Then, if $\bar{\lambda}_{1}<0$, we have $u_{1,n}\rightarrow \bar{u}_{1}$ in $H_{r}^{1}$. Similarly, if $\bar{\lambda}_{2}<0$, we have $u_{2,n}\rightarrow \bar{u}_{2}$ in $H_{r}^{1}$, which finishes the proof. □

Lemma 3.7

At least one of $\lambda _{i}$, $i=1,2$ is negative.

Proof

Notice that $(u_{1,n},u_{2,n})\in \mathcal{P}$, $u_{1,n}^{-}$, $u_{2,n}^{-}\rightarrow 0$ in $H_{r}^{1}$ and Lemma 3.2. There exists $\delta >0$ such that

$$\begin{aligned}& \bar{\lambda}_{1}a_{1}^{2}+ \bar{\lambda}_{2}a_{2}^{2} \\& \quad = \lambda _{1,n}a_{1}^{2}+\lambda _{2,n}a_{2}^{2}+o(1) \\& \quad = \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}+ \vert \nabla u_{2,n} \vert ^{2}- \int _{\mathbb{R}^{3}}\mu _{1}u_{1,n}^{4}+\mu _{2} u_{2,n}^{4}+2 \beta u_{1,n}^{2}u_{2,n}^{2}-2 \int _{\mathbb{R}^{3}}\kappa u_{1,n}^{+}u_{2,n}^{+}+o(1) \\& \quad \leq -\frac{1}{3} \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}+ \vert \nabla u_{2,n} \vert ^{2}+o(1) \\& \quad < -\frac{1}{6}\delta +o(1). \end{aligned}$$

Then we have $\bar{\lambda}_{1}<0$ or $\bar{\lambda}_{2}<0$, which finishes the proof. □

We need some Liouville type theorems to ensure that $\bar{\lambda}_{1}$ and $\bar{\lambda}_{2}$ are both negative.

Lemma 3.8

If $\bar{\lambda}_{1}<0$ and $\bar{\lambda}_{2}\geq 0$, then there exists $R\gg 0$ such that $\bar{u}_{1}$ and $\bar{u}_{2}$ are decreasing in $B_{R}^{c}(0)$. Moreover, $\bar{u}_{1}\leq \frac{C}{r^{2}}$ for some $C>0$ in $B_{R}^{c}(0)$.

Proof

When $u\in H^{1}_{rad}(\mathbb{R}^{3})$, for simplicity we take $u(x)=u(r)$, where $r=|x|$. We have

$$ \Delta u= \bigl(r^{2} u'(r)\bigr)'{r^{-2}} $$

and

$$ \bigl\vert u(x) \bigr\vert \leq C \vert u \vert _{2}^{\frac{1}{2}} \vert \nabla u \vert _{2}^{\frac{1}{2}} \vert x \vert ^{-1}, $$

(3.9)

then there exists $R_{1}>0$ such that $x\in B^{c}_{R_{1}}(0)$

$$ \begin{aligned} \Delta \bar{u}_{2}(x)&=-\bigl( \mu _{2}\bar{u}_{2}^{3}(x)+\beta \bar{u}_{1}^{2}(x)\bar{u}_{2}(x)+\kappa \bar{u}_{1}(x)+\bar{\lambda}_{2} \bar{u}_{2}(x)\bigr) \\ &\leq -\bigl(\beta \bar{u}_{1}^{2}(x)\bar{u}_{2}(x)+ \kappa \bar{u}_{1}(x)\bigr) \\ &\leq -\frac{\kappa}{2} \bar{u}_{1}(x). \end{aligned} $$

(3.10)

Taking $r_{1}$, $r_{2}> R_{1}$ in (3.10), we have

$$ \int _{r_{1}}^{r_{2}}\bigl(r^{2}\bar{u}'_{2}(r)\bigr)'\,dr\leq - \int _{r_{1}}^{r_{2}} \kappa \bar{u}_{2}(r)r^{2}\,dr, $$

i.e.,

$$ r^{2}\bar{u}'_{2}(r_{2})-r_{1}^{2}u'_{2}(r_{1})+ \int _{r_{1}}^{r_{2}}r^{2}u_{1}(r)\,dr \leq 0. $$

(3.11)

We claim that there exists $R_{2}>0$ such that when $r>R_{2}$ we have $\bar{u}'_{2}(r)<0$. If not, it is obvious that there exists a sequence ${r_{n}}$ such that $r_{n}\rightarrow \infty$ and $\bar{u}'_{2}(r_{n})=0$. Then from (3.11) we have

$$ \kappa \int _{r_{n}}^{r_{n+1}}r^{2}\bar{u}_{1}(r)\,dr\leq 0, $$

which is impossible because from the maximum principle we know that $\bar{u}_{1}$, $\bar{u}_{2}>0$. Then we have $\bar{u}_{2}$ is decreasing when $r>R_{2}$. From [11] we know that there exists $R_{3}>0$ such that $\bar{u}_{1}(r)$ is decreasing when $r>R_{3}$.

Next, we take $r_{2}=2r_{1}>R:=\max \{R_{1},R_{2},R_{3}\}$, and noticing (3.9), we have

$$ - \int _{r_{1}}^{r_{2}}\bigl(r^{2}\bar{u}'_{2}(r)\bigr)\leq C \int _{r_{1}}^{r_{2}} \frac{1}{r}+r\,dr\leq C\bigl(\ln 2+r_{2}^{2}\bigr) $$

(3.12)

for some constant $C>0$. Moreover, notice that $\bar{u}_{2}(r)$ is decreasing when $r>R$ and (3.12), then we have

$$ \bigl\vert \bar{u}'_{2}(r) \bigr\vert \leq \frac{C}{r}, $$

(3.13)

where $r>R$. Finally, from (3.11) and (3.13), we have

$$ \bar{u}_{1}(r)\leq \frac{C}{r^{2}}, $$

when $r>R$. □

Lemma 3.9

If $\bar{\lambda}_{1}<0$ and $\bar{\lambda}_{2}\geq 0$, then we have $\bar{u}_{2}\equiv 0$.

Proof

We assume that $\bar{u}_{2}\not \equiv 0$. From the maximum principle, we have $\bar{u}_{2}>0$. First we have

$$ -\Delta \bar{u}_{2}-\bar{\lambda}_{2} \bar{u}_{2}=\mu _{2}\bar{u}_{2}^{3}+ \beta \bar{u}_{1}^{2} \bar{u}_{2}+\kappa \bar{u}_{1}. $$

Take $\bar{u}_{2}=w$ and $c(x)=-|\beta |\bar{u}_{1}^{2}(x)$, then we have

$$ \bigl\vert c(x) \bigr\vert \leq \frac{C}{ \vert x \vert ^{4}} $$

and

$$ -\Delta w+c(x)w\geq 0, $$

where $x\in B_{R}^{c}(0)$. For $\phi \in (1,\frac{3}{2}]$, we take $V=r^{-\phi}$, then we have

$$ \begin{aligned} -\Delta V+c(x)V&=-\bigl(\phi ^{2}-\phi \bigr) \vert x \vert ^{-\phi -2}+c(x) \vert x \vert ^{- \phi} \\ &\leq -\bigl(\phi ^{2}-\phi \bigr) \vert x \vert ^{-\phi -2}+C \vert x \vert ^{-\phi -4} \\ &< 0 \end{aligned} $$

(3.14)

when $x\in B_{R'}^{c}(0)$ for some $R'>0$. Take $\varphi :=w-C_{0}r^{-\phi}$, where $C_{0}=\frac{w(\bar{R})}{\bar{R}^{-\phi}}$ and $\bar{R}:=\max \{R,R'\}$, we have

$$ -\Delta \varphi +c(x)\varphi \geq 0 $$

in $B_{\bar{R}}^{c}(0)$. From the maximum principle, we have $w\geq C_{0}r^{-\phi}$ in $B_{\bar{R}}^{c}(0)$, but it is easy to show that $r^{-\phi}\notin L^{2}(B_{\bar{R}}^{c}(0))$, which contradicts $w\in L^{2}(\mathbb{R}^{3})$, so we must have $\bar{u}_{2}\equiv 0$. □

Proof of Theorem 1.1

If $\bar{\lambda}_{1}<0$, $\bar{\lambda}_{2}\geq 0$, we have $\bar{u}_{2}\equiv 0$ and $\bar{u}_{1}>0$. From the structure of system (3.7), we know that $(\bar{u}_{1}, 0)$ cannot be the solution of system (3.7). If $\bar{\lambda}_{1}\geq 0$, $\bar{\lambda}_{2}<0$, we have $\bar{u}_{1}\equiv 0$ and $\bar{u}_{2}>0$. From the structure of system (3.7), we know that $(\bar{0}, u_{2})$ cannot be the solution of system (3.7). So we must have $\bar{\lambda}_{1}$, $\bar{\lambda}_{2}<0$. From Lemma 3.6 we have $(\bar{\lambda}_{1},\bar{\lambda}_{2},\bar{u}_{1},\bar{u}_{2})$ is a solution of system (1.1). Moreover, $\bar{\lambda}_{1}$, $\bar{\lambda}_{2}<0$ and $\bar{u}_{1}$, $\bar{u}_{2}>0$, which finishes the proof of Theorem 1.1. □

4 Nonautonomous systems

In this section, we prove Theorem 1.2.

The corresponding energy functional of system (1.1) on $\mathcal{S}$ is defined by

$$ J(u_{1},u_{2})=\frac{1}{2} \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}- \frac{1}{4} \int _{\mathbb{R}^{3}}\mu _{1} u_{1}^{4}+ \mu _{2} u_{2}^{4}+2\beta u_{1}^{2}u_{2}^{2}- \int _{\mathbb{R}^{3}} \kappa (x) u_{1}u_{2}, $$

where $\mu _{1}$, $\mu _{2}$, $\kappa >0$ and $\beta <0$.

Firstly, $J|_{\mathcal{S}}$ is unbounded from below, so we cannot achieve $\inf_{\mathcal{S}}J(u_{1},u_{2})$. Secondly, (1.1) is a nonautonomous system. The Pohožeav identity of system (1.1) involves the gradient of $\kappa (x)$, and it is hard to figure out whether J is bounded below on the Pohožeav manifold. To get the critical point of $J|_{\mathcal{S}}$, we will try to find a minimax value of $J|_{\mathcal{S}}$ by constructing a minimax structure on $\mathcal{S}$. For this purpose, we introduce the following two sets, where $K_{2}>K_{1}>0$:

$$\begin{aligned}& A_{K_{1}}:=\biggl\{ (u_{1},u_{2})\in \mathcal{S}: \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}\leq K_{1}\biggr\} ,\\& B_{K_{2}}:=\biggl\{ (u_{1},u_{2})\in \mathcal{S}: \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}=K_{2} \biggr\} . \end{aligned}$$

By Lemma 3.1 we have

$$ \int _{\mathbb{R}^{3}}\mu _{1} u_{1}^{4}+\mu _{2} u_{2}^{4}+2\beta u_{1}^{2}u_{2}^{2} \leq C_{a_{1},a_{2}}\biggl( \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2} \biggr)^{\frac{3}{2}}, $$

where $C_{a_{1},a_{2}}=\max \{a_{1}\mu _{1}G^{4},a_{2}\mu _{2}G^{4}\}$, and $G>0$ denotes the best constant for the Gagliardo–Nirenberg inequality in $\mathbb{R}^{3}$.

Lemma 4.1

There exists $C_{1}>0$, where $C:=C(\kappa ,a_{1},a_{2})$ and $K_{1}>0$ such that for any $(u_{1},u_{2})\in A_{K_{1}}$,

$$ J(u_{1},u_{2})>-C. $$

(4.1)

Proof

We let $K_{1}<\frac{4}{C_{a_{1},a_{2}}^{2}}$, where $\frac{4}{C_{a_{1},a_{2}}^{2}}$ is the biggest zero point of the function

$$ \frac{1}{2}x-\frac{C_{a_{1},a_{2}}}{4}x^{\frac{3}{2}}. $$

Then we have

$$ \begin{aligned} J(u_{1},u_{2})&=\frac{1}{2} \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}- \frac{1}{4} \int _{\mathbb{R}^{3}} \mu _{1} u_{1}^{4}+\mu _{2} u_{2}^{4}+\beta u_{1}^{2}u_{2}^{2}- \int _{ \mathbb{R}^{3}}\kappa u_{1}u_{2} \\ &\geq \frac{1}{2} \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}- \frac{C_{a_{1},a_{2}}}{4}\biggl( \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2} \biggr)^{\frac{3}{2}}-\kappa a_{1}a_{2} \\ &\geq -\kappa a_{1}a_{2}, \end{aligned} $$

then we take $C=\kappa a_{1}a_{2}$ to get (4.1). □

Lemma 4.2

Assume $K_{2}=\frac{16}{9C_{a_{1},a_{2}}^{2}}$ and $\kappa <\frac{5}{18C_{a_{1},a_{2}}^{2}}$. If $K_{1}$ is small enough, then we have

$$ \sup_{A_{K_{1}}}J(u_{1},u_{2})< \inf_{B_{K_{2}}}J(u_{1},u_{2}) $$

(4.2)

and

$$ \inf_{B_{K_{2}}}J(u_{1},u_{2})>0. $$

(4.3)

Proof

Take $(v_{1},v_{2})\in B_{K_{2}}$, $(u_{1},u_{2})\in A_{K_{1}}$, notice that $K_{2}>0$ is the maximum point of the function

$$ \frac{1}{2}x-\frac{C_{a_{1},a_{2}}}{4}x^{\frac{3}{2}}, $$

$\kappa (x)<\frac{5}{18C_{a_{1},a_{2}}^{2}a_{1}a_{2}}$. Note that $-\sqrt{\mu _{1}\mu _{2}}<\beta <0$ and choose $K_{1}$ small enough, then we have

$$ \begin{aligned} &J(v_{1},v_{2})-J(u_{1},u_{2}) \\ &\quad = \frac{1}{2} \int _{ \mathbb{R}^{3}} \vert \nabla v_{1} \vert ^{2}+ \vert \nabla v_{2} \vert ^{2}- \frac{1}{2} \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2} \\ &\qquad {}-\frac{1}{4} \int _{\mathbb{R}^{3}} \mu _{1}v_{1}^{4}+\mu _{2} v_{2}^{4}+2 \beta v_{1}^{2} v_{2}^{2}+\frac{1}{4} \int _{\mathbb{R}^{3}} \mu _{1} u_{1}^{4}+ \mu _{2} u_{2}^{4}+2\beta u_{1}^{2}u_{2}^{2} \\ &\qquad {}- \int _{\mathbb{R}^{3}} \kappa v_{1}v_{2}+ \int _{\mathbb{R}^{3}} \kappa u_{1}u_{2} \\ &\quad \geq \frac{1}{2} \int _{\mathbb{R}^{3}} \vert \nabla v_{1} \vert ^{2}+ \vert \nabla v_{2} \vert ^{2}- \frac{1}{2} \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2} \\ &\qquad {}-\frac{1}{4} \int _{\mathbb{R}^{3}} \mu _{1}v_{1}^{4}+\mu _{2} v_{2}^{4}+2 \beta v_{1}^{2} v_{2}^{2}-2\kappa a_{1}a_{2} \\ &\quad \geq \frac{1}{2} K_{2}-\frac{C_{a_{1},a_{2}}}{4}(K_{2})^{ \frac{3}{2}}- \frac{1}{2}K_{1}-2\kappa a_{1}a_{2} \\ &\quad >0. \end{aligned} $$

(4.4)

Take $(u_{1},u_{2})\in B_{K_{2}}$, similarly to (4.4), we have

$$\begin{aligned} J(u_{1},u_{2})&=\frac{1}{2} \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}- \frac{1}{4} \int \mu _{1}u_{1}^{4}+ \mu _{2} u_{2}^{4}+2\beta u_{1}^{2}u_{2}^{2}- \int _{\mathbb{R}^{3}} \kappa u_{1}u_{2} \\ &\geq \frac{1}{2}K_{2}-\frac{C_{a_{1},a_{2}}}{4}(K_{2})^{\frac{3}{2}}- \kappa a_{1}a_{2} \\ &>0. \end{aligned}$$

This finishes the proof. □

We fix a point $(v_{1},v_{2})\in A_{K_{1}}$ both nonnegative, and we try to find a point $(w_{1},w_{2})$ such that $J(w_{1},w_{2})$ is negative enough, and $\int _{\mathbb{R}^{3}} |\nabla w_{1}|^{2}+|\nabla w_{2}|^{2}$ is large enough. Then any path from $(v_{1},v_{2})$ to $(w_{1},w_{2})$ must pass through $B_{K_{2}}$, so we get a mountain pass structure on manifold $\mathcal{S}$. To do this, we use the translation, which was firstly mentioned in [11]:

$$ s\star u:=e^{\frac{3s}{2}}u\bigl(e^{s}x\bigr). $$

By direct calculation we have

$$ \vert s\star u \vert _{2}^{2}= \vert u \vert _{2}^{2} $$

and

$$ \bigl\vert \nabla (s\star u) \bigr\vert _{2}^{2}=e^{2s} \vert \nabla u \vert _{2}^{2}. $$

Moreover, we have

$$ \begin{aligned} &J(s\star v_{1},s\star v_{2}) \\ &\quad ={}\frac{e^{2s}}{2} \int _{\mathbb{R}^{3}} \vert \nabla v_{1} \vert ^{2}+ \vert \nabla v_{2} \vert ^{2}- \frac{e^{3s}}{4} \int _{\mathbb{R}^{3}} \mu _{1} v_{1}^{4}+ \mu _{2} v_{2}^{4}+2\beta v_{1}^{2}v_{2}^{2}- \int _{\mathbb{R}^{3}} \kappa (x) (s\star v_{1}) (s\star v_{2}) \\ &\quad \leq{} \frac{e^{2s}}{2} \int _{\mathbb{R}^{3}} \vert \nabla v_{1} \vert ^{2}+ \vert \nabla v_{2} \vert ^{2}- \frac{e^{3s}}{4} \int _{\mathbb{R}^{3}} \mu _{1} v_{1}^{4}+ \mu _{2} v_{2}^{4}+2\beta v_{1}^{2}v_{2}^{2}+ \kappa a_{1}a_{2}. \end{aligned} $$

If s is large enough, then we have $J(s\star v_{1},s\star v_{2})<-C_{1}$, where $C_{1}$ is defined in (4.1), and we take $(w_{1},w_{2}):=(s\star v_{1},s\star v_{2})$.

Then we can get a mountain pass structure of J on manifold $\mathcal{S}$:

$$ \Gamma :=\bigl\{ \gamma (t)=\bigl(\gamma _{1}(t),\gamma _{2}(t)\bigr):\gamma (0)=(v_{1},v_{2}), \gamma (1)=(w_{1},w_{2})\bigr\} , $$

(4.5)

and the mountain pass value is

$$ c:=\inf_{\gamma \in \Gamma}\sup_{t\in [0,1]}J \bigl(\gamma (t)\bigr)\geq \inf_{B_{K_{2}}}J(u_{1},u_{2})>0. $$

(4.6)

To obtain the boundedness of the P.S. sequence at mountain pass value c, we use the following notations:

$$ \tilde{J}(s,u_{1},u_{2}):=J(s\star u_{1},s\star u_{2})=\tilde {J}(0,s \star u_{1},s\star u_{2}). $$

(4.7)

The corresponding minimax structure of J̃ on $\mathbb{R}\times \mathcal{S}$ is as follows:

$$ \tilde {\Gamma}:=\bigl\{ \tilde {\gamma}(t)=\bigl(s(t),\gamma _{1}(t), \gamma _{2}(t)\bigr): \tilde{\gamma}(0)=(0,v_{1},v_{2}), \tilde{\gamma}(1)=(0,w_{1},w_{2})\bigr\} , $$

and its minimax value is

$$ \tilde{c}=\inf_{\tilde{\gamma} \in \tilde{\Gamma}}\sup_{t\in [0,1]} \tilde{J}\bigl( \tilde{\gamma}(t)\bigr). $$

First we claim that $\tilde{c}=c$.

In fact, from $\tilde{\Gamma}\supset \Gamma$ we have $\tilde{c}\leq c$. On the other hand, for any

$$ \tilde{\gamma}(t)=\bigl(s(t),\gamma _{1}(t),\gamma _{2}(t) \bigr), $$

by definition we have

$$ \tilde{J}\bigl(\tilde{\gamma}(t)\bigr)=J\bigl(s(t)\star \gamma (t)\bigr), $$

and $s(t)\star \gamma (t)\in \Gamma$ is obvious, then

$$ \sup_{t\in [0,1]}\tilde{J}\bigl(\tilde{\gamma}(t)\bigr)\geq \inf _{\gamma \in \Gamma}\sup_{t\in [0,1]}J\bigl(\gamma (t)\bigr). $$

By the definition of c̃, we have $\tilde{c}\geq c$, then $\tilde{c}=c$. Because

$$ \tilde{J}(s,u_{1},u_{2})=\tilde {J}(0,s\star u_{1},s\star u_{2}), $$

we take a sequence $\tilde{\gamma}_{n}=(0,\gamma _{1,n},\gamma _{2,n})\in \tilde{\Gamma}$ such that

$$ c=\lim_{n\rightarrow \infty}\sup_{t\in [0,1]}\tilde{J}\bigl( \tilde{\gamma}_{n}(t)\bigr). $$

Moreover, using the fact that $\kappa (x)>0$, we have

$$ \tilde{J}\bigl(s, \vert u_{1} \vert , \vert u_{2} \vert \bigr)\leq \tilde{J}(s,u_{1},u_{2}), $$

then we can assume $\gamma _{1,n}$, $\gamma _{2,n}\geq 0$. By Theorem 3.2 in [10] (it is easy to check that the conditions of Theorem 3.2 in [10] are satisfied by Lemma 4.2), we can get a P.S. sequence $(s_{n},\tilde{u}_{1,n},\tilde{u}_{2,n})$ of J̃ on $\mathbb{R}\times \mathcal{S}$ at level c. Moreover,

$$ \lim_{n\rightarrow \infty} \vert s_{n} \vert +\operatorname{dist}_{\mathcal{H}}\bigl((\tilde{u}_{1,n}, \tilde{u}_{2,n}),( \gamma _{1,n},\gamma _{2,n})\bigr)=0. $$

So, we have $s_{n}\rightarrow 0$ and $\tilde{u}_{1,n}^{-}$, $\tilde{u}_{2,n}^{-}\rightarrow 0$ in $H^{1}_{r}$. Then, taking

$$ (u_{1,n},u_{2,n}):=(s_{n}\star \tilde{u}_{1,n},s_{n}\star \tilde{u}_{2,n}), $$

we have the following lemma.

Lemma 4.3

$(u_{1,n},u_{2,n})$ is a P.S. sequence of $J(u_{1},u_{2})$ at level c on $\mathcal{S}$.

Proof

First we know that $(s_{n},\tilde{u}_{1,n},\tilde{u}_{2,n})$ is a P.S. sequence of $\tilde{J}(s,u_{1},u_{2})$, then for any $(\phi _{1},\phi _{2})\in H_{r}^{1}\times H_{r}^{1}$ we have

$$\begin{aligned}& \bigl(\partial _{\textbf{u}}\tilde{J}(s_{n},\tilde{u}_{1,n}, \tilde{u}_{2,n}),(\phi _{1},\phi _{2})\bigr) \\& \quad ={} e^{2s_{n}} \int _{\mathbb{R}^{3}}\nabla \tilde{u}_{1,n}\cdot \nabla \phi _{1}+\nabla \tilde{u}_{2,n}\cdot \nabla \phi _{2} \\& \qquad {}-e^{3s_{n}} \int _{\mathbb{R}^{3}} \mu _{1}\tilde{u}_{1,n}^{3} \phi _{1}+ \mu _{2}\tilde{u}_{2,n}^{3}+\beta \tilde{u}_{1,n}\phi _{1}\tilde{u}_{2,n}^{2}+ \beta \tilde{u}_{1,n}^{2}\tilde{u}_{2,n}\phi _{2} \\& \qquad {}- \int _{\mathbb{R}^{3}}\kappa \bigl(e^{-s_{n}}x\bigr)\tilde{u}_{1,n}\phi _{2}- \int _{\mathbb{R}^{3}}\kappa \bigl(e^{-s_{n}}x\bigr)\tilde{u}_{2,n}\phi _{1} \\& \quad ={} \int _{\mathbb{R}^{3}} \nabla u_{1,n}\cdot \nabla (s_{n}\star \phi _{1})+\nabla u_{2,n}\cdot \nabla (s_{n}\star \phi _{2}) \\& \qquad {}- \int _{\mathbb{R}^{3}} \mu _{1} u_{1,n}^{3}(s_{n} \star \phi _{1})+ \mu _{2} u_{2,n}^{3}(s_{n} \star \phi _{1})+\beta u_{1,n}^{2} u_{2,n}(s_{n} \star \phi _{2})+\beta u_{1,n}u_{2,n}^{2}(s_{n}\star \phi _{1}) \\& \qquad {}- \int _{\mathbb{R}^{3}}\kappa (x)u_{1,n}(s_{n}\star \phi _{2})- \int _{ \mathbb{R}^{3}}\kappa (x)u_{2,n}(s_{n}\star \phi _{1}) \\& \quad ={} \bigl(dJ(u_{1,n},u_{2,n}),(s_{n}\star \phi _{1},s_{n}\star \phi _{2})\bigr), \end{aligned}$$

where $\textbf{u}=(u_{1},u_{2})$. Notice that $-s\star (s\star \phi )=\phi$, $\forall s\in \mathbb{R}$, we have

$$ \partial _{\textbf{u}}\tilde{J}(s_{n},\tilde{u}_{1,n},\tilde{u}_{2,n}) (-s_{n} \star \phi _{1},-s_{n} \star \phi _{2})= dJ(u_{1,n},u_{2,n}) ( \phi _{1}, \phi _{2}). $$

It is obvious that $(\phi _{1},\phi _{2})\in T_{(u_{1,n},u_{2,n})}\mathcal{S}$ if and only if $(-s_{n}\star \phi _{1},-s_{n}\star \phi _{2})\in T_{(\tilde{u}_{1,n}, \tilde{u}_{2,n})}\mathcal{S}$, see [6]. Since $s_{n}\rightarrow 0$, we have $-s_{n}\star \phi _{i}\rightarrow \phi _{i}$, $i=1,2$, as $n\rightarrow \infty$ in $H^{1}_{r}$. Then, for n large enough, there exist $A_{1}>0$ and $A_{2}>0$ such that

$$ A_{1}< \frac{ \Vert (\phi _{1},\phi _{2}) \Vert }{ \Vert (-s_{n}\star \phi _{1},-s_{n}\star \phi _{2}) \Vert }< A_{2}, $$

(4.8)

where $(\phi _{1},\phi _{2})\neq (0,0)$. Let $\|\cdot \|_{\star}$ be the norm of the cotangent space $(T_{(u_{1},u_{2})}\mathcal{S})^{\star}$. Thus, for any $(\phi _{1},\phi _{2})\in T_{(u_{1,n},u_{2,n})}\mathcal{S}$ and $(\phi _{1},\phi _{2})\neq (0,0)$, we have

$$ \begin{aligned} \biggl\vert dJ|_{\mathcal{S}}(u_{1,n},u_{2,n}) \frac{(\phi _{1},\phi _{2})}{ \Vert (-s_{n}\star \phi _{1},-s_{n}\star \phi _{2}) \Vert } \biggr\vert & \leq \bigl\Vert (\partial _{\textbf{u}} \tilde{J}|_{\mathcal{S}} ) (s_{n},\tilde{u}_{1,n}, \tilde{u}_{2,n}) \bigr\Vert _{\star}\rightarrow 0 \end{aligned} $$

as $n\rightarrow \infty$. Take the supremum on both sides and notice (4.8), we have

$$ A_{1} \bigl\Vert dJ|_{\mathcal{S}}(u_{1,n},u_{2,n}) \bigr\Vert _{\star} \leq \bigl\Vert ( \partial _{\textbf{u}}\tilde{J}|_{\mathcal{S}} ) (s_{n},\tilde{u}_{1,n}, \tilde{u}_{2,n}) \bigr\Vert _{\star}\rightarrow 0 \quad \text{as } n \rightarrow \infty . $$

From the fact that $A_{1}>0$, we have

$$ \bigl\Vert dJ|_{\mathcal{S}}(u_{1,n},u_{2,n}) \bigr\Vert _{\star}\rightarrow 0 \quad \text{as } n\rightarrow \infty . $$

On the other hand, we have

$$ J(u_{1,n},u_{2,n})=\tilde{J}(s_{n},\tilde{u}_{1,n},\tilde{u}_{2,n}) \rightarrow c \quad \text{as } n\rightarrow \infty . $$

This finishes the proof. □

Lemma 4.4

If $\kappa (x)$ and $\nabla \kappa (x)\cdot x$ is bounded in $\mathbb{R}^{3}$, then the P.S. sequence $(u_{1,n},u_{2,n})$ obtained in Lemma 4.3of $J(u_{1},u_{2})$ on $\mathcal{S}$ at level c is bounded in $\mathcal{H}$.

Proof

Since $(s_{n},\tilde{u}_{1,n},\tilde{u}_{2,n})$ is a P.S. sequence for J̃, we have

$$ \frac{\partial}{\partial s}\tilde{J}(s_{n},\tilde{u}_{1,n},\tilde{u}_{2,n}) \rightarrow 0, $$

i.e.,

$$ \begin{aligned} & \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}+ \vert \nabla u_{2,n} \vert ^{2}- \frac{3}{4} \int _{\mathbb{R}^{3}} \mu _{1} u_{1,n}^{4}+ \mu _{2} u_{2,n}^{4}+2 \beta u_{1,n}^{2}u_{2,n}^{2} \\ &\quad {}+ \int _{\mathbb{R}^{3}} \nabla \kappa \bigl(e^{-s_{n}}x\bigr)\cdot e^{-s_{n}}x \tilde{u}_{1,n}\tilde{u}_{2,n}\rightarrow 0. \end{aligned} $$

(4.9)

On the other hand, notice that $\tilde{J}(s_{n},\tilde{u}_{1,n},\tilde{u}_{2,n})=J(u_{1,n},u_{2,n})$, we obtain

$$ \begin{aligned} J(u_{1,n},u_{2,n})={}& \frac{1}{2} \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}+ \vert \nabla u_{2,n} \vert ^{2}- \frac{1}{4} \int _{\mathbb{R}^{3}} \mu _{1} u_{1,n}^{4}+ \mu _{2} u_{2,n}^{4}+2\beta u_{1,n}^{2}u_{2,n}^{2} \\ &{}- \int _{\mathbb{R}^{3}}\kappa \bigl(e^{-s_{n}}x\bigr)\tilde{u}_{1,n}\tilde{u}_{2,n} \rightarrow c. \end{aligned} $$

(4.10)

Then, using the boundedness of $\kappa (x)$, $\nabla \kappa (x)\cdot x$, $(\tilde{u}_{1,n},\tilde{u}_{2,n})\in \mathcal{S}$, (4.9) and (4.10), we can deduce that $\int _{\mathbb{R}^{3}} |\nabla u_{1,n}|^{2}+|\nabla u_{2,n}|^{2}$ is bounded. Notice that $(u_{1,n},u_{2,n})\in \mathcal{S}$, we get $( u_{1,n}, u_{2,n})$ is bounded in $H^{1}_{r}\times H^{1}_{r}$. □

Because $(u_{1,n},u_{2,n})$ is bounded in $H^{1}_{r}\times H^{1}_{r}$, there exists $(\bar{u}_{1},\bar{u}_{2})\in H^{1}_{r}\times H^{1}_{r}$ such that

$$ (u_{1,n},u_{2,n})\rightharpoonup (\bar{u}_{1},\bar{u}_{2})\quad \text{in } H^{1}_{r} \times H^{1}_{r}. $$

Lemma 4.5

Under the assumptions of Lemma 4.4, and we assume $\frac{1}{3}\nabla \kappa (x)\cdot x+\kappa (x)\geq 0$, then there exists $C>0$ such that for n large we have

$$ \vert \nabla u_{1,n} \vert _{2}^{2}+ \vert \nabla u_{2,n} \vert _{2}^{2}\geq C. $$

Proof

By (4.9) and $\tilde{u}_{1,n}^{-},\tilde{u}_{2,n}^{-}\rightarrow 0$ in $H^{1}_{r}$, we have

$$ \begin{aligned} & \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}+ \vert \nabla u_{2,n} \vert ^{2}- \frac{3}{4} \int _{\mathbb{R}^{3}} \mu _{1} u_{1,n}^{4}+ \mu _{2} u_{2,n}^{4}+2 \beta u_{1,n}^{2}u_{2,n}^{2} \\ &\quad {}+ \int _{\mathbb{R}^{3}} \nabla \kappa \bigl(e^{-s_{n}}x\bigr)\cdot e^{-s_{n}}x \tilde{u}_{1,n}^{+}\tilde{u}_{2,n}^{+}=o(1) \end{aligned} $$

and

$$ \begin{aligned}& \frac{1}{2} \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}+ \vert \nabla u_{2,n} \vert -\frac{1}{4} \int _{\mathbb{R}^{3}} \mu _{1} u_{1,n}^{4}+ \mu _{2} u_{2,n}^{4}+2\beta u_{1,n}^{2}u_{2,n}^{2} \\ &\quad {}- \int _{\mathbb{R}^{3}}\kappa \bigl(e^{-s_{n}}x\bigr)\tilde{u}_{1,n}^{+} \tilde{u}_{2,n}^{+}= c+o(1). \end{aligned} $$

By (4.6), we have $c>0$, thus

$$\begin{aligned} c+o(1)&=J(u_{1,n},u_{2,n}) \\ &= \frac{1}{6} \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}+ \vert \nabla u_{2,n} \vert ^{2}- \int _{\mathbb{R}^{3}} \biggl(\frac{1}{3} \nabla \kappa \bigl(e^{-s_{n}}x\bigr)\cdot e^{-s_{n}}x+ \kappa \bigl(e^{-s_{n}}x \bigr)\biggr)\tilde{u}_{1,n}^{+}\tilde{u}_{2,n}^{+} \\ &\leq \frac{1}{6} \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}+ \vert \nabla u_{2,n} \vert ^{2}, \end{aligned}$$

then, for n large enough and taking $C=3c$, this finishes the proof. □

Because $(u_{1,n},u_{2,n})$ is a P.S. sequence of J on $\mathcal{S}$, for any $(\phi _{1},\phi _{2})\in H^{1}_{r}\times H^{1}_{r}$, there exist $\lambda _{1,n}$, $\lambda _{2,n}$ such that, as $n\rightarrow \infty$,

$$ \begin{aligned} &\bigl(dJ|_{\mathcal{S}}(u_{1,n},u_{2,n}),( \phi _{1},\phi _{2})\bigr) \\ &\quad ={} \int _{\mathbb{R}^{3}} \nabla u_{1,n}\nabla \phi _{1}+ \int _{ \mathbb{R}^{3}} \nabla u_{2,n}\nabla \phi _{2}- \mu _{1} \int _{ \mathbb{R}^{3}} u_{1,n}^{3}\phi _{1}- \mu _{2} \int _{\mathbb{R}^{3}} u_{2,n}^{3} \phi _{2} \\ &\qquad {}-\beta \int _{\mathbb{R}^{3}} u_{1,n}u_{2,n}^{2}\phi _{1}-\beta \int _{\mathbb{R}^{3}} u_{1,n}^{2}u_{2,n}\phi _{2}- \int _{\mathbb{R}^{3}} \kappa (x)u_{1,n}\phi _{2} \\ &\qquad {}- \int _{\mathbb{R}^{3}}\kappa (x)u_{2,n}\phi _{1}-\lambda _{1,n} \int _{\mathbb{R}^{3}} u_{1,n}\phi _{1}-\lambda _{2,n} \int _{ \mathbb{R}^{3}} u_{2,n}\phi _{2} \\ &\quad ={}o\bigl( \bigl\Vert (\phi _{1},\phi _{2}) \bigr\Vert \bigr). \end{aligned} $$

(4.11)

From Sect. 3 we have

$$\begin{aligned}& \lambda _{1,n}a_{1}^{2}= \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}- \mu _{1} \int _{\mathbb{R}^{3}} u_{1,n}^{4}-\beta \int _{\mathbb{R}^{3}} u_{1,n}^{2}u_{2,n}^{2}- \int _{\mathbb{R}^{3}} \kappa (x) u_{1,n}u_{2,n}, \end{aligned}$$

(4.12)

$$\begin{aligned}& \lambda _{2,n}a_{2}^{2}= \int _{\mathbb{R}^{3}} \vert \nabla u_{2,n} \vert ^{2}- \mu _{2} \int _{\mathbb{R}^{3}} u_{2,n}^{4}-\beta \int _{\mathbb{R}^{3}} u_{1,n}^{2}u_{2,n}^{2}- \int _{\mathbb{R}^{3}} \kappa (x) u_{1,n}u_{2,n}, \end{aligned}$$

(4.13)

then it is easy to deduce that $\{\lambda _{1,n}\}$ and $\{\lambda _{2,n}\}$ are bounded. So we may assume

$$\begin{aligned}& \lambda _{1,n}\rightarrow \bar{\lambda}_{1},\\& \lambda _{2,n}\rightarrow \bar{\lambda}_{2} \end{aligned}$$

by choosing subsequence if necessary.

Lemma 4.6

Under the conditions of Lemma 4.5, assume $\frac{2}{3}\nabla \kappa (x)\cdot x+\kappa (x)\geq 0$ and $\kappa (x)>0$, then at least one of $\bar{\lambda}_{i}$, $i=1,2$, is negative.

Proof

Notice that $\tilde{u}_{1,n}^{-}\rightarrow 0$, $\tilde{u}_{2,n}^{-}\rightarrow 0$ in $H^{1}_{r}$, (4.12), (4.13), and (4.9), we have

$$\begin{aligned}& \bar{\lambda}_{1}a_{1}^{2}+ \bar{\lambda}_{2}a_{2}^{2} \\& \quad ={} \lambda _{1,n}a_{1}^{2}+\lambda _{2,n}a_{2}^{2}+o(1) \\& \quad ={} \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}+ \vert \nabla u_{2,n} \vert ^{2}- \int _{\mathbb{R}^{3}} \mu _{1} u_{1,n}^{4}+ \mu _{2} u_{2,n}^{4}+2 \beta u_{1,n}^{2}u_{2,n}^{2} \\& \qquad {}-2 \int _{\mathbb{R}^{3}} \kappa (x)u_{1,n}u_{2,n}+o(1) \\& \quad ={} -\frac{1}{3} \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}+ \vert \nabla u_{2,n} \vert ^{2} \\& \qquad {}-\biggl( \int _{\mathbb{R}^{3}} \biggl(\frac{4}{3}\nabla \kappa \bigl(e^{-s_{n}}x\bigr)\cdot e^{-s_{n}}x+2 \kappa \bigl(e^{-s_{n}}x\bigr)\biggr)\tilde{u}_{1,n}^{+}\tilde{u}_{2,n}^{+}\biggr)+o(1) \\& \quad \leq{} -\frac{1}{3} \int _{\mathbb{R}^{3}} \vert \nabla u_{1,n} \vert ^{2}+ \vert \nabla u_{2,n} \vert ^{2}+o(1) \\& \quad < {} -\frac{1}{3}C+o(1), \end{aligned}$$

then one of $\bar{\lambda}_{1}$, $\bar{\lambda}_{2}$ is negative. □

Proof of Theorem 1.2.

From the standard argument we can conclude that $(\bar{\lambda}_{1},\bar{\lambda}_{2},\bar{u}_{1},\bar{u}_{2})$ is a solution of the system

$$ \textstyle\begin{cases} -\Delta u_{1}-\lambda _{1} u_{1}=\mu _{1} u_{1}^{3}+\beta u_{1}u_{2}^{2}+ \kappa (x)u_{2} \quad\text{in }\mathbb{R}^{3}, \\ -\Delta u_{2}-\lambda _{2} u_{2}=\mu _{2} u_{2}^{3}+\beta u_{1}^{2} u_{2}+ \kappa (x)u_{1} \quad\text{in }\mathbb{R}^{3}, \end{cases} $$

(4.14)

we just need to prove $(u_{1,n},u_{2,n})\rightarrow (\bar{u}_{1},\bar{u}_{2})$ strongly in $\mathcal{H}$. From Lemma 3.6, it is sufficient to prove that $\bar{\lambda}_{1}<0$ and $\bar{\lambda}_{2}<0$, Lemma 4.6, Lemma 3.8, and Lemma 3.8 make sure that $\bar{\lambda}_{1}<0$ and $\bar{\lambda}_{2}<0$, which finishes the proof. □