Skip to main content

Sign-changing solutions for coupled Schrödinger system

Abstract

In this paper we study the following nonlinear Schrödinger system:

$$ \textstyle\begin{cases} -\Delta u+\alpha u = \vert u \vert ^{p-1}u+\frac{2}{q+1} \lambda \vert u \vert ^{ \frac{p-3}{2}}u \vert v \vert ^{\frac{q+1}{2}},\quad x \in \mathbb{R}^{3}, \\ -\Delta v+\beta v = \vert v \vert ^{q-1}v+\frac{2}{p+1} \lambda \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}v ,\quad x \in \mathbb{R}^{3}, \\ u(x)\rightarrow 0,\qquad v(x)\rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty , \end{cases} $$

where \(3\leq p, q<5\), α, β are positive parameters. We show that there exists \(\lambda _{k}>0\) such that the equation has at least k radially symmetric sign-changing solutions and at least k seminodal solutions for each \(k\in \mathbb{N}\) and \(\lambda \in (0, \lambda _{k})\). Moreover, we show the existence of a least energy radially symmetric sign-changing solution for each \(\lambda \in (0, \lambda _{0})\) where \(\lambda _{0}\in (0, \lambda _{1}]\).

1 Background and main results

Consider the following nonlinear coupled Schrödinger system:

$$ \textstyle\begin{cases} -\Delta u+\alpha u = \vert u \vert ^{p-1}u+\frac{2}{q+1} \lambda \vert u \vert ^{ \frac{p-3}{2}}u \vert v \vert ^{\frac{q+1}{2}} ,\quad x \in \Omega , \\ -\Delta v+\beta v = \vert v \vert ^{q-1}v+\frac{2}{p+1} \lambda \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}v ,\quad x \in \Omega , \\ u=v=0\quad \text{on } \partial \Omega . \end{cases} $$
(1.1)

Here \(\Omega =\mathbb{R}^{N}\) or Ω is a smooth bounded domain in \(\mathbb{R}^{N}\), α, β are positive parameters and \(\lambda \neq 0\) is a coupling constant.

In the case \(p=q=3\), system (1.1) becomes the cubic system:

$$ \textstyle\begin{cases} -\Delta u +\alpha u= u^{3}+\lambda u v^{2}, \quad x\in \Omega , \\ -\Delta v +\beta v = v^{3}+\lambda u^{2} v, \quad x\in \Omega , \\ u=v=0 \quad \text{on }\partial \Omega ,\end{cases} $$
(1.2)

which arises in the study of many physical phenomena like nonlinear optics and Bose–Einstein condensation (cf. [15, 17]). Therefore, in the last decades, system (1.2) has received great interest from mathematicians. When Ω is the entire space \(\mathbb{R}^{N}\), the existence of least energy and other finite energy solutions of (1.2) was studied in [2, 11, 12, 18, 21, 22, 27] and the references therein. In particular, when \(\lambda >0\) is sufficiently large, infinitely many radially symmetric sign-changing solutions of (1.2) were obtained in [23]. Liu and Wang [20] studied a general m-coupled system (\(m\ge 2\)) and proved that system (1.2) has infinitely many nontrivial solutions, but whether solutions obtained in [20] are positive or sign-changing cannot be determined there (see also [21]). When \(\Omega \subset \mathbb{R}^{N}\) (\(N=2, 3\)) is a smooth bounded domain, there are also many papers studying (1.2). Lin and Wei [18] proved that a least energy solution of (1.2) exists within an appropriate range of λ. Dancer, Wei, and Weth [14] and Noris and Ramos [24] proved the existence of infinitely many positive solutions of (1.2). When Ω is a ball, a multiplicity result on positive radially symmetric solutions was given in [29]. Later, by using a global bifurcation approach, the result of [29] was reproved by [4] without requiring the symmetric condition. Under some more general assumptions, Sato and Wang [26] proved that system (1.2) has infinitely many semipositive solutions (i.e., at least one component is positive). In [14], the authors proved the existence of unbounded sequence solutions for \(N\leq 3\) and \(\lambda \leq -1\). As pointed out above, for \(\lambda \leq -1\), Wei and Weth [29] proved that (1.2) has a radially symmetric solution, which turns out to be a positive solution.

We remark that the existence of infinitely many sign-changing solutions or seminodal solutions to (1.2) was solved by Chen, Lin, and Zou [10] and Liu, Liu, and Wang [19] independently, where \(N\leq 3\) and \(\lambda <0\).

To the best of our knowledge, the existence of sign-changing solutions to (1.1) has not ever been studied in the literature when \(\Omega =\mathbb{R}^{3}\) and \(3\leq p, q<5\). The main goal of this paper is to study the existence of sign-changing solutions, seminodal solutions, and least energy sign-changing solutions to problem (1.1) when \(\lambda >0\) is small. This will complement the study made in [14, 19, 21, 22, 29].

Definition 1.1

A solution \((u, v)\) is called nontrivial if \(u\not \equiv 0\) and \(v\not \equiv 0\), a solution \((u, v)\) is semitrivial if \((u, v)\) is type of \((u, 0)\) or \((0, v)\). We call a solution \((u, v)\) positive if \(u>0\) and \(v>0\) in \(\mathbb{R}^{N}\), a solution \((u, v)\) sign-changing if both u and v change sign, a solution \((u, v)\) seminodal if one changes sign and the other one is positive.

The first main result of the current paper is as follows.

Theorem 1.1

Assume \(\alpha , \beta >0\). Then for any \(k\in \mathbb{N}\) there exists \(\lambda _{k}>0\) such that system (1.1) possesses at least k radially symmetric sign-changing solutions for each fixed \(\lambda \in (0, \lambda _{k})\).

We can also study some further properties of the sign-changing solutions obtained in Theorem 1.1. It is well known that a nontrivial solution \((u, v)\in H^{1}(\mathbb{R}^{N})\times H^{1}(\mathbb{R}^{N})\) is called a least energy solution if its energy is minimal among the energy of all nontrivial solutions. A sign-changing solution is called a least energy sign-changing solution if it has the least energy among all sign-changing solutions. Precisely, we have the following theorem.

Theorem 1.2

Assume \(\alpha , \beta >0\). Then there exists \(\lambda _{0}\in (0, \lambda _{1}]\) such that system (1.1) possesses a least energy radially symmetric sign-changing solution for each fixed \(\lambda \in (0, \lambda _{0})\).

Theorem 1.3

Assume \(\alpha , \beta >0\). Then for any \(k\in \mathbb{N}\) there exists \(\lambda _{k}>0\) such that system (1.1) possesses at least k seminodal solutions for each fixed \(\lambda \in (0, \lambda _{k})\).

Remark 1.1

We can prove that system (1.1) possesses at least k seminodal solutions with the first component positive and the second component radially symmetric sign-changing or the first component radially symmetric sign-changing and the second component positive.

The structure of this paper is as follows. In Sect. 2 we prove the existence of at least k radially symmetric sign-changing solutions. The main tool will be the use of a new notion of vector genus by [28] and a new constrained problem by [10], which will be used to construct minimax values. Remark that the ideas in [10, 28] cannot be used directly, and here we will give some new ideas. The crucial idea in this paper is turning to study a new problem with two constraints to obtain sign-changing solutions of (1.1). This idea has never been used for (1.1) in the literature up to our knowledge. We will give all the necessary details of the proof. Section 3 is then dedicated to the proof of Theorem 1.2 by using a minimizing argument. Finally in Sect. 4 we will present the proof of Theorem 1.3 applying the arguments in Sect. 2 and Sect. 3.

We give some notations here. Throughout this paper, we denote the norm of \(L^{p}(\mathbb{R}^{N})\) by \(|u|_{p}= (\int _{\mathbb{R}^{N}} |u|^{p} \,dx )^{\frac{1}{p}}\), the norm of \(H^{1}(\mathbb{R}^{N})\) by \(\|u\|^{2}=\int _{\mathbb{R}^{N}}(|\nabla u|^{2}+ |u|^{2}) \,dx \), and positive constants (possibly different in different places) by C. Define \(H_{r}:= H_{r}^{1}(\mathbb{R}^{N})\times H_{r}^{1}(\mathbb{R}^{N})\) as a subspace of \(H:= H^{1}(\mathbb{R}^{N})\times H^{1}(\mathbb{R}^{N})\) with norm \(\|(u, v)\|_{H_{r}}^{2}:=\|u\|_{\alpha}^{2}+\|v\|_{\beta}^{2}\) where

$$ \begin{aligned}&H_{r}^{1}\bigl(\mathbb{R}^{N}\bigr):=\bigl\{ u\in H^{1}\bigl(\mathbb{R}^{N}\bigr): u \text{ is radially symmetric}\bigr\} , \\ & \Vert u \Vert _{\alpha}^{2}:= \int _{\mathbb{R}^{N}}\bigl( \vert \nabla u \vert ^{2}+ \alpha \vert u \vert ^{2}\bigr) \,dx .\end{aligned} $$

2 Proof of Theorem 1.1

In this section, we assume that \(N=3\), \(3\leq p, q<2^{*}-1=5\) and \(\alpha , \beta >0\). Without loss of generality, we assume \(p\leq q\). Let \(\lambda \in (0, 1)\). For any \(k\in \mathbb{N}\), let \(X_{k+1}\subset H_{r}^{1}(\mathbb{R}^{3})\), \(\operatorname{dim}X_{k+1}=k+1\), and there exists \(u_{0}\in X_{k+1}\) and \(u_{0}>0\). Then there exists \(m>0\) such that for any \((u, v)\in X_{k+1}\times X_{k+1}\) satisfying \(|u|_{p+1}^{p+1}, |v|_{q+1}^{q+1}<2\), we have

$$ \Vert u \Vert _{\alpha}^{2}< m,\qquad \Vert v \Vert _{\beta}^{2} < m. $$
(2.1)

Without loss of generality, we can assume \(m>1\). Obviously, the sign-changing solutions of system (1.1) are the critical points of the \(C^{2}\) functional \(\Phi _{\lambda}: H_{r}\rightarrow \mathbb{R}\) given by

$$ \begin{aligned}\Phi _{\lambda}(u, v) &:= \frac{1}{2} \bigl( \Vert u \Vert _{\alpha}^{2}+ \Vert v \Vert _{\beta}^{2}\bigr)-\frac{1}{p+1} \vert u \vert _{p+1}^{p+1}-\frac{1}{q+1} \vert v \vert _{q+1}^{q+1} \\ &\quad{}-\frac{4\lambda}{(p+1)(q+1)} \int _{ \mathbb{R}^{3}} \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q+1}{2}} \,dx . \end{aligned} $$
(2.2)

We will look for solutions of Eq. (1.1) as critical points of the functional \(\Phi _{\lambda}\) restricted to the sphere

$$ \mathcal{A}:=\bigl\{ (u, v)\in H_{r}: \vert u \vert _{p+1}=1, \vert v \vert _{q+1}=1 \bigr\} . $$

To obtain at least k sign-changing critical points, we need to define several minimax energy levels using a new definition of vector genus introduced by [28]. As in [28], we recall vector genus and take the transformations

$$ \sigma _{i}: \mathcal{A}\rightarrow \mathcal{A},\qquad \sigma _{1}(u, v)=(-u, v),\qquad \sigma _{2}(u, v)=(u, -v), \quad i=1, 2. $$

Consider the class of sets

$$ \mathcal{F}=\bigl\{ A\subset \mathcal{A}: A \text{ is a closed set and } \sigma _{i}(u, v)\in A, \forall (u, v)\in A, i=1, 2\bigr\} $$

and for each \(A\in \mathcal{F}\) and \(k_{1}, k_{2}\in \mathbb{N}\), the class of functions

$$ \begin{aligned}F_{(k_{1}, k_{2})}(A)={}&\Biggl\{ f=(f_{1}, f_{2}): A\rightarrow \prod_{i=1}^{2} \mathbb{R}^{k_{i}-1}: f_{i}: A\rightarrow \mathbb{R}^{k_{i}-1}\text{ continuous}, \\ &{}f_{i}\bigl(\sigma _{i}(u, v)\bigr)=-f_{i}(u, v) \text{ for each } i, f_{i}\bigl( \sigma _{j}(u, v) \bigr)=f_{i}(u, v) \text{ for } i\neq j \Biggr\} . \end{aligned} $$

where \(\mathbb{R}^{0}:=\{0\}\).

Definition 2.1

(Vector genus, see [28]) For every nonempty and closed set \(A\subset H_{0}^{1}(\Omega )\) such that \(-A=A\), we define

$$ \gamma (A):=\inf \bigl\{ k: \text{there exists } h:A \rightarrow \mathbb{R}^{k} \backslash \{0\} \text{ continuous and odd}\bigr\} $$

and \(\gamma (A):=\infty \) if no such k exists.

Let \(A\in \mathcal{F}\) and take any \(k_{1}, k_{2}\in \mathbb{N}\). We say that \(\gamma (A)\geq (k_{1}, k_{2})\) if for every \(f\in F_{(k_{1}, k_{2})}(A)\) there exists \((u, v)\in A\) such that \(f(u, v)=(f_{1}(u, v), f_{2}(u, v))=(0, 0)\). We denote

$$ \Gamma ^{(k_{1}, k_{2})}:=\bigl\{ A\in \mathcal{F}: \gamma (A)\geq (k_{1}, k_{2})\bigr\} . $$

Remark 2.1

Note that Definition 2.1 does not actually define the quantity \(\gamma (A)\) but gives the meaning of \(\gamma (A)\geq (k_{1}, k_{2})\) only. A different notation of genus was introduced by Chang, Wang, and Zhang in [8].

Lemma 2.1

(see [28])

Let \(f=(f_{1}, f_{2}): \prod_{i=1}^{2} S^{k_{i}}\rightarrow \prod_{i=1}^{2} \mathbb{R}^{k_{i}}\) be a continuous function such that \(f_{i}(\sigma _{i}(u, v))=-f_{i}(u, v)\), \(f_{i}(\sigma _{j}(u, v))=f_{i}(u, v)\) for any \(i, j=1, 2\), \(i\neq j\), then there exists \((u_{0}, v_{0})\in \prod_{i=1}^{2} S^{k_{i}}\) such that \(f(u_{0}, v_{0})=(0, \ldots ,0)\).

Lemma 2.2

(see [28])

The following properties hold.

  1. (1)

    Take \(A_{1}\times A_{2}\subset \mathcal{A}\) and let \(\eta _{i}: S^{k_{i}-1}\rightarrow A_{i}\) be a homeomorphism such that \(\eta _{i}(-x)=-\eta _{i}(x)\) for every \(x\in S^{k_{i}-1}\), \(i=1, 2\). Then \(A_{1}\times A_{2} \in \Gamma ^{(k_{1}, k_{2})}\), where \(S^{k_{i}-1}=\{x\in \mathbb{R}^{k_{i}}: |x|=1\}\).

  2. (2)

    We have \(\overline{\eta (A)}\in \Gamma ^{(k_{1}, k_{2})}\) whenever \(A \in \Gamma ^{(k_{1}, k_{2})}\) and a continuous map \(\eta : A\rightarrow \mathcal{A}\) is such that \(\eta \circ \sigma _{i}=\sigma _{i}\circ \eta \), \(\forall i=1, 2\).

Together with the notation of vector genus, to obtain sign-changing solutions, we will use cones of positive or negative functions based on the works such as [5, 13, 30]. We define the cone

$$ \mathcal{P}_{1}:=\bigl\{ (u, v)\in H_{r}: u\geq 0\bigr\} ,\qquad \mathcal{P}_{2}:= \bigl\{ (u, v)\in H_{r}: v\geq 0\bigr\} , $$

and take \(\mathcal{P}:=\bigcup_{i=1}^{2} (\mathcal{P}_{i}\cup - \mathcal{P}_{i} )\). Moreover, for any \(\delta >0\), we define

$$ \mathcal{P}_{\delta}:=\bigl\{ (u, v)\in H_{r}: \operatorname{dist}\bigl((u, v), \mathcal{P}\bigr)< \delta \bigr\} , $$

where

$$\begin{aligned}& \begin{aligned} \operatorname{dist} \bigl((u, v), \mathcal{P}\bigr):={}& \min \bigl\{ \operatorname{dist}_{p+1} (u, \mathcal{P}_{1}) , \operatorname{dist}_{p+1} (u, -\mathcal{P}_{1}), \\ &\operatorname{dist}_{q+1} (v, \mathcal{P}_{2}) , \operatorname{dist}_{q+1} (v, -\mathcal{P}_{2})\bigr\} , \end{aligned}\\& \operatorname{dist}_{p+1} (u, \pm \mathcal{P}_{1}) :=\inf _{\omega \in \pm \mathcal{P}_{1}} \vert u-\omega \vert _{p+1}= \bigl\vert u^{\mp} \bigr\vert _{p+1},\\& \operatorname{dist}_{q+1} (v, \pm \mathcal{P}_{2}) :=\inf _{\omega \in \pm \mathcal{P}_{2}} \vert v-\omega \vert _{q+1}= \bigl\vert v^{\mp} \bigr\vert _{q+1}, \end{aligned}$$

where \(u^{\pm}:=\max \{0, \pm u\}\).

Lemma 2.3

For any \(0<\delta <2^{-\frac{1}{p+1}}\), there holds \(A\backslash \mathcal{P}_{\delta }\neq \emptyset \) whenever \(A \in \Gamma ^{(k_{1}, k_{2})}\) with \(k_{1}, k_{2}\geq 2\).

Proof

For any \(A \in \Gamma ^{(k_{1}, k_{2})}\), define \(f=(f_{1}, f_{2})\) by

$$\begin{aligned}& f_{1}(u, v)= \biggl( \int _{ \mathbb{R}^{3}} \vert u \vert ^{p}u \,dx , 0, \ldots , 0 \biggr),\\& f_{2}(u, v)= \biggl( \int _{ \mathbb{R}^{3}} \vert v \vert ^{q}v \,dx , 0, \ldots , 0 \biggr), \end{aligned}$$

then \(f\in F_{(k_{1}, k_{2})}(A)\), so by Definition 2.1, there exists \((u_{0}, v_{0})\in A\) such that \(f(u_{0}, v_{0})=(0, \ldots ,0)\). By \(A\in \mathcal{A}\), we deduce that

$$\begin{aligned}& \int _{ \mathbb{R}^{3}} \bigl(u_{0}^{+} \bigr)^{p+1} \,dx = \int _{ \mathbb{R}^{3}} \bigl(u_{0}^{-} \bigr)^{p+1} \,dx =\frac{1}{2},\\& \int _{ \mathbb{R}^{3}} \bigl(v_{0}^{+} \bigr)^{q+1} \,dx = \int _{ \mathbb{R}^{3}} \bigl(v_{0}^{-} \bigr)^{q+1} \,dx =\frac{1}{2}, \end{aligned}$$

therefore, \(\operatorname{dist} ((u_{0}, v_{0}), \mathcal{P})=2^{-\frac{1}{p+1}}\), and so \((u_{0}, v_{0})\in A \backslash \mathcal{P}_{\delta}\) for any \(0<\delta <2^{-\frac{1}{p+1}}\). □

For technical reasons, we will work on the neighborhood of \(\mathcal{A}\) in \(H_{r}^{1}(\mathbb{R}^{3})\),

$$ \mathcal{A}^{*}:=\biggl\{ (u, v)\in H_{r}: \frac{1}{2}< \vert u \vert _{p+1}^{p+1}< 2, \frac{1}{2}< \vert v \vert _{q+1}^{q+1}< 2 \biggr\} , $$
(2.3)

when \(u\in \mathcal{A}^{*}\), \((u, v)\not \equiv (0, 0)\). Define

$$\begin{aligned}& \mathcal{B}_{m}^{*}:=\bigl\{ (u, v)\in \mathcal{A}^{*} : \Vert u \Vert _{\alpha}^{2}< m, \Vert v \Vert _{\beta}^{2}< m \bigr\} , \end{aligned}$$
(2.4)
$$\begin{aligned}& \mathcal{B}_{m}:=\bigl\{ (u, v)\in \mathcal{A} : \Vert u \Vert _{\alpha}^{2}< m, \Vert v \Vert _{\beta}^{2}< m \bigr\} , \end{aligned}$$
(2.5)
$$\begin{aligned}& \mathcal{C}_{m}:= \bigl\{ (u, v)\in \mathcal{A} : \Vert u \Vert _{\alpha}^{2}=m, \Vert v \Vert _{\beta}^{2}=m \bigr\} . \end{aligned}$$
(2.6)

Let \(S_{p}\) and \(S_{q}\) be the sharp constants of the Sobolev embedding \(H_{r}^{1}(\mathbb{R}^{3})\hookrightarrow L^{p+1}(\mathbb{R}^{3})\) and \(H_{r}^{1}(\mathbb{R}^{3})\hookrightarrow L^{q+1}(\mathbb{R}^{3})\), respectively,

$$ \Vert u \Vert _{\alpha}^{2}\geq S_{p} \vert u \vert _{p+1}^{2},\qquad \Vert v \Vert _{\beta}^{2} \geq S_{q} \vert v \vert _{q+1}^{2},\quad \forall u, v\in H_{r}^{1}\bigl( \mathbb{R}^{3}\bigr). $$
(2.7)

For any \((u, v)\in H_{r}\backslash \{(0, 0)\}\), we have

$$ \sup_{t, s\geq 0} \Phi _{\lambda}(tu, sv)= \Phi _{\lambda}(t_{u,v, \lambda}u, s_{u,v,\lambda}v)=:\Psi _{\lambda}(u, v), $$
(2.8)

where \(t_{u,v,\lambda}, s_{u,v,\lambda}\geq 0\) satisfy

$$ \frac{\partial}{\partial t}\Phi _{\lambda}(tu, sv)|_{(t_{u,v,\lambda}, s_{u,v,\lambda})}= \frac{\partial}{\partial s}\Phi _{\lambda}(tu, sv)|_{(t_{u,v, \lambda}, s_{u,v,\lambda})}=0. $$

Note that for \(t, s\geq 0\),

$$ \begin{aligned}\Phi _{\lambda}(tu, sv) :={}& \frac{1}{2} \bigl(t^{2} \Vert u \Vert _{ \alpha}^{2}+s^{2} \Vert v \Vert _{\beta}^{2}\bigr)-\frac{t^{p+1}}{p+1} \vert u \vert _{p+1}^{p+1}- \frac{s^{q+1}}{q+1} \vert v \vert _{q+1}^{q+1} \\ &{}-\frac{4\lambda}{(p+1)(q+1)} t^{\frac{p+1}{2}} s^{ \frac{q+1}{2}} \int _{ \mathbb{R}^{3}} \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{ \frac{q+1}{2}} \,dx . \end{aligned} $$
(2.9)

Define

$$ \begin{aligned}F(u, v, \lambda ; t, s)&:= t \Vert u \Vert _{\alpha}^{2}-t^{p} \vert u \vert _{p+1}^{p+1}- \frac{2}{q+1} t^{\frac{p-1}{2}} s^{\frac{q+1}{2}} \lambda \int _{ \mathbb{R}^{3}} \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{\frac{q+1}{2}} \,dx \\ &:= t F_{1}(u, v, \lambda ; t, s) \end{aligned} $$

and

$$ \begin{aligned}G(u, v, \lambda ; t, s)&:= s \Vert v \Vert _{\beta}^{2}-s^{q} \vert v \vert _{q+1}^{q+1}- \frac{2}{p+1} t^{\frac{p+1}{2}} s^{\frac{q-1}{2}} \lambda \int _{ \mathbb{R}^{3}} \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{\frac{q+1}{2}} \,dx \\ &:=s G_{1}(u, v, \lambda ; t, s), \end{aligned} $$

which implies

$$ F_{1}(u, v, \lambda ; t_{u,v,\lambda}, s_{u,v,\lambda})=G_{1}(u, v, \lambda ; t_{u,v,\lambda}, s_{u,v,\lambda})=0. $$
(2.10)

Since \(F_{1}(u, v, \lambda ; t, s)\) and \(G_{1}(u, v, \lambda ; t, s)\) are decreasing with respect to \(t>0\) and \(s>0\), respectively, \(F_{1}(u, v, \lambda ; 0, 0)>0\), \(G_{1}(u, v, \lambda ; 0, 0)>0\), so \(t_{u,v,\lambda}\), \(s_{u,v,\lambda}\) are unique. Note that for \(t, s\geq 0\), \(3\leq p, q<5\), by (2.9), we can choose some positive constant T such that \(\Phi _{\lambda}(tu, sv)<0\) for any \(t, s>T\), therefore, \(t_{u,v,\lambda}, s_{u,v,\lambda}\in [0, T]\).

Define

$$ \widetilde{m}> \biggl[(q+1) S_{p}\biggl( \frac{1}{2}\biggr)^{\frac{2}{p+1}} \biggr]^{ \frac{2}{p+q-2}} + \frac{4(p+1)(q+1)}{(p-1)(\frac{S_{p}}{8})^{\frac{2}{p-1}}}m^{ \frac{p+1}{p-1}}+m. $$
(2.11)

Then \(B_{m}\subset B_{\widetilde{m}}\), \(B_{m}^{*}\subset B_{\widetilde{m}}^{*}\).

Lemma 2.4

For any \(k\in \mathbb{N}\), there exist \(\widetilde{\lambda}\in (0, 1)\) and \(T_{1}>T_{2}>0\) such that for any \(\lambda \in (0, \widetilde{\lambda})\) and \((u, v)\in B_{\widetilde{m}}^{*}\), we have

$$ T_{2}\leq t_{u,v,\lambda}, s_{u,v,\lambda} \leq T_{1}. $$
(2.12)

Furthermore, there exist \(\lambda _{k}\in (0, \widetilde{\lambda}]\) and \(c_{k}>0\) such that for any \(\lambda \in (0, \lambda _{k})\), we have

$$ \sup_{(u,v)\in \mathcal{B}_{m}} \sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv)< c_{k}\leq \inf_{(u,v)\in \mathcal{C}_{\widetilde{m }}} \sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv) . $$
(2.13)

Proof

We see from (2.9) and (2.10) that

$$ \begin{aligned}\sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv)&= \Phi _{\lambda}(t_{u,v, \lambda}u, s_{u,v,\lambda}v) \\ &=\biggl(\frac{1}{2}-\frac{1}{p+1}\biggr)t_{u,v,\lambda}^{2} \Vert u \Vert _{\alpha}^{2}+\biggl( \frac{1}{2}-\frac{1}{q+1}\biggr)s_{u,v,\lambda}^{q+1} \vert v \vert _{q+1}^{q+1} \\ &\quad{}+\frac{(q-1)}{(p+1)(q+1)} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{ \mathbb{R}^{3}} \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q+1}{2}} \,dx . \end{aligned} $$
(2.14)

Firstly, we claim that there exist \(\widetilde{\lambda}\in (0, 1)\) and \(T_{1}>T_{2}>0\) such that for any \(\lambda \in (0, \widetilde{\lambda})\) and \((u, v)\in B_{\widetilde{m}}^{*}\), we have

$$ T_{2}\leq t_{u,v,\lambda}, s_{u,v,\lambda}\leq T_{1}. $$

By (2.10),

$$\begin{aligned}& t_{u,v,\lambda}\leq \biggl(\frac{ \Vert u \Vert _{\alpha}^{2}}{ \vert u \vert _{p+1}^{p+1}} \biggr)^{\frac{1}{p-1}}< (2 \widetilde{m})^{\frac{1}{p-1}}< 2\widetilde{m},\\& s_{u,v,\lambda}\leq \biggl(\frac{ \Vert v \Vert _{\beta}^{2}}{ \vert v \vert _{q+1}^{q+1}} \biggr)^{\frac{1}{q-1}}< (2 \widetilde{m})^{\frac{1}{q-1}}< 2\widetilde{m}. \end{aligned}$$

Thus, we obtain that

$$ t_{u,v,\lambda}, s_{u,v,\lambda}< 2\widetilde{m}=:T_{1}. $$

Define

$$ \widetilde{\lambda}= \frac{(q+1) S_{p} (\frac{1}{2})^{\frac{2}{p+1}}}{8 (2\widetilde{m} )^{\frac{p+q-2}{2}}}. $$

We see from (2.11) that \(\widetilde{\lambda}\in (0, 1)\). Moreover, by (2.7) and (2.10), for any \(\lambda \in (0, \widetilde{\lambda})\), we have

$$\begin{aligned} t_{u,v,\lambda}^{p-1} \vert u \vert _{p+1}^{p+1} &= \Vert u \Vert _{\alpha}^{2}- \frac{2}{q+1} t_{u,v,\lambda}^{\frac{p-3}{2}} s_{u,v,\lambda}^{ \frac{q+1}{2}} \lambda \int _{ \mathbb{R}^{3}} \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{ \frac{q+1}{2}} \,dx \\ &> S_{p} \biggl(\frac{1}{2}\biggr)^{\frac{2}{p+1}} - \frac{2}{q+1} (2 \widetilde{m})^{\frac{p+q-2}{2}}\lambda \vert u \vert _{p+1}^{\frac{p+1}{2}} \vert v \vert _{q+1}^{\frac{q+1}{2}} \\ &> S_{p} \biggl(\frac{1}{2}\biggr)^{\frac{2}{p+1}} - \frac{4}{q+1} (2 \widetilde{m})^{\frac{p+q-2}{2}}\lambda \\ &> \frac{1}{2} S_{p} \biggl(\frac{1}{2} \biggr)^{\frac{2}{p+1}}> \frac{S_{p}}{4}. \end{aligned}$$

Then we get \(t_{u,v,\lambda}> (\frac{S_{p}}{8} )^{\frac{1}{p-1}}\). Similarly, we have \(s_{u,v,\lambda}> (\frac{S_{q}}{8} )^{\frac{1}{q-1}}\). Thus, we get

$$ t_{u,v,\lambda}, s_{u,v,\lambda}>\min \biggl\{ \biggl( \frac{S_{p}}{8} \biggr)^{ \frac{1}{p-1}}, \biggl(\frac{S_{q}}{8} \biggr)^{\frac{1}{q-1}} \biggr\} =:T_{2}. $$

This completes \(T_{2}\leq t_{u,v,\lambda}\leq T_{1}\).

Now we prove the existence of \(\lambda _{k}\) and \(c_{k}\). For any \((u, v)\in \overline{B}_{\widetilde{m}}\) and \(\lambda \in (0, \widetilde{\lambda}]\), by (2.14), there holds

$$ \begin{aligned}& \biggl\vert \sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv)-\biggl( \frac{1}{2}-\frac{1}{p+1} \biggr)t_{u,v,\lambda}^{2} \Vert u \Vert _{\alpha}^{2}- \biggl( \frac{1}{2}-\frac{1}{q+1}\biggr)s_{u,v,\lambda}^{q+1} |v|_{q+1}^{q+1} \biggr\vert \\ &\quad = \biggl\vert \frac{(q-1)}{(p+1)(q+1)} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{ \mathbb{R}^{3}} \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q+1}{2}} \,dx \biggr\vert \leq C\lambda . \end{aligned} $$

Hence,

$$ \begin{aligned} & \sup_{(u,v)\in B_{m}} \sup _{t,s\geq 0} \Phi _{ \lambda}(tu, sv) \\ &\quad \leq \sup_{(u,v)\in B_{m}} \biggl[ \biggl(\frac{1}{2}- \frac{1}{p+1}\biggr)t_{u,v, \lambda}^{2} \Vert u \Vert _{\alpha}^{2}+\biggl(\frac{1}{2}- \frac{1}{q+1}\biggr)s_{u,v, \lambda}^{q+1} \vert v \vert _{q+1}^{q+1} \biggr]+C\lambda \\ &\quad \leq \sup_{(u,v)\in B_{m}} \biggl[ \biggl(\frac{1}{2}- \frac{1}{p+1}\biggr) \biggl( \frac{ \Vert u \Vert _{\alpha}^{2}}{ \vert u \vert _{p+1}^{p+1} } \biggr)^{\frac{2}{p-1}} \Vert u \Vert _{\alpha}^{2}+\biggl(\frac{1}{2}- \frac{1}{q+1}\biggr) \biggl( \frac{ \Vert v \Vert _{\beta}^{2} }{ \vert v \vert _{q+1}^{q+1}} \biggr)^{\frac{q+1}{q-1}} \biggr]+C\lambda \\ &\quad \leq \biggl(\frac{1}{2}-\frac{1}{p+1}\biggr) m^{\frac{p+1}{p-1}} +\biggl(\frac{1}{2}- \frac{1}{q+1} \biggr)m^{\frac{q+1}{q-1}}+C\lambda \\ &\quad \leq 2 \biggl(\frac{1}{2}-\frac{1}{q+1}\biggr)m^{\frac{p+1}{p-1}}+C \lambda < (q+1) m^{\frac{p+1}{p-1}}+C\lambda , \end{aligned} $$

and

$$ \begin{aligned} &\inf_{(u,v)\in \mathcal{C}_{\widetilde{m }}} \sup _{t,s\geq 0} \Phi _{\lambda}(tu, sv) \\ &\quad \geq \inf_{(u,v)\in \mathcal{C}_{\widetilde{m }}} \biggl[ \biggl(\frac{1}{2}- \frac{1}{p+1}\biggr)t_{u,v,\lambda}^{2} \Vert u \Vert _{ \alpha}^{2}+\biggl(\frac{1}{2}- \frac{1}{q+1}\biggr)s_{u,v,\lambda}^{q+1} \vert v \vert _{q+1}^{q+1} \biggr]-C\lambda \\ &\quad >\inf_{(u,v)\in \mathcal{C}_{\widetilde{m }}} \biggl(\frac{1}{2}- \frac{1}{p+1}\biggr)t_{u,v,\lambda}^{2} \Vert u \Vert _{\alpha}^{2}-C \lambda \\ &\quad \geq \biggl(\frac{1}{2}-\frac{1}{p+1}\biggr) \biggl( \frac{S_{p}}{8} \biggr)^{ \frac{2}{p-1}}\widetilde{m}-C\lambda , \end{aligned} $$

then by (2.11), we can choose

$$\begin{aligned}& \lambda _{k}=\min \biggl\{ \frac{q+1}{2C} m^{\frac{p+1}{p-1}} , \widetilde{\lambda} \biggr\} ,\\& c_{k}=\biggl(\frac{1}{2}-\frac{1}{p+1}\biggr) \biggl(\frac{S_{p}}{8} \biggr)^{ \frac{2}{p-1}}\widetilde{m}-C\lambda _{k} \end{aligned}$$

such that \(c_{k}>0\) for any \(0<\lambda <\lambda _{k}\) the conclusion holds. □

For any \((u, v)\in B_{\widetilde{m}}^{*}\), the following linear problem

$$ \textstyle\begin{cases} -\Delta \varphi +\alpha \varphi -\frac{2}{q+1} t_{u,v,\lambda}^{ \frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{ \frac{p-3}{2}}\varphi \vert v \vert ^{\frac{q+1}{2}}=t_{u,v,\lambda}^{p-1} \vert u \vert ^{p-1}u, \\ - \Delta \psi +\beta \psi -\frac{2}{p+1} t_{u,v,\lambda}^{ \frac{p+1}{2}} s_{u,v,\lambda}^{\frac{q-3}{2}} \lambda \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}\psi =s_{u,v,\lambda}^{q-1} \vert v \vert ^{q-1}v, \\ \varphi (x)\rightarrow 0,\qquad \psi (x)\rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty , \end{cases} $$
(2.15)

has a unique solution \((\varphi , \psi )\in H_{r}\setminus \{ (0, 0)\}\). Then we can choose \(\lambda _{k}\) small enough such that for any \(\varphi , \psi \in H_{r}^{1}(\mathbb{R}^{3})\),

$$ \begin{aligned} \int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u \varphi \,dx &= \frac{ \Vert \varphi \Vert _{\alpha}^{2}- \frac{2}{q+1} t_{u,v,\lambda}^{\frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{\frac{p-3}{2}}\varphi ^{2} \vert v \vert ^{\frac{q+1}{2}} \,dx }{t_{u,v,\lambda}^{p-1}} \\ &\geq \frac{\frac{1}{2} \Vert \varphi \Vert _{\alpha}^{2}}{t_{u,v,\lambda}^{p-1}}>0 \end{aligned} $$

and

$$ \begin{aligned} \int _{\mathbb{R}^{3}} \vert v \vert ^{q-1}v \psi \,dx &= \frac{ \Vert \psi \Vert _{\beta}^{2}- \frac{2}{p+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v,\lambda}^{\frac{q-3}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}\psi ^{2} \,dx }{s_{u,v,\lambda}^{q-1}} \\ &\geq \frac{\frac{1}{2} \Vert \psi \Vert _{\beta}^{2}}{s_{u,v,\lambda}^{q-1}}>0. \end{aligned} $$

Define

$$ \mu :=\frac{1}{\int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u \varphi \,dx },\qquad \nu :=\frac{1}{\int _{\mathbb{R}^{3}} \vert v \vert ^{q-1}v \psi \,dx }, $$

then \(\mu >0\), \(\nu >0\) and \((\widetilde{\varphi}, \widetilde{\psi}):=(\mu \varphi , \nu \psi )\) is the unique solution of

$$ \textstyle\begin{cases} -\Delta \widetilde{\varphi}+\alpha \widetilde{\varphi}-\frac{2}{q+1} t_{u,v,\lambda}^{\frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{\frac{p-3}{2}}\widetilde{\varphi} \vert v \vert ^{ \frac{q+1}{2}}=\mu t_{u,v,\lambda}^{p-1} \vert u \vert ^{p-1}u, \\ - \Delta \widetilde{\psi}+\beta \widetilde{ \psi}-\frac{2}{p+1} t_{u,v, \lambda}^{\frac{p+1}{2}} s_{u,v,\lambda}^{\frac{q-3}{2}} \lambda \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}\widetilde{\psi } =\nu s_{u,v, \lambda}^{q-1} \vert v \vert ^{q-1}v, \\ \int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u\widetilde{\varphi} \,dx =\int _{ \mathbb{R}^{3}} \vert v \vert ^{q-1}v\widetilde{\psi} \,dx =1, \\ \widetilde{\varphi}(x)\rightarrow 0,\qquad \widetilde{\psi}(x) \rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty . \end{cases} $$
(2.16)

Fixed any \(k\in \mathbb{N}\), we define

$$ A_{1}:=\bigl\{ u\in X_{k+1}: \vert u \vert _{p+1}=1 \bigr\} ,\qquad A_{2}:=\bigl\{ v\in X_{k+1}: \vert v \vert _{q+1}=1\bigr\} . $$

There is an odd homeomorphism from \(S^{k}\) to \(A_{1}\) and \(A_{2}\). By Lemma \(2.2 (1)\), \(A : =A_{1}\times A_{2}\in \Gamma ^{(k+1, k+1)}\). Observe that from (2.1) we deduce that \(A\subset B_{m}\), and so by (2.13),

$$ \sup_{(u,v)\in A} \sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv)< c_{k}. $$

Define

$$ \Gamma _{\lambda}^{(k_{1}, k_{2})}:=\Bigl\{ A\in \Gamma ^{(k_{1}, k_{2})} : A\subset B_{\widetilde{m}}, \sup_{(u,v)\in A} \sup _{t,s\geq 0} \Phi _{\lambda}(tu, sv)< c_{k} \Bigr\} . $$

Observe that \(\Gamma _{\lambda}^{(k_{1}, k_{2})}\neq \emptyset \), \(\Gamma _{\lambda}^{(k_{1}, k_{2})}\subset \Gamma _{\lambda}^{(k_{1}', k_{2}')}\) when \(k_{1}\geq k_{1}'\) and \(k_{2}\geq k_{2}'\). We are now ready to define a sequence of minimax energy levels which will turn out to be critical levels for \(\Phi _{\lambda}\) over \(\mathcal{A}\). For every \(k_{1}, k_{2}\in [2, k+1]\) and \(0<\delta <2^{-\frac{1}{p+1}}\), define

$$ d_{\lambda ,\delta}^{k_{1},k_{2}}:=\inf_{A\in \Gamma _{\lambda}^{(k_{1},k_{2})}} \sup_{A\backslash \mathcal{P}_{\delta}}\sup_{t,s\geq 0} \Phi _{ \lambda}(tu, sv). $$
(2.17)

It is easy to see that

$$ d_{\lambda ,\delta}^{k_{1},k_{2}}< c_{k}\quad \text{for any } 0< \delta < 2^{-\frac{1}{p+1}}, 2\leq k_{1}, k_{2}\leq k+1. $$
(2.18)

As a step towards to the proof of Theorem 1.1, we will prove that \(d_{\lambda ,\delta}^{k_{1},k_{2}}\) is indeed a critical level of \(\Phi _{\lambda}\) for δ sufficiently small. To prove Theorem 1.1, it is necessary to find a pseudogradient for \(\Phi _{\lambda}\) over \(\mathcal{A}\) for which \(\mathcal{P}_{\delta}\) is positively invariant for the associated flow. We can now define the operator

$$ K: B_{\widetilde{m}}^{*}\rightarrow H_{r};\quad (u, v)\mapsto ( \widetilde{\varphi}, \widetilde{\psi}), $$

that is, for any \((u, v)\in B_{\widetilde{m}}^{*}\), \(K(u, v)=(\widetilde{\varphi}, \widetilde{\psi})\) is the unique solution of (2.16). It is easy to prove that \(K(\sigma _{i}(u, v))=\sigma _{i}(K(u, v))\), \(i=1, 2\).

Now, we give some property of the operator K. We can now prove that K is a compact \(C^{1}\) operator.

Lemma 2.5

The operator K is of class \(C^{1}\).

Proof

Define \(C^{1}\) maps \(J_{i}: B_{\widetilde{m}}^{*}\times H_{r}^{1}(\mathbb{R}^{3})\times \mathbb{R}\rightarrow H_{r}^{1}(\mathbb{R}^{3})\times \mathbb{R}\), \(i=1, 2\), by

$$ \begin{aligned} &J_{1} \bigl((u, v), \omega , \gamma \bigr) \\ &= \biggl(\omega -(-\Delta +\alpha )^{-1} \biggl( \frac{2}{q+1} t_{u,v, \lambda}^{\frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{\frac{p-3}{2}}\omega \vert v \vert ^{\frac{q+1}{2}} +\gamma t_{u,v, \lambda}^{p-1} \vert u \vert ^{p-1}u\biggr), \\ &\qquad \int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u \omega \,dx -1 \biggr) \end{aligned} $$

and

$$ \begin{aligned} &J_{2} \bigl((u, v), \omega , \gamma \bigr) \\ &\quad = \biggl(\omega -(-\Delta +\beta )^{-1}\biggl( \frac{2}{p+1} t_{u,v,\lambda}^{ \frac{p+1}{2}} s_{u,v,\lambda}^{\frac{q-3}{2}} \lambda \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}} \omega +\gamma s_{u,v,\lambda}^{q-1} \vert v \vert ^{q-1}v\biggr), \\ &\quad \quad \int _{\mathbb{R}^{3}} \vert v \vert ^{q -1}v \omega \,dx -1 \biggr) \end{aligned} $$

then by (2.16), \(J_{1} ((u, v), \widetilde{\varphi}, \mu )=J_{2} ((u, v), \widetilde{\psi}, \nu )=0\). Moreover, the derivatives of \(J_{1}\) and \(J_{2}\) with respect to \((\omega , \gamma )\) at the point \(((u, v), \widetilde{\varphi}, \mu )\) and \(((u, v), \widetilde{\psi}, \nu )\) in the direction \((\omega _{0}, \gamma _{0})\), respectively, are

$$ \begin{aligned}& D_{\omega ,\gamma}J_{1} \bigl((u, v), \widetilde{\varphi}, \mu \bigr) (\omega _{0}, \gamma _{0}) \\ &\quad = \biggl(\omega _{0}-(-\Delta +\alpha )^{-1} \biggl( \frac{2}{q+1} t_{u,v, \lambda}^{\frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{\frac{p-3}{2}}\omega _{0} \vert v \vert ^{\frac{q+1}{2}} +\gamma _{0} t_{u,v, \lambda}^{p-1} \vert u \vert ^{p-1}u\biggr), \\ &\qquad \int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u \omega _{0} \,dx \biggr) \end{aligned} $$

and

$$ \begin{aligned} &D_{\omega ,\gamma}J_{2} \bigl((u, v), \widetilde{\psi}, \nu \bigr) (\omega _{0}, \gamma _{0}) \\ &\quad = \biggl(\omega _{0}-(-\Delta +\beta )^{-1}\biggl( \frac{2}{p+1} t_{u,v, \lambda}^{\frac{p+1}{2}} s_{u,v,\lambda}^{\frac{q-3}{2}} \lambda \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}} \omega _{0} +\gamma _{0} s_{u,v,\lambda}^{q-1} \vert v \vert ^{q-1}v\biggr), \\ &\qquad \int _{\mathbb{R}^{3}} \vert v \vert ^{q -1}v \omega _{0} \,dx \biggr). \end{aligned} $$

We claim that \(D_{\omega ,\gamma}J_{1} ((u, v), \widetilde{\varphi}, \mu )\) and \(D_{\omega ,\gamma}J_{2} ((u, v), \widetilde{\psi}, \nu )\) are bijective maps. In fact, for any \((\omega , \gamma )\in H_{r}^{1}(\mathbb{R}^{3})\times \mathbb{R}\), the following linear problems

$$\begin{aligned}& -\Delta \omega _{1}+\alpha \omega _{1}- \frac{2}{q+1} t_{u,v,\lambda}^{ \frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{ \frac{p-3}{2}}\omega _{1} \vert v \vert ^{\frac{q+1}{2}}=-\Delta \omega +\alpha \omega ,\\& -\Delta \omega _{2}+\alpha \omega _{2}- \frac{2}{q+1} t_{u,v,\lambda}^{ \frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{ \frac{p-3}{2}}\omega _{2} \vert v \vert ^{\frac{q+1}{2}}=t_{u,v,\lambda}^{p-1} \vert u \vert ^{p-1}u, \end{aligned}$$

have unique solutions \(\omega _{1}, \omega _{2}\in H_{r}^{1}(\mathbb{R}^{3})\), \(\omega _{2}\neq 0\) by \(u\in B_{\widetilde{m}}^{*}\) and (2.12), then we define

$$ \gamma _{0}= \frac{\gamma -\int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u \omega _{1} \,dx }{\int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u \omega _{2} \,dx }, $$

we have

$$ D_{\omega ,\gamma}J_{1} \bigl((u, v), \widetilde{\varphi}, \mu \bigr) ( \omega _{1}+\gamma _{0} \omega _{2}, \gamma _{0})=(\omega , \gamma ), $$

that is, \(D_{\omega ,\gamma}J_{1} ((u, v), \widetilde{\varphi}, \mu )\) is surjective. Similarly, \(D_{\omega ,\gamma}J_{2} ((u, v), \widetilde{\psi}, \nu )\) is surjective.

If \(D_{\omega ,\gamma}J_{1} ((u, v), \widetilde{\varphi}, \mu )(\omega _{0}, \gamma _{0})=(0, 0)\), then

$$ \textstyle\begin{cases} -\Delta \omega _{0}+\alpha \omega _{0} = \frac{2}{q+1} t_{u,v, \lambda}^{\frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{\frac{p-3}{2}}\omega _{0} \vert v \vert ^{\frac{q+1}{2}} +\gamma _{0} t_{u,v, \lambda}^{p-1} \vert u \vert ^{p-1}u, \\ \int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u \omega _{0} \,dx =0, \end{cases} $$

so \(\omega _{0}\equiv 0\), \(\gamma _{0} t_{u,v,\lambda}^{p-1}|u|^{p-1}u\equiv 0\), by \(t_{u,v,\lambda}>0\), \(u\in B_{\widetilde{m}}^{*}\), we have \(\gamma _{0}=0\), this implies \(D_{\omega ,\gamma}J_{1} ((u, v), \widetilde{\varphi}, \mu )\) is injective. Therefore, \(D_{\omega ,\gamma}J_{1} ((u, v), \widetilde{\varphi}, \mu )\) is bijective. Similarly, \(D_{\omega ,\gamma}J_{2} ((u, v), \widetilde{\psi}, \nu )\) is a bijective map. Then we can apply the implicit theorem to the \(C^{1}\) maps \(D_{\omega ,\gamma}J_{1} ((u, v), \widetilde{\varphi}, \mu )\) and \(D_{\omega ,\gamma}J_{2} ((u, v), \widetilde{\psi}, \nu )\), we have the conclusions. □

Lemma 2.6

Let \(\{(u_{n}, v_{n})\}_{n\geq 1}\subset B_{\widetilde{m}}\). For any \(0<\lambda <\lambda _{k}\), there exists \((\widetilde{\varphi}_{0}, \widetilde{\psi}_{0})\in H_{r}\) such that, up to a subsequence,

$$ K(u_{n}, v_{n})\rightarrow (\widetilde{ \varphi}_{0}, \widetilde{\psi}_{0}),\quad \textit{strongly in } H_{r}. $$

Proof

Since \(\{(u_{n}, v_{n})\}_{n\geq 1}\subset B_{\widetilde{m}}\), we have

$$\begin{aligned}& (u_{n}, v_{n})\rightharpoonup (u_{0}, v_{0})\quad \text{weakly in } H_{r},\\& u_{n}\rightarrow u_{0},\quad \text{strongly in } L^{p+1}\bigl( \mathbb{R}^{3}\bigr),\\& v_{n}\rightarrow v_{0},\quad \text{strongly in } L^{q+1}\bigl( \mathbb{R}^{3}\bigr), \end{aligned}$$

and \(|u_{0}|_{p+1}=|v_{0}|_{q+1}=1\). By (2.12), we also have

$$ t_{u_{n},v_{n},\lambda}\rightarrow t_{u_{0},v_{0},\lambda}>0,\qquad s_{u_{n},v_{n}, \lambda} \rightarrow s_{u_{0},v_{0},\lambda}>0. $$

Then by (2.3), (2.7), (2.12), and (2.15),

$$ \begin{aligned}\frac{1}{2} \Vert \varphi _{n} \Vert ^{2}_{\alpha }&\leq \Vert \varphi _{n} \Vert ^{2}_{\alpha}- \frac{2}{q+1} t_{u_{n},v_{n},\lambda}^{ \frac{p-3}{2}} s_{u_{n},v_{n},\lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{\frac{p-3}{2}}\varphi _{n}^{2} \vert v_{n} \vert ^{ \frac{q+1}{2}} \,dx \\ & =t_{u_{n},v_{n},\lambda}^{p-1} \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{p-1}u_{n} \varphi _{n} \,dx \\ &\leq C \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{p} \vert \varphi _{n} \vert \,dx \\ &\leq C \vert u_{n} \vert _{p+1}^{p} \vert \varphi _{n} \vert _{p+1} \leq C \Vert \varphi _{n} \Vert _{\alpha}. \end{aligned} $$

Similar estimates hold for \(\psi _{n}\), we get \(\|\psi _{n}\|^{2}_{\beta}\leq C \| \psi _{n}\|_{\beta}\), so \(\{(\varphi _{n}, \psi _{n})\}_{n\geq 1}\subset H_{r}\) are bounded. Thus

$$\begin{aligned}& (\varphi _{n}, \psi _{n})\rightharpoonup (\varphi _{0}, \psi _{0}) \quad \text{weakly in } H_{r},\\& \varphi _{n}\rightarrow \varphi _{0},\quad \text{strongly in } L^{p+1}\bigl( \mathbb{R}^{3} \bigr),\\& \psi _{n}\rightarrow \psi _{0},\quad \text{strongly in } L^{q+1}\bigl( \mathbb{R}^{3} \bigr). \end{aligned}$$

Then by (2.15) and Hölder’s inequality,

$$ \begin{aligned}& \int _{\mathbb{R}^{3}} \bigl(\nabla \varphi _{n} \nabla ( \varphi _{n}-\varphi _{0})+\alpha \varphi _{n} (\varphi _{n}- \varphi _{0}) \bigr) \,dx \\ &\quad = \frac{2}{q+1} t_{u_{n},v_{n},\lambda}^{\frac{p-3}{2}} s_{u_{n},v_{n}, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{ \frac{p-3}{2}}\varphi _{n}(\varphi _{n}-\varphi _{0}) \vert v_{n} \vert ^{ \frac{q+1}{2}} \,dx \\ &\quad \quad{}+ t_{u_{n},v_{n},\lambda}^{p-1} \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{p-1}u_{n} (\varphi _{n}-\varphi _{0}) \,dx \\ &\quad \rightarrow 0,\quad \text{as } n\rightarrow \infty . \end{aligned} $$

Hence,

$$ \Vert \varphi _{n} \Vert ^{2}_{\alpha }= \int _{\mathbb{R}^{3}} (\nabla \varphi _{n} \nabla \varphi _{0}+\alpha \varphi _{n} \varphi _{0} ) \,dx +o(1)= \Vert \varphi _{0} \Vert ^{2}_{\alpha}+o(1). $$

Similarly, we have \(\|\psi _{n}\|^{2}_{\beta}=\|\psi _{0}\|^{2}_{\beta}+o(1)\). Therefore, we have \((\varphi _{n}, \psi _{n})\rightarrow (\varphi _{0}, \psi _{0})\) strongly in \(H_{r}\) and \((\varphi _{0}, \psi _{0})\) satisfies

$$ \textstyle\begin{cases} -\Delta \varphi _{0}+\alpha \varphi _{0}-\frac{2}{q+1} t_{u_{0},v_{0}, \lambda}^{\frac{p-3}{2}} s_{u_{0},v_{0},\lambda}^{\frac{q+1}{2}} \lambda \vert u_{0} \vert ^{\frac{p-3}{2}}\varphi _{0} \vert v_{0} \vert ^{\frac{q+1}{2}}=t_{u_{0},v_{0}, \lambda}^{p-1} \vert u_{0} \vert ^{p-1}u_{0}, \\ -\Delta \psi _{0}+\beta \psi _{0}-\frac{2}{p+1} t_{u_{0},v_{0}, \lambda}^{\frac{p+1}{2}} s_{u_{0},v_{0},\lambda}^{\frac{q-3}{2}} \lambda \vert u_{0} \vert ^{\frac{p+1}{2}} \vert v_{0} \vert ^{\frac{q-3}{2}}\psi _{0} =s_{u_{0},v_{0}, \lambda}^{q-1} \vert v_{0} \vert ^{q-1}v_{0}, \\ \varphi _{0}(x)\rightarrow 0,\qquad \psi _{0}(x)\rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty , \end{cases} $$

since \(|u_{0}|_{p+1}=|v_{0}|_{q+1}=1\), so \(\varphi _{0}\neq 0\), \(\psi _{0}\neq 0\) and

$$\begin{aligned}& \mu _{n}:= \frac{1}{\int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{p-1}u_{n}\varphi _{n} \,dx } \rightarrow \frac{1}{\int _{\mathbb{R}^{3}} \vert u_{0} \vert ^{p-1}u_{0}\varphi _{0} \,dx }=: \mu _{0},\\& \nu _{n}:= \frac{1}{\int _{\mathbb{R}^{3}} \vert v_{n} \vert ^{q-1}v_{n}\psi _{n} \,dx } \rightarrow \frac{1}{\int _{\mathbb{R}^{3}} \vert v_{0} \vert ^{q-1}v_{0}\psi _{0} \,dx }=: \nu _{0}. \end{aligned}$$

We see that

$$ (\widetilde{\varphi}_{n}, \widetilde{\psi}_{n})=(\mu _{n} \varphi _{n}, \nu _{n}\psi _{n})\rightarrow (\mu _{0} \varphi _{0}, \nu _{0} \psi _{0})=:(\widetilde{\varphi}_{0}, \widetilde{\psi}_{0}),\quad \text{strongly in } H_{r}. $$

This completes the proof. □

Define

$$ B_{\widetilde{m}, \lambda}:=\Bigl\{ (u, v)\in B_{\widetilde{m}}: \sup _{t,s \geq 0} \Phi _{\lambda}(tu, sv)< c_{k}\Bigr\} , $$

then by (2.13) we obtain \(B_{m}\subset B_{\widetilde{m}, \lambda}\).

Lemma 2.7

For any \(0<\delta <2^{-\frac{1}{p+1}}\) sufficiently small, we have that

$$ \operatorname{dist} \bigl(K(u, v), \mathcal{P} \bigr)< \frac{\delta}{2},\quad \forall (u, v)\in B_{\widetilde{m}, \lambda},\qquad \operatorname{dist} \bigl((u, v), \mathcal{P} \bigr)< \delta . $$

Proof

Suppose by contradiction that there exist \(\delta _{n}\rightarrow 0\) and \((u_{n}, v_{n})\in B_{\widetilde{m}, \lambda}\) satisfying \(\operatorname{dist} ((u_{n}, v_{n}), \mathcal{P} )<\delta _{n}\) and \(\operatorname{dist} (K(u_{n}, v_{n}), \mathcal{P} )\geq \frac{\delta _{n}}{2}\). We suppose that \(\operatorname{dist} ((u_{n}, v_{n}), \mathcal{P} )=|u_{n}^{-}|_{p+1}\) without loss of generality. Let \((\widetilde{\varphi}_{n}, \widetilde{\psi}_{n})=K(u_{n}, v_{n})\) and \(\widetilde{\varphi}_{n}=\mu _{n}\varphi _{n}\), \(\widetilde{\psi}_{n}=\nu _{n} \psi _{n}\). By a similar proof as in Lemma 2.6, we have that \(\mu _{n}\) and \(\nu _{n}\) are uniformly bounded. By (2.12), we can take \(\lambda _{k}\) smaller if necessary such that for any \(\lambda \in (0, \lambda _{k})\) and \((u, v)\in B_{\widetilde{m}}^{*}\), we get

$$ \frac{1}{2} \bigl\Vert \widetilde{\varphi}_{n}^{-} \bigr\Vert _{\alpha}^{2}\leq \bigl\Vert \widetilde{ \varphi}_{n}^{-} \bigr\Vert _{\alpha}^{2}- \frac{2}{q+1} t_{u_{n},v_{n}, \lambda}^{\frac{p-3}{2}} s_{u_{n},v_{n},\lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{\frac{p-3}{2}}\bigl({ \widetilde{\varphi}_{n}^{-}} \bigr)^{2} \vert v_{n} \vert ^{\frac{q+1}{2}} \,dx . $$

This together with (2.7) and (2.16) allows us to get

$$\begin{aligned}& \bigl\vert \widetilde{\varphi}_{n}^{-} \bigr\vert ^{2}_{p+1}\leq \frac{1}{S_{p}} \bigl\Vert \widetilde{\varphi}_{n}^{-} \bigr\Vert _{\alpha}^{2} \\& \quad \leq C \biggl( \bigl\Vert \widetilde{\varphi}_{n}^{-} \bigr\Vert _{\alpha}^{2}- \frac{2}{q+1} t_{u_{n},v_{n},\lambda}^{\frac{p-3}{2}} s_{u_{n},v_{n}, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{ \frac{p-3}{2}}\bigl({\widetilde{\varphi}_{n}^{-}} \bigr)^{2} \vert v_{n} \vert ^{ \frac{q+1}{2}} \,dx \biggr) \\& \quad = -C\mu _{n} t_{u_{n},v_{n},\lambda}^{p-1} \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{p-1}u_{n} \widetilde{\varphi}_{n}^{-} \,dx \\& \quad \leq C \int _{\mathbb{R}^{3}}\bigl(u_{n} ^{-} \bigr)^{p} \widetilde{\varphi}_{n}^{-} \,dx \leq C \bigl\vert u_{n} ^{-} \bigr\vert ^{p}_{p+1} \bigl\vert \widetilde{\varphi}_{n}^{-} \bigr\vert _{p+1} \leq C\delta _{n}^{p} \bigl\vert \widetilde{\varphi}_{n}^{-} \bigr\vert _{p+1}, \end{aligned}$$

and hence \(\operatorname{dist} (K(u_{n}, v_{n}), \mathcal{P} )\leq | \widetilde{\varphi}_{n}^{-}|_{p+1}\leq C\delta _{n}^{p} < \frac{\delta _{n}}{2}\) for n sufficiently large, which is a contradiction. This completes the proof. □

Now define a map

$$ V: B_{\widetilde{m}}^{*}\rightarrow H_{r};\qquad (u, v)\mapsto (u, v)-K(u, v). $$

It is easy to prove that \(V(\sigma _{i}(u, v))=\sigma _{i}(V(u, v))\), \(i=1, 2\). We will prove that if \((u, v)\in B_{\widetilde{m}}\backslash \mathcal{P}\), \(V(u, v)=0\), then \((t_{u,v,\lambda}u, s_{u,v,\lambda}v)\) is a sign-changing solution of Eq. (1.1). Firstly, we prove that V satisfies the Palais–Smale type condition and V is a pseudogradient for \(\sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv)\) over \(B_{\widetilde{m}}\). Denote \(\Psi _{\lambda}(u, v):=\sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv)\).

Lemma 2.8

(Palais–Smale type condition) Let \((u_{n}, v_{n})\in B_{\widetilde{m}}\) be such that

$$ \Psi _{\lambda}(u_{n}, v_{n})\rightarrow c< c_{k}\quad \textit{and} \quad V(u_{n}, v_{n})\rightarrow 0\quad \textit{strongly in } H_{r}. $$

Then there exists \((u_{0}, v_{0})\in B_{\widetilde{m}}\) such that \((u_{n}, v_{n})\rightarrow (u_{0}, v_{0})\) strongly in \(H_{r}\), up to a subsequence, and \(V(u_{0}, v_{0})=0\). We also have

$$ \textit{For any } (u, v)\in B_{\widetilde{m}},\quad \bigl\langle \nabla \Psi _{\lambda}(u, v), V(u, v)\bigr\rangle _{H_{r}}\geq \frac{T_{2}^{2}}{2} \bigl\Vert V(u, v) \bigr\Vert _{H_{r}}^{2}. $$

Proof

Similar as Lemma 2.6, we have, up to a subsequence,

$$\begin{aligned}& (u_{n}, v_{n})\rightharpoonup (u_{0}, v_{0})\quad \text{weakly in } H_{r},\\& K(u_{n}, v_{n})\rightarrow (\widetilde{ \varphi}_{0}, \widetilde{\psi}_{0})\quad \text{strongly in } H_{r}. \end{aligned}$$

Then we have, as \(n\rightarrow \infty \),

$$ \begin{aligned}o(1)&=\bigl\langle V(u_{n}, v_{n}), (u_{n}-u_{0}, v_{n}-v_{0}) \bigr\rangle _{H_{r}} \\ &= \langle u_{n}-\widetilde{\varphi}_{n}, u_{n}-u_{0} \rangle _{H_{r}}+ \langle v_{n}-\widetilde{\psi}_{n}, v_{n}-v_{0} \rangle _{H_{r}} \\ &= \langle u_{n}, u_{n}-u_{0} \rangle _{H_{r}}-\langle \widetilde{\varphi}_{n}, u_{n}-u_{0} \rangle _{H_{r}}+\langle v_{n}, v_{n}-v_{0}\rangle _{H_{r}}-\langle \widetilde{\psi}_{n}, v_{n}-v_{0} \rangle _{H_{r}} \end{aligned} $$

whence

$$ \langle u_{n}, u_{n}-u_{0} \rangle _{H_{r}}+\langle v_{n}, v_{n}-v_{0} \rangle _{H_{r}}=o(1). $$

Then \((u_{n}, v_{n})\rightarrow (u_{0}, v_{0})\) strongly in \(H_{r}\) and \((u_{0}, v_{0})\in \overline{B}_{\widetilde{m}}\),

$$ \Phi _{\lambda}(t_{u_{0},v_{0},\lambda}u_{0}, s_{u_{0},v_{0},\lambda}v_{0})= \lim_{n\rightarrow \infty}\Phi _{\lambda}(t_{u_{n},v_{n},\lambda}u_{n}, s_{u_{n},v_{n},\lambda}v_{n})=c< c_{k}, $$

then by (2.13), we have \((u_{0}, v_{0})\in B_{\widetilde{m}}\), \(V(u_{0}, v_{0})=\lim_{n\rightarrow \infty}V(u_{n}, v_{n})=0\).

Finally, we prove that V is a pseudogradient for \(\Psi _{\lambda}(u, v)\) over \(B_{\widetilde{m}}\). By (2.9) and (2.10) we can prove that

$$\begin{aligned}& \begin{aligned}\bigl\langle \nabla \Psi _{\lambda}(u, v), (\omega , 0) \bigr\rangle _{H_{r}}&=t_{u,v,\lambda}^{2} \int _{\mathbb{R}^{3}} ( \nabla u \nabla \omega +\alpha u \omega ) \,dx \\ &\quad{}-\frac{2\lambda}{q+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \int _{\mathbb{R}^{3}} \vert u \vert ^{\frac{p-3}{2}}u \omega \vert v \vert ^{\frac{q+1}{2}} \,dx, \end{aligned} \end{aligned}$$
(2.19)
$$\begin{aligned}& \begin{aligned}\bigl\langle \nabla \Psi _{\lambda}(u, v), (0, \omega ) \bigr\rangle _{H_{r}}&=s_{u,v,\lambda}^{2} \int _{\mathbb{R}^{3}} ( \nabla v \nabla \omega +\beta v \omega ) \,dx \\ &\quad{}-\frac{2\lambda}{p+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \int _{\mathbb{R}^{3}} \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{ \frac{q-3}{2}}v\omega \,dx \end{aligned} \end{aligned}$$
(2.20)

hold for any \((u, v)\in \mathcal{B}_{\widetilde{m}}\) and \(\omega \in H_{r}^{1}(\mathbb{R}^{3})\). We can take \(\lambda _{k}\) smaller if necessary such that for any \(\lambda \in (0, \lambda _{k})\) by (2.19), (2.20), (2.12), and (2.16)

$$ \begin{aligned} &\bigl\langle \nabla \Psi _{\lambda}(u, v), V(u, v) \bigr\rangle _{H_{r}} \\ &\quad = t_{u,v,\lambda}^{2} \int _{\mathbb{R}^{3}} \bigl( \nabla u \nabla (u- \widetilde{\varphi} )+ \alpha u (u-\widetilde{\varphi} ) \bigr) \,dx \\ &\quad \quad{}+s_{u,v,\lambda}^{2} \int _{\mathbb{R}^{3}} \bigl( \nabla v \nabla (v-\widetilde{\psi} )+ \beta v (v-\widetilde{\psi} ) \bigr) \,dx \\ &\quad \quad{}-\frac{2}{q+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{ \frac{p-3}{2}}u(u-\widetilde{ \varphi} ) \vert v \vert ^{\frac{q+1}{2}} \,dx \\ &\quad \quad{}-\frac{2}{p+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}v(v-\widetilde{\psi}) \,dx \\ &\quad =t_{u,v,\lambda}^{2} \Vert u-\widetilde{\varphi} \Vert _{\alpha}^{2}+ s_{u,v, \lambda}^{2} \Vert v-\widetilde{\psi} \Vert _{\beta}^{2} \\ &\quad \quad{}+t_{u,v,\lambda}^{2} \int _{\mathbb{R}^{3}} \bigl( \nabla \widetilde{\varphi} \nabla (u- \widetilde{\varphi} )+\alpha \widetilde{\varphi}(u-\widetilde{\varphi} ) \bigr) \,dx \\ &\quad \quad{}+s_{u,v,\lambda}^{2} \int _{\mathbb{R}^{3}} \bigl( \nabla \widetilde{ \psi} \nabla (u- \widetilde{\psi} )+\alpha \widetilde{\psi}(v-\widetilde{\psi} ) \bigr) \,dx \\ &\quad \quad{}-\frac{2}{q+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{ \frac{p-3}{2}}u(u-\widetilde{ \varphi} ) \vert v \vert ^{\frac{q+1}{2}} \,dx \\ &\quad \quad{}-\frac{2}{p+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}v(v-\widetilde{\psi}) \,dx \\ &\quad =t_{u,v,\lambda}^{2} \Vert u-\widetilde{\varphi} \Vert _{\alpha}^{2}+ s_{u,v, \lambda}^{2} \Vert v-\widetilde{\psi} \Vert _{\beta}^{2} \\ &\quad \quad{}-\frac{2}{q+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{ \frac{p-3}{2}}(u-\widetilde{ \varphi} )^{2} \vert v \vert ^{\frac{q+1}{2}} \,dx \\ &\quad \quad{}-\frac{2}{p+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}(v-\widetilde{\psi})^{2} \,dx \\ &\quad \geq \frac{t_{u,v,\lambda}^{2}}{2} \Vert u-\widetilde{\varphi} \Vert _{ \alpha}^{2}+ \frac{s_{u,v,\lambda}^{2}}{2} \Vert v-\widetilde{ \psi} \Vert _{ \beta}^{2} \\ &\quad \geq \frac{T_{2}^{2}}{2} \bigl( \Vert u-\widetilde{\varphi} \Vert _{\alpha}^{2}+ \Vert v-\widetilde{\psi} \Vert _{\beta}^{2} \bigr)=\frac{T_{2}^{2}}{2} \bigl\Vert V(u, v) \bigr\Vert ^{2}_{H_{r}}. \end{aligned} $$

This completes the proof. □

Lemma 2.9

There exists a unique global solution \(\eta =(\eta _{1}, \eta _{2}): \mathbb{R}^{+}\times B_{\widetilde{m}, \lambda}\rightarrow H_{r}\) for the initial value problem

$$ \textstyle\begin{cases} \frac{d}{dt}\eta (t, (u, v))=-V (\eta (t, (u, v)) ), \\ \eta (0, (u, v))=(u, v)\in B_{\widetilde{m},\lambda}. \end{cases} $$
(2.21)

Moreover,

  1. (1)

    For any \(t>0\) and \((u, v)\in B_{\widetilde{m},\lambda}\), there holds \(\eta (t, (u, v))\in B_{\widetilde{m},\lambda}\);

  2. (2)

    For any \(t>0\), \((u, v)\in B_{\widetilde{m},\lambda}\), there holds \(\eta (t, \sigma _{i}(u, v))=\sigma _{i} ( \eta (t, (u, v)) )\), \(i=1, 2\);

  3. (3)

    For any \((u, v)\in B_{\widetilde{m},\lambda}\), \(\Psi _{\lambda} ( \eta (t, (u, v)) )\) is nonincreasing in t;

  4. (4)

    There exists \(\delta _{0}\in (0, 2^{-\frac{1}{p+1}})\) such that, for any \(0<\delta <\delta _{0}\), \((u, v)\in B_{\widetilde{m},\lambda}\cap \mathcal{P}_{\delta}\) and \(t>0\), there holds \(\eta (t, (u, v))\in \mathcal{P}_{\delta}\).

Proof

It follows from Lemma 2.5 that \(V\in C^{1}(B_{\widetilde{m}}^{*}, H_{r})\). As \(B_{\widetilde{m},\lambda}\subset B_{\widetilde{m}}\subset B_{ \widetilde{m}}^{*}\), we get that \(V\in C^{1}(B_{\widetilde{m},\lambda}, H_{r})\). Then there exists a solution \(\eta : [0, T_{\max})\times B_{\widetilde{m},\lambda}\rightarrow H_{r}\), where \(T_{\max}\) is the maximal time such that (2.21) has a solution \(\eta \in B_{\widetilde{m}}^{*}\).

For any \((u, v)\in B_{\widetilde{m},\lambda}\) and \(t\in (0, T_{\max})\), there holds

$$ \begin{aligned}&\frac{d}{dt} \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(t, (u, v)\bigr) \bigr\vert ^{p+1} \,dx \\ &\quad = -(p+1) \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(t, (u, v)\bigr) \bigr\vert ^{p-1}\eta _{1}\bigl(t, (u, v) \bigr)V_{1} \bigl(\eta \bigl(t, (u, v)\bigr) \bigr) \,dx \\ &\quad = -(p+1) \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(t, (u, v)\bigr) \bigr\vert ^{p-1}\eta _{1}\bigl(t, (u, v)\bigr) \bigl[\eta _{1}\bigl(t, (u, v)\bigr)-K_{1} \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)\bigr] \,dx \\ &\quad = (p+1)-(p+1) \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(t, (u, v)\bigr) \bigr\vert ^{p+1} \,dx , \end{aligned} $$

so we have

$$ \frac{d}{dt}\biggl[e^{(p+1)t} \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(t, (u, v)\bigr) \bigr\vert ^{p+1} \,dx -1 \biggr)\biggr] =0. $$

Then

$$ \begin{aligned}e^{(p+1)t} \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(t, (u, v)\bigr) \bigr\vert ^{p+1} \,dx -1 \biggr) &= \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(0, (u, v)\bigr) \bigr\vert ^{p+1} \,dx -1 \\ &= \int _{\mathbb{R}^{3}} \vert u \vert ^{p+1} \,dx -1\equiv 0. \end{aligned} $$

Similarly, there holds

$$ \begin{aligned}e^{(q+1)t} \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \eta _{2}\bigl(t, (u, v)\bigr) \bigr\vert ^{q+1} \,dx -1 \biggr) &= \int _{\mathbb{R}^{3}} \bigl\vert \eta _{2}\bigl(0, (u, v)\bigr) \bigr\vert ^{q+1} \,dx -1 \\ &= \int _{\mathbb{R}^{3}} \vert v \vert ^{q+1} \,dx -1\equiv 0, \end{aligned} $$

we deduce that for any \((u, v)\in B_{\widetilde{m},\lambda}\) and \(t\in [0, T_{\max})\),

$$ \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(t, (u, v)\bigr) \bigr\vert ^{p+1} \,dx \equiv \int _{ \mathbb{R}^{3}} \bigl\vert \eta _{2}\bigl(t, (u, v) \bigr) \bigr\vert ^{q+1} \,dx \equiv 1. $$

Thus, for any \(t\in [0, T_{\max})\), \((u, v)\in B_{\widetilde{m}}\), we have \(\eta (t, (u, v))\in B_{\widetilde{m}}^{*}\cap \mathcal{A}=B_{ \widetilde{m}}\). If \(T_{\max}<+\infty \), then \(\eta (T_{\max}, (u, v))\in \mathcal{C}_{\widetilde{m}}\). There holds \(\Psi _{\lambda} (\eta (T_{\max}, (u, v)) )\geq c_{k}\) by (2.13). Moreover,

$$ \begin{aligned}\frac{d}{dt}\Psi _{\lambda} \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)&= \biggl\langle \nabla \Psi _{\lambda}\bigl(\eta \bigl(t, (u, v)\bigr) \bigr), \frac{d}{dt} \eta \bigl(t, (u, v)\bigr)\biggr\rangle _{H_{r}} \\ &= -\bigl\langle \nabla \Psi _{\lambda}\bigl(\eta \bigl(t, (u, v)\bigr) \bigr), V \bigl( \eta \bigl(t, (u, v)\bigr) \bigr)\bigr\rangle _{H_{r}} \\ &\leq -\frac{T_{2}^{2}}{2} \bigl\Vert V \bigl(\eta \bigl(t, (u, v)\bigr) \bigr) \bigr\Vert ^{2}_{H_{r}} \leq 0. \end{aligned} $$
(2.22)

On the other hand, we see from \((u, v)\in B_{\widetilde{m},\lambda}\) and (2.22),

$$ \Psi _{\lambda} \bigl(\eta \bigl(T_{\max}, (u, v)\bigr) \bigr) \leq \Psi _{\lambda} \bigl(\eta \bigl(0, (u, v)\bigr) \bigr)=\Psi _{\lambda}(u, v)< c_{k}, $$

it yields a contradiction, so \(T_{\max}=+\infty \), \(\eta (t, (u, v))\in B_{\widetilde{m},\lambda}\) and \((1)(3)\) hold.

Since \(V(\sigma _{i}(u, v))=\sigma _{i}(V(u, v))\), \(i=1, 2\), then \((2)\) holds.

Take \(\delta _{0}>0\) as in Lemma 2.7, note that as \(t\rightarrow 0\),

$$ \begin{aligned}\eta \bigl(t, (u, v)\bigr)&= (u, v)+t \frac{d}{dt}\eta \bigl(t, (u, v)\bigr)|_{t=0}+o(t) \\ &= (u, v)-t V(u, v)+o(t)=(1-t) (u, v)+t K(u, v)+o(t), \end{aligned} $$

hence for any \(0<\delta <\delta _{0}\), \((u, v)\in B_{\widetilde{m},\lambda}\cap \mathcal{P}_{\delta}\), we have

$$ \begin{aligned}\operatorname{dist} \bigl(\eta \bigl(t, (u, v)\bigr), \mathcal{P} \bigr)&= \operatorname{dist} \bigl((1-t) (u, v)+t K(u, v)+o(t), \mathcal{P} \bigr) \\ &\leq (1-t)\operatorname{dist} \bigl((u, v), \mathcal{P} \bigr)+t \operatorname{dist} \bigl(K(u, v), \mathcal{P} \bigr)+o(t) \\ &< (1-t)\delta +\frac{t\delta}{2}+o(t)< \delta , \end{aligned} $$

for sufficiently small \(t>0\), and \((4)\) holds. This completes the proof. □

To prove Theorem 1.1, we will give that \(d_{\lambda ,\delta}^{k_{1},k_{2}}\) is indeed critical energy level for \(\delta >0\) sufficiently small.

Lemma 2.10

For any \(k\in \mathbb{N}\), \(k_{1}, k_{2}\in [2, k+1]\), \(0<\delta <\delta _{0}\), and \(0<\lambda <\lambda _{k}\), there exists \((\widetilde{u}_{0}, \widetilde{v}_{0})\in H_{r}\) such that \((\widetilde{u}_{0}, \widetilde{v}_{0})\) is a sign-changing solution of Eq. (1.1) and \(\Phi _{\lambda}(\widetilde{u}_{0}, \widetilde{v}_{0})=d_{\lambda , \delta}^{k_{1},k_{2}}\).

Proof

By (2.18) we see that \(d_{\lambda ,\delta}^{k_{1},k_{2}}< c_{k}\). Assume that there is small \(0<\varepsilon <1\) such that for any \((u, v)\in B_{\widetilde{m},\lambda}\), \(|\Psi _{\lambda}(u, v)-d_{\lambda ,\delta}^{k_{1},k_{2}}|\leq 2 \varepsilon \), \(\operatorname{dist} ((u, v), \mathcal{P} )\geq \delta \), there holds \(\|V(u, v)\|_{H_{r}}^{2}\geq \varepsilon \). By (2.17), there exists \(A\in \Gamma _{\lambda}^{(k_{1},k_{2})}\) such that

$$ \sup_{A\backslash{\mathcal{P}_{\delta}}} \Psi _{\lambda}(u, v) < d_{ \lambda ,\delta}^{k_{1},k_{2}}+\varepsilon , $$
(2.23)

then \(\sup_{A} \Psi _{\lambda}(u, v)< c_{k}\), \(A\subset B_{\widetilde{m},\lambda}\). Thus we consider the set \(A_{0}=\eta (\frac{4}{T_{2}^{2}}, A)\), \(A_{0}\in B_{\widetilde{m},\lambda}\) by Lemma 2.9(1). From Lemma 2.2(2), Lemma 2.3, and Lemma 2.9(3), we get

$$ \sup_{A_{0}} \Psi _{\lambda}(u, v)\leq \sup _{A} \Psi _{\lambda}(u, v)< c_{k}, $$

so \(A_{0}\in \Gamma _{\lambda}^{(k_{1},k_{2})}\) and \(A_{0}\backslash{\mathcal{P}_{\delta}}\neq \emptyset \). Then, by (2.15), (2.19), and Lemma 2.9(3), for the \(\varepsilon >0\), \(t\in [0, \frac{4}{T_{2}^{2}}]\), there exists \((u, v)\in A\) such that \(\eta (\frac{4}{T_{2}^{2}}, (u, v))\in A_{0}\backslash {\mathcal{P}_{ \delta}}\) satisfying

$$ \begin{aligned}d_{\lambda ,\delta}^{k_{1},k_{2}}&\leq \sup _{A_{0} \backslash {\mathcal{P}_{\delta}}} \Psi _{\lambda}(u, v)< \Psi _{ \lambda} \biggl(\eta \biggl(\frac{4}{T_{2}^{2}}, (u, v)\biggr) \biggr)+\varepsilon \\ &\leq \Psi _{\lambda} \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)+\varepsilon \leq \Psi _{\lambda} (u, v )+\varepsilon < d_{\lambda ,\delta}^{k_{1},k_{2}}+2 \varepsilon . \end{aligned} $$
(2.24)

We conclude that \(\|V (\eta (t, (u, v)) )\|^{2}_{H_{r}}\geq \varepsilon \) for any \(t\in [0, \frac{4}{T_{2}^{2}}]\) and

$$ \begin{aligned}\frac{d}{dt}\Psi _{\lambda} \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)&=- \bigl\langle \nabla \Psi _{\lambda}\bigl( \eta \bigl(t, (u, v)\bigr) \bigr), V \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)\bigr\rangle _{H_{r}} \\ &\leq -\frac{T_{2}^{2}}{2} \bigl\Vert V \bigl(\eta \bigl(t, (u, v)\bigr) \bigr) \bigr\Vert ^{2}_{H_{r}} \leq -\frac{T_{2}^{2}}{2} \varepsilon . \end{aligned} $$

Therefore, by integrating over 0 to \(\frac{4}{T_{2}^{2}}\) and (2.24), we have

$$ \begin{aligned}\bigl(d_{\lambda ,\delta}^{k_{1},k_{2}}-\varepsilon \bigr)-\bigl(d_{ \lambda ,\delta}^{k_{1},k_{2}}+\varepsilon \bigr)&< \Psi _{\lambda} \biggl( \eta \biggl(\frac{4}{T_{2}^{2}}, (u, v)\biggr) \biggr)-\Psi _{\lambda}(u, v) \\ &\leq -\frac{T_{2}^{2}}{2}\varepsilon \int _{0}^{\frac{4}{T_{2}^{2}}} \,dt =-2\varepsilon , \end{aligned} $$

it yields a contradiction, and therefore, for any \(\varepsilon =\frac{1}{n}>0\), there exists \((u_{n}, v_{n})\in B_{\widetilde{m},\lambda}\) such that

$$ \bigl\vert \Psi _{\lambda}(u_{n}, v_{n})-d_{\lambda ,\delta}^{k_{1},k_{2}} \bigr\vert \leq 2\varepsilon ,\qquad \bigl\Vert V(u_{n}, v_{n}) \bigr\Vert ^{2}_{H_{r}}\leq \varepsilon \quad \text{and}\quad \operatorname{dist} \bigl((u_{n}, v_{n}), \mathcal{P} \bigr)\geq \delta . $$

By Lemma 2.8, there exists \((u_{0}, v_{0})\in B_{\widetilde{m},\lambda}\) such that \((u_{n}, v_{n})\rightarrow (u_{0}, v_{0})\) strongly in \(H_{r}\), up to a subsequence. Hence, we have

$$ \Psi _{\lambda}(u_{0}, v_{0})=d_{\lambda ,\delta}^{k_{1},k_{2}}, \qquad V(u_{0}, v_{0})=0\quad \text{and}\quad \operatorname{dist} \bigl((u_{0}, v_{0}), \mathcal{P} \bigr)\geq \delta . $$

We conclude that \((u_{0}, v_{0})\) is sign-changing and \((u_{0}, v_{0})=K(u_{0}, v_{0})=(\widetilde{\varphi}_{0}, \widetilde{\psi}_{0})\). It follows from (2.16) that \((u_{0}, v_{0})\) satisfies

$$ \textstyle\begin{cases} -\Delta u_{0}+\alpha u_{0}=\mu t_{u_{0},v_{0},\lambda}^{p-1} \vert u_{0} \vert ^{p-1}u_{0} +\frac{2}{q+1} t_{u_{0},v_{0},\lambda}^{\frac{p-3}{2}} s_{u_{0},v_{0}, \lambda}^{\frac{q+1}{2}} \lambda \vert u_{0} \vert ^{\frac{p-3}{2}}u_{0} \vert v_{0} \vert ^{ \frac{q+1}{2}}, \\ -\Delta v_{0}+\beta v_{0}=\nu s_{u_{0},v_{0},\lambda}^{q-1} \vert v_{0} \vert ^{q-1}v_{0} +\frac{2}{p+1} t_{u_{0},v_{0},\lambda}^{\frac{p+1}{2}} s_{u_{0},v_{0}, \lambda}^{\frac{q-3}{2}} \lambda \vert u_{0} \vert ^{\frac{p+1}{2}} \vert v_{0} \vert ^{ \frac{q-3}{2}}v_{0}, \\ u_{0}(x)\rightarrow 0,\qquad v_{0}(x)\rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty . \end{cases} $$
(2.25)

On the other hand, \(t_{u_{0},v_{0},\lambda}\) and \(s_{u_{0},v_{0},\lambda}\) satisfy

$$\begin{aligned}& \Vert u_{0} \Vert _{\alpha}^{2}=t_{u_{0},v_{0},\lambda}^{p-1} \vert u_{0} \vert _{p+1}^{p+1}+ \frac{2}{q+1} t_{u_{0},v_{0},\lambda}^{\frac{p-3}{2}} s_{u_{0},v_{0}, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u_{0} \vert ^{ \frac{p+1}{2}} \vert v_{0} \vert ^{\frac{q+1}{2}} \,dx ,\\& \Vert v_{0} \Vert _{\beta}^{2}=s_{u_{0},v_{0},\lambda}^{q-1} \vert v_{0} \vert _{q+1}^{q+1}+ \frac{2}{p+1} t_{u_{0},v_{0},\lambda}^{\frac{p+1}{2}} s_{u_{0},v_{0}, \lambda}^{\frac{q-3}{2}} \lambda \int _{ \mathbb{R}^{3}} \vert u_{0} \vert ^{ \frac{p+1}{2}} \vert v_{0} \vert ^{\frac{q+1}{2}} \,dx , \end{aligned}$$

then we have \(\mu =\nu =1\). Hence, we have that \((t_{u_{0},v_{0},\lambda} u_{0}, s_{u_{0},v_{0},\lambda} v_{0})\) is a sign-changing solution of Eq. (1.1) by problem (2.25) and

$$ \Phi _{\lambda}(\widetilde{u}_{0}, \widetilde{v}_{0}):= \Phi _{ \lambda}(t_{u_{0},v_{0},\lambda} u_{0}, s_{u_{0},v_{0},\lambda} v_{0})= \Psi _{\lambda}(u_{0}, v_{0})=d_{\lambda ,\delta}^{k_{1},k_{2}}. $$

This completes the proof. □

Proof of Theorem 1.1

Observe that from Lemma 2.10 we know that for any \(k\in \mathbb{N}\), \(k_{1}, k_{2}\in [2, k+1]\), \(0<\delta <\delta _{0}\), and \(0<\lambda <\lambda _{k}\), there exists a sign-changing solution \((\widetilde{u}_{0}, \widetilde{v}_{0})\) with \(\Phi _{\lambda}(\widetilde{u}_{0}, \widetilde{v}_{0})=d_{\lambda , \delta}^{k_{1},k_{2}}\). For any fixed \(k_{1}\in [2, k+1]\), we have

$$ d_{\lambda ,\delta}^{k_{1},2}\leq d_{\lambda ,\delta}^{k_{1},3} \leq \cdots \leq d_{\lambda ,\delta}^{k_{1},k}\leq d_{\lambda ,\delta}^{k_{1},k+1}< c_{k}. $$

Suppose that problem (1.1) has at most \(k-1\) sign-changing solutions by contradiction, then there exists \(k_{2}\in [2, k]\) satisfying

$$ d:=d_{\lambda ,\delta}^{k_{1},k_{2}}=d_{\lambda ,\delta}^{k_{1},k_{2}+1}< c_{k}. $$

Now define

$$ \mathcal{M}:=\bigl\{ (u, v)\in B_{\widetilde{m}}: (u, v) \text{ sign-changing}, \Psi _{\lambda}(u, v)=d, V(u, v)=0\bigr\} , $$

then \(\mathcal{M}\subset \mathcal{F}\) is finite. So there exist \(N\in [1, k-1]\) and \(\{(u_{n}, v_{n})\}_{1\leq n\leq N}\subset \mathcal{M}\) such that

$$ \mathcal{M}=\bigl\{ \bigl\{ (u_{n}, v_{n})\bigr\} \cup \bigl\{ (-u_{n}, v_{n})\bigr\} \cup \bigl\{ (u_{n}, -v_{n})\bigr\} \cup \bigl\{ (-u_{n}, -v_{n})\bigr\} \bigr\} _{1\leq n\leq N}. $$

For any \(1\leq n\leq N\), there exist open neighborhoods \(\Omega _{n}^{1}\), \(\Omega _{n}^{2}\), \(\Omega _{n}^{3}\), \(\Omega _{n}^{4}\) of \(\{(u_{n}, v_{n})\}\), \(\{(-u_{n}, v_{n})\}\), \(\{(u_{n}, -v_{n})\}\), \(\{(-u_{n}, -v_{n})\}\), respectively, such that

$$\begin{aligned}& \Omega _{n}^{1}\cap \Omega _{n}^{2} \cap \Omega _{n}^{3}\cap \Omega _{n}^{4}= \emptyset ,\\& \mathcal{M}\subset \bigcup_{n=1}^{3} \bigl( \Omega _{n}^{1}\cup \Omega _{n}^{2} \cup \Omega _{n}^{3}\cup \Omega _{n}^{4} \bigr)=: \Omega . \end{aligned}$$

Define

$$ \mathcal{M}_{\rho}:=\bigl\{ (u, v)\in B_{\widetilde{m}}: \operatorname{dist}_{H_{r}} \bigl((u, v), \mathcal{M} \bigr)< \rho \bigr\} , $$

we can choose \(\rho >0\) small enough such that \(\mathcal{M}_{2\rho}\subset \Omega \). Since \(\mathcal{M}\) is finite, then there is \(\varepsilon _{0}\in (0, \frac{c_{k}-d}{2})\) such that for any \((u, v)\in B_{\widetilde{m}}\backslash (\mathcal{P}_{\delta }\cup \mathcal{M}_{\rho})\), \(|\Psi _{\lambda}(u, v)-d|\leq 2 \varepsilon _{0}\), we have

$$ \bigl\Vert V(u, v) \bigr\Vert _{H_{r}}^{2} \geq \varepsilon _{0}. $$
(2.26)

In fact, if for any \(\varepsilon =\frac{1}{n}>0\) there exists \((u_{n}, v_{n})\in B_{\widetilde{m}}\backslash (\mathcal{P}_{\delta } \cup \mathcal{M}_{\rho})\) satisfying \(|\Psi _{\lambda}(u_{n}, v_{n})-d|\leq 2 \varepsilon \), then there holds \(\|V(u_{n}, v_{n})\|_{H_{r}}^{2}\leq \varepsilon \). Then, by Lemma 2.8, there exists \((u_{0}, v_{0})\in B_{\widetilde{m}}\backslash (\mathcal{P}_{\delta } \cup \mathcal{M}_{\rho})\) such that \((u_{n}, v_{n})\rightarrow (u_{0}, v_{0})\) strongly in \(H_{r}\), up to a subsequence, \(\Psi _{\lambda}(u_{0}, v_{0})=d\) and \(V(u_{0}, v_{0})=0\). Therefore, \((u_{0}, v_{0})\in \mathcal{M}_{\rho}\). It yields a contradiction.

Moreover, for \((u, v)\in \mathcal{M}\), \(V(u, v)=0\), then for \(\rho >0\) small enough, there exists \(T_{0}>0\) such that for any \((u, v)\in \overline{\mathcal{M}}_{2\rho}\),

$$ \bigl\Vert V(u, v) \bigr\Vert _{H_{r}}\leq T_{0}. $$
(2.27)

Let

$$ T:=\frac{1}{2}\min \biggl\{ 1, \frac{\rho T_{2}^{2}}{4T_{0}} \biggr\} . $$
(2.28)

By (2.17), for \(\varepsilon _{0}>0\), there exists \(A\in \Gamma _{\lambda}^{(k_{1},k_{2}+1)}\) such that

$$ \sup_{A\backslash {\mathcal{P}_{\delta}}} \Psi _{\lambda}(u, v)< d_{ \lambda ,\delta}^{k_{1},k_{2}+1} +\frac{T\varepsilon _{0}}{2}=d+ \frac{T\varepsilon _{0}}{2}. $$
(2.29)

Let \(B:=A\backslash {\mathcal{M}_{2\rho}}\), then \(B\subset \mathcal{F}\).

We claim that \(\gamma (B)\geq (k_{1}, k_{2})\). In view of a contradiction, suppose that \(\gamma (B)<(k_{1}, k_{2})\). From Definition 2.1, we know that there exists \(f\in F_{(k_{1},k_{2})}(B)\) such that \(f(u, v)= (f_{1}(u, v), f_{2}(u, v) )\neq (0, 0)\) for any \((u, v)\in B\). Take \(\widetilde{f}=(\widetilde{f}_{1}, \widetilde{f}_{2})\in C (H_{r}, \mathbb{R}^{k_{1}-1}\times \mathbb{R}^{k_{2}-1} )\) such that \(\widetilde{f}|_{B}=f\) by Tietze’s extension theorem. Define

$$\begin{aligned}& F_{1}(u, v):=\widetilde{f}_{1}(u, v)+ \widetilde{f}_{1}\bigl(\sigma _{2}(u, v)\bigr) - \widetilde{f}_{1}\bigl(\sigma _{1}(u, v)\bigr)- \widetilde{f}_{1}(-u, -v),\\& F_{2}(u, v):=\widetilde{f}_{2}(u, v)+ \widetilde{f}_{2}\bigl(\sigma _{1}(u, v)\bigr) - \widetilde{f}_{2}\bigl(\sigma _{2}(u, v)\bigr)- \widetilde{f}_{2}(-u, -v), \end{aligned}$$

then \(F:=(F_{1}, F_{2})\in C (H_{r}, \mathbb{R}^{k_{1}-1}\times \mathbb{R}^{k_{2}-1} )\), \(F|_{B}=4\widetilde{f}\), \(F_{i}(\sigma _{i}(u, v))=-4\widetilde{f}_{i}(u, v)=-F_{i}(u, v)\) and \(F_{i}(\sigma _{j}(u, v))=4\widetilde{f}_{i}(u, v)=F_{i}(u, v)\), \(i\neq j\), \(i, j=1, 2\).

Define the continuous function

$$ g(u, v):= \textstyle\begin{cases} 1,& (u, v)\in \bigcup_{n=1}^{3} ( \overline{\Omega _{n}^{1}}\cup \overline{\Omega _{n}^{2}} ), \\ -1,& (u, v)\in \bigcup_{n=1}^{3} ( \overline{\Omega _{n}^{3}}\cup \overline{\Omega _{n}^{4}} ) \end{cases} $$

and \(g(\sigma _{1}(u, v))=g(u, v)\), \(g(\sigma _{2}(u, v))=-g(u, v)\). Take \(\widetilde{g}\in C(H_{r}, \mathbb{R})\) such that \(\widetilde{g}|_{\Omega}=g\) by Tietze’s extension theorem. Define

$$ G(u, v):=\widetilde{g}(u, v)+\widetilde{g}\bigl(\sigma _{1}(u, v) \bigr) - \widetilde{g}\bigl(\sigma _{2}(u, v)\bigr)-\widetilde{g}(-u, -v), $$

then \(G\in C(H_{r}, \mathbb{R})\), \(G|_{\Omega}=4\widetilde{g}\), \(G(\sigma _{1}(u, v))=G(u, v)\), and \(G(\sigma _{2}(u, v))=-G(u, v)\). Therefore, we can define

$$\begin{aligned}& H_{1}(u, v):=F_{1}(u, v)\in \mathbb{R}^{k_{1}-1},\\& H_{2}(u, v):= \bigl(F_{2}(u, v), G(u, v) \bigr)\in \mathbb{R}^{k_{2}}, \end{aligned}$$

then \(H:=(H_{1}, H_{2})\in C (A, \mathbb{R}^{k_{1}-1}\times \mathbb{R}^{k_{2}} )\) and \(H\in F_{(k_{1},k_{2}+1)}(A)\). Since \(A\in \Gamma _{\lambda}^{(k_{1},k_{2}+1)}\), \(\gamma (A)\geq (k_{1}, k_{2}+1)\), so there exists \((u, v)\in A\) such that \(H(u, v)=(0, 0)\). If \((u, v)\in B=A\backslash \mathcal{M}_{2\rho}\), then

$$ F(u, v)=4\widetilde{f}(u, v)=4f(u, v)\neq (0, 0), $$

a contradiction. Thus \((u, v)\in \mathcal{M}_{2\rho}\), then

$$ G(u, v)=4\widetilde{g}(u, v)=4g(u, v)\neq (0, 0), $$

a contradiction. Therefore, \(\gamma (B)\geq (k_{1}, k_{2})\).

Since \(B\subset A\subset B_{\widetilde{m}}\), \(\sup_{B} \Psi _{\lambda}(u, v)\leq \sup_{A} \Psi _{ \lambda}(u, v)< c_{k}\), then we have \(B\subset B_{\widetilde{m},\lambda}\) and \(B\in \Gamma _{\lambda}^{(k_{1},k_{2})}\). Define \(B_{0}:=\eta (\frac{\rho}{2T_{0}}, B)\), then \(B_{0}\subset B_{\widetilde{m},\lambda}\), \(B_{0}\in \Gamma ^{(k_{1},k_{2})}\), \(B_{0}\backslash P_{\delta}\neq \emptyset \), and \(\sup_{B_{0}} \Psi _{\lambda}(u, v)\leq \sup_{B} \Psi _{\lambda}(u, v)< c_{k}\) by Lemma 2.2(2) and Lemma 2.3, so \(B_{0}\in \Gamma _{\lambda}^{(k_{1},k_{2})}\). Thus \(\sup_{B_{0}\backslash \mathcal{P}_{\delta}} \Psi _{\lambda}(u, v)\geq d_{\lambda ,\delta}^{k_{1},k_{2}}\) by (2.17).

We claim that \(\eta (t, (u, v))\notin \mathcal{M}_{\rho}\) for any \(t\in (0, \frac{\rho}{2T_{0}})\), \((u, v)\in B\). In view of a contradiction, if there exists \(t_{0}\in (0, \frac{\rho}{2T_{0}})\) such that \(\eta (t_{0}, (u, v))\in \mathcal{M}_{\rho}\), for \((u, v)\in B=A\backslash \mathcal{M}_{2\rho}\), by the continuity of η, there exists \(0\leq t_{1}< t_{2}\leq t_{0}\) satisfying \(\eta (t_{1}, (u, v))\in \partial \mathcal{M}_{2\rho}\), \(\eta (t_{2}, (u, v))\in \partial \mathcal{M}_{\rho}\), and \(\eta (t, (u, v))\in \mathcal{M}_{2\rho}\backslash \mathcal{M}_{ \rho} \) for any \(t\in (t_{1}, t_{2})\). Then by (2.27) we have

$$ \rho \leq \bigl\Vert \eta \bigl(t_{1}, (u, v)\bigr)-\eta \bigl(t_{2}, (u, v)\bigr) \bigr\Vert _{H_{r}}= \biggl\Vert \int _{t_{1}}^{t_{2}} V \bigl( \eta \bigl(t, (u, v) \bigr) \bigr) \biggr\Vert _{H_{r}}\leq 2T_{0}(t_{2}-t_{1}), $$

so \(\frac{\rho}{2T_{0}}\leq t_{2}-t_{1}\leq t_{0}-0<\frac{\rho}{2T_{0}}\), this yields a contradiction.

For \(\varepsilon _{0}>0\), there exists \((u, v)\in B\) such that \(\eta (\frac{\rho}{2T_{0}}, (u, v))\in B_{0}\backslash \mathcal{P}_{ \delta}\) satisfies

$$ d_{\lambda ,\delta}^{k_{1},k_{2}}\leq \sup_{B_{0}\backslash { \mathcal{P}_{\delta}}}\Psi _{\lambda}(u, v)< \Psi _{\lambda} \biggl( \eta \biggl( \frac{\rho}{2T_{0}}, (u, v)\biggr) \biggr)+\frac{T\varepsilon _{0}}{2}. $$

Moreover, \(\eta (t, (u, v))\in B_{\widetilde{m},\lambda}\) for any \(t\geq 0\), then by Lemma 2.9(4), \(\eta (t, (u, v))\notin \mathcal{P}_{\delta}\) for any \(t\in [0, \frac{\rho}{2T_{0}}]\). Therefore,

$$ \eta \bigl(t, (u, v)\bigr)\in B_{\widetilde{m}}\backslash ( \mathcal{P}_{ \delta}\cup \mathcal{M}_{\rho } ). $$
(2.30)

In particular, \((u, v)\notin P_{\delta}\). Moreover, by (2.29) and Lemma 2.9\((3)\), we get

$$ \begin{aligned}d_{\lambda ,\delta}^{k_{1},k_{2}}&\leq \sup _{B_{0} \backslash {\mathcal{P}_{\delta}}} \Psi _{\lambda}(u, v)< \Psi _{ \lambda} \biggl(\eta \biggl(\frac{\rho}{2T_{0}}, (u, v)\biggr) \biggr)+ \frac{T\varepsilon _{0}}{2} \\ &\leq \Psi _{\lambda} \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)+ \frac{T\varepsilon _{0}}{2} \\ &\leq \Psi _{\lambda} (u, v )+\frac{T\varepsilon _{0}}{2} < d_{ \lambda ,\delta}^{k_{1},k_{2}+1}+ \frac{T\varepsilon _{0}}{2}+ \frac{T\varepsilon _{0}}{2}, \end{aligned} $$
(2.31)

that is,

$$ \bigl\vert \Psi _{\lambda}(u, v)-d \bigr\vert \leq \frac{T\varepsilon _{0}}{2}< 2 \varepsilon _{0}. $$

So we see from (2.26) and Lemma 2.8 that

$$ \begin{aligned}\frac{d}{dt}\Psi _{\lambda} \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)&=- \bigl\langle \nabla \Psi _{\lambda}\bigl(\eta \bigl(t, (u, v)\bigr) \bigr), V \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)\bigr\rangle _{H_{r}} \\ &\leq -\frac{T_{2}^{2}}{2} \bigl\Vert V \bigl(\eta \bigl(t, (u, v)\bigr) \bigr) \bigr\Vert ^{2}_{H_{r}} \leq -\frac{T_{2}^{2}}{2} \varepsilon _{0}. \end{aligned} $$
(2.32)

Finally, we deduce from (2.28), (2.31), and (2.32) that

$$\begin{aligned} d_{\lambda ,\delta}^{k_{1},k_{2}}& < \Psi _{\lambda} \biggl(\eta \biggl( \frac{\rho}{2T_{0}}, (u, v)\biggr) \biggr)+\frac{T\varepsilon _{0}}{2} \\ & \leq \Psi _{\lambda} (u, v )+\frac{T\varepsilon _{0}}{2}- \int _{0}^{\frac{\rho}{2T_{0}}}\frac{T_{2}^{2}}{2}\varepsilon _{0} \,dt \\ & < d_{\lambda ,\delta}^{k_{1},k_{2}}+\frac{T\varepsilon _{0}}{2}+ \frac{T\varepsilon _{0}}{2}-\frac{T_{2}^{2}}{2}\varepsilon _{0} \frac{\rho}{2T_{0}} \\ & = d_{\lambda ,\delta}^{k_{1},k_{2}}+\frac{\varepsilon _{0}}{2}\biggl(2T- \frac{T_{2}^{2}\rho}{2T_{0}}\biggr)\leq d_{\lambda ,\delta}^{k_{1},k_{2}}, \end{aligned}$$

this yields a contradiction. This completes the proof. □

3 Proof of Theorem 1.2

Using Theorem 1.1, for \(k=1\), there exists \(\lambda _{1}>0\) such that system (1.1) has a radially symmetric sign-changing solution \((u_{1}, v_{1})\) for any \(\lambda \in (0, \lambda _{1})\) and for \(k_{1}=k_{2}=2\),

$$ \Phi _{\lambda}(u_{1}, v_{1})=d_{\lambda ,\delta}^{2,2}< c_{1}. $$

Let

$$ U_{\lambda}:=\bigl\{ (u, v)\in H_{r}: (u, v) \text{ is a sign-changing solution of } (1.1)\bigr\} , $$

then \(U_{\lambda}\neq \emptyset \) by Theorem 1.1, we can define

$$ d_{\lambda}:=\inf_{(u,v)\in U_{\lambda}} \Phi _{\lambda}(u, v) $$

and \(d_{\lambda}< c_{1}\). Let \((u_{n}, v_{n})\in U_{\lambda}\) be a minimizing sequence of \(d_{\lambda}\) with \(\Phi _{\lambda}(u_{n}, v_{n})\rightarrow d_{\lambda}\), \(\Phi _{\lambda}(u_{n}, v_{n})< c_{1}\) and \(\Phi _{\lambda}'(u_{n}, v_{n})=0\). Then

$$ \begin{aligned} & \biggl(\frac{1}{2}- \frac{1}{p+1}\biggr) \bigl( \Vert u_{n} \Vert _{\alpha}^{2}+ \Vert v_{n} \Vert ^{2}_{\beta}\bigr) \\ &\quad \leq \biggl(\frac{1}{2}-\frac{1}{p+1}\biggr) \bigl( \Vert u_{n} \Vert _{\alpha}^{2}+ \Vert v_{n} \Vert ^{2}_{ \beta}\bigr) +\biggl( \frac{1}{p+1}-\frac{1}{q+1}\biggr) \vert v_{n} \vert _{q+1}^{q+1} \\ &\quad \quad{}+\frac{2}{p+1}\biggl(\frac{1}{p+1}-\frac{1}{q+1} \biggr) \lambda \int _{ \mathbb{R}^{3}} \vert u_{n} \vert ^{\frac{p+1}{2}} \vert v_{n} \vert ^{\frac{q+1}{2}} \,dx \\ &\quad =\Phi _{\lambda}(u_{n}, v_{n})- \frac{1}{p+1}\Phi _{\lambda}'(u_{n}, v_{n}) (u_{n}, v_{n})< c_{1}. \end{aligned} $$
(3.1)

Observe that \(\{(u_{n}, v_{n})\}_{n\geq 1}\) is bounded in \(H_{r}\), we may assume that, up to a subsequence,

$$\begin{aligned}& (u_{n}, v_{n})\rightharpoonup (u_{0}, v_{0})\quad \text{weakly in } H_{r},\\& u_{n}\rightarrow u_{0},\quad \text{strongly in } L^{p+1}\bigl( \mathbb{R}^{3}\bigr),\\& v_{n}\rightarrow v_{0},\quad \text{strongly in } L^{q+1}\bigl( \mathbb{R}^{3}\bigr). \end{aligned}$$

Since \(\Phi _{\lambda}'(u_{n}, v_{n})=0\), it is standard to prove that

$$ (u_{n}, v_{n})\rightarrow (u_{0}, v_{0})\quad \text{strongly in } H_{r}, $$

and \(\Phi _{\lambda}'(u_{0}, v_{0})=0\), \(\Phi _{\lambda}(u_{0}, v_{0})=d_{\lambda}\).

Moreover, \(\Phi _{\lambda}'(u_{n}, v_{n})(u_{n}^{\pm}, 0)=0\) and \(\Phi _{\lambda}'(u_{n}, v_{n})(0, v_{n}^{\pm})=0\), we deduce from (2.7) and (3.1) that

$$ \begin{aligned}S_{p} \bigl\vert u_{n}^{\pm} \bigr\vert _{p+1}^{2}&\leq \bigl\Vert u_{n}^{\pm} \bigr\Vert _{\alpha}^{2}= \bigl\vert u_{n}^{ \pm} \bigr\vert _{p+1}^{p+1}+ \frac{2}{q+1} \lambda \int _{ \mathbb{R}^{3}} \bigl\vert u_{n}^{ \pm} \bigr\vert ^{\frac{p+1}{2}} \vert v_{n} \vert ^{\frac{q+1}{2}} \,dx \\ &\leq \bigl\vert u_{n}^{\pm} \bigr\vert _{p+1}^{p+1}+\frac{2}{q+1} \lambda \bigl\vert u_{n}^{\pm} \bigr\vert _{p+1}^{ \frac{p+1}{2}} \vert v_{n} \vert _{q+1}^{\frac{q+1}{2}} \\ &< \bigl\vert u_{n}^{\pm} \bigr\vert _{p+1}^{p+1}+\frac{2}{q+1} \biggl[ \frac{c_{1}}{(\frac{1}{2}-\frac{1}{p+1})S_{q}} \biggr]^{\frac{q+1}{4}} \lambda \bigl\vert u_{n}^{\pm} \bigr\vert _{p+1}^{\frac{p+1}{2}}. \end{aligned} $$

We can choose \(0<\lambda _{0}<\lambda _{1}\) small enough such that for any \(\lambda \in (0, \lambda _{0})\) we have

$$ S_{p} \bigl\vert u_{n}^{\pm} \bigr\vert _{p+1}^{2}< 2 \bigl\vert u_{n}^{\pm} \bigr\vert _{p+1}^{p+1}, $$

which implies \(|u_{n}^{\pm}|_{p+1}\geq \xi _{1}>0\) for any \(n\geq 1\). Similarly, \(|v_{n}^{\pm}|_{q+1}\geq \xi _{2}>0\) for any \(n\geq 1\). Therefore, \(|u_{0}^{\pm}|_{p+1}\geq \xi _{1}>0\), \(|v_{0}^{\pm}|_{q+1}\geq \xi _{2}>0\), and so Eq. (1.1) has a least energy sign-changing solution \((u_{0}, v_{0})\). This completes the proof.  □

4 The proof of Theorem 1.3

In this section, we obtain seminodal solutions \((u, v)\) such that u is positive, v is sign-changing and use the same notations as in Sect. 2 for convenience. Define the \(C^{1}\) functional

$$ \begin{aligned}\Phi _{\lambda}(u, v) &:= \frac{1}{2} \bigl( \Vert u \Vert _{\alpha}^{2}+ \Vert v \Vert _{\beta}^{2}\bigr)-\frac{1}{p+1} \bigl\vert u^{+} \bigr\vert _{p+1}^{p+1}- \frac{1}{q+1} \vert v \vert _{q+1}^{q+1} \\ &\quad{}-\frac{4\lambda}{(p+1)(q+1)} \int _{ \mathbb{R}^{3}} \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q+1}{2}} \,dx , \end{aligned} $$

where \((u, v)\in \widetilde{H}_{r}:=\{(u, v)\in H_{r}: u^{+}\neq 0, v \neq 0\}\),

$$\begin{aligned}& \mathcal{A}:=\bigl\{ (u, v)\in H_{r}: \bigl\vert u^{+} \bigr\vert _{p+1}=1, \vert v \vert _{q+1}=1 \bigr\} ,\\& \mathcal{A}^{*}:=\biggl\{ (u, v)\in H_{r}: \frac{1}{2}< \bigl\vert u^{+} \bigr\vert _{p+1}^{p+1}< 2, \frac{1}{2}< \vert v \vert _{q+1}^{q+1}< 2 \biggr\} ,\\& \mathcal{B}_{m}^{*}:=\bigl\{ (u, v)\in \mathcal{A}^{*} : \Vert u \Vert _{\alpha}^{2}< m, \Vert v \Vert _{\beta}^{2}< m \bigr\} ,\qquad \mathcal{B}_{m}:= \mathcal{B}_{m}^{*} \cap \mathcal{A}. \end{aligned}$$

As in Sect. 2, for any \((u, v)\in \mathcal{A}\), we define

$$ \sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv)= \Phi _{\lambda}(t_{u,v, \lambda}u, s_{u,v,\lambda}v)=:\Psi _{\lambda}(u, v). $$
(4.1)

It is easy to prove that Lemma 2.4 also holds in this section by trivial modifications. Then define

$$ B_{\widetilde{m}, \lambda}:=\Bigl\{ (u, v)\in B_{\widetilde{m}}: \sup _{t,s \geq 0} \Phi _{\lambda}(tu, sv) < c_{k} \Bigr\} . $$

For any \((u, v)\in \mathcal{B}_{\widetilde{m}}^{*}\), \(\lambda \in (0, \lambda _{k})\), we consider the following linear problem:

$$ \textstyle\begin{cases} -\Delta \varphi +\alpha \varphi -\frac{2}{q+1} t_{u,v,\lambda}^{ \frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{ \frac{p-3}{2}}\varphi \vert v \vert ^{\frac{q+1}{2}}=t_{u,v,\lambda}^{p-1} (u^{+})^{p}, \\ -\Delta \psi +\beta \psi -\frac{2}{p+1} t_{u,v,\lambda}^{ \frac{p+1}{2}} s_{u,v,\lambda}^{\frac{q-3}{2}} \lambda \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}\psi =s_{u,v,\lambda}^{q-1} \vert v \vert ^{q-1}v, \\ \varphi (x)\rightarrow 0,\qquad \psi (x)\rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty , \end{cases} $$
(4.2)

then (4.2) has a unique solution \((\varphi , \psi )\in H_{r}\backslash \{(0, 0)\}\). Define

$$ \mu :=\frac{1}{\int _{\mathbb{R}^{3}}(u^{+})^{p} \varphi \,dx }>0, \qquad \nu :=\frac{1}{\int _{\mathbb{R}^{3}} \vert v \vert ^{q-1}v \psi \,dx }>0. $$

Then \((\widetilde{\varphi}, \widetilde{\psi}):=(\mu \varphi , \nu \psi )\) is the unique solution of the following problem:

$$ \textstyle\begin{cases} -\Delta \widetilde{\varphi}+\alpha \widetilde{\varphi}-\frac{2}{q+1} t_{u,v,\lambda}^{\frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{\frac{p-3}{2}}\widetilde{\varphi} \vert v \vert ^{ \frac{q+1}{2}}=\mu t_{u,v,\lambda}^{p-1} (u^{+})^{p}, \\ -\Delta \widetilde{\psi}+\beta \widetilde{ \psi}-\frac{2}{p+1} t_{u,v, \lambda}^{\frac{p+1}{2}} s_{u,v,\lambda}^{\frac{q-3}{2}} \lambda \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}\widetilde{\psi } =\nu s_{u,v, \lambda}^{q-1} \vert v \vert ^{q-1}v, \\ \int _{\mathbb{R}^{3}}(u^{+})^{p} \widetilde{\varphi} \,dx =\int _{ \mathbb{R}^{3}} \vert v \vert ^{q-1}v\widetilde{\psi} \,dx =1, \\ \widetilde{\varphi}(x)\rightarrow 0,\qquad \widetilde{\psi}(x) \rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty . \end{cases} $$
(4.3)

We can now also define the operator

$$\begin{aligned}& K: B_{\widetilde{m}}^{*}\rightarrow H_{r};\qquad (u, v)\mapsto ( \widetilde{\varphi}, \widetilde{\psi}), \\& K\bigl(\sigma _{2}(u, v)\bigr)=\sigma _{2}\bigl(K(u, v)\bigr). \end{aligned}$$
(4.4)

Then, by similar proofs as in Lemma 2.5 and Lemma 2.6, we have that \(K\in C^{1}(B_{\widetilde{m}}^{*}, H_{r})\) and K satisfies the Palais–Smale type condition. Define the map

$$ V: B_{\widetilde{m}}^{*}\rightarrow H_{r};\qquad (u, v)\mapsto (u, v)-K(u, v). $$

Consider the class of sets

$$ \mathcal{F}=\bigl\{ A\in \mathcal{A}: A \text{ is a closed set and } \sigma _{2}(u, v)\in A, \forall (u, v)\in A\bigr\} $$
(4.5)

for each \(A\in \mathcal{F}\) and \(k_{2}\geq 2\), the class of functions

$$ F_{(1, k_{2})}(A)=\bigl\{ f: A\rightarrow \mathbb{R}^{k_{2}-1}: f \text{ continuous and } f\bigl(\sigma _{2}(u, v)\bigr)=-f(u, v) \bigr\} . $$
(4.6)

To obtain seminodal solutions, we should also define a cone of positive functions, that is,

$$\begin{aligned}& \mathcal{P}_{2}:=\bigl\{ (u, v)\in H_{r}: v\geq 0\bigr\} ,\qquad \mathcal{P}= \mathcal{P}_{2}\cup -\mathcal{P}_{2}, \\& \operatorname{dist}_{q+1} \bigl((u, v), \mathcal{P} \bigr):=\min \bigl\{ \operatorname{dist}_{q+1} (v, \mathcal{P}_{2}) , \operatorname{dist}_{q+1} (v, -\mathcal{P}_{2}) \bigr\} , \end{aligned}$$
(4.7)

thus, v is sign-changing if \(\operatorname{dist}_{q+1} ((u, v), \mathcal{P})>0\).

Under the new definitions (4.4)–(4.6), we define vector genus, slightly different from Definition 2.1.

Definition 4.1

Let \(A\in \mathcal{F}\) and take any \(k_{2}\in \mathbb{N}\) with \(k_{2}\geq 2\). We say that \(\gamma (A)\geq (1, k_{2})\) if for every \(f\in F_{(1, k_{2})}(A)\) there exists \((u, v)\in A\) such that \(f(u, v)=0\). We denote

$$ \Gamma ^{(1, k_{2})}:=\bigl\{ A\in \mathcal{F}: \gamma (A)\geq (1, k_{2}) \bigr\} . $$

Lemma 4.1

  1. (1)

    Take \(A:=A_{1}\times A_{2}\subset \mathcal{A}\) and let \(\eta : S^{k_{2}-1}\rightarrow A_{2}\) be a homeomorphism such that \(\eta (-x)=-\eta (x)\) for every \(x\in S^{k_{2}-1}\). Then \(A\in \Gamma ^{(1, k_{2})}\);

  2. (2)

    We have \(\overline{\eta (A)}\in \Gamma ^{(1, k_{2})}\) whenever \(A \in \Gamma ^{(1, k_{2})}\) and a continuous map \(\eta : A\rightarrow \mathcal{A}\) is such that \(\eta \circ \sigma _{2}=\sigma _{2}\circ \eta \).

Proof

\((1)\) For every \(f\in F_{(1, k_{2})}(A)\) and \(u\in A_{1}\), we define a map

$$ h: S^{k_{2}-1}\rightarrow \mathbb{R}^{k_{2}-1};\qquad h(x):=f \bigl(u, \eta (x)\bigr), $$

then by (4.6) it is easy to see that h is continuous and

$$ h(-x)=f\bigl(u, \eta (-x)\bigr)=f\bigl(u, -\eta (x)\bigr)=-f\bigl(u, \eta (x) \bigr)=-h(x). $$

Then Borsuk–Ulam theorem yields \(x_{0}\in S^{k_{2}-1}\) such that \(h(x_{0})=f(u, \eta (x_{0}))=0\). By Definition 4.1, we have \(A\in \Gamma ^{(1, k_{2})}\).

\((2)\) Fix any \(f\in F_{(1, k_{2})}(\overline{\eta (A)})\), then by (4.6) we have \(f\circ \eta \in F_{(1, k_{2})}(A)\). Since \(A\in \Gamma ^{(1, k_{2})}\), there exists \((u_{0}, v_{0})\in A\) such that \(f\circ \eta (u_{0}, v_{0})=0\). Then by \(\eta (u_{0}, v_{0})\in \overline{\eta (A)}\) we have \(\gamma (\overline{\eta (A)})\geq (1, k_{2})\), that is, \(\overline{\eta (A)}\in \Gamma ^{(1, k_{2})}\). This completes the proof. □

Lemma 4.2

Assume \(k_{2}\geq 2\). Then, for any \(0<\delta <2^{-\frac{1}{q+1}}\) and \(A\in \Gamma ^{(1, k_{2})}\), we have \(A\backslash \mathcal{P}_{\delta }\neq \emptyset \).

Proof

For any \(A\in \Gamma ^{(1, k_{2})}\), define f by

$$ f(u, v)= \biggl( \int _{\mathbb{R}^{3}} \vert v \vert ^{q}v \,dx , 0, \ldots , 0\biggr), $$

then \(f\in F_{(1, k_{2})}(A)\), so by Definition 4.1, there exists \((u_{0}, v_{0})\in A\) such that \(f(u_{0}, v_{0})=0\). We deduce from \(A\in \mathcal{A}\) that

$$ \int _{\mathbb{R}^{3}}\bigl(v_{0}^{+} \bigr)^{q+1} \,dx = \int _{\mathbb{R}^{3}}\bigl(v_{0}^{-} \bigr)^{q+1} \,dx =\frac{1}{2}. $$

Therefore, \(\operatorname{dist}_{q+1} ((u_{0}, v_{0}), \mathcal{P})=2^{-\frac{1}{q+1}}\), and so \((u_{0}, v_{0})\in A\backslash \mathcal{P}_{\delta}\) for any \(0<\delta <2^{-\frac{1}{q+1}}\). This completes the proof. □

Fixed any \(k\in \mathbb{N}\), we define

$$ A_{1}:=\biggl\{ cu_{0}: c=\frac{1}{ \vert u_{0} \vert _{p+1}}, u_{0}>0\biggr\} ,\qquad A_{2}:= \bigl\{ v\in X_{k+1}: \vert v \vert _{q+1}=1\bigr\} . $$

By Lemma 4.1(1), \(A:=A_{1}\times A_{2}\in \Gamma ^{(1, k+1)}\), \(A\subset B_{\widetilde{m}}\), and \(\sup_{A} \Psi _{\lambda}(u, v)< c_{k}\). Then we can define

$$ \Gamma _{\lambda}^{(1, k_{2})}:=\Bigl\{ A\in \Gamma ^{(1, k_{2})}: A \subset B_{\widetilde{m}}, \sup_{A} \Psi _{\lambda}(u, v)< c_{k} \Bigr\} . $$

For any \(k_{2}\in [2, k+1]\) and \(0<\delta <2^{-\frac{1}{q+1}}\), we define a sequence of minimax energy level:

$$ d_{\lambda ,\delta}^{1,k_{2}}:=\inf_{A\in \Gamma _{\lambda}^{(1,k_{2})}} \sup _{A\backslash \mathcal{P}_{\delta}}\sup_{t,s\geq 0} \Phi _{ \lambda}(tu, sv). $$

It is easy to see that

$$ d_{\lambda ,\delta}^{1,k_{2}}< c_{k}\quad \text{for any } 0< \delta < 2^{-\frac{1}{q+1}} \text{ and } 2\leq k_{2}\leq k+1. $$

Lemma 2.7 and Lemma 2.8 also hold in Sect. 4.

Lemma 4.3

There exists a unique global solution \(\eta : \mathbb{R}^{+}\times B_{\widetilde{m},\lambda}\rightarrow H_{r}\) for the initial value problem

$$ \textstyle\begin{cases} \frac{d}{dt}\eta (t, (u, v))=-V (\eta (t, (u, v)) ), \\ \eta (0, (u, v))=(u, v)\in B_{\widetilde{m},\lambda}. \end{cases} $$
(4.8)

Moreover, (1), (3), (4) of Lemma 2.9hold and

  1. (2)

    For any \(t>0\), \((u, v)\in B_{\widetilde{m},\lambda}\), \(\eta (t, \sigma _{2}(u, v))=\sigma _{2} ( \eta (t, (u, v)) )\).

Proof

From the above discussion, we see that \(V\in C^{1}(B_{\widetilde{m}}^{*}, H_{r})\). As \(B_{\widetilde{m},\lambda}\subset B_{\widetilde{m}}\subset B_{ \widetilde{m}}^{*}\), we get that \(V\in C^{1}(B_{\widetilde{m},\lambda}, H_{r})\), then there exists a solution \(\eta : [0, T_{\max})\times B_{\widetilde{m},\lambda}\rightarrow H_{r}\), where \(T_{\max}\) is the maximal time such that (4.8) has s solution \(\eta \in B_{\widetilde{m}}^{*}\).

For any \((u, v)\in B_{\widetilde{m},\lambda}\) and \(t\in (0, T_{\max})\), there holds

$$ \begin{aligned}& \frac{d}{dt} \int _{\mathbb{R}^{3}}\bigl(\eta _{1}^{+} \bigl(t, (u, v)\bigr)\bigr)^{p+1} \,dx \\ &\quad = -(p+1) \int _{\mathbb{R}^{3}}\bigl(\eta _{1}^{+} \bigl(t, (u, v)\bigr)\bigr)^{p}V \bigl( \eta _{1}^{+} \bigl(t, (u, v)\bigr) \bigr) \,dx \\ &\quad = -(p+1) \int _{\mathbb{R}^{3}}\bigl(\eta _{1}^{+} \bigl(t, (u, v)\bigr)\bigr)^{p}\bigl[\eta _{1}^{+} \bigl(t, (u, v)\bigr)-K_{1} \bigl(\eta ^{+}\bigl(t, (u, v)\bigr) \bigr)\bigr] \,dx \\ &\quad = (p+1)-(p+1) \int _{\mathbb{R}^{3}}\bigl(\eta _{1}^{+} \bigl(t, (u, v)\bigr)\bigr)^{p+1} \,dx , \end{aligned} $$

so we have

$$ \frac{d}{dt}\biggl[e^{(p+1)t} \biggl( \int _{\mathbb{R}^{3}}\bigl(\eta _{1}^{+} \bigl(t, (u, v)\bigr)\bigr)^{p+1} \,dx -1 \biggr)\biggr] =0. $$

Since \(\int _{\mathbb{R}^{3}}(\eta _{1}^{+}(0, (u, v)))^{p+1} \,dx =\int _{ \mathbb{R}^{3}}(u^{+})^{p+1} \,dx =1\), then for any \(t\in [0, T_{\max})\),

$$ \int _{\mathbb{R}^{3}}\bigl(\eta _{1}^{+} \bigl(t, (u, v)\bigr)\bigr)^{p+1} \,dx \equiv 1. $$

The rest of the proof is the same as Lemma 2.9. This completes the proof. □

Proof of Theorem 1.2

Observe that from Lemma 2.10, for any \(k_{2}\in [2, k+1]\), \(0<\delta <\delta _{0}\) small, there exists \((u_{0}, v_{0})\in B_{\widetilde{m}}\) such that

$$ \Psi _{\lambda}(u_{0}, v_{0})=d_{\lambda ,\delta}^{1,k_{2}}, \qquad V(u_{0}, v_{0})=0\quad \text{and}\quad \operatorname{dist}_{q+1} \bigl((u_{0}, v_{0}), \mathcal{P} \bigr)\geq \delta . $$

We conclude that \(v_{0}\) is sign-changing and \((u_{0}, v_{0})=K(u_{0}, v_{0})=(\widetilde{\varphi}_{0}, \widetilde{\psi}_{0})\). It follows from (4.3) that \((u_{0}, v_{0})\) satisfies

$$ \textstyle\begin{cases} -\Delta u_{0}+\alpha u_{0}=\mu t_{u,v,\lambda}^{p-1} (u_{0}^{+})^{p} +\frac{2}{q+1} t_{u_{0},v_{0},\lambda}^{\frac{p-3}{2}} s_{u_{0},v_{0}, \lambda}^{\frac{q+1}{2}} \lambda \vert u_{0} \vert ^{\frac{p-3}{2}}u_{0} \vert v_{0} \vert ^{ \frac{q+1}{2}}, \\ -\Delta v_{0}+\beta v_{0}=\nu s_{u_{0},v_{0},\lambda}^{q-1} \vert v_{0} \vert ^{q-1}v_{0} +\frac{2}{p+1} t_{u_{0},v_{0},\lambda}^{\frac{p+1}{2}} s_{u_{0},v_{0}, \lambda}^{\frac{q-3}{2}} \lambda \vert u_{0} \vert ^{\frac{p+1}{2}} \vert v_{0} \vert ^{ \frac{q-3}{2}}v_{0}, \\ u_{0}(x)\rightarrow 0,\qquad v_{0}(x)\rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty , \end{cases} $$
(4.9)

and \(|u_{0}^{+}|_{p+1}=|v_{0}|_{q+1}=1\), then by (4.1) we have \(\mu =\nu =1\). Moreover, (4.9) yields

$$ \bigl\Vert u_{0}^{-} \bigr\Vert _{\alpha}^{2}=\frac{2}{q+1} t_{u_{0},v_{0},\lambda}^{ \frac{p-3}{2}} s_{u_{0},v_{0},\lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u_{0} \vert ^{\frac{p-3}{2}}\bigl(u^{-}_{0}\bigr)^{2} \vert v_{0} \vert ^{ \frac{q+1}{2}}. $$

We can take \(\lambda _{k}\) small enough if necessary such that for any \(\lambda \in (0, \lambda _{k})\) and \((u_{0}, v_{0})\in \mathcal{B}_{\widetilde{m}}^{*}\),

$$ \bigl\Vert u_{0}^{-} \bigr\Vert _{\alpha}^{2}-\frac{2}{q+1} t_{u_{0},v_{0},\lambda}^{ \frac{p-3}{2}} s_{u_{0},v_{0},\lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u_{0} \vert ^{\frac{p-3}{2}}\bigl(u^{-}_{0}\bigr)^{2} \vert v_{0} \vert ^{ \frac{q+1}{2}} \geq \frac{1}{2} \bigl\Vert u_{0}^{-} \bigr\Vert _{\alpha}^{2}, $$

then \(\|u_{0}^{-}\|_{\alpha}^{2}=0\), so \(u_{0}\geq 0\). By the strong maximum principle, \(u_{0}>0\). Hence we have that \((t_{u_{0},v_{0},\lambda}u_{0}, s_{u_{0},v_{0},\lambda}v_{0})\) is a seminodal solution of (1.1) with \(t_{u_{0},v_{0},\lambda}u_{0}\) positive and \(s_{u_{0},v_{0},\lambda}v_{0}\) sign-changing,

$$ \Phi _{\lambda}(t_{u_{0},v_{0},\lambda}u_{0}, s_{u_{0},v_{0},\lambda}v_{0}) =\Psi _{\lambda}(u_{0}, v_{0})=d_{\lambda ,\delta}^{1,k_{2}}. $$

By similar proof as Theorem 1.1, we complete the proof. □

Data Availability

No datasets were generated or analysed during the current study.

References

  1. Akhmediev, N., Ankiewicz, A.: Novel soliton states and bifurcation phenomena in nonlinear fiber couplers. Phys. Rev. Lett. 70, 2395–2398 (1993)

    Article  MathSciNet  Google Scholar 

  2. Ambrosetti, A., Colorado, E.: Standing waves of some coupled nonlinear Schrödinger equations. J. Lond. Math. Soc. 75, 67–82 (2007)

    Article  MathSciNet  Google Scholar 

  3. Atkinson, F.V., Brézis, H., Peletier, L.A.: Nodal solutions of elliptic equations with critical Sobolev exponents. J. Differ. Equ. 85, 151–170 (1990)

    Article  MathSciNet  Google Scholar 

  4. Bartsch, T., Dancer, N., Wang, Z.Q.: A Liouville theorem, a priori bounds, and bifurcating branches of positive solutions for a nonlinear elliptic system. Calc. Var. Partial Differ. Equ. 37, 345–361 (2010)

    Article  MathSciNet  Google Scholar 

  5. Bartsch, T., Liu, Z., Weth, T.: Sign changing solutions of superlinear Schrödinger equations. Commun. Partial Differ. Equ. 29, 25–42 (2004)

    Article  Google Scholar 

  6. Brézis, H., Nirenberg, L.: Positive solutions of nonlinear elliptic equations involving critical Sobolev exponents. Commun. Pure Appl. Math. 36, 437–477 (1983)

    Article  MathSciNet  Google Scholar 

  7. Cerami, G., Solimini, S., Struwe, M.: Some existence results for superlinear elliptic boundary value problems involving critical exponents. J. Funct. Anal. 69, 289–306 (1986)

    Article  MathSciNet  Google Scholar 

  8. Chang, K.C., Wang, Z.Q., Zhang, T.: On a new index theory and non semi-trivial solutions for elliptic systems. Discrete Contin. Dyn. Syst. 28, 809–826 (2010)

    Article  MathSciNet  Google Scholar 

  9. Chen, Z., Lin, C., Zou, W.: Multiple sign-changing and semi-nodal solutions for coupled Schrödinger equations. J. Differ. Equ. 255, 4289–4311 (2013)

    Article  Google Scholar 

  10. Chen, Z., Lin, C., Zou, W.: Infinitely many sign-changing and seminodal solutions for a nonlinear Schrödinger system. Ann. Sc. Norm. Super. Pisa, Cl. Sci. 15, 859–897 (2016)

    MathSciNet  Google Scholar 

  11. Chen, Z., Zou, W.: Positive least energy solutions and phase separation for coupled Schrödinger equations with critical exponent. Arch. Ration. Mech. Anal. 205, 515–551 (2012)

    Article  MathSciNet  Google Scholar 

  12. Chen, Z., Zou, W.: An optimal constant for the existence of least energy solutions of a coupled Schrödinger system. Calc. Var. Partial Differ. Equ. (2012). https://doi.org/10.1007/s00526-012-0568-2

    Article  Google Scholar 

  13. Conti, M., Merizzi, L., Terracini, S.: Remarks on variational methods and lower-upper solutions. Nonlinear Differ. Equ. Appl. 6, 371–393 (1999)

    Article  MathSciNet  Google Scholar 

  14. Dancer, N., Wei, J., Weth, T.: A priori bounds versus multiple existence of positive solutions for a nonlinear Schrödinger systems. Ann. Inst. Henri Poincaré, Anal. Non Linéaire 27, 953–969 (2010)

    Article  MathSciNet  Google Scholar 

  15. Frantzeskakis, D.J.: Dark solitons in atomic Bose-Einstein condensates: from theory to experiments. J. Phys. A 43, 213001 (2010)

    Article  MathSciNet  Google Scholar 

  16. Kim, S.: On vector solutions for coupled nonlinear Schrödinger equations with critical exponents. Commun. Pure Appl. Anal. 12, 1259–1277 (2013)

    Article  MathSciNet  Google Scholar 

  17. Kivshar, Y.S., Luther-Davies, B.: Dark optical solitons: physics and applications. Phys. Rep. 298, 81–197 (1998)

    Article  Google Scholar 

  18. Lin, T., Wei, J.: Ground state of N coupled nonlinear Schrodinger equations in \(\mathbb{R}^{3}\), \(n\leq 3\). Commun. Math. Phys. 255, 629–653 (2005)

    Article  Google Scholar 

  19. Liu, J., Liu, X., Wang, Z.Q.: Multiple mixed states of nodal solutions for nonlinear Schrödinger systems. Calc. Var. Partial Differ. Equ. 52, 565–586 (2015)

    Article  Google Scholar 

  20. Liu, Z., Wang, Z.-Q.: Ground states and bound states of a nonlinear Schrödinger system. Adv. Nonlinear Stud. 10, 175–193 (2010)

    Article  MathSciNet  Google Scholar 

  21. Liu, Z., Wang, Z.Q.: Multiple bound states of nonlinear Schrödinger systems. Commun. Math. Phys. 282, 721–731 (2008)

    Article  Google Scholar 

  22. Maia, L., Montefusco, E., Pellacci, B.: Positive solutions for a weakly coupled nonlinear Schrödinger systems. J. Differ. Equ. 229, 743–767 (2006)

    Article  Google Scholar 

  23. Maia, L., Montefusco, E., Pellacci, B.: Infinitely many nodal solutions for a weakly coupled nonlinear Schrödinger system. Commun. Contemp. Math. 10, 651–669 (2008)

    Article  MathSciNet  Google Scholar 

  24. Noris, B., Ramos, M.: Existence and bounds of positive solutions for a nonlinear Schrödinger system. Proc. Am. Math. Soc. 138, 1681–1692 (2010)

    Article  Google Scholar 

  25. Quittner, P., Souplet, P.: Optimal Liouville-type theorems for noncooperative elliptic Schrödinger systems and applications. Commun. Math. Phys. 311, 1–19 (2012)

    Article  Google Scholar 

  26. Sato, Y., Wang, Z.: On the multiple existence of semi-positive solutions for a nonlinear Schrödinger system. Ann. Inst. Henri Poincaré, Anal. Non Linéaire 30, 1–22 (2013)

    Article  MathSciNet  Google Scholar 

  27. Sirakov, B.: Least energy solitary waves for a system of nonlinear Schrödinger equations in \(\mathbb{R}^{3}\). Commun. Math. Phys. 271, 199–221 (2007)

    Article  Google Scholar 

  28. Tavares, H., Terracini, S.: Sign-changing solutions of competition diffusion elliptic systems and optimal partition problems. Ann. Inst. Henri Poincaré, Anal. Non Linéaire 29, 279–300 (2012)

    Article  MathSciNet  Google Scholar 

  29. Wei, J., Weth, T.: Radial solutions and phase separation in a system of two coupled Schrödinger equations. Arch. Ration. Mech. Anal. 190, 83–106 (2008)

    Article  MathSciNet  Google Scholar 

  30. Zou, W.: Sign-Changing Critical Points Theory. Springer, New York (2008)

    Google Scholar 

Download references

Acknowledgements

The author is thankful to the reviewers for their careful reading and suggestions.

Funding

This work was supported by the NSFC (Grant No. 12101162).

Author information

Authors and Affiliations

Authors

Contributions

Jing Zhang wrote the main manuscript text.

Corresponding author

Correspondence to Jing Zhang.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J. Sign-changing solutions for coupled Schrödinger system. Bound Value Probl 2024, 69 (2024). https://doi.org/10.1186/s13661-024-01881-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13661-024-01881-z