Least energy sign-changing solutions for the fractional Schrödinger–Poisson systems in $\mathbb{R}^{3}$

Abstract

In this paper, we study the following nonlinear fractional Schrödinger–Poisson system

$$ \textstyle\begin{cases} (-\Delta )^{s}u+V(x)u+\phi u=K(x)f(u),&x\in \mathbb{R}^{3}, \\ (-\Delta )^{t} \phi =u^{2},&x\in {\mathbb{R}}^{3}. \end{cases} $$

(0.1)

where $s\in (\frac{3}{4},1)$, $t\in (0,1)$, $V,K : \mathbb{R}^{3}\rightarrow \mathbb{R}$ are continuous functions verifying some conditions about zero mass. By using the constraint variational method and the quantitative deformation lemma, we obtain the existence of least energy sign-changing solution to this system.

1 Introduction and main results

In this article, we are interested in the existence of the sign-changing solutions for the following fractional Schrödinger–Poisson system

$$ \textstyle\begin{cases} (-\Delta )^{s}u+V(x)u+\phi u=K(x)f(u),&x\in \mathbb{R}^{3}, \\ (-\Delta )^{t} \phi =u^{2},&x\in {\mathbb{R}}^{3}, \end{cases} $$

(1.1)

where $s\in (\frac{3}{4},1)$, $t\in (0,1)$. The fractional Laplacian operator $(-\Delta )^{s}$ is defined by

$$\begin{aligned} (-\Delta )^{s}u(x)=C_{N,s}P.V. \int _{\mathbb{R}^{N}} \frac{u(x)-u(y)}{ \vert x-y \vert ^{N+2s}}\,\mathrm{d}y,\quad u\in \mathbb{S} \bigl(\mathbb{R} ^{N}\bigr), \end{aligned}$$

where $P.V$. stands for the Cauchy principal value, $C_{N,s}$ is a normalized constant, $\mathbb{S}(\mathbb{R}^{N})$ is the Schwartz space of rapidly decaying functions.

Throughout this paper, as in [3], we say that $(V,K)\in \mathcal{K}$ if continuous functions $V, K:\mathbb{R}^{3}\rightarrow \mathbb{R}$ satisfy the following conditions:

$(H_{0})$ :

$V(x),K(x)>0$ for all $x\in \mathbb{R}^{3}$ and $K\in L^{\infty }(\mathbb{R}^{3})$;

$(H_{1})$ :

If $\{A_{n}\}\subset \mathbb{R}^{3}$ is a sequence of Borel sets such that the Lebesgue measure $|A_{n}|\leq R$, for all n and some $R>0$, then

$$ \lim_{r\rightarrow +\infty } \int _{A_{n}\bigcap B^{c}_{r}(0)}K(x) \,\mathrm{d}x=0, \quad \text{uniformly in }n=1,2,\ldots ; $$

One of the following conditions occurs:

$(H_{2})$ :

$K/V\in L^{\infty }(\mathbb{R}^{3})$; or

$(H_{3})$ :

There exists $p\in (2,2^{*}_{s})$ such that

$$ \frac{K(x)}{V(x)^{\frac{2^{*}_{s}-p}{2^{*}_{s}-2}}}\rightarrow 0 \quad \text{as } \vert x \vert \rightarrow + \infty , $$

where $2^{*}_{s}=\frac{6}{3-2s}$ is the fractional critical exponent.

As for the function f, we assume $f\in C^{1}(\mathbb{R},\mathbb{R})$ and:

$(f_{1})$ :

$\lim_{t\rightarrow 0}f(t)/|t|=0$, if $(H_{2})$ holds;

$(f_{2})$ :

$\lim_{t\rightarrow 0}f(t)/|t|^{p-1}=A\in \mathbb{R}$, if $(H_{3})$ holds;

$(f_{3})$ :

f has a “quasicritical growth”, namely, $\lim_{|t|\rightarrow \infty }f(t)/|t|^{2^{*}_{s}-1}=0$;

$(f_{4})$ :

$\lim_{|t|\rightarrow \infty }F(t)/t^{4}=\infty$, where $F(t)=\int ^{t}_{0}f(s)\,\mathrm{d}s$;

$(f_{5})$ :

The map $t\mapsto f(t)/|t|^{3}$ is nondecreasing on $(- \infty ,0)$ and $(0,\infty )$.

Remark 1.1

Similar conditions as hypotheses $(H_{0})$–$(H_{3})$ on functions V and K were firstly introduced in [3] and characterize a class of Schrödinger–Poisson problems as zero-mass problem.

When $s=t=1$, $K(x)\equiv 1$, system (1.1) reduces to the following Schrödinger–Poisson system

$$ \textstyle\begin{cases} -\Delta u+V(x)+\lambda \phi (x)u=f(u),&x\in \mathbb{R}^{3}, \\ -\Delta \phi =u^{2},&x\in \mathbb{R}^{3}. \end{cases} $$

(1.2)

System (1.2) comes from time-dependent Schrödinger–Poisson equation, which describes quantum (nonrelativistic) particles interacting with the electromagnetic field generated by the motion. For more details on the mathematical and physical background of system (1.2), we refer the readers to the papers [10, 11] and the references therein. Since the so-called nonlocal term $\lambda \phi _{u}(x) u$ is involved, system (1.2) is called a nonlocal problem. The appearance of the nonlocal term in the equations not only makes it important in many physical applications but also causes some difficulties and challenges from a mathematical point of view. Therefore, in the past several decades, there has been an increasing attention toward systems (1.2) or similar problems, and the existence of positive, multiple, bound state, multi-bump, as well as semiclassical state solutions has been investigated; see, for example, [4, 7, 9, 10, 13, 16, 21, 24, 29, 37, 38, 43, 47,48,49, 60]. Besides, He and Zou [23] considered multiplicity of concentrating positive solutions for a class of double parameter perturbed Schrödinger–Poisson equation with critical growth.

For sign-changing solutions, Alves and Souto [2] proved that system (1.2) possesses a least-energy sign-changing solution, in which $\mathbb{R}^{3}$ is replaced by bounded domains with smooth boundary. Via a constraint variational method combined with the Brouwer degree theory, Wang and Zhou [50] investigated the existence of least-energy sign-changing solutions for the system (1.2) when $f(u)=|u|^{p-1}u$, $p\in (3,5)$. By using the constraint variational methods and the quantitative deformation lemma, Shuai and Wang [42] studied the existence and the asymptotic behavior of least energy sign-changing solution for system (1.2). Latter, under some more weak assumptions on f, Chen and Tang [15] improved and generalized some results obtained in [42]. For other work about sign-changing solution of system (1.2) or similar problems, we refer the reader to [9, 25, 26, 30, 32, 61] and the reference therein.

The nonlinear fractional Schrödinger–Poisson systems (1.1) also come from the following fractional Schrödinger equation

$$\begin{aligned} (-\Delta )^{s}u+V(x)u=f(x,u) \quad\text{in } \mathbb{R}^{N}. \end{aligned}$$

(1.3)

Equation (1.3) has been first proposed by Laskin [27, 28] as a result of expanding the Feynman path integral from the Brownian-like to the Lévy-like quantum mechanical paths. There are many interesting papers which considered the existence, multiplicity, uniqueness, regularity and asymptotic decay properties of the solutions to fractional Schrödinger equation (1.3), see [1, 5, 12, 18, 22, 35, 39, 40, 46, 59] and references therein. Besides, some more complicated fractional equations and systems were also studied, and indeed some interesting results were obtained, see [19, 45, 53,54,55] and references therein. Furthermore, there is a very interesting book [36], in which nonlocal fractional problems are systematic investigated. For sign-changing solutions, since the fractional Laplacian operator is nonlocal, there are important structural differences between the classical and the fractional Laplacian. In fact, for $u\in H^{1}(\mathbb{R}^{N})$, we have

$$ \int _{\mathbb{R}^{N}} \vert \nabla u \vert ^{2}\, \mathrm{d}x= \int _{\mathbb{R}^{N}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,\mathrm{d}x+ \int _{\mathbb{R}^{N}} \bigl\vert \nabla u^{-} \bigr\vert ^{2} \,\mathrm{d}x. $$

However, for $u\in H^{s}(\mathbb{R}^{N})$, we have that

$$ \begin{aligned} \int _{\mathbb{R}^{N}} \bigl\vert (-\triangle )^{\frac{s}{2}} u \bigr\vert ^{2}\,\mathrm{d}x&= \int _{\mathbb{R}^{N}} \bigl\vert (-\triangle )^{\frac{s}{2}} u^{+} \bigr\vert ^{2} \,\mathrm{d}x+ \int _{\mathbb{R}^{N}} \bigl\vert (-\triangle )^{\frac{s}{2}} u^{-} \bigr\vert ^{2} \,\mathrm{d}x\\ &\quad {}+2 \int _{\mathbb{R}^{N}}(-\triangle )^{\frac{s}{2}}u^{+}(- \triangle )^{\frac{s}{2}} u^{-}\,\mathrm{d}x. \end{aligned} $$

This fact makes the study of sign-changing solutions to fractional Schrödinger equation (1.3) particularly interesting, and indeed some interesting results were obtained; see, for example, [6, 14, 17, 51] and the references therein.

Before presenting our main result, let us first recall some Sobolev space as follows. We denote $D^{s,2}(\mathbb{R}^{3})$ by the closure of function space $C^{\infty }_{c}(\mathbb{R}^{3})$ with respect to the so-called Gagliardo seminorm

$$ [u]^{2}:= \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y. $$

Since (1.1) is a zero-mass problem, it seems that the appropriate working space should be

$$ X=\biggl\{ u\in D^{s,2}\bigl(\mathbb{R}^{3}\bigr): \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{2}(x)u^{2}(y)}{ \vert x-y \vert ^{3-2t}}\,\mathrm{d}x \,\mathrm{d}y< \infty \biggr\} $$

with the norm

$$ \Vert u \Vert ^{2}_{X}= \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y+ C_{t} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{u^{2}(x)u^{2}(y)}{ \vert x-y \vert ^{3-2t}}\,\mathrm{d}x\,\mathrm{d}y. $$

Then, we know that $(X,\|\cdot \|_{X})$ is a Banach space.

By the Lax–Milgram Theorem, for any $u\in X$, there exists a unique $\phi _{u}^{t}\in D^{t,2}(\mathbb{R}^{3})$ such that

$$ \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{(\phi ^{t}_{u}(x)-\phi ^{t}_{u}(y))(v(x)-v(y))}{ \vert x-y \vert ^{3+2t}}\,\mathrm{d}x\,\mathrm{d}y= \int _{\mathbb{R}^{3}}u^{2}v\,\mathrm{d}x $$

for any $v\in D^{t,2}(\mathbb{R}^{3})$, that is, $\phi _{u}^{t}$ is a weak solution of

$$ (-\Delta )^{t} \phi =u^{2},\quad x\in {\mathbb{R}}^{3}. $$

In fact, we have that

$$ \phi ^{t}_{u}(x)=C_{(t)} \int _{\mathbb{R}^{3}} \frac{u^{2}(y)}{ \vert x-y \vert ^{3-2t}}\,\mathrm{d}y,\quad x \in \mathbb{R}^{3} $$

(1.4)

where $C_{(t)}=\pi ^{-\frac{3}{2}}2^{-2t}\frac{\varGamma (\frac{3-2t}{2})}{ \varGamma (t)}$.

Using the expression of (1.4), we obtain that system (1.1) is merely a single equation for u:

$$\begin{aligned} (-\Delta )^{s}u+V(x)u+\phi _{u}^{t}u=K(x)f(u) \quad\text{in } \mathbb{R}^{3}. \end{aligned}$$

(1.5)

The condition $(V,K)\in \mathcal{K}$ is fascinating. It can be used to certify that the space E given by

$$ E=\biggl\{ u\in D^{s,2}\bigl(\mathbb{R}^{3}\bigr): \int _{\mathbb{R}^{3}}V(x) \vert u \vert ^{2} \,\mathrm{d}x< + \infty \biggr\} $$

endowed with the norm

$$ \Vert u \Vert ^{2}_{E}= \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y+ \int _{\mathbb{R}^{3}}V(x) \vert u \vert ^{2}\,\mathrm{d}x $$

is compactly embedded into the weighted Lebesgue space

$$ L^{q}_{K}\bigl(\mathbb{R}^{3}\bigr)=\biggl\{ u:u \text{ is measurable on }\mathbb{R}^{3}\text{ and } \int _{\mathbb{R}^{3}}K \vert u \vert ^{q}< \infty \biggr\} $$

for some $q\in (2,2^{*}_{s})$, see Proposition 2.2 below.

However, because of the zero-mass situation, we need to consider a new space

$$ H=\biggl\{ u\in D^{s,2}\bigl(\mathbb{R}^{3}\bigr): \int _{\mathbb{R}^{3}}V(x) \vert u \vert ^{2} \,\mathrm{d}x< \infty , \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u ^{2}(x)u^{2}(y)}{ \vert x-y \vert ^{3-2t}}\,\mathrm{d}x\,\mathrm{d}y< \infty \biggr\} $$

with the norm

$$ \Vert u \Vert ^{2}= \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y+ \int _{\mathbb{R}^{3}}V(x) \vert u \vert ^{2}\,\mathrm{d}x+ C_{t} \int _{\mathbb{R} ^{3}} \int _{\mathbb{R}^{3}}\frac{u^{2}(x)u^{2}(y)}{ \vert x-y \vert ^{3-2t}} \,\mathrm{d}x\,\mathrm{d}y. $$

Since $(X,\|\cdot \|_{X})$ and $(E,\|\cdot \|_{E})$ are both Banach, it follows from $(H_{0})$ that $(H,\|\cdot \|)$ is also a Banach space. Denote the usual norm of $L^{p}(\mathbb{R}^{3})$ by $|\cdot |_{p}$. By Sobolev embedding theorem, the embedding $E\hookrightarrow D^{s,2} \hookrightarrow L^{2^{*}_{s}}(\mathbb{R}^{3})$ is continuous. Let $S'>0$ be the embedding constant, i.e.,

$$ \vert u \vert _{2_{s}^{*}}^{2}\leq S^{\prime -1} \Vert u \Vert _{E}^{2},\quad u\in E. $$

(1.6)

The energy functional associated with system (1.1) is defined by

$$ \begin{aligned}[b] J(u)&=\frac{1}{2} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y + \frac{1}{2} \int _{\mathbb{R}^{3}}V(x)u^{2}\,\mathrm{d}x+\frac{1}{4} \int _{\mathbb{R}^{3}}\phi ^{t}_{u}u^{2}\, \mathrm{d}x \\ &\quad{}- \int _{\mathbb{R} ^{3}}K(x)F(u)\,\mathrm{d}x,\quad u\in H, \end{aligned} $$

where $F(u)=\int ^{u}_{0}f(t)\,dt$.

Moreover, under our conditions, J belongs to $C^{1}(H,\mathbb{R})$, and the Fréchet derivative of J is

$$ \begin{aligned}[b] \bigl\langle J'(u),v\bigr\rangle &= \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{(u(x)-u(y))(v(x)-v(y))}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y + \int _{\mathbb{R}^{3}}V(x)uv\,\mathrm{d}x \\ &\quad{}+ \int _{\mathbb{R}^{3}}\phi ^{t}_{u}uv\,\mathrm{d}x- \int _{\mathbb{R}^{3}}K(x)f(u)v\,\mathrm{d}x,\quad u, v \in H. \end{aligned} $$

As is well known, a critical point of J is a weak solution of system (1.1). Furthermore, if $u\in H$ is a weak solution of system (1.1) and $u^{\pm }\neq 0$, we say that u is a sign-changing solution of system (1.1), where

$$ u^{+}(x)=\max \bigl\{ u(x),0\bigr\} ,\qquad u^{-}(x)=\min\bigl\{ u(x),0\bigr\} . $$

Since fractional Schrodinger equation is coupled with a fractional Poisson term $\phi (x)u$, the existence of multiple nonlocal terms causes some mathematical difficulties and makes the study of system (1.1) very interesting. In recent years, several scholars paid their attention to the existence of positive, ground state, semiclassical and other solutions to fractional Schrödinger–Poisson system (1.1) or similar problems; see [33, 41, 44, 56,57,58] and references therein. Besides, in [34], Luo and Tang considered a class of doubly singularly perturbed fractional Schrödinger–Poisson system with critical Sobolev exponent, and proved the existence of ground state and multiple solutions for this system. However, to the best of our knowledge, few papers considered sign-changing solutions to fractional Schrödinger–Poisson system (1.1) or similar problems. Via the quantitative deformation lemma and degree theory, Guo [20] studied the existence and asymptotic behavior of sign-changing solutions for system (1.1).

Our goal in this paper is to seek the least energy sign-changing solutions to system (1.1). As in [3, 15, 42, 50], to overcome the difficulties and challenges stemming from the nonlocal term, we borrow some ideas from [8]. Specifically, we first try to seek a minimizer of the energy functional J under the following constraint:

$$ \mathcal{M}=\bigl\{ u\in H:u^{\pm }\neq 0, \bigl\langle J'(u),u^{+}\bigr\rangle = \bigl\langle J'(u),u^{-}\bigr\rangle =0\bigr\} , $$

and then we will prove that the minimizer is a sign-changing solution of system (1.1).

The main result can be stated as follows.

Theorem 1.1

Suppose that $(V,K)\in \mathcal{K}$ and f satisfies $(f_{1})$–$(f _{5})$. Then system (1.1) possesses at least one least-energy sign-changing solution.

2 Preliminary results

In this section we give some propositions and lemmas for convenience.

Proposition 2.1

([6])

Assume $(V,K)\in \mathcal{K}$. If $(H_{2})$ holds, then E is continuously embedded in $L^{q}_{K}(\mathbb{R}^{3})$ for every $q\in [2,2^{*}_{s}]$; if $(H_{3})$ holds, then E is continuously embedded in $L^{p}_{K}(\mathbb{R}^{3})$.

Proposition 2.2

([6])

Assume $(V,K)\in \mathcal{K}$. If $(H_{2})$ holds, then E is compactly embedded in $L^{q}_{K}(\mathbb{R}^{3})$ for every $q\in (2,2^{*}_{s})$; if $(H_{3})$ holds, then E is compactly embedded in $L^{p}_{K}( \mathbb{R}^{3})$.

Proposition 2.3

([6])

Suppose that f satisfies $(f_{1})$–$(f_{3})$ and $(V,K)\in \mathcal{K}$. Let $\{v_{n}\}$ be such that $v_{n}\rightharpoonup v$ in E. Then

$$ \int _{\mathbb{R}^{3}}K(x)F(v_{n})\,\mathrm{d}x\rightarrow \int _{\mathbb{R}^{3}}K(x)F(v)\,\mathrm{d}x,\qquad \int _{\mathbb{R}^{3}}K(x)f(v _{n})v_{n}\,\mathrm{d}x \rightarrow \int _{\mathbb{R}^{3}}K(x)f(v)v \,\mathrm{d}x. $$

Similarly as in [30], we have following lemmas.

Lemma 2.1

Assume that $(V,K)\in \mathcal{K}$ and f satisfies $(f_{1})$–$(f _{5})$. Then, for any $u\in E\setminus \{0\}$,

$$ \lim_{|t|\rightarrow \infty } \int _{\mathbb{R}^{3}}\frac{Kf(tu)u}{t ^{3}}=\infty . $$

Lemma 2.2

Assume that $(V,K)\in \mathcal{K}$ and f satisfies $(f_{1})$–$(f _{5})$. Then, for any $u\in E\setminus \{0\}$,

$$ \lim_{|t|\rightarrow \infty } \int _{\mathbb{R}^{3}} \frac{KF(tu)}{t^{4}}=\infty . $$

Lemma 2.3

Assume that $(V,K)\in \mathcal{K}$ and f satisfies $(f_{1})$–$(f _{5})$. Then, for any $u\in E\setminus \{0\}$,

$$ \lim_{t\rightarrow 0} \int _{\mathbb{R}^{3}}\frac{Kf(tu)u}{t}=0. $$

3 Technical lemmas

In this section, we prove some technical lemmas related to the existence of sign-changing solutions of system (1.1).

For $u\in H$ with $u^{\pm }\neq 0$, we define $G_{u}: \mathbb{R}^{2} _{+}\rightarrow \mathbb{R}$ by $G_{u}(\alpha ,\beta )=J(\alpha u^{+}+ \beta u^{-})$.

Lemma 3.1

Assume that $(V,K)\in \mathcal{K}$ and $(f_{1})$–$(f_{5})$ hold. Then,

• (i) the pair $(\alpha ,\beta )$ is a critical point of $G_{u}$ with $\alpha ,\beta >0$ if and only if $\alpha u^{+}+\beta u ^{-}\in \mathcal{M}$;

• (ii) the map $G_{u}$ has a unique critical point $(\alpha _{+},\beta _{-})$, with $\alpha _{+}=\alpha _{+}(u)>0$ and $\beta _{-}= \beta _{-}(u)>0$, which is the unique maximum point of $G_{u}$.

Proof

By definition of $G_{u}$, we have that

$$\begin{aligned} \nabla G_{u}(\alpha ,\beta )&=\bigl(\bigl\langle J' \bigl(\alpha u^{+}+\beta u^{-}\bigr),u ^{+}\bigr\rangle ,\bigl\langle J'\bigl(\alpha u^{+}+\beta u^{-}\bigr),u^{-}\bigr\rangle \bigr) \\ &=\biggl(\frac{1}{\alpha }\bigl\langle J'\bigl(\alpha u^{+}+\beta u^{-}\bigr),\alpha u^{+} \bigr\rangle ,\frac{1}{\beta }\bigl\langle J'\bigl(\alpha u^{+}+\beta u^{-}\bigr),\beta u ^{-}\bigr\rangle \biggr) \\ &=\biggl(\frac{1}{\alpha }g_{u}(\alpha ,\beta ), \frac{1}{\beta }h_{u}( \alpha ,\beta )\biggr), \end{aligned}$$

where

$$\begin{aligned}& \begin{aligned}[b] g_{u}(\alpha ,\beta )&= \alpha ^{2} \bigl\Vert u^{+} \bigr\Vert _{E}^{2}-\alpha \beta \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad{}+\alpha ^{4} \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x+\alpha ^{2} \beta ^{2} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2} \,\mathrm{d}x \\ &\quad{}- \int _{\mathbb{R}^{3}}K(x)f\bigl(\alpha u^{+}\bigr)\alpha u^{+} \,\mathrm{d}x, \end{aligned} \end{aligned}$$

(3.1)

$$\begin{aligned}& \begin{aligned}[b] h_{u}(\alpha ,\beta )&= \beta ^{2} \bigl\Vert u^{-} \bigr\Vert _{E}^{2}-\alpha \beta \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad{}+\beta ^{4} \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x+\alpha ^{2} \beta ^{2} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{+}} \bigl\vert u^{-} \bigr\vert ^{2} \,\mathrm{d}x \\ &\quad{}- \int _{\mathbb{R}^{3}}K(x)f\bigl(\beta u^{-}\bigr)\beta u^{-} \,\mathrm{d}x. \end{aligned} \end{aligned}$$

(3.2)

From above facts, item (i) is obvious.

In the following, we prove (ii).

Firstly, we assert that $\mathcal{M}\neq \emptyset$ .

In fact, to this end, we just prove the existence of a critical point of $G_{u}$. Letting $u\in H$ with $u^{\pm }\neq 0$ and $\beta _{0} \geq 0$ fixed, from (3.1), we obtain

$$\begin{aligned}& \begin{aligned}[b] g_{u}(\alpha ,\beta _{0})&= \alpha ^{2}\biggl( \bigl\Vert u^{+} \bigr\Vert _{E}^{2}-\frac{\beta _{0}}{\alpha } \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u ^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad{}+ \alpha ^{2} \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x+ \beta _{0}^{2} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2} \,\mathrm{d}x \\ &\quad{}- \int _{\mathbb{R}^{3}}\frac{K(x)f(\alpha u^{+})u^{+}}{ \alpha }\,\mathrm{d}x\biggr), \end{aligned} \\& \begin{aligned}[b] g_{u}(\alpha ,\beta _{0})&= \alpha ^{4}\biggl(\frac{1}{\alpha ^{2}} \bigl\Vert u^{+} \bigr\Vert _{E}^{2}-\frac{\beta _{0}}{\alpha ^{3}} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{u^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x \,\mathrm{d}y \\ &\quad{}+ \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{+} \bigr\vert ^{2} \,\mathrm{d}x+\frac{\beta _{0}^{2}}{\alpha ^{2}} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad{}- \int _{\mathbb{R}^{3}}\frac{K(x)f( \alpha u^{+})u^{+}}{\alpha ^{3}}\,\mathrm{d}x\biggr). \end{aligned} \end{aligned}$$

Then, according to Lemmas 2.1 and 2.3, we have that

$$\begin{aligned} g_{u}(\alpha ,\beta _{0})>0 \quad \text{for }\alpha \text{ small enough};\qquad g_{u}(\alpha ,\beta _{0})< 0 \quad \text{for } \alpha \text{ large enough}. \end{aligned}$$

Since $g_{u}(\alpha ,\beta _{0})$ is continuous, there exists $\alpha _{0}>0$ such that $g_{u}(\alpha _{0},\beta _{0})=0$. We assert $\alpha _{0}$ is unique. In fact, supposing by contradiction, that there exist $0<\alpha _{1}<\alpha _{2}$ such that $g_{u}(\alpha _{1},\beta _{0})=g _{u}(\alpha _{2},\beta _{0})$, we then have

$$ \begin{aligned}[b]&\biggl(\frac{1}{\alpha _{1}^{2}}-\frac{1}{\alpha _{2}^{2}} \biggr) \biggl( \bigl\Vert u^{+} \bigr\Vert _{E} ^{2}+\beta _{0}^{2} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2} \,\mathrm{d}x\biggr) \\ &\qquad {}+\biggl(\frac{\beta _{0}}{\alpha _{2}^{3}}-\frac{\beta _{0}}{ \alpha _{1}^{3}}\biggr) \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u ^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad = \int _{\mathbb{R}^{3}}K(x)\biggl[\frac{f(\alpha _{1} u^{+})}{(\alpha _{1} u ^{+})^{3}}-\frac{f(\alpha _{2} u^{+})}{(\alpha _{2} u^{+})^{3}} \biggr]\bigl(u^{+}\bigr)^{4} \,\mathrm{d}x. \end{aligned} $$

Therefore, in view of $(f_{5})$ and $0<\alpha _{1}<\alpha _{2}$, we obtain a contradiction. That is, there exists a unique $\alpha _{0}>0$ such that $g_{u}(\alpha _{0},\beta _{0})=0$.

Thus, we can define a function $\varphi _{1}:\mathbb{R}_{+}\rightarrow (0,\infty )$ by

$$ \varphi _{1}(\beta )=\alpha (\beta ), $$

where $\alpha (\beta )$ satisfies $g_{u}(\alpha (\beta ),\beta )=0$.

By the same arguments as above, we can define functions $\varphi _{2}: \mathbb{R}_{+}\rightarrow (0,\infty )$ by $\varphi _{2}(\alpha )= \beta (\alpha )$ which satisfies $h_{u}(\alpha ,\beta (\alpha ))=0$.

Furthermore, the functions $\varphi _{i}$, $i=1,2$, have the following three good properties:

• (a) $\varphi _{i}$ are continuous on $\mathbb{R}_{+}$.

• (b) $\varphi _{1}(\beta )>0$, $\varphi _{2}(\alpha )>0$ for any $\alpha ,\beta \in \mathbb{R}_{+}$.

• (c) $\varphi _{1}(\beta )<\beta$ and $\varphi _{2}(\alpha )<\alpha$ for α, β large.

By (c), there exists $C_{1}>0$ such that $\varphi _{1}(\beta )\leq \beta$ and $\varphi _{2}(\alpha )\leq \alpha$, respectively, when $\alpha ,\beta >C_{1}$. Let $C_{2}=\max \{ \max_{\beta \in [0,C_{1}]}\varphi _{1}(\beta ),\max_{\alpha \in [0,C _{1}]}\varphi _{2}(\alpha )\}$, $C=\max \{C_{1},C_{2}\}$, and define $T:[0,C]\times [0,C]\rightarrow \mathbb{R}^{2}_{+}$ by

$$ T(\alpha ,\beta )=\bigl(\varphi _{1}(\beta ),\varphi _{2}( \alpha )\bigr). $$

It is easy to see that $T(\alpha ,\beta )\in [0,C]\times [0,C]$ for all $(\alpha ,\beta )\in [0,C]\times [0,C]$. In fact,

$$ \textstyle\begin{cases}\varphi _{2}(\alpha )\leq \alpha \leq C_{1},&\alpha >C_{1},\\ \varphi _{2}(\alpha )\leq \max_{\alpha \in [0,C_{1}]}\varphi _{2}( \alpha )\leq C_{2},&\alpha \leq C_{1}, \end{cases} $$

that is to say, $\varphi _{2}(\alpha )\leq C$. Similarly, we have $\varphi _{1}(\beta )\leq C$. Since T is continuous, using Brouwer fixed point theorem, there exists $(\alpha _{+}\beta _{-})\in [0,C] \times [0,C]$ such that

$$ \begin{aligned}[b] \bigl(\varphi _{1}(\beta _{-}),\varphi _{2}(\alpha _{+})\bigr)=(\alpha _{+},\beta _{-}). \end{aligned} $$

(3.3)

It follows from $\varphi _{i}>0$, (3.3) that $\alpha _{+},\beta _{-}>0$. According to the definition, we have

$$ \frac{\partial G_{u}}{\partial \alpha }(\alpha _{+},\beta _{-})= \frac{ \partial G_{u}}{\partial \beta }(\alpha _{+},\beta _{-})=0. $$

We next prove the uniqueness of $(\alpha _{+},\beta _{-})$.

Case 1. $u\in \mathcal{M}$.

Supposing that $u\in \mathcal{M}$, one has

$$ \begin{aligned}[b] \nabla G_{u}(1,1)&=\biggl( \frac{\partial G_{u}}{\partial \alpha }(1,1),\frac{ \partial G_{u}}{\partial \beta }(1,1)\biggr) \\ &=\bigl(\bigl\langle J'\bigl(u^{+}+u^{-} \bigr),u ^{+}\bigr\rangle ,\bigl\langle J' \bigl(u^{+}+u^{-}\bigr),u^{-}\bigr\rangle \bigr)=(0,0), \end{aligned} $$

which shows that $(1,1)$ is a critical point of $G_{u}$. Now, we need to prove that $(1,1)$ is the unique critical point of $G_{u}$ with positive coordinates. Let $(\alpha _{0},\beta _{0})$ be a critical point of $G_{u}$ such that $0<\alpha _{0}\leq \beta _{0}$. So, one has that

$$ \begin{aligned}[b] &\alpha _{0}^{2} \bigl\Vert u^{+} \bigr\Vert ^{2}_{E}-\alpha _{0} \beta _{0} \int _{\mathbb{R} ^{3}} \int _{\mathbb{R}^{3}} \frac{u^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x \,\mathrm{d}y \\ &\qquad {}+\alpha _{0}^{4} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{+}} \bigl\vert u ^{+} \bigr\vert ^{2}\,\mathrm{d}x+\alpha _{0}^{2}\beta _{0}^{2} \int _{\mathbb{R}^{3}} \phi ^{t}_{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad = \int _{\mathbb{R}^{3}}K(x)f\bigl( \alpha _{0}u^{+} \bigr) \bigl(\alpha _{0}u^{+}\bigr)\,\mathrm{d}x \end{aligned} $$

(3.4)

and

$$ \begin{aligned}[b] &\beta _{0}^{2} \bigl\Vert u^{-} \bigr\Vert ^{2}_{E}-\alpha _{0}\beta _{0} \int _{\mathbb{R} ^{3}} \int _{\mathbb{R}^{3}} \frac{u^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x \,\mathrm{d}y \\ &\qquad {}+\beta _{0}^{4} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{-}} \bigl\vert u ^{-} \bigr\vert ^{2}\,\mathrm{d}x+\alpha _{0}^{2}\beta _{0}^{2} \int _{\mathbb{R}^{3}} \phi ^{t}_{u^{+}} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad = \int _{\mathbb{R}^{3}}K(x)f\bigl( \beta _{0}u^{-} \bigr) \bigl(\beta _{0}u^{-}\bigr)\,\mathrm{d}x. \end{aligned} $$

(3.5)

Thanks to $0<\alpha _{0}\leq \beta _{0}$ and (3.5), we have that

$$ \begin{aligned}[b] &\frac{ \Vert u^{-} \Vert ^{2}_{E}}{\beta _{0}^{2}}- \frac{1}{\beta ^{2}_{0}} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y + \int _{\mathbb{R}^{3}}\phi ^{t}_{u} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad \geq \int _{\mathbb{R}^{3}}K(x) \frac{f(\beta _{0}u^{-})}{(\beta _{0}u^{-})^{3}}\bigl(u^{-} \bigr)^{4}\,\mathrm{d}x. \end{aligned} $$

(3.6)

On the other hand, for $u\in \mathcal{M}$, we have

$$ \begin{aligned}[b] & \bigl\Vert u^{-} \bigr\Vert ^{2}_{E}- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u ^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y + \int _{\mathbb{R}^{3}}\phi ^{t}_{u} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad = \int _{\mathbb{R}^{3}}K(x)\frac{f(u^{-})}{(u^{-})^{3}}\bigl(u^{-} \bigr)^{4} \,\mathrm{d}x. \end{aligned} $$

(3.7)

Combining (3.6) with (3.7), one has that

$$ \begin{aligned}[b] &\biggl(\frac{1}{\beta _{0}^{2}}-1\biggr) \bigl\Vert u^{-} \bigr\Vert ^{2}_{E}+\biggl(1- \frac{1}{\beta _{0} ^{2}}\biggr) \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad \geq \int _{\mathbb{R}^{3}}K(x)\biggl[ \frac{f(\beta _{0}u^{-})}{(\beta _{0}u^{-})^{3}}- \frac{f(u^{-})}{(u^{-})^{3}} \biggr]\bigl(u^{-}\bigr)^{4}\,\mathrm{d}x. \end{aligned} $$

By the definitions of $u^{\pm }$ and $u^{\pm }\neq 0$, we know that $\int _{\mathbb{R}^{3}}\int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{-}(x)u^{+}(y)}{|x-y|^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y<0$. So, if $\beta _{0}>1$, the left-hand side of the above inequality is negative, which is absurd because the right-hand side is nonnegative by condition $(f_{5})$. Therefore, we obtain that $0<\alpha _{0}\leq \beta _{0} \leq 1$.

Similarly, by (3.4) and $0<\alpha _{0}\leq \beta _{0}$, one has that

$$ \begin{aligned}[b] &\biggl(\frac{1}{\alpha _{0}^{2}}-1\biggr) \bigl\Vert u^{+} \bigr\Vert ^{2}_{E}+\biggl(1- \frac{1}{\alpha _{0} ^{2}}\biggr) \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad \leq \int _{\mathbb{R}^{3}}K(x)\biggl[ \frac{f(\alpha _{0}u^{+})}{(\alpha _{0}u^{+})^{3}}-\frac{f(u^{+})}{(u ^{+})^{3}} \biggr]\bigl(u^{+}\bigr)^{4}\,\mathrm{d}x. \end{aligned} $$

Therefore, by condition $(f_{5})$, we must have $\alpha _{0}\geq 1$. Consequently, $\alpha _{0}=\beta _{0}=1$, which indicates that $(1,1)$ is the unique critical point of $G_{u}$ with positive coordinates.

Case 2. $u\notin \mathcal{M}$.

Let $u\in H$, $u^{\pm }\neq 0$ and $(\alpha _{1},\beta _{1})$, $(\alpha _{2}, \beta _{2})$ be the critical points of $G_{u}$ with positive coordinates. In view of (i), one has that

$$ u_{1}=\alpha _{1}u^{+}+\beta _{1}u^{-}\in \mathcal{M},\qquad u_{2}=\alpha _{2}u ^{+}+\beta _{2}u^{-}\in \mathcal{M}. $$

So,

$$ u_{2}=\biggl(\frac{\alpha _{2}}{\alpha _{1}}\biggr)\alpha _{1}u^{+}+ \biggl(\frac{\beta _{2}}{ \beta _{1}}\biggr)\beta _{1}u^{-}=\biggl( \frac{\alpha _{2}}{\alpha _{1}}\biggr)u_{1}^{+}+\biggl(\frac{ \beta _{2}}{\beta _{1}} \biggr)u_{1}^{-}\in \mathcal{M}. $$

It follows from $u_{1}\in H$ with $u_{1}^{\pm }\neq 0$ that $(\frac{\alpha _{2}}{\alpha _{1}},\frac{\beta _{2}}{\beta _{1}})$ is a critical point of the map $G_{u_{1}}$ with positive coordinates. Thanks to $u_{1}\in \mathcal{M}$, one has that

$$ \frac{\alpha _{2}}{\alpha _{1}}=\frac{\beta _{2}}{\beta _{1}}=1, $$

Hence, $\alpha _{1}=\alpha _{2}$, $\beta _{1}=\beta _{2}$.

Finally, we prove that the unique critical point is the unique maximum point of $G_{u}$.

In fact, using Lemma (2.2), we have that

$$ G_{u}(\alpha ,\beta )\rightarrow -\infty \quad \text{as } \bigl\vert (\alpha , \beta ) \bigr\vert \rightarrow \infty . $$

Hence, the maximum point of $G_{u}(\alpha ,\beta )$ cannot be achieved on the boundary of $\mathbb{R}^{2}_{+}$. Without loss of generality, we may assume that $(0,\overline{\beta })$ is a maximum point of $G_{u}$. But, according to Lemma (2.3), it is obvious that

$$ \begin{aligned}[b] &G_{u}(\alpha ,\overline{\beta })\\ &\quad = \frac{\alpha ^{2}}{2} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{ \vert u^{+}(x)-u^{+}(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y+ \frac{\overline{ \beta }^{2}}{2} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{ \vert u^{-}(x)-u ^{-}(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\qquad{}-\alpha \overline{ \beta } \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{+}(y)u^{-}(x)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y+ \frac{\alpha ^{2}}{2} \int _{\mathbb{R}^{3}}V(x) \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x \\ &\qquad{}+\frac{\overline{\beta }^{2}}{2} \int _{\mathbb{R}^{3}}V(x) \bigl\vert u^{-} \bigr\vert ^{2} \,\mathrm{d}x+\frac{\alpha ^{2}}{4} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{+}} \bigl\vert u ^{+} \bigr\vert ^{2}\,\mathrm{d}x+\frac{\overline{\beta }^{2}}{4} \int _{\mathbb{R} ^{3}}\phi ^{t}_{u^{-}} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x \\ &\qquad{}+\frac{\alpha \overline{ \beta }}{4} \int _{\mathbb{R}^{3}}\bigl(\phi ^{t}_{u^{+}} \bigl\vert u^{-} \bigr\vert ^{2}+\phi ^{t} _{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2}\bigr)\,\mathrm{d}x- \int _{\mathbb{R}^{3}}K(x)F\bigl(\alpha u ^{+}\bigr)\, \mathrm{d}x- \int _{\mathbb{R}^{3}}K(x)F\bigl(\overline{\beta } u^{-}\bigr) \,\mathrm{d}x, \end{aligned} $$

which is an increasing function with respect to α, if α is small enough. Hence, the pair $(0,\overline{\beta })$ is not a maximum point of $G_{u}$ in $\mathbb{R}_{+}^{2}$. □

Lemma 3.2

Suppose that $(V,K)\in \mathcal{K}$ and $(f_{1})$–$(f_{5})$ hold. If $u\in H$ with $u^{\pm }\neq 0$ is such that $g_{u}(1,1)\leq 0$ and $h_{u}(1,1)\leq 0$, where $g_{u}(\alpha ,\beta )$, $h_{u}(\alpha ,\beta )$ are given by (3.1) and (3.2), then the unique pair $(\alpha _{+},\beta _{-})$ obtained in Lemma (2.1) satisfies $0<\alpha _{+}$, $\beta _{-}\leq 1$.

Proof

Suppose $\alpha _{+}\geq \beta _{-}>0$. Since $\alpha_{+} u^{+}+\beta _{-} u^{-} \in \mathcal{M}$, we have

$$ \begin{aligned}[b] &\alpha _{+}^{2} \bigl\Vert u^{+} \bigr\Vert _{E}^{2}+ \alpha _{+}^{4} \int _{\mathbb{R}^{3}} \phi _{u^{+}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x+\alpha _{+}^{4} \int _{\mathbb{R} ^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x \\ &\qquad{}-\alpha _{+}^{2} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{+}(y)u^{-}(x)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad \geq\alpha _{+}^{2} \bigl\Vert u^{+} \bigr\Vert _{E}^{2}+\alpha _{+}^{4} \int _{\mathbb{R}^{3}}\phi _{u ^{+}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x+\alpha _{+}^{2}\beta _{-}^{2} \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x \\ &\qquad{}-\alpha _{+}\beta _{-} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u ^{-}(y)+u^{+}(y)u^{-}(x)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad = \int _{\mathbb{R}^{3}}K(x)f\bigl(\alpha _{+}u^{+} \bigr) \bigl(\alpha _{+}u^{+}\bigr) \,\mathrm{d}x. \end{aligned} $$

(3.8)

Thanks to $g_{u}(1,1)\leq 0$, we have that

$$ \begin{aligned}[b] &\bigl\Vert u^{+} \bigr\Vert _{E}^{2}+ \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{+} \bigr\vert ^{2} \,\mathrm{d}x+ \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{+} \bigr\vert ^{2} \,\mathrm{d}x \\ &\qquad{}- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u ^{-}(y)+u^{+}(y)u^{-}(x)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ & \quad \leq \int _{\mathbb{R}^{3}}K(x)f\bigl(u^{+}\bigr) \bigl(u^{+}\bigr)\,\mathrm{d}x. \end{aligned} $$

(3.9)

From (3.8) and (3.9), we have that

$$ \begin{aligned}[b] &\biggl(\frac{1}{\alpha _{+}^{2}}-1\biggr) \bigl\Vert u^{+} \bigr\Vert ^{2}_{E}+\biggl(1- \frac{1}{\alpha _{+} ^{2}}\biggr) \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad \geq \int _{\mathbb{R}^{3}}K(x)\biggl[ \frac{f(\alpha _{+}u^{+})}{(\alpha _{+}u^{+})^{3}}-\frac{f(u^{+})}{(u ^{+})^{3}} \biggr]\bigl(u^{+}\bigr)^{4}. \end{aligned} $$

By $(f_{5})$, we must have $\alpha _{+}\leq 1$. Then the proof is completed. □

Next, we consider the following minimization problem

$$ m=\inf \bigl\{ J(u):u\in \mathcal{M}\bigr\} . $$

(3.10)

Lemma 3.3

Suppose that $(V,K)\in \mathcal{K}$ and f satisfies $(f_{1})$–$(f _{5})$. Then $m>0$ can be achieved.

Proof

Firstly, we prove $m>0$.

For every $u\in \mathcal{M}$, we have $\langle J'(u),u\rangle =0$.

First of all, suppose that $(H_{2})$ is true. It follows from $(f_{1})$ and $(f_{3})$ that for any given $\varepsilon >0$, there exists $C_{\varepsilon }>0$ such that

$$ \bigl\vert f(t) \bigr\vert \leq \varepsilon \vert t \vert +C_{\varepsilon } \vert t \vert ^{2^{*}_{s}-1},\quad t\in \mathbb{R}. $$

So, we have that

$$ \begin{aligned}[b] \Vert u \Vert ^{2}_{E}&\leq \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{u(x)-u(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y+ \int _{\mathbb{R} ^{3}}V(x)u^{2}\,\mathrm{d}x+ \int _{\mathbb{R}^{3}}\phi ^{t}_{u}u^{2} \, \mathrm{d}x \\ &= \int _{\mathbb{R}^{3}}K(x)f(u)u\,\mathrm{d}x\leq \varepsilon \vert K/V \vert _{\infty } \Vert u \Vert ^{2}_{E}+C_{\varepsilon } \vert K \vert _{\infty }(S^{\prime})^{-\frac{2^{*}_{s}}{2}} \Vert u \Vert ^{2^{*}_{s}}_{E}. \end{aligned} $$

(3.11)

Choosing $\varepsilon <1/|K/V|_{\infty }$, there exists a constant $\theta _{1}>0$ such that $\|u\|^{2}_{E}\geq \theta _{1}$.

Next, suppose that $(H_{3})$ holds. By the discussion of [6], there is a constant $C_{p}>0$, for every given $\varepsilon \in (0,C _{p})$, there exists $R>0$ large enough leading to

$$\begin{aligned} \int _{ \vert x \vert \geq R}K(x) \vert u \vert ^{p}\,dx\leq \varepsilon \int _{ \vert x \vert \geq R}\bigl(Vu^{2}+u ^{2^{*}_{s}}\bigr)\,dx,\quad u \in E. \end{aligned}$$

(3.12)

From $(f_{2})$ and $(f_{3})$, there are $C_{1},C_{2}>0$ such that

$$ \bigl\vert f(t) \bigr\vert \leq C_{1} \vert t \vert ^{p-1}+C_{2} \vert t \vert ^{{2^{*}_{s}-1}},\quad t\in \mathbb{R}. $$

So, by Hölder’s inequality and Soblev inequalities, we have that

$$ \begin{aligned}[b] \Vert u \Vert ^{2}_{E}&\leq \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{u(x)-u(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y+ \int _{\mathbb{R} ^{3}}V(x)u^{2}\,\mathrm{d}x+ \int _{\mathbb{R}^{3}}\phi ^{t}_{u}u^{2} \, \mathrm{d}x \\ &= \int _{\mathbb{R}^{3}}K(x)f(u)u\,\mathrm{d}x\leq C_{1} \varepsilon \Vert u \Vert ^{2}_{E}+C_{1} \bigl(S'\bigr)^{\frac{-2}{2^{*}_{s}}}\bigl(\varepsilon +C_{2} \vert K \vert _{\infty }\bigr) \Vert u \Vert ^{2^{*}_{s}}_{E} \\ &\quad{}+C_{2} \vert K \vert _{L^{2^{*}_{s}/(2^{*} _{s}-p)}B_{R}(0)}\bigl(S' \bigr)^{\frac{-p}{2}} \Vert u \Vert ^{p}_{E}. \end{aligned} $$

(3.13)

Choosing $\varepsilon <1/C_{1}$, there exists a constant $\theta _{2}>0$ such that $\|u\|^{2}_{E}\geq \theta _{2}$.

Consequently, we conclude that $\|u\|^{2}_{E}\geq \theta$ for any $u\in \mathcal{M}$, where $\theta =\max \{\theta _{1},\theta _{2}\}>0$.

On the other hand, by condition $(f_{5})$, we have

$$ H(t):=f(t)t-4F(t)\geq 0,\quad t\in \mathbb{R}, $$

(3.14)

and $H(t)$ is increasing when $t>0$ and decreasing when $t<0$. Hence,

$$ \begin{aligned}[b] J(u)&=J(u)-\frac{1}{4}\bigl\langle J'(u),u\bigr\rangle \\ &=\frac{1}{4} \Vert u \Vert _{E}^{2}+ \frac{1}{4} \int _{\mathbb{R}^{3}}K(x) \bigl(f(u)u-4F(u)\bigr)\,\mathrm{d}x \\ &\geq \frac{1}{4} \Vert u \Vert _{E}^{2} \\ &\geq \frac{1}{4}\theta . \end{aligned} $$

This implies that $m\geq \frac{\theta }{4}>0$.

In the following, we prove that m is achieved.

Let $\{u_{n}\}\subset \mathcal{M}$ such that $J(u_{n})\rightarrow m$. Then $\|u_{n}\|_{E}\leq C$. Hence, we may assume that there exists $u\in E$ such that $u_{n}\rightharpoonup u$, $u_{n}^{\pm }\rightharpoonup u^{\pm }$ in E.

By Proposition (2.3), we know that

$$ \int _{\mathbb{R}^{3}}K(x)F(u_{n})\,dx\rightarrow \int _{\mathbb{R}^{3}}K(x)F(u)\,dx. $$

Hence $\{\int _{\mathbb{R}^{3}}K(x)F(u_{n})\,dx\}$ is bounded.

By definition of J, we get

$$ \frac{1}{4} \int _{\mathbb{R}^{3}}\phi _{u_{n}}^{t}u_{n}^{2} \,\mathrm{d}x+ \frac{1}{2} \Vert u_{n} \Vert _{E}^{2}=J(u_{n})+ \int _{\mathbb{R}^{3}}KF(u_{n}) \,\mathrm{d}x, $$

which implies that $\{u_{n}\}$ is bounded in H.

Hence, by the uniqueness of the convergence, we get $u_{n}\rightharpoonup u$ and $u_{n}^{\pm }\rightharpoonup u^{\pm }$ in H. Thanks to $u_{n}\in \mathcal{M}$, we have that

$$ \begin{aligned}[b] \bigl\langle J'(u_{n}),u_{n}^{\pm } \bigr\rangle &= \bigl\Vert u_{n}^{\pm } \bigr\Vert _{E}^{2}- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u_{n}^{+}(x)u_{n}^{-}(y)+u _{n}^{-}(x)u_{n}^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad {}+ \int _{\mathbb{R}^{3}}\phi ^{t}_{u_{n}} \bigl\vert u_{n}^{\pm } \bigr\vert ^{2}\,\mathrm{d}x- \int _{\mathbb{R}^{3}}Kf\bigl(u_{n}^{\pm } \bigr)u_{n}^{\pm }\,\mathrm{d}x\\ &=0. \end{aligned} $$

(3.15)

Together (3.15) with Proposition 2.3, we get

$$ \begin{aligned}[b] 0&< \theta\leq \bigl\Vert u_{n}^{\pm } \bigr\Vert _{E}^{2}\\ &\leq \bigl\Vert u_{n}^{\pm } \bigr\Vert _{E}^{2}- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u_{n}^{+}(x)u_{n}^{-}(y)+u _{n}^{-}(x)u_{n}^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y + \int _{\mathbb{R}^{3}}\phi ^{t}_{u_{n}} \bigl\vert u_{n}^{\pm } \bigr\vert ^{2}\,\mathrm{d}x \\ &= \int _{\mathbb{R}^{3}}K(x)f\bigl(u_{n}^{\pm } \bigr)u_{n}^{\pm }\,\mathrm{d}x= \int _{\mathbb{R}^{3}}K(x)f\bigl(u^{\pm }\bigr)u^{\pm }\, \mathrm{d}x+o(1). \end{aligned} $$

(3.16)

Thus $u^{\pm }\neq 0$.

On the other hand, combining (1.4) with the Hardy–Littlewood–Sobolev inequality [31], we have that

$$ \lim_{n\rightarrow \infty }\inf \int _{\mathbb{R}^{3}}\phi ^{t}_{u_{n}} \bigl\vert u _{n}^{\pm } \bigr\vert ^{2}\,\mathrm{d}x= \int _{\mathbb{R}^{3}}\phi ^{t}_{u} \bigl\vert u^{ \pm } \bigr\vert ^{2}\,\mathrm{d}x. $$

Then, by the weak lower semicontinuity of norm and Fatou’s lemma, we have

$$ \begin{aligned}[b] & \bigl\Vert u^{\pm } \bigr\Vert _{E}^{2}- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u ^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y + \int _{\mathbb{R}^{3}}\phi ^{t}_{u} \bigl\vert u^{\pm } \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad \leq \lim_{n\rightarrow \infty }\inf \biggl[ \bigl\Vert u_{n}^{\pm } \bigr\Vert _{E}^{2}- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u_{n}^{+}(x)u_{n}^{-}(y)+u _{n}^{-}(x)u_{n}^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\qquad{}+ \int _{\mathbb{R}^{3}}\phi ^{t}_{u_{n}} \bigl\vert u_{n}^{\pm } \bigr\vert ^{2}\,\mathrm{d}x\biggr]. \end{aligned} $$

(3.17)

Then, by (3.16) and (3.17), we have that

$$ \begin{aligned}[b] & \bigl\Vert u^{\pm } \bigr\Vert _{E}^{2}- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u ^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y + \int _{\mathbb{R}^{3}}\phi ^{t}_{u} \bigl\vert u^{\pm } \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad \leq \int _{\mathbb{R}^{3}}K(x)f\bigl(u^{\pm }\bigr)u^{\pm }\, \mathrm{d}x. \end{aligned} $$

(3.18)

According to Lemmas 3.1 and 3.2, there exists $(\overline{\alpha },\overline{\beta })\in (0,1]\times (0,1]$ such that

$$ \overline{u}:=\overline{\alpha }u^{+}+\overline{\beta }u^{-}\in \mathcal{M}. $$

Thanks to (3.14), we have that

$$ \begin{aligned}[b] m&\leq J(\overline{u})-\frac{1}{4}\bigl\langle J'(\overline{u}), \overline{u}\bigr\rangle \\ &=\frac{1}{4} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{ \vert \overline{u}(x)-\overline{u}(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x \,\mathrm{d}y+ \frac{1}{4} \int _{\mathbb{R}^{3}}V(x) \vert \overline{u} \vert ^{2} \, \mathrm{d}x+\frac{1}{4} \int _{\mathbb{R}^{3}}K(x) \bigl(f(\overline{u}) \overline{u}-4F( \overline{u})\bigr)\,\mathrm{d}x \\ &=\frac{\overline{\alpha }^{2}}{4} \bigl\Vert u^{+} \bigr\Vert _{E}^{2}- \frac{\overline{\alpha }\overline{\beta }}{2} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{u^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x \,\mathrm{d}y+ \frac{\overline{\beta }^{2}}{4} \bigl\Vert u^{-} \bigr\Vert _{E}^{2} \\ &\quad{}+ \frac{1}{4} \int _{\mathbb{R}^{3}}K(x) \bigl(f\bigl(\overline{\alpha }u^{+} \bigr)\overline{ \alpha }u^{+}-4F\bigl(\overline{\alpha }u^{+}\bigr)\bigr)\,\mathrm{d}x +\frac{1}{4} \int _{\mathbb{R}^{3}}K(x) \bigl(f\bigl(\overline{\beta }u^{-} \bigr)\overline{\beta }u ^{-}-4F\bigl(\overline{\beta }u^{-} \bigr)\bigr)\,\mathrm{d}x \\ &\leq \frac{1}{4} \Vert u \Vert ^{2}_{E}+ \frac{1}{4} \int _{\mathbb{R}^{3}}K(x) \bigl(f(u)u-4F(u)\bigr)\,\mathrm{d}x \\ &\leq \lim_{n\rightarrow \infty }\biggl[J(u_{n})-\frac{1}{4} \bigl\langle J'(u _{n}),u_{n}\bigr\rangle \biggr]=m. \end{aligned} $$

Consequently, $\overline{\alpha }=\overline{\beta }=1$. Thus $\overline{u}=u$ and $J(u)=m$. □

4 Proof of the main result

In this section, we will prove Theorem 1.1. In fact, we just prove that the minimizer u for (3.10) is indeed a sign-changing solution of system (1.1).

Proof

Since $u\in \mathcal{M}$, we have $\langle J'(u),u^{+}\rangle =0= \langle J'(u),u^{-}\rangle$. By Lemma 3.1, for $(\alpha , \beta )\in \mathbb{R}^{2}_{+}$ and $(\alpha ,\beta )\neq (1,1)$, we have

$$ J\bigl(\alpha u^{+}+\beta u^{-}\bigr)< J \bigl(u^{+}+u^{-}\bigr)=m. $$

(4.1)

Set $\xi _{1}=|u^{+}|_{2_{s}^{*}}$, $\xi _{2}=|u^{-}|_{2_{s}^{*}}$ and $\xi =\min \{\xi _{1},\xi _{2}\}$. We denote S̃ the imbedding constant of $H\hookrightarrow L^{2^{*}_{s}}(\mathbb{R}^{3})$, that is, $|u|_{2_{s}^{*}}\leq \widetilde{S}\|u\|$, $u\in H$.

If $J'(u)\neq 0$, then there exist $r, \mu >0$ such that

$$ \bigl\Vert J'(v) \bigr\Vert \geq \mu ,\qquad \Vert v-u \Vert \leq r. $$

(4.2)

Choose $\delta \in (0,\min \{\xi /(2\widetilde{S}),r/3\})$ and $\sigma \in (0,\min \{1/2,\delta /(\sqrt{2}\|u\|)\})$. Let $D=(1-\sigma , 1+\sigma )\times (1-\sigma , 1+\sigma )$ and $\psi (\alpha ,\beta ):=\alpha u^{+}+\beta u^{-}$, $(\alpha ,\beta ) \in D$. In view of (4.1), it is easy to see that

$$ \overline{m}:=\max_{\partial D}J\circ \psi < m. $$

(4.3)

Let $0<\varepsilon <\min \{(m-\overline{m})/2,\mu \delta /8\}$ and $S_{\delta }:=\{v\in H,\|v-u\|<\delta \}$, according to quantitative deformation lemma [52], there exists a deformation $\eta \in ([0,1] \times H, H)$ satisfying

• (a) $\eta (1,u)=u$ if $u\notin J^{-1}([m-2\varepsilon , m+2\varepsilon ])\cap S_{2\delta }$;

• (b) $\eta (1,J^{m+\varepsilon }\cap S_{\delta })\subset J^{m-\varepsilon }$;

• (c) $\|\eta (1,u)-u\|\leq \delta$ for all $u\in H$, where $J^{m+ \varepsilon }:=\{x|J(x)\leq m+\varepsilon \}$.

Firstly, we need to prove that

$$ \max_{(\alpha ,\beta )\in \overline{D}}J\bigl(\eta \bigl(1,\psi (\alpha , \beta )\bigr)\bigr)< m. $$

(4.4)

In fact, it is follows from Lemma 3.1 that $J(\psi (\alpha , \beta ))\leq m< m+\varepsilon$. That is, $\psi (\alpha ,\beta )\in J ^{m+\varepsilon }$.

On the other hand, we have

$$ \begin{aligned}[b] \bigl\Vert \psi (\alpha ,\beta )-u \bigr\Vert ^{2}&\leq 2\bigl((\alpha -1)^{2} \bigl\Vert u^{+} \bigr\Vert ^{2}+( \beta -1)^{2} \bigl\Vert u^{-} \bigr\Vert ^{2}\bigr) \\ &\leq 2\sigma \Vert u \Vert ^{2} \\ &< \delta ^{2}, \end{aligned} $$

which shows that $\psi (\alpha ,\beta )\in S_{\delta }$ for all $(\alpha ,\beta )\in \overline{D}$.

Therefore, according to (b), we have $J(\eta (1,\psi (\alpha ,\beta )))< m- \varepsilon$. Hence (4.4) holds.

In the following, we prove that

$$ \eta \bigl(1,\psi (D)\bigr)\cap \mathcal{M}\neq \emptyset , $$

(4.5)

which is a contradiction to the definition of m.

Let $\gamma (\alpha ,\beta ):=\eta (1,\psi (\alpha ,\beta ))$ and

$$\begin{aligned}& \begin{aligned}[b] \varPhi _{0}(\alpha ,\beta )&:=\bigl(\bigl\langle J'\bigl(\psi (\alpha ,\beta )\bigr),u^{+} \bigr\rangle ,\bigl\langle J'\bigl(\psi (\alpha ,\beta ) \bigr),u^{-}\bigr\rangle \bigr) \\ &=\bigl(\bigl\langle J'\bigl(\alpha u^{+}+\beta u^{-}\bigr),u^{+}\bigr\rangle ,\bigl\langle J' \bigl(\alpha u^{+}+ \beta u^{-}\bigr),u^{+}\bigr\rangle \bigr), \end{aligned} \\& \begin{aligned}[b] \varPhi _{1}(\alpha ,\beta )&:=\biggl( \frac{1}{\alpha }\bigl\langle J'\bigl(\gamma ( \alpha ,\beta ) \bigr),\gamma ^{+}(\alpha ,\beta )\bigr\rangle ,\frac{1}{\beta } \bigl\langle J'\bigl(\gamma (\alpha ,\beta )\bigr),\gamma ^{-}(\alpha ,\beta )\bigr\rangle \biggr). \end{aligned} \end{aligned}$$

Thanks to $(f_{5})$, we have that $f'(u)u^{2}\geq 3f(u)u$. Then, by direct calculation, we get

$$\begin{aligned}& \begin{aligned}[b] A_{11}&:=\frac{\partial \langle J'(\alpha u^{+}+\beta u^{-}),u^{+} \rangle }{\partial \alpha }\bigg|_{(1,1)} \\ &= \bigl\Vert u^{+} \bigr\Vert ^{2}_{E}+3 \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x+ \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x- \int _{\mathbb{R}^{3}}K(x)f'\bigl(u^{+}\bigr) \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x \\ &=-2 \bigl\Vert u ^{+} \bigr\Vert ^{2}_{E}+3 \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u ^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y-2 \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad{}- \int _{\mathbb{R}^{3}}K(x) \bigl(f'\bigl(u^{+} \bigr) \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x-3f \bigl(u^{+}\bigr)u ^{+}\bigr)\,\mathrm{d}x \\ &< \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u ^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y-2 \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x< 0, \end{aligned} \\& \begin{aligned}[b] A_{12}&:=\frac{\partial \langle J'(\alpha u^{+}+\beta u^{-}),u^{+} \rangle }{\partial \beta }\bigg|_{(1,1)} \\ &=- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{u^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x \,\mathrm{d}y+2 \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{+} \bigr\vert ^{2} \,\mathrm{d}x, \end{aligned} \\& \begin{aligned}[b] A_{21}&:=\frac{\partial \langle J'(\alpha u^{+}+\beta u^{-}),u^{-} \rangle }{\partial \alpha }\bigg|_{(1,1)} \\ &=- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{u^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x \,\mathrm{d}y+2 \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{-} \bigr\vert ^{2} \,\mathrm{d}x, \end{aligned} \end{aligned}$$

and

$$ \begin{aligned}[b] A_{22}&:=\frac{\partial \langle J'(\alpha u^{+}+\beta u^{-}),u^{-} \rangle }{\partial \beta }\bigg|_{(1,1)} \\ &= \bigl\Vert u^{-} \bigr\Vert ^{2}_{E}+3 \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x+ \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x- \int _{\mathbb{R}^{3}}K(x)f'\bigl(u^{-}\bigr) \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x \\ &=-2 \bigl\Vert u ^{-} \bigr\Vert ^{2}_{E}+3 \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u ^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y-2 \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad{}- \int _{\mathbb{R}^{3}}K(x) \bigl(f'\bigl(u^{-} \bigr) \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x-3f \bigl(u^{-}\bigr)u ^{-}\bigr)\,\mathrm{d}x \\ &< \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u ^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y-2 \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x< 0. \end{aligned} $$

Thus

$$J_{{\Phi }_0(1,1)}={\left|\begin{array}{cc}\frac{\partial 〈J^{\prime }(\alpha u^{+}+\beta u^{-}),u^{+}〉}{\partial \alpha }& \frac{\partial 〈J^{\prime }(\alpha u^{+}+\beta u^{-}),u^{+}〉}{\partial \beta }\\ \frac{\partial 〈J^{\prime }(\alpha u^{+}+\beta u^{-}),u^{-}〉}{\partial \alpha }& \frac{\partial 〈J^{\prime }(\alpha u^{+}+\beta u^{-}),u^{-}〉}{\partial \beta }\end{array}\right|}_{(1,1)}=A_{11}{A}_{22}-A_{12}{A}_{21}>0,$$

which implies that $\operatorname{deg}(\varPhi _{0},D,0)=1$.

So, combining (4.3) with (a), we obtain $\psi =\gamma$ on ∂D. Consequently, we obtain $\operatorname{deg}(\varPhi _{1},D,0)= \operatorname{deg}(\varPhi _{0},D,0)=1$. Therefore, $\varPhi _{1}(\alpha _{0},\beta _{0})=0$ for some $(\alpha _{0},\beta _{0})\in D$, so that $\eta (1, \psi (\alpha _{0},\beta _{0}))=\gamma (\alpha _{0},\beta _{0})\in \mathcal{M}$, which contradicts (4.4).

From the above discussion, we conclude that u is a sign-changing solution for system (1.1). □