Multiple positive solutions for a fractional elliptic system with critical nonlinearities

Abstract

In this paper, we study the multiplicity results of positive solutions for a fractional elliptic system involving both concave-convex and critical growth terms. With the help of the Nehari manifold and the Ljusternik-Schnirelmann category, we prove that the problem admits at least $\operatorname{cat}(\Omega)+1$ distinct positive solutions.

1 Introduction and the main result

In this paper, we are concerned with the number of positive solutions of the fractional elliptic system:

$$ (\mathrm{E}_{\lambda,\mu})\quad \left \{ \textstyle\begin{array}{l@{\quad}l} (-\Delta)^{\frac{s}{2}} u=\lambda|u|^{q-2}u+\frac{\alpha}{\alpha +\beta}|u|^{\alpha-2}u|v|^{\beta} & \text{in } \Omega, \\ (-\Delta)^{\frac{s}{2}} v=\mu|v|^{q-2}v+\frac{\beta}{\alpha+\beta }|u|^{\alpha} |v|^{\beta-2}v & \text{in } \Omega, \\ u=v=0& \text{on } \partial\Omega, \end{array}\displaystyle \right . $$

where Ω is a bounded set in $\mathbb{R}^{N}$ with smooth boundary, $N>s$ with $s\in(0,2)$ fixed, $1< q<2$, $\lambda,\mu>0$, $\alpha,\beta>1$ satisfy $\alpha+\beta=2_{s}^{*}=\frac{2N}{N-s}$, $2_{s}^{*}$ is the fractional Sobolev critical exponent, and $(-\Delta )^{\frac{s}{2}}$ is the fractional Laplacian. These types of operators are the infinitesimal generators of Lévy stable diffusion process and arise in anomalous diffusions in plasmas, flames propagation and chemical reactions in Liquids, population dynamics, geophysical fluid dynamics, and American options in finance; see [1, 2].

In recent years, a great deal of attention has been focused on studying of problems involving fractional Sobolev spaces and corresponding nonlocal equations, both from a pure mathematical point of view and for concrete applications. We refer to [3–7] for the subcritical case and to [8–12] for the critical case. In particular, set $\alpha+\beta =p\leq2_{s}^{*}$, $\lambda=\mu$, and $u=v$, ($\mathrm{E}_{\lambda,\mu}$) reduces to the following fractional elliptic equation with concave-convex nonlinearities:

$$ (\mathrm{E}_{\lambda})\quad \left \{ \textstyle\begin{array}{l@{\quad}l} (-\Delta)^{\frac{s}{2}} u=\lambda|u|^{q-2}u+|u|^{p-2}u& \text{in } \Omega, \\ u=0 &\text{on } \partial\Omega, \end{array}\displaystyle \right . $$

Goyal and Sreenadh [13] studied the existence and multiplicity of non-negative solutions to ($\mathrm{E}_{\lambda}$). Moreover, by Nehari manifold and fibering maps, Chen and Deng [9] obtained the existence of multiple solutions to ($\mathrm{E}_{\lambda}$) for subcritical case and critical case. For the fractional Laplacian system with concave-convex nonlinearities, He et al. [14] proved that ($\mathrm{E}_{\lambda ,\mu}$) permits at least two positive solutions when the pair of parameters $(\lambda,\mu)$ belongs to a certain subset of $\mathbb {R}^{2}$. Similar results were taken by Chen and Deng [15]. The tool of them is the decomposition of the Nehari manifold.

Motivated by the results mentioned above, the purpose of this article is to get a better information on the number of positive solutions of ($\mathrm{E}_{\lambda,\mu}$), for $\lambda,\mu>0$ small enough, via the tools of the variational theory and the Ljusternik-Schnirelmann category theory. We refer the reader to [16–21] for similar results to ($\mathrm{E}_{\lambda,\mu}$) for Laplacian operator. Our main result can be stated as follows.

Theorem 1.1

There exists $\Lambda_{*}>0$ such that if $\lambda,\mu\in(0,\Lambda _{*})$, ($\mathrm{E}_{\lambda,\mu}$) has at least $\operatorname{cat}(\Omega)+1$ distinct positive solutions. Here $\operatorname{cat}(\Omega)$ denotes the Ljusternik-Schnirelmann category of Ω in itself.

Remark 1.1

If Ω is a general domain, $\operatorname{cat}(\Omega)\geq1$, and Theorem 1.1 is the main result of [14, 15].

Remark 1.2

Concerning regularity, one can get an a priori estimate for the solutions to ($\mathrm{E}_{\lambda,\mu}$) and hence obtain, as in [22], Proposition 5.2, $u,v\in C^{\infty}(\overline{\Omega})$ for $s=1$, $u,v\in C^{0,s}(\overline{\Omega})$ if $0< s<1$ and $u,v\in C^{1,s-1}$ if $1< s<2$.

This paper is organized as follows: In Section 2, we introduce some notations and preliminaries. In Section 3, we give some technical results which are crucial to the proof of Theorem 1.1. In Section 4, we give the proof of Theorem 1.1.

2 Notations and preliminaries

In this section, we collect preliminary facts for future reference. First of all, let us write the standard notations which we will use in this paper. We denote the upper half-space in $\mathbb {R}^{N+1}_{+}$ by

$$ \mathbb{R}^{N+1}_{+}:=\bigl\{ (x,y);(x_{1},x_{2}, \ldots x_{N},y)\in\mathbb {R}^{N+1},y>0\bigr\} . $$

Denote the half cylinder with base Ω by $C_{\Omega}=\Omega \times(0,\infty)\subset\mathbb{R}^{N+1}_{+}$ and its lateral boundary by $\partial_{L}C_{\Omega}=\partial\Omega\times[0,\infty)$. We shall use C ($C_{i}$, $i=1,2,\ldots$) to denote any positive constant.

Let $\varphi_{j}$, $\lambda_{j}$ be the eigenfunctions and eigenvectors of −Δ in Ω with zero Dirichlet boundary data. The fractional Laplacian $(-\Delta)^{\frac{s}{2}}$ is defined in the space of functions

$$ H_{0}^{\frac{s}{2}}(\Omega):=\Biggl\{ u= \sum _{j=1}^{\infty}a_{j}\varphi_{j}\in L^{2}(\Omega);\|u\|_{ H_{0}^{\frac{s}{2}}(\Omega)}= \Biggl( \sum _{j=1}^{\infty}a^{2}_{j} \lambda^{\frac{s}{2}}_{j} \Biggr)^{\frac{1}{2}}< \infty\Biggr\} , $$

and $\|u\|_{ H_{0}^{\frac{s}{2}}(\Omega)}=\|(-\Delta)^{\frac{s}{4}}u\| _{L^{2}(\Omega)}$. The dual space $H^{-\frac{s}{2}}(\Omega)$ is defined in the standard way as well as the inverse operator $(-\Delta )^{-\frac{s}{2}}$.

Definition 2.1

We say that $(u,v)\in H_{0}^{\frac{s}{2}}(\Omega)\times H_{0}^{\frac {s}{2}}(\Omega)$ is a solution of ($\mathrm{E}_{\lambda,\mu}$) if the identity

$$\begin{aligned}& \int_{\Omega}(-\Delta)^{\frac{s}{4}}u(-\Delta)^{\frac {s}{4}} \varphi_{1} +(-\Delta)^{\frac{s}{4}}v(-\Delta)^{\frac {s}{4}} \varphi_{2}\,dx \\& \quad = \int_{\Omega}\bigl(\lambda |u|^{q-2}u\varphi_{1}+ \mu|v|^{q-2}v\varphi_{2}\bigr)\,dx +\frac{\alpha}{\alpha+\beta} \int_{\Omega}|u|^{\alpha-2}u|v|^{\beta}\varphi_{1}\,dx \\& \qquad {}+\frac{\beta}{\alpha+\beta } \int_{\Omega}|u|^{\alpha}|v|^{\beta-2}v\varphi _{2}\,dx \end{aligned}$$

holds for all $(\varphi_{1},\varphi_{2})\in H_{0}^{\frac{s}{2}}(\Omega )\times H_{0}^{\frac{s}{2}}(\Omega)$.

Associated with ($\mathrm{E}_{\lambda,\mu}$) we consider the energy functional

$$\begin{aligned} J_{\lambda,\mu}(u,v) :=&\frac{1}{2} \int_{\Omega}\bigl(\bigl\vert (-\Delta)^{\frac{s}{4}}u\bigr\vert ^{2}+\bigl\vert (-\Delta)^{\frac{s}{4}}v\bigr\vert ^{2}\bigr)\,dx \\ &{}-\frac {1}{q} \int_{\Omega}\bigl(\lambda|u|^{q}+\mu|v|^{q} \bigr)\,dx-\frac {1}{2_{s}^{*}} \int_{\Omega}|u|^{\alpha}|v|^{\beta}\,dx. \end{aligned}$$

This functional is well defined in $H_{0}^{\frac{s}{2}}(\Omega)\times H_{0}^{\frac{s}{2}}(\Omega)$, and, moreover, the critical points of $J_{\lambda,\mu}$ correspond to weak solutions of ($\mathrm{E}_{\lambda,\mu}$).

To treat the nonlocal problem ($\mathrm{E}_{\lambda,\mu}$), we will study a corresponding extension problem, which allows us to investigate ($\mathrm{E}_{\lambda,\mu}$) by studying a local problem via classical variational methods. We define the extension operator and fractional Laplacian for functions in $H_{0}^{\frac{s}{2}}(\Omega)$.

Definition 2.2

Given a function $u\in H_{0}^{\frac{s}{2}}(\Omega)$, we define its s-harmonic extension $\omega=E_{s}(u)$ to the cylinder $C_{\Omega}$ as a solution to the problem

$$ \left \{ \textstyle\begin{array}{l@{\quad}l} \operatorname{div}(y^{1-s}\nabla\omega)=0 & \text{in } C_{\Omega}, \\ \omega=0 & \text{on } \partial_{L} C_{\Omega}, \\ \omega=u & \text{on } \Omega\times\{0\}, \end{array}\displaystyle \right . $$

and

$$ (-\Delta)^{\frac{s}{2}}u(x)=-K_{s} \lim_{y\rightarrow 0^{+}}y^{1-s} \frac{\partial\omega}{\partial y}(x,y), $$

where $K_{s}$ is a normalization constant.

The extension function $\omega(x,y)$ belongs to the space $H_{0,L}^{s}(C_{\Omega})=\overline{C_{0}^{\infty}(\Omega\times[0,\infty ))}$, with

$$ \|\omega\|_{H_{0,L}^{s}(C_{\Omega})}= \biggl(K_{s} \int _{C_{\Omega}} y^{1-s}|\nabla\omega|^{2}\,dx\,dy \biggr)^{\frac{1}{2}}. $$

The extension operator is an isometry between $H_{0}^{\frac{s}{2}}(\Omega )$ and $H_{0,L}^{s}(C_{\Omega})$, namely,

$$ \|\omega\|_{H_{0,L}^{s}(C_{\Omega})}=\|u\|_{H_{0}^{\frac{s}{2}}(\Omega )}, \quad \forall u\in H_{0}^{\frac{s}{2}}(\Omega). $$

(2.1)

With this extension, we can transform ($\mathrm{E}_{\lambda,\mu}$) into the following local problem:

$$ (\widehat{\mathrm{E}}_{\lambda,\mu})\quad \left \{ \textstyle\begin{array}{l@{\quad}l} -\operatorname{div}(y^{1-s}\nabla\omega_{1})=0, \qquad -\operatorname{div}(y^{1-s}\nabla\omega_{1})=0& \text{in } C_{\Omega}, \\ \omega_{1}=\omega_{2}=0 &\text{on } \partial_{L} C_{\Omega}, \\ \frac{\partial\omega_{1}}{\partial\nu^{s}}=\lambda|\omega _{1}|^{q-2}\omega_{1}+\frac{\alpha}{\alpha+\beta}|\omega_{1}|^{\alpha -2}\omega_{1}|\omega_{2}|^{\beta} & \text{on } C_{\Omega}\times\{0\}, \\ \frac{\partial\omega_{2}}{\partial\nu^{s}}=\lambda|\omega _{2}|^{q-2}\omega_{1}+\frac{\beta}{\alpha+\beta}|\omega_{1}|^{\alpha}|\omega_{2}|^{\beta-2}\omega_{2} & \text{on } C_{\Omega}\times\{0\}, \\ \omega_{1}=u, \qquad \omega_{2}=v&\text{on } C_{\Omega}\times\{0\}, \end{array}\displaystyle \right . $$

where

$$ \frac{\partial\omega_{i}}{\partial\nu^{s}}:=-K_{s} \lim_{y\rightarrow0^{+}}y^{1-s} \frac{\partial\omega_{i}}{\partial y}, \quad i=1,2. $$

In the following, we will study ($\widehat{\mathrm{E}}_{\lambda,\mu}$) in the framework of the Sobolev space $H=H_{0,L}^{s}(C_{\Omega})\times H_{0,L}^{s}(C_{\Omega})$ using the standard norm

$$ \bigl\Vert (\omega_{1},\omega_{2})\bigr\Vert _{H}= \biggl(K_{s} \int_{\Omega}y^{1-s}\bigl(|\nabla\omega_{1}|^{2}+| \nabla\omega_{2}|^{2}\bigr)\,dx\,dy \biggr)^{\frac{1}{2}}. $$

An energy solution to ($\widehat{\mathrm{E}}_{\lambda,\mu}$) is a function $(\omega_{1},\omega_{2})\in H$ satisfying

$$\begin{aligned}& K_{s} \int_{C_{\Omega}} y^{1-s}\nabla\omega_{1}\nabla \varphi_{1} \,dx\,dy+ K_{s} \int_{C_{\Omega}} y^{1-s}\nabla \omega_{2}\nabla \varphi_{2} \,dx\,dy \\& \quad = \int_{\Omega\times\{0\}}\bigl(\lambda|\omega _{1}|^{q-2} \omega_{1}\varphi_{1}+\mu|\omega_{2}|^{q-2} \omega_{2}\varphi_{2}\bigr)\,dx \\& \qquad {}+\frac{\alpha}{\alpha+\beta} \int_{\Omega \times\{0\}} |\omega_{1}|^{\alpha-2} \omega_{1}|\omega_{2}|^{\beta}\varphi _{1} \,dx+\frac{\beta}{\alpha+\beta} \int_{\Omega\times\{ 0\}} |\omega_{1}|^{\alpha}| \omega_{2}|^{\beta-2}\omega_{2}\varphi _{2}\,dx \end{aligned}$$

for all $(\varphi_{1},\varphi_{2})\in H$.

If $(\omega_{1},\omega_{2})$ satisfies ($\widehat{\mathrm{E}}_{\lambda,\mu}$), then the trace $(u,v)=(\omega_{1}(\cdot,0),\omega_{2}(\cdot,0))$ is a solution of ($\mathrm{E}_{\lambda,\mu}$). The converse is also true. Therefore, both formulations are equivalent.

The associated energy functional to ($\widehat{\mathrm{E}}_{\lambda,\mu}$) is

$$\begin{aligned} I_{\lambda,\mu}(\omega_{1},\omega_{2}) =&\frac{1}{2} \bigl\Vert (\omega_{1},\omega _{2})\bigr\Vert _{H}^{2}-\frac{1}{q} \int_{\Omega\times\{0\}}\bigl(\lambda |\omega_{1}|^{q}+ \mu|\omega_{2}|^{q}\bigr)\,dx \\ &{}-\frac{1}{2_{s}^{*}} \int _{\Omega\times\{0\}} |\omega_{1}|^{\alpha}| \omega_{2}|^{\beta}\,dx. \end{aligned}$$

Clearly, critical points of $I_{\lambda,\mu}$ in H correspond to critical points of $J_{\lambda,\mu}$ in $H_{0}^{\frac{s}{2}}(\Omega )\times H_{0}^{\frac{s}{2}}(\Omega)$.

In the following lemmas, we will list some relevant inequalities from [14, 15].

Lemma 2.1

For every $1\leq r\leq2_{s}^{*}$, and every $\omega\in H_{0,L}^{s}(C_{\Omega})$, we have

$$ \biggl( \int_{\Omega\times\{0\}}|\omega|^{r}\,dx \biggr)^{\frac{2}{r}}\leq C \int_{C_{\Omega}} y^{1-s}|\nabla \omega|^{2}\,dx\,dy $$

(2.2)

for some positive constant C. Furthermore, the space $H_{0,L}^{s}(C_{\Omega})$ is compactly embedded into $L^{r}(\Omega)$, for every $r<2_{s}^{*}$.

Remark 2.1

When $r=2_{s}^{*}$, the best constant is denoted by $S(s,N)$, that is,

$$ S(s,N):= \inf_{\omega\in H_{0,L}^{s}(C_{\Omega})\backslash \{0\}}\frac{ \int_{C_{\Omega}} y^{1-s}|\nabla\omega |^{2}\,dx\,dy}{ ( \int_{\Omega\times\{0\}} |\omega |^{2_{s}^{*}}\,dx )^{\frac{2}{2_{s}^{*}}}}. $$

(2.3)

It is not achieved in any bounded domain and, for all $\omega\in H^{s}(\mathbb{R}_{+}^{N+1})$,

$$ S(s,N) \biggl( \int_{\mathbb{R}^{N}\times\{0\}} |\omega |^{2_{s}^{*}}\,dx \biggr)^{\frac{1}{2_{s}^{*}}} \leq \int_{\mathbb {R}_{+}^{N+1}} y^{1-s}|\nabla\omega|^{2}\,dx\,dy, $$

(2.4)

$S(s,N)$ is achieved for $\Omega=\mathbb{R}^{N}$ by the functions $\omega_{\varepsilon}$ which are the s-harmonic extensions of

$$ u_{\varepsilon}(x):=\frac{\varepsilon^{\frac{(N-s)}{2}}}{(\varepsilon ^{2}+|x|^{2})^{\frac{(N-s)}{2}}},\quad \varepsilon>0, x\in \mathbb{R}^{N}. $$

(2.5)

The constant $S(s,N)$ given in (2.3) takes the exact value

$$ S(s,N)=\frac{2\pi^{\frac{s}{2}}\Gamma(\frac{2-s}{2})\Gamma(\frac {N+s}{2})(\Gamma(\frac{N}{2}))^{\frac{s}{N}}}{\Gamma(\frac {s}{2})\Gamma(\frac{N-s}{2})(\Gamma(N))^{\frac{s}{N}}}, $$

and it is achieved for $\Omega=\mathbb{R}^{N}$ by the functions $\omega_{\varepsilon}=E_{s}(u_{\varepsilon})$.

We consider the following minimization problem:

$$ S_{s,\alpha,\beta}:= \inf_{(\omega_{1},\omega_{2})\in H\backslash\{(0,0)\}}\frac{ \int_{C_{\Omega}} y^{1-s}(|\nabla\omega_{1}|^{2}+|\nabla\omega_{2}|^{2})\,dx\,dy}{ ( \int_{\Omega\times\{0\}} |\omega_{1}|^{\alpha}|\omega _{2}|^{\beta}\,dx )^{\frac{2}{2_{s}^{*}}}}. $$

(2.6)

From [14], we establish a relationship between $S(s,N)$ and $S_{s,\alpha,\beta}$.

Lemma 2.2

For the constants $S(s,N)$ and $S_{s,\alpha,\beta}$ introduced in (2.3) and (2.6), we have

$$ S_{s,\alpha,\beta}= \biggl( \biggl(\frac{\alpha}{\beta} \biggr)^{\frac{\beta}{\alpha+\beta}}+ \biggl(\frac{\beta}{\alpha} \biggr)^{\frac{\alpha}{\alpha+\beta}} \biggr)S(s,N). $$

In particular, the constant $S_{s,\alpha,\beta}$ is achieved for $\Omega=\mathbb{R}^{N}$.

To proceed, we introduce the Nehari manifold of $I_{\lambda,\mu}$ by setting

$$ N_{\lambda,\mu}=\bigl\{ (\omega_{1},\omega_{2})\in H \backslash\bigl\{ (0,0)\bigr\} ;I'_{\lambda,\mu}( \omega_{1},\omega_{2}) (\omega_{1}, \omega_{2})=0\bigr\} . $$

This enables us to construct homotopies between Ω and certain levels of $I_{\lambda,\mu}$. Clearly, $(\omega_{1},\omega_{2})\in N_{\lambda,\mu}$ if and only if

$$ \bigl\Vert (\omega_{1},\omega_{2})\bigr\Vert _{H}^{2}= \int_{\Omega\times\{0\} }\bigl(\lambda|\omega_{1}|^{q}+\mu| \omega_{2}|^{q}\bigr)\,dx+ \int _{\Omega\times\{0\}}|\omega_{1}|^{\alpha}| \omega_{2}|^{\beta}\,dx. $$

On the Nehari manifold $N_{\lambda,\mu}$, from Lemma 2.1 and the Young inequality, we have

$$\begin{aligned} I_{\lambda,\mu}(\omega_{1},\omega_{2})&= \biggl( \frac{1}{2}-\frac {1}{2_{s}^{*}} \biggr)\bigl\Vert (\omega_{1}, \omega_{2})\bigr\Vert _{H}^{2}- \biggl( \frac {1}{q}-\frac{1}{2_{s}^{*}} \biggr) \int_{\Omega\times\{0\} }\bigl(\lambda|\omega_{1}|^{q}+\mu| \omega_{2}|^{q}\bigr)\,dx \\ &\geq \biggl(\frac{1}{2}-\frac{1}{2_{s}^{*}} \biggr)\bigl\Vert ( \omega_{1},\omega _{2})\bigr\Vert _{H}^{2}- \biggl(\frac{1}{q}-\frac{1}{2_{s}^{*}} \biggr) (\lambda+\mu)C\bigl\Vert ( \omega_{1},\omega_{2})\bigr\Vert _{H}^{q} \end{aligned}$$

(2.7)

$$\begin{aligned} &\geq-(\lambda+\mu)^{2/(2-q)}C, \end{aligned}$$

(2.8)

where C denotes positive constants (possibly different) independent of $(\omega_{1},\omega_{2})\in H$. Let

$$\begin{aligned} \psi_{\lambda,\mu}(\omega_{1},\omega_{2})&:=I'_{\lambda,\mu}( \omega _{1},\omega_{2}) (\omega_{1}, \omega_{2}) \\ &=\bigl\Vert (\omega_{1},\omega_{2})\bigr\Vert _{H}^{2}- \int_{\Omega\times\{ 0\}}\bigl(\lambda|\omega_{1}|^{q}+\mu| \omega_{2}|^{q}\bigr)\,dx- \int _{\Omega\times\{0\}}|\omega_{1}|^{\alpha}| \omega_{2}|^{\beta}\,dx. \end{aligned}$$

(2.9)

Then, for $(\omega_{1},\omega_{2})\in N_{\lambda,\mu}$,

$$\begin{aligned} \psi'_{\lambda,\mu}(\omega_{1},\omega_{2}) ( \omega_{1},\omega _{2}) =&(2-q)\bigl\Vert ( \omega_{1},\omega_{2})\bigr\Vert _{H}^{2}- \bigl(2_{s}^{*}-q\bigr) \int _{\Omega\times\{0\}}|\omega_{1}|^{\alpha}| \omega_{2}|^{\beta}\,dx \end{aligned}$$

(2.10)

$$\begin{aligned} =&\bigl(2-2_{s}^{*}\bigr)\bigl\Vert (\omega_{1}, \omega_{2})\bigr\Vert _{H}^{2} \\ &{}+ \bigl(2_{s}^{*}-q\bigr) \int _{\Omega\times\{0\}}\bigl(\lambda|\omega_{1}|^{q}+ \mu|\omega_{2}|^{q}\bigr)\,dx. \end{aligned}$$

(2.11)

Similar to the method used in [14, 15], we split $N_{\lambda,\mu}$ into three parts:

$$\begin{aligned}& N_{\lambda,\mu}^{+} =\bigl\{ (\omega_{1},\omega_{2})\in N_{\lambda,\mu}; \psi '_{\lambda,\mu}(\omega_{1}, \omega_{2}) (\omega_{1},\omega_{2})>0\bigr\} ; \\& N_{\lambda,\mu}^{0} =\bigl\{ (\omega_{1}, \omega_{2})\in N_{\lambda,\mu}; \psi'_{\lambda,\mu}( \omega_{1},\omega_{2}) (\omega_{1}, \omega_{2})=0\bigr\} ; \\& N_{\lambda,\mu}^{-} =\bigl\{ (\omega_{1},\omega_{2})\in N_{\lambda,\mu}; \psi'_{\lambda,\mu}(\omega_{1}, \omega_{2}) (\omega_{1},\omega_{2})< 0\bigr\} . \end{aligned}$$

In the sequel, we shall use $\Lambda_{*}$ to denote different small parameters. Then we have the following results.

Lemma 2.3

Suppose that $(\omega_{1},\omega_{2})$ is a local minimizer for $I_{\lambda,\mu}$ on $N_{\lambda,\mu}$. Then, if $(\omega_{1},\omega _{2})\notin N_{\lambda,\mu}^{0}$, $(\omega_{1},\omega_{2})$ is a critical point of $I_{\lambda,\mu}$.

Lemma 2.4

There exists $\Lambda_{*}>0$ such that, for each $\lambda,\mu\in (0,\Lambda_{*})$, we have $N_{\lambda,\mu}^{0}=\emptyset$.

By Lemma 2.4, for $\lambda,\mu\in(0,\Lambda_{*})$, we write $N_{\lambda,\mu}=N_{\lambda,\mu}^{+}\cup N_{\lambda,\mu}^{-}$ and define

$$ \alpha_{\lambda,\mu}^{+}= \inf_{(\omega_{1},\omega_{2})\in N_{\lambda,\mu}^{+}} I_{\lambda,\mu}( \omega_{1},\omega_{2});\qquad \alpha _{\lambda,\mu}^{-}= \inf _{(\omega_{1},\omega_{2})\in N_{\lambda,\mu}^{-}} I_{\lambda,\mu}(\omega_{1}, \omega_{2}). $$

Set

$$ t_{\mathrm{max}}= \biggl(\frac{(2-q)\|(\omega_{1},\omega_{2})\| _{H}^{2}}{(2_{s}^{*}-q) \int_{\Omega\times\{0\}}|\omega _{1}|^{\alpha}|\omega_{2}|^{\beta}\,dx} \biggr)^{\frac{1}{2_{s}^{*}-2}}>0. $$

Then we have the following result.

Lemma 2.5

For each $(\omega_{1},\omega_{2})\in H$ with $\int_{\Omega \times\{0\}}|\omega_{1}|^{\alpha}|\omega_{2}|^{\beta}\,dx>0$, there exist unique $0< t^{+}< t_{\mathrm{max}}< t^{-}$ such that $(t^{+}\omega_{1},t^{+}\omega_{2})\in N_{\lambda,\mu}^{+}$, $(t^{+}\omega_{1},t^{+}\omega_{2})\in N_{\lambda,\mu }^{+}$, $(t^{-}\omega_{1},t^{-}\omega_{2})\in N_{\lambda,\mu}^{-}$ and

$$ I_{\lambda,\mu}\bigl(t^{+}\omega_{1},t^{+}\omega_{2}\bigr)= \inf_{0\leq t\leq t_{\mathrm{max}}} I_{\lambda,\mu}(t\omega_{1},t \omega_{2});\qquad I_{\lambda ,\mu}\bigl(t^{-}\omega_{1},t^{-} \omega_{2}\bigr)= \sup_{t\geq0} I_{\lambda,\mu}(t \omega_{1},t\omega_{2}). $$

Lemma 2.6

If $\lambda,\mu\in(0,\Lambda_{*})$, then

• (i) $\alpha_{\lambda,\mu}^{+}<0$,

• (ii) $\alpha_{\lambda,\mu}^{-}\geq\delta$ for some $\delta>0$.

For the proofs of Lemmas 2.3-2.6, the reader is referred to [14, 15] for similar proofs.

Remark 2.2

From Lemmas 2.5 and 2.6, it is easy to know that if $(\omega_{1},\omega _{2})\in N_{\lambda,\mu}^{-}$,

$$ \int_{\Omega\times\{0\}}|\omega_{1}|^{\alpha}|\omega _{2}|^{\beta}\,dx>0. $$

Next we establish that $I_{\lambda,\mu}$ satisfies the $(\mathit{PS})_{c}$-condition under some restriction on the level of $(\mathit{PS})_{c}$-sequences in the following.

Lemma 2.7

For each $\lambda,\mu\in(0,\Lambda_{*})$, $I_{\lambda,\mu}$ satisfies the $(\mathit{PS})_{c}$-condition for $c\in(-\infty,\alpha_{\lambda ,\mu}^{+}+\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s})$.

Proof

Let $\{(\omega_{1,n},\omega_{2,n})\}\subset H$ be a $(\mathit{PS})_{c}$-sequence for $I_{\lambda,\mu}$ and $c\in(-\infty,\alpha_{\lambda,\mu }^{+}+\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s})$. Note (2.7), it is easy to see that $\{(\omega_{1,n},\omega_{2,n})\}$ is bounded in H. Thus, there exists a subsequence still denoted by $\{ (\omega_{1,n},\omega_{2,n})\}$ and $(\omega_{1},\omega_{2})\in H$ such that $(\omega_{1,n},\omega_{2,n})\rightharpoonup(\omega _{1},\omega_{2})$ weakly in H. Furthermore, we get

$$\begin{aligned} \bullet&\quad \int_{\Omega\times\{0\}}\bigl(\lambda|\omega _{1,n}|^{q}+ \mu|\omega_{2,n}|^{q}\bigr)\,dx= \int_{\Omega\times\{ 0\}}\bigl(\lambda|\omega_{1}|^{q}+\mu| \omega_{2}|^{q}\bigr)\,dx+o(1); \\ \bullet&\quad\bigl\Vert (\omega_{1,n}-\omega_{1}, \omega_{2,n}-\omega_{2})\bigr\Vert _{H}^{2}= \bigl\Vert (\omega_{1,n},\omega_{2,n})\bigr\Vert _{H}^{2}-\bigl\Vert (\omega_{1}, \omega_{2})\bigr\Vert _{H}^{2}+o(1); \\ \bullet&\quad \int_{\Omega\times\{0\}}|\omega_{1,n}-\omega _{1}|^{\alpha}| \omega_{2,n}-\omega_{2}|^{\beta}\,dx \\ &\qquad = \int _{\Omega\times\{0\}}|\omega_{1,n}|^{\alpha}| \omega_{2,n}|^{\beta}\,dx- \int_{\Omega\times\{0\}}|\omega_{1}|^{\alpha}|\omega _{2}|^{\beta}\,dx+o(1). \end{aligned}$$

Moreover, we can obtain $I'_{\lambda,\mu}(\omega_{1},\omega_{2})=0$. Since $I_{\lambda,\mu}(\omega_{1,n},\omega_{2,n})=c+o(1)$ and $I'_{\lambda,\mu}(\omega_{1,n},\omega_{2,n})=o(1)$, we deduce that

$$\begin{aligned}& \frac{1}{2}\bigl\Vert (\omega_{1,n}- \omega_{1},\omega_{2,n}-\omega_{2})\bigr\Vert _{H}^{2}-\frac{1}{2_{s}^{*}} \int_{\Omega\times\{0\}}|\omega _{1,n}-\omega_{1}|^{\alpha}| \omega_{2,n}-\omega_{2}|^{\beta}\,dx \\& \quad =c-I_{\lambda,\mu}( \omega_{1},\omega_{2})+o(1) \end{aligned}$$

(2.12)

and

$$\begin{aligned} \begin{aligned} o(1)&=I'_{\lambda,\mu}(\omega_{1,n}, \omega_{2,n}) (\omega _{1,n}-\omega_{1}, \omega_{2,n}-\omega_{2}) \\ &=\bigl(I'_{\lambda,\mu}(\omega_{1,n}, \omega_{2,n})-I'_{\lambda,\mu }(\omega_{1}, \omega_{2})\bigr) (\omega_{1,n}-\omega_{1}, \omega_{2,n}-\omega _{2}) \\ &=\bigl\Vert (\omega_{1,n}-\omega_{1},\omega_{2,n}- \omega_{2})\bigr\Vert _{H}^{2}- \int_{\Omega\times\{0\}}|\omega_{1,n}-\omega _{1}|^{\alpha}| \omega_{2,n}-\omega_{2}|^{\beta}\,dx+o(1). \end{aligned} \end{aligned}$$

Now we may assume that

$$\begin{aligned}& \bigl\Vert (\omega_{1,n}-\omega_{1},\omega_{2,n}- \omega_{2})\bigr\Vert _{H}^{2}\rightarrow l \quad \text{and} \\& \int_{\Omega\times\{0\}}|\omega _{1,n}-\omega_{1}|^{\alpha}| \omega_{2,n}-\omega_{2}|^{\beta}\,dx\rightarrow l\quad \text{as } n\rightarrow\infty \end{aligned}$$

for some $l\in[0,+\infty)$.

Suppose $l\neq0$. Using (2.6) and passing to the limit as $n\rightarrow\infty$, we have

$$ l\geq K_{s}S_{s,\alpha,\beta}l^{\frac{2}{2_{s}^{*}}}, $$

that is,

$$ l\geq (K_{s}S_{s,\alpha,\beta} )^{N/s}. $$

(2.13)

Then by (2.12)-(2.13) and $(\omega_{1},\omega_{2})\in N_{\lambda,\mu }\cup\{0\}$,

$$ c=I_{\lambda,\mu}(\omega_{1},\omega_{2})+ \biggl( \frac{1}{2}-\frac {1}{2_{s}^{*}} \biggr)l\geq\alpha_{\lambda,\mu}^{+}+ \frac {s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}, $$

which contradicts the definition of c. Hence $l=0$, and the proof is completed. □

Lemma 2.8

For $\lambda,\mu\in(0,\Lambda_{*})$, the functional $I_{\lambda,\mu }$ has a minimizer $((\omega_{1})_{\lambda,\mu}^{+},(\omega_{2})_{\lambda ,\mu}^{+})\in N_{\lambda,\mu}^{+}$ and it satisfies:

• (i) $I_{\lambda,\mu}((\omega_{1})_{\lambda,\mu}^{+},(\omega _{2})_{\lambda,\mu}^{+})=\alpha_{\lambda,\mu}^{+}$;

• (ii) $((\omega_{1})_{\lambda,\mu}^{+},(\omega_{2})_{\lambda ,\mu}^{+})$ is a positive solution of ($\widehat{\mathrm{E}}_{\lambda,\mu}$);

• (iii) $I_{\lambda,\mu}((\omega_{1})_{\lambda,\mu }^{+},(\omega_{2})_{\lambda,\mu}^{+})\rightarrow0$ as $\lambda,\mu \rightarrow0$;

• (iv) $\lim_{\lambda,\mu\rightarrow0}\| ((\omega_{1})_{\lambda,\mu}^{+},(\omega_{2})_{\lambda,\mu}^{+})\|_{H}=0$.

Proof

(i)-(ii) are consequences of [14]. It follows from (2.8) and Lemma 2.6 that

$$ 0>I_{\lambda,\mu}\bigl((\omega_{1})_{\lambda,\mu}^{+},( \omega_{2})_{\lambda ,\mu}^{+}\bigr)\geq-(\lambda+\mu)^{2/(2-q)}C. $$

We obtain $I_{\lambda,\mu}((\omega_{1})_{\lambda,\mu}^{+},(\omega _{2})_{\lambda,\mu}^{+})\rightarrow0$ as $\lambda,\mu\rightarrow0$.

Now we show (iv). By (i)-(iii),

$$\begin{aligned} 0 =& \lim_{\lambda,\mu\rightarrow0}I_{\lambda,\mu }\bigl(( \omega_{1})_{\lambda,\mu}^{+},(\omega_{2})_{\lambda,\mu}^{+} \bigr) \\ =& \lim_{\lambda,\mu\rightarrow0} \biggl(\frac {s}{2N}\bigl\Vert \bigl(( \omega_{1})_{\lambda,\mu}^{+},(\omega_{2})_{\lambda,\mu}^{+} \bigr)\bigr\Vert _{H}^{2} \\ &{}- \biggl(\frac{1}{q}- \frac{1}{2_{s}^{*}} \biggr) \int _{\Omega\times\{0\}}\bigl(\lambda\bigl\vert (\omega_{1})_{\lambda,\mu}^{+} \bigr\vert ^{q}+\mu \bigl\vert (\omega_{1})_{\lambda,\mu}^{+} \bigr\vert ^{q}\bigr)\,dx \biggr). \end{aligned}$$

(2.14)

Since $I_{\lambda,\mu}$ is coercive and bounded below on $N_{\lambda ,\mu}$, $((\omega_{1})_{\lambda,\mu}^{+},(\omega_{2})_{\lambda,\mu }^{+})$ is bounded in H and so that

$$ \lim_{\lambda,\mu\rightarrow0} \int _{\Omega\times\{0\}}\bigl(\lambda\bigl\vert (\omega_{1})_{\lambda,\mu}^{+} \bigr\vert ^{q}+\mu \bigl\vert (\omega_{1})_{\lambda,\mu}^{+} \bigr\vert ^{q}\bigr)\,dx=0. $$

(2.15)

Therefore, we obtain the desired result. □

3 Some technical results

In this section, we shall introduce some useful results which are crucial for the proof of Theorem 1.1.

Lemma 3.1

Let $\{(\omega_{1,n},\omega_{2,n})\}\subset H$ be a non-negative function sequence with

$$ \int_{\Omega\times\{0\}}|\omega_{1,n}|^{\alpha}|\omega _{2,n}|^{\beta}\,dx=1\quad \textit{and}\quad \bigl\Vert ( \omega_{1,n},\omega_{2,n})\bigr\Vert _{H}^{2} \rightarrow K_{s}S_{s,\alpha,\beta}. $$

Then there exists a sequence $\{(y_{n},\varepsilon_{n})\}\subset\mathbb {R}^{N}\times\mathbb{R}^{+}$ such that

$$ \bigl(W_{1,n}(x),W_{2,n}(x)\bigr):=\bigl(E_{s}\bigl( \varepsilon_{n}^{\frac{N-s}{2}}\omega _{1,n}( \varepsilon_{n}x+y_{n},0)\bigr),E_{s}\bigl( \varepsilon_{n}^{\frac {N-s}{2}}\omega_{2,n}( \varepsilon_{n}x+y_{n},0)\bigr)\bigr) $$

contains a convergent subsequence denoted again by $\{ (W_{1,n}(x),W_{2,n}(x))\}$ such that

$$ \bigl(W_{1,n}(x),W_{2,n}(x)\bigr)\rightarrow(W_{1},W_{2}) \quad \textit{in } H. $$

Moreover, we have $\varepsilon_{n}\rightarrow0$ and $y_{n}\rightarrow y\in\overline{\Omega}$ as $n\rightarrow\infty$.

Proof

Let $Z_{n,1}(x)=\omega_{1,n}(x,0)$, $Z_{n,2}(x)=\omega_{2,n}(x,0)$, we have

$$ \int_{\Omega}|Z_{n,1}|^{\alpha}|Z_{n,2}|^{\beta}\,dx=1\quad \text{and}\quad \| Z_{n,1}\|^{2}_{H_{0}^{s}(\Omega)}+ \|Z_{n,1}\|^{2}_{H_{0}^{s}(\Omega)}\rightarrow K_{s}S_{\alpha,\beta} \quad \text{as } n\rightarrow\infty. $$

By the proof of Lemma 2.2, we know that $\{Z_{n,1}\}$ and $\{Z_{n,2}\}$ are minimizing sequences for the critical Sobolev inequality in the form (2.3). Thus we deduce from [23], Theorem 3 and [23], Theorem 5, that there exist a sequence of points $\{y_{n}\}\subseteq\mathbb{R}^{N}$ and a sequence of numbers $\{\varepsilon_{n}\}\subset(0,\infty)$ such that $\widehat{Z}_{n,1}(x)=\varepsilon_{n}^{\frac {N-s}{2}}Z_{n,1}(\varepsilon_{n}x+y_{n})\rightarrow\widehat{Z}_{1}(x)$ and $\widehat{Z}_{n,2}(x)=\varepsilon_{n}^{\frac {N-s}{2}}Z_{n,2}(\varepsilon_{n}x+y_{n})\rightarrow\widehat{Z}_{2}(x)$ in $H^{s}(\mathbb{R}^{N})$ as $n\rightarrow\infty$. Moreover, we have $\varepsilon_{n}\rightarrow0$ and $y_{n}\rightarrow y\in\overline {\Omega}$ as $n\rightarrow\infty$. Denote $W_{1,n}=E_{s}(\widehat {Z}_{n,1})$, $W_{2,n}=E_{s}(\widehat{Z}_{n,2})$ and $W_{1}=E_{s}(\widehat {Z}_{1})$, $W_{2}=E_{s}(\widehat{Z}_{2})$. Then we obtain the result. □

Lemma 3.2

Suppose that X is a Hilbert manifold and $F\in C^{1}(X,\mathbb{R})$. Assume that, for $c_{0}\in\mathbb{R}$ and $K\in\mathbb{N}$:

• (i) $F(x)$ satisfies the $(\mathit{PS})_{c}$ condition for $c\leq c_{0}$,

• (ii) $\operatorname{cat}(\{x\in X;F(x)\leq c_{0}\})\geq K$.

Then $F(x)$ has at least K critical points in $\{x\in X;F(x)\leq c_{0}\}$.

Proof

See [24], Theorem 2.3. □

Up to translations, we may assume that $0\in\Omega$. Moreover, in the following, we fix $r>0$ such that $B_{r}=\{x\in\mathbb{R}^{N};|x|< r\} \subset\Omega$ and the sets

$$ \Omega_{r}^{+}:=\bigl\{ x\in\mathbb{R}^{N}; \operatorname{dist}(x,\Omega)< r\bigr\} , \qquad \Omega_{r}^{-}:=\bigl\{ x \in\Omega;\operatorname{dist}(x,\partial\Omega)>r\bigr\} $$

are both homotopically equivalent to Ω.

Noting Remark 2.2, below we can define the continuous map $\Phi :N_{\lambda,\mu}^{-}\rightarrow\mathbb{R}^{N}$ by setting

$$ \Phi(\omega_{1},\omega_{2}):=\frac{ \int_{\Omega\times\{ 0\}} x|\omega_{1}|^{\alpha}|\omega_{2}|^{\beta}\,dx}{ \int _{\Omega\times\{0\}} |\omega_{1}|^{\alpha}|\omega_{2}|^{\beta}\,dx}. $$

Denote

$$ c_{\lambda,\mu}:=\alpha_{\lambda,\mu}^{+}+\frac {s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s} $$

and

$$ N_{\lambda,\mu}^{-}(c_{\lambda,\mu}):=\bigl\{ (\omega_{1}, \omega_{2})\in N_{\lambda,\mu}^{-};I_{\lambda,\mu}(\omega_{1}, \omega_{2})\leq c_{\lambda,\mu}\bigr\} . $$

Lemma 3.3

There exists $\Lambda_{*}>0$ such that if $\lambda,\mu\in(0,\Lambda _{*})$ and $(\omega_{1},\omega_{2})\in N_{\lambda,\mu}^{-}(c_{\lambda,\mu})$,

$$ \Phi(\omega_{1},\omega_{2})\in\Omega_{r}^{+}. $$

Proof

By way of contradiction, let $\lambda_{n}, \mu_{n}\rightarrow0$, $\{ (\omega_{1,n},\omega_{2,n})\}\subset N_{\lambda_{n},\mu _{n}}^{-}(c_{\lambda_{n},\mu_{n}})$ and $\Phi(\omega_{1,n}, \omega_{2,n})\notin\Omega_{r}^{+}$. From (2.7), we see that $\{(\omega_{1,n},\omega_{2,n})\}$ is bounded and $\int_{\Omega\times\{0\}}(\lambda_{n}|\omega _{1,n}|^{q}+ \mu_{n}|\omega_{2,n}|^{q})\,dx\rightarrow0$. Thus,

$$\begin{aligned} \lim_{n\rightarrow\infty}I_{\lambda_{n},\mu_{n}}(\omega _{1,n},\omega_{2,n})&= \lim_{n\rightarrow\infty} \frac {s}{2N}\bigl\Vert (\omega_{1,n},\omega_{2,n})\bigr\Vert _{H}^{2} \\ &= \lim_{n\rightarrow\infty}\frac {s}{2N} \int_{\Omega\times\{0\}}|\omega_{1,n}|^{\alpha}| \omega_{2,n}|^{\beta}\,dx \\ &\leq\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}. \end{aligned}$$

(3.1)

Defining

$$ (W_{1,n},W_{2,n})= \biggl(\frac{\omega_{1,n}}{ ( \int_{\Omega\times\{0\}}|\omega_{1,n}|^{\alpha}|\omega_{2,n}|^{\beta}\,dx )^{1/(\alpha+\beta)}}, \frac{\omega_{2,n}}{ ( \int_{\Omega\times\{0\}}|\omega_{1,n}|^{\alpha}|\omega_{2,n}|^{\beta}\,dx )^{1/(\alpha+\beta)}} \biggr), $$

we see that $\int_{\Omega\times\{0\}}|W_{1,n}|^{\alpha}|W_{2,n}|^{\beta}\,dx=1$. By (3.1) and the definition of $S_{s,\alpha ,\beta}$, we obtain

$$ \lim_{n\rightarrow\infty}\bigl\Vert (W_{1,n},W_{2,n}) \bigr\Vert _{H}^{2}=K_{s}S_{s,\alpha,\beta}. $$

By Lemma 3.1, there is a sequence $\{(y_{n},\varepsilon_{n})\}\in\mathbb {R}^{N}\times\mathbb{R}^{+}$ such that $\varepsilon_{n}\rightarrow0$, $y_{n}\rightarrow y\in\overline{\Omega }$ and $(E_{s}(\varepsilon_{n}^{\frac{N-s}{2}}W_{1,n}(\varepsilon _{n}x+y_{n})),E_{s}(\varepsilon_{n}^{\frac{N-s}{2}}W_{2,n}(\varepsilon _{n}x+y_{n})))\rightarrow(W_{1},W_{2})$ in H as $n\rightarrow\infty$.

Considering $\varphi\in C_{0}^{\infty}(\mathbb{R}^{N})$ such that $\varphi (x)=x$ in Ω, we infer

$$\begin{aligned}& \Phi(\omega_{1,n},\omega_{2,n}) \\& \quad =\frac{ \int_{\Omega \times\{0\}} x|\omega_{1,n}|^{\alpha}|\omega_{2,n}|^{\beta}\,dx}{ \int_{\Omega\times\{0\}} |\omega_{1,n}|^{\alpha}|\omega_{2,n}|^{\beta}\,dx}= \frac{ \int_{\mathbb {R}^{N}\times\{0\}} \varphi(x)|W_{1,n}|^{\alpha}|W_{2,n}|^{\beta}\,dx}{ \int_{\mathbb{R}^{N}\times\{0\}} |W_{1,n}|^{\alpha}|W_{2,n}|^{\beta}\,dx} \\& \quad =\frac{ \int_{\mathbb{R}^{N}\times\{0\}}\varphi (\varepsilon_{n}x+y_{n})|E_{s}(\varepsilon_{n}^{\frac {N-s}{2}}W_{1,n}(\varepsilon_{n}x+y_{n}))|^{\alpha}|E_{s}(\varepsilon _{n}^{\frac{N-s}{2}}W_{1,n}(\varepsilon_{n}x+y_{n}))|^{\beta}\,dx}{ \int_{\mathbb{R}^{N}\times\{0\}}|E_{s}(\varepsilon _{n}^{\frac{N-s}{2}}W_{1,n}(\varepsilon_{n}x+y_{n}))|^{\alpha}|E_{s}(\varepsilon_{n}^{\frac{N-s}{2}}W_{1,n}(\varepsilon _{n}x+y_{n}))|^{\beta}\,dx}. \end{aligned}$$

Moreover, by the Lebesgue theorem, we have

$$ \frac{ \int_{\mathbb{R}^{N}\times\{0\}}\varphi (\varepsilon_{n}x+y_{n})|E_{s}(\varepsilon_{n}^{\frac {N-s}{2}}W_{1,n}(\varepsilon_{n}x+y_{n}))|^{\alpha}|E_{s}(\varepsilon _{n}^{\frac{N-s}{2}}W_{1,n}(\varepsilon_{n}x+y_{n}))|^{\beta}\,dx}{ \int_{\mathbb{R}^{N}\times\{0\}}|E_{s}(\varepsilon _{n}^{\frac{N-s}{2}}W_{1,n}(\varepsilon_{n}x+y_{n}))|^{\alpha}|E_{s}(\varepsilon_{n}^{\frac{N-s}{2}}W_{1,n}(\varepsilon _{n}x+y_{n}))|^{\beta}\,dx}\rightarrow y\in\overline{\Omega}, $$

as $n\rightarrow\infty$, so that $\lim_{n\rightarrow\infty}\Phi(\omega _{1,n},\omega_{2,n})=y\in\overline{\Omega}$, in contradiction with $\Phi(\omega_{1,n},\omega_{2,n})\notin\Omega_{r}^{+}$. □

Next, we will use $\omega_{\varepsilon}=E_{s}(u_{\varepsilon})$, the family of minimizers to the inequality (2.4), where $u_{\varepsilon}$ is given in (2.5). Let $\eta\in C^{\infty}(C_{\Omega})$, $0\leq\eta (x,y)\leq1$ and for small fixed ρ,

$$ \eta(x,y)=\left \{ \textstyle\begin{array}{l@{\quad}l} 1, & (x,y)\in B_{\frac{\rho}{2}}^{+}:=\{(x_{1},x_{2},\ldots,x_{N},y);\sqrt {x_{1}^{2}+x_{2}^{2}+\cdots+x_{N}^{2}+y^{2}}< \frac{\rho}{2},y>0\}, \\ 0, & (x,y)\notin B_{\rho}^{+}:=\{(x_{1},x_{2},\ldots,x_{N},y);\sqrt {x_{1}^{2}+x_{2}^{2}+\cdots+x_{N}^{2}+y^{2}}< \rho,y>0\}. \end{array}\displaystyle \right . $$

We take $\rho<\frac{r}{4}$ small enough such that

$$ \overline{B_{\rho}^{+}}(x-z,y)\subset\overline{C_{\Omega}} $$

for all $z\in\Omega_{r}^{-}$, where

$$\begin{aligned} \overline{B_{\rho}^{+}}(x-z,y) :=&\bigl\{ (x_{1},x_{2}, \ldots,x_{N},y); \\ &\sqrt {(x_{1}-z_{1})^{2}+(x_{2}-z_{2})^{2}+ \cdots+(x_{N}-z_{N})^{2}+y^{2}}\leq\rho,y \geq0\bigr\} . \end{aligned}$$

Assume

$$ v_{\varepsilon,z}=\eta(x-z,y)\omega_{\varepsilon}(x-z,y)=\eta (x-z,y)E_{s}\bigl(u_{\varepsilon}(x-z)\bigr),\quad z\in \Omega_{r}^{-}, $$

where $u_{\varepsilon}$ is defined in (2.5). We obtain from [11]

$$\begin{aligned}& \|v_{\varepsilon,z}\|_{H^{s}_{0,L}(C_{\Omega})}^{2} =K_{s} \int _{\mathbb{R}_{+}^{N+1}}y^{1-s}|\nabla\omega_{\varepsilon}|^{2}\,dx\,dy+O\bigl(\varepsilon^{N-s}\bigr), \end{aligned}$$

(3.2)

$$\begin{aligned}& \int_{\Omega\times\{0\}}|v_{\varepsilon ,z}|^{2_{s}^{*}}\,dx = \int_{\mathbb{R}^{N}\times\{0\}}|\omega _{\varepsilon}|^{2_{s}^{*}}\,dx+O\bigl( \varepsilon^{N}\bigr)= \int_{\mathbb {R}^{N}} \biggl(\frac{\varepsilon}{\varepsilon^{2}+|x|^{2}} \biggr)^{N}\, dx+O\bigl(\varepsilon^{N}\bigr). \end{aligned}$$

(3.3)

Then we have the following.

Lemma 3.4

There exist $\varepsilon_{0},\sigma(\varepsilon)>0$ such that, for $\varepsilon\in(0,\varepsilon_{0})$ and $\sigma\in(0,\sigma (\varepsilon))$, we have

$$ \sup_{t\geq0}I_{\lambda,\mu}\bigl((\omega_{1})_{\lambda ,\mu}^{+}+t \sqrt{\alpha}v_{\varepsilon,z},(\omega_{2})_{\lambda,\mu }^{+}+t\sqrt{ \beta}v_{\varepsilon,z}\bigr)< c_{\lambda,\mu}-\sigma \quad \textit{uniformly in } z\in\Omega_{r}^{-}, $$

where $((\omega_{1})_{\lambda,\mu}^{+},(\omega_{2})_{\lambda,\mu}^{+})$ is a local minimum in Lemma  2.8. Furthermore, there exists $t_{(\lambda,\mu,\varepsilon,z)}^{-}>0$ such that

$$ \bigl((\omega_{1})_{\lambda,\mu}^{+}+t_{(\lambda,\mu,\varepsilon ,z)}^{-}\sqrt{ \alpha}v_{\varepsilon,z},(\omega_{2})_{\lambda,\mu }^{+}+t_{(\lambda,\mu,\varepsilon,z)}^{-} \sqrt{\beta}v_{\varepsilon ,z}\bigr)\in N_{\lambda,\mu}^{-}(c_{\lambda,\mu}-\sigma) $$

and

$$ \Phi\bigl((\omega_{1})_{\lambda,\mu}^{+}+t_{(\lambda,\mu,\varepsilon ,z)}^{-}\sqrt{ \alpha}v_{\varepsilon,z},(\omega_{2})_{\lambda,\mu }^{+}+t_{(\lambda,\mu,\varepsilon,z)}^{-} \sqrt{\beta}v_{\varepsilon ,z}\bigr)\in\Omega_{r}^{+}. $$

Proof

Since

$$\begin{aligned}& I_{\lambda,\mu}\bigl((\omega_{1})_{\lambda,\mu}^{+}+t \sqrt{\alpha }v_{\varepsilon,z},(\omega_{2})_{\lambda,\mu}^{+}+t\sqrt{ \beta }v_{\varepsilon,z}\bigr) \\& \quad =\frac{K_{s}}{2} \int_{C_{\Omega}} y^{1-s}\bigl(\bigl\vert \nabla \bigl(( \omega_{1})_{\lambda,\mu}^{+}+t\sqrt{\alpha}v_{\varepsilon ,z}\bigr)\bigr\vert ^{2}+\bigl\vert \nabla\bigl((\omega_{2})_{\lambda,\mu}^{+}+t \sqrt{\beta }v_{\varepsilon,z}\bigr)\bigr\vert ^{2} \bigr)\,dx\,dy \\& \qquad {}-\frac{1}{q} \int_{\Omega\times\{0\}}\bigl(\lambda \bigl\vert (\omega_{1})_{\lambda,\mu}^{+}+t \sqrt{\alpha}\eta (x-z,0)u_{\varepsilon}(x-z)\bigr\vert ^{q} \\& \qquad {}+\mu \bigl\vert (\omega_{2})_{\lambda,\mu}^{+}+t\sqrt {\beta} \eta(x-z,0)u_{\varepsilon}(x-z)\bigr\vert ^{q}\bigr)\,dx \\& \qquad {}-\frac{1}{2_{s}^{*}} \int_{\Omega\times\{0\}}\bigl\vert (\omega _{1})_{\lambda,\mu}^{+}+t \sqrt{\alpha}\eta(x-z,0)u_{\varepsilon}(x-z)\bigr\vert ^{\alpha} \\& \qquad {}\times\bigl\vert (\omega_{2})_{\lambda,\mu}^{+}+t\sqrt{\beta}\eta (x-z,0)u_{\varepsilon}(x-z)\bigr\vert ^{\beta}\,dx \\& \quad =\frac{1}{2}\bigl\| \bigl((\omega_{1})_{\lambda,\mu}^{+},( \omega_{2})_{\lambda ,\mu}^{+}\bigr)\bigr\| _{H}^{2}+t^{2} \frac{\alpha+\beta}{2}\|v_{\varepsilon,z}\| _{H_{0,L}^{s}(C_{\Omega})}^{2} \\& \qquad {}+tK_{s} \biggl( \int_{C_{\Omega}}\nabla(\omega _{1})_{\lambda,\mu}^{+}\nabla( \sqrt{\alpha}v_{\varepsilon ,z})\,dx\,dy+ \int_{C_{\Omega}}\nabla(\omega_{2})_{\lambda ,\mu}^{+}\nabla( \sqrt{\beta}v_{\varepsilon,z})\,dx\,dy \biggr) \\& \qquad {}-\frac{1}{q} \int_{\Omega\times\{0\}}\bigl(\lambda \bigl\vert (\omega_{1})_{\lambda,\mu}^{+}+t \sqrt{\alpha}\eta (x-z,0)u_{\varepsilon}(x-z)\bigr\vert ^{q} \\& \qquad {}+\mu \bigl\vert (\omega_{2})_{\lambda,\mu}^{+}+t\sqrt {\beta} \eta(x-z,0)u_{\varepsilon}(x-z)\bigr\vert ^{q}\bigr)\,dx \\& \qquad {}-\frac{1}{2_{s}^{*}} \int_{\Omega\times\{0\}}\bigl\vert (\omega _{1})_{\lambda,\mu}^{+}+t \sqrt{\alpha}\eta(x-z,0)u_{\varepsilon}(x-z)\bigr\vert ^{\alpha} \\& \qquad {}\times\bigl\vert (\omega_{2})_{\lambda,\mu}^{+}+t\sqrt{\beta}\eta (x-z,0)u_{\varepsilon}(x-z)\bigr\vert ^{\beta}\,dx \\& \quad \leq I_{\lambda,\mu}\bigl((\omega_{1})_{\lambda,\mu}^{+},(\omega _{2})_{\lambda,\mu}^{+}\bigr)+t^{2}\frac{\alpha+\beta}{2} \|v_{\varepsilon ,z}\|_{H_{0,L}^{s}(C_{\Omega})}^{2} \\& \qquad {}-\frac{1}{2_{s}^{*}} \int_{\Omega\times\{0\}}\bigl\vert (\omega _{1})_{\lambda,\mu}^{+}+t \sqrt{\alpha}\eta(x-z,0)u_{\varepsilon}(x-z)\bigr\vert ^{\alpha} \\& \qquad {}\times\bigl\vert (\omega_{2})_{\lambda,\mu}^{+}+t\sqrt{\beta}\eta (x-z,0)u_{\varepsilon}(x-z)\bigr\vert ^{\beta}\,dx +\frac{1}{2_{s}^{*}} \int_{\Omega\times\{0\}}\bigl\vert (\omega _{1})_{\lambda,\mu}^{+} \bigr\vert ^{\alpha}\bigl\vert (\omega_{2})_{\lambda,\mu}^{+} \bigr\vert ^{\beta}\,dx \\& \qquad {}+\frac{1}{2_{s}^{*}} \int_{\Omega\times\{0\}}\alpha \bigl\vert (\omega_{1})_{\lambda,\mu}^{+} \bigr\vert ^{\alpha-1}\bigl\vert (\omega_{2})_{\lambda,\mu }^{+} \bigr\vert ^{\beta}\bigl(t\sqrt{\alpha}\eta(x-z,0)u_{\varepsilon}(x-z) \bigr) \,dx \\& \qquad {}+\frac{1}{2_{s}^{*}} \int_{\Omega\times\{0\}}\beta \bigl\vert (\omega_{1})_{\lambda,\mu}^{+} \bigr\vert ^{\alpha}\bigl\vert (\omega_{2})_{\lambda,\mu }^{+} \bigr\vert ^{\beta-1}\bigl(t\sqrt{\beta}\eta(x-z,0)u_{\varepsilon}(x-z) \bigr)\,dx \\& \quad =\alpha_{\lambda,\mu}^{+}+K(t), \end{aligned}$$

(3.4)

where

$$\begin{aligned} K(t) =&t^{2}\frac{\alpha+\beta}{2}\|v_{\varepsilon,z}\| _{H_{0,L}^{s}(C_{\Omega})}^{2} \\ &{}-\frac{1}{2_{s}^{*}} \int_{\Omega\times\{0\}}\bigl\vert (\omega _{1})_{\lambda,\mu}^{+}+t \sqrt{\alpha}\eta(x-z,0)u_{\varepsilon}(x-z)\bigr\vert ^{\alpha}\\ &{}\times\bigl\vert (\omega_{2})_{\lambda,\mu}^{+}+t\sqrt{\beta}\eta (x-z,0)u_{\varepsilon}(x-z)\bigr\vert ^{\beta}\,dx \\ &{}+\frac{1}{2_{s}^{*}} \int_{\Omega\times\{0\}}\bigl\vert (\omega _{1})_{\lambda,\mu}^{+} \bigr\vert ^{\alpha}\bigl\vert (\omega_{2})_{\lambda,\mu}^{+} \bigr\vert ^{\beta}\,dx \\ &{}+\frac{1}{2_{s}^{*}} \int_{\Omega\times\{0\}}\alpha \bigl\vert (\omega_{1})_{\lambda,\mu}^{+} \bigr\vert ^{\alpha-1}\bigl\vert (\omega_{2})_{\lambda,\mu }^{+} \bigr\vert ^{\beta}\bigl(t\sqrt{\alpha}\eta(x-z,0)u_{\varepsilon}(x-z) \bigr) \,dx \\ &{}+\frac{1}{2_{s}^{*}} \int_{\Omega\times\{0\}}\beta \bigl\vert (\omega_{1})_{\lambda,\mu}^{+} \bigr\vert ^{\alpha}\bigl\vert (\omega_{2})_{\lambda,\mu }^{+} \bigr\vert ^{\beta-1}\bigl(t\sqrt{\beta}\eta(x-z,0)u_{\varepsilon}(x-z) \bigr)\,dx. \end{aligned}$$

In the following we shall show that

$$ \sup_{t\geq0}K(t)< \frac{s}{2N}(K_{s}S_{s,\alpha,\beta })^{N/s} \quad \text{for } \varepsilon>0 \text{ small enough} . $$

It is easy to see that

$$ \lim_{t\rightarrow0}K(t)=0. $$

Thus, for all ε sufficiently small, there exists $t_{0}>0$ such that

$$ K(t)< \frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s} \quad \text{for all } t\in(0,t_{0}]. $$

(3.5)

From [25], Lemma 4.1, we see that there exist $C_{1}(\alpha),C_{2}(\beta )>0$ such that

$$\begin{aligned} (a+b)^{\alpha}(c+d)^{\beta} \geq& a^{\alpha}c^{\beta}+a^{\alpha}d^{\beta}+b^{\alpha}c^{\beta}+b^{\alpha}d^{\beta}+C_{1}( \alpha)a^{\alpha-1}bc^{\beta}+C_{1}(\alpha)a^{\alpha-1}bd^{\beta}\\ &{}+C_{2}(\beta)b^{\alpha}c^{\beta-1}d+C_{2}( \beta)a^{\alpha}c^{\beta -1}d+C_{1}(\alpha)C_{2}( \beta)a^{\alpha-1}bc^{\beta-1}d \end{aligned}$$

for any $a,b,c,d>0$. Consequently,

$$ K(t)\leq t^{2}\frac{2_{s}^{*}}{2}\|v_{\varepsilon,z} \|_{H_{0,L}^{s}(C_{\Omega})}^{2}-\frac{\alpha^{\frac{\alpha}{2}}\beta^{\frac{\beta }{2}}}{2_{s}^{*}}t^{2_{s}^{*}} \int_{\Omega\times\{0\} }|v_{\varepsilon,z}|^{2_{s}^{*}}\,dx-C_{1} \int_{\Omega\times\{ 0\}}|v_{\varepsilon,z}|^{2_{s}^{*}-1}\,dx $$

(3.6)

with some constant $C_{1}>0$. Note that

$$\begin{aligned}& \int_{\Omega\times\{0\}}|v_{\varepsilon ,z}|^{2_{s}^{*}-1}\,dx \\& \quad = \int_{\Omega\times\{0\}}\bigl\vert \eta(x-z,0)u_{\varepsilon}(x-z)\bigr\vert ^{2_{s}^{*}-1}\,dx \\& \quad = \int_{B_{2\rho}} \biggl[\frac{\eta(x,0)\varepsilon ^{\frac{N-s}{2}}}{(\varepsilon^{2}+|x|^{2})^{\frac{N-s}{2}}} \biggr]^{\frac {N+s}{N-s}}\,dx \\& \quad \geq \int_{B_{\rho}}\frac{\varepsilon^{\frac {N+s}{2}}}{\varepsilon^{N+s}(\varepsilon^{2}+|x|^{2})^{\frac {N+s}{2}}}\varepsilon^{N}\,dx \\& \quad =C_{2}\varepsilon^{\frac{N-s}{2}} \int_{0}^{\rho}\frac {r^{N-1}}{(1+r^{2})^{\frac{N+s}{2}}}\,dr \\& \quad =C_{3}\varepsilon^{\frac{N-s}{2}} \end{aligned}$$

(3.7)

for some $C_{2},C_{3}>0$. It follows from (3.2)-(3.7) that

$$\begin{aligned} K(t)&\leq t^{2}\frac{2_{s}^{*}}{2}\|v_{\varepsilon,z} \|_{H_{0,L}^{s}(C_{\Omega})}^{2}-\frac{\alpha^{\frac{\alpha}{2}}\beta^{\frac{\beta }{2}}}{2_{s}^{*}}t^{2_{s}^{*}} \int_{\Omega\times\{0\} }|v_{\varepsilon,z}|^{2_{s}^{*}} \,dx-C_{4}\varepsilon^{\frac{N-s}{2}} \\ &\leq\frac{s}{2N} \biggl(\frac{(\alpha+\beta)\|v_{\varepsilon ,z}\|_{H_{0,L}^{s}(C_{\Omega})}^{2}}{ ( \int_{\Omega \times\{0\}}\alpha^{\frac{\alpha}{2}}\beta^{\frac{\beta }{2}}|v_{\varepsilon,z}|^{2_{s}^{*}}\,dx )^{\frac{2}{2_{s}^{*}}}} \biggr)^{\frac{N}{s}}-C_{4} \varepsilon^{\frac{N-s}{2}} \\ &=\frac{s}{2N} \biggl( \biggl( \biggl(\frac{\alpha}{\beta} \biggr)^{\frac{\beta}{\alpha+\beta}}+ \biggl(\frac{\beta}{\alpha} \biggr)^{\frac{\alpha}{\alpha+\beta}} \biggr) \frac{K_{s} \int _{\mathbb{R}_{+}^{N+1}}y^{1-s}|\nabla\omega_{\varepsilon}|^{2}\,dx\,dy+O(\varepsilon^{N-s})}{ ( \int_{\mathbb {R}^{N}} (\frac{\varepsilon}{\varepsilon^{2}+|x|^{2}} )^{N}\,dx+O(\varepsilon^{N}) )^{\frac{2}{2_{s}^{*}}}} \biggr)^{\frac {N}{s}}-C_{4}\varepsilon^{\frac{N-s}{2}} \\ &=\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{\frac{N}{s}}+O\bigl( \varepsilon ^{N-s}\bigr)-C_{4}\varepsilon^{\frac{N-s}{2}} \\ &< \frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{\frac{N}{s}} \end{aligned}$$

(3.8)

for ε sufficiently small and $t\in[t_{0},+\infty)$. Noting the compactness of $\overline{\Omega_{r}^{-}}$, it follows from (3.4)-(3.5) and (3.8) that there exist $\varepsilon_{0},\sigma (\varepsilon)>0$ such that, for $\varepsilon\in(0,\varepsilon_{0})$ and $\sigma\in(0,\sigma(\varepsilon))$, we have

$$ \sup_{t\geq0}I_{\lambda,\mu}\bigl((\omega_{1})_{\lambda ,\mu}^{+}+t \sqrt{\alpha}v_{\varepsilon,z},(\omega_{2})_{\lambda,\mu }^{+}+t\sqrt{ \beta}v_{\varepsilon,z}\bigr)< c_{\lambda,\mu}-\sigma\quad \text{uniformly in } z \in\Omega_{r}^{-}. $$

Arguing as the proof of [11], Lemma 4.4, we conclude that there exists $t_{(\lambda,\mu,\varepsilon,z)}^{-}>0$ such that

$$ \bigl((\omega_{1})_{\lambda,\mu}^{+}+t_{(\lambda,\mu,\varepsilon ,z)}^{-}\sqrt{ \alpha}v_{\varepsilon,z},(\omega_{2})_{\lambda,\mu }^{+}+t_{(\lambda,\mu,\varepsilon,z)}^{-} \sqrt{\beta}v_{\varepsilon ,z}\bigr)\in N_{\lambda,\mu}^{-}(c_{\lambda,\mu}- \sigma). $$

Moreover, we obtain from Lemma 3.3

$$ \Phi\bigl((\omega_{1})_{\lambda,\mu}^{+}+t_{(\lambda,\mu,\varepsilon ,z)}^{-}\sqrt{ \alpha}v_{\varepsilon,z},(\omega_{2})_{\lambda,\mu }^{+}+t_{(\lambda,\mu,\varepsilon,z)}^{-} \sqrt{\beta}v_{\varepsilon ,z}\bigr)\in\Omega_{r}^{+} $$

for $\lambda,\mu\in(0,\Lambda_{*})$. □

From Lemma 3.4, we can define the map $\gamma:\Omega_{r}^{-}\rightarrow N_{\lambda,\mu}^{-}(c_{\lambda,\mu}-\sigma)$ defined by

$$ \gamma(z):=\bigl((\omega_{1})_{\lambda,\mu}^{+}+t_{(\lambda,\mu ,\varepsilon,z)}^{-} \sqrt{\alpha}v_{\varepsilon,z},(\omega _{2})_{\lambda,\mu}^{+}+t_{(\lambda,\mu,\varepsilon,z)}^{-} \sqrt{\beta }v_{\varepsilon,z}\bigr). $$

Furthermore, by Lemma 2.6 and Lemma 2.8(iv), we can define the map $\Phi_{\lambda,\mu}: N_{\lambda,\mu}^{-}(c_{\lambda,\mu}-\sigma )\rightarrow\mathbb{R}^{N}$ by setting

$$ \Phi_{\lambda,\mu}(\omega_{1},\omega_{2}):= \frac{ \int _{\Omega\times\{0\}} x|\omega_{1}-(\omega_{1})_{\lambda,\mu }^{+}|^{\alpha}|\omega_{2}-(\omega_{2})_{\lambda,\mu}^{+}|^{\beta}\,dx}{ \int_{\Omega\times\{0\}} |\omega_{1}-(\omega _{1})_{\lambda,\mu}^{+}|^{\alpha}|\omega_{2}-(\omega_{2})_{\lambda,\mu }^{+}|^{\beta}\,dx}. $$

Then, for each $z\in\Omega_{r}^{-}$, note that $u_{\varepsilon}(x)$ is radial, we have

$$ (\Phi_{\lambda,\mu}\circ\gamma) (z)=z. $$

Next we define the map $H_{\lambda,\mu}:[0,1]\times N_{\lambda,\mu }^{-}(c_{\lambda,\mu}-\sigma)\rightarrow\mathbb{R}^{N}$ by

$$ H_{\lambda,\mu}\bigl(t,(\omega_{1},\omega_{2})\bigr)=t \Phi_{\lambda,\mu }(\omega_{1},\omega_{2})+(1-t) \Phi_{\lambda,\mu}(\omega_{1},\omega_{2}). $$

Lemma 3.5

For each $\varepsilon\in(0,\varepsilon_{0})$, there exists $\Lambda_{*}>0$ such that if $\lambda,\mu,\sigma\in(0,\Lambda_{*})$,

$$ H_{\lambda,\mu}\bigl([0,1]\times N_{\lambda,\mu}^{-}(c_{\lambda,\mu }-\sigma) \bigr)\subset\Omega_{r}^{+}. $$

Proof

Suppose by contradiction that there exist $t_{n}\in[0,1]$, $\lambda _{n},\mu_{n},\sigma_{n}\rightarrow0$, and $(\omega_{1,n},\omega _{2,n})\in N_{\lambda_{n},\mu_{n}}^{-}(c_{\lambda_{n},\mu_{n}}-\sigma_{n})$ such that

$$ H_{\lambda_{n},\mu_{n}}\bigl(t_{n},(\omega_{1,n}, \omega_{2,n})\bigr)\notin\Omega _{r}^{+}\quad \text{for all } n. $$

Furthermore, we can assume that $t_{n}\rightarrow t_{0}\in[0,1]$. Then by Lemma 2.8(iv) and argue as in the proof of Lemma 3.3, we have

$$ H_{\lambda_{n},\mu_{n}}\bigl(t_{n},(\omega_{1,n}, \omega_{2,n})\bigr)\rightarrow z\in\overline{\Omega}\quad \text{as } n \rightarrow\infty, $$

which is a contradiction. □

4 Proof of Theorem 1.1

We begin with the following lemma.

Lemma 4.1

If $(\omega_{1},\omega_{2})$ is a critical point of $I_{\lambda,\mu}$ on $N_{\lambda,\mu}^{-}$, then it is a critical point of $I_{\lambda ,\mu}$ in H.

Proof

Assume $(\omega_{1},\omega_{2})\in N_{\lambda,\mu}^{-}$, then $I'_{\lambda,\mu}(\omega_{1},\omega_{2})(\omega_{1},\omega_{2})=0$. On the other hand,

$$ I'_{\lambda,\mu}(\omega_{1}, \omega_{2})=\theta\psi'_{\lambda,\mu }( \omega_{1},\omega_{2}) $$

(4.1)

for some $\theta\in\mathbb{R}$, where $\psi_{\lambda,\mu}$ is defined in (2.9).

Remark that $(\omega_{1},\omega_{2})\in N_{\lambda,\mu}^{-}$, and so $\psi'_{\lambda,\mu}(\omega_{1},\omega_{2})(\omega_{1},\omega_{2})<0$. Thus by (4.1),

$$ 0=\theta\psi'_{\lambda,\mu}(\omega_{1}, \omega_{2}) (\omega_{1},\omega_{2}), $$

which implies that $\theta=0$, consequently $I'_{\lambda,\mu }(\omega_{1},\omega_{2})=0$. □

Below we denote by $I_{N_{\lambda,\mu}^{-}}$ the restriction of $I_{\lambda,\mu}$ on $N_{\lambda,\mu}^{-}$.

Lemma 4.2

Any sequence $\{(\omega_{1,n},\omega_{2,n})\}\subset N_{\lambda,\mu }^{-}$ such that $I_{N_{\lambda,\mu}^{-}}(\omega_{1,n},\omega _{2,n})\rightarrow c\in(-\infty,c_{\lambda,\mu})$ and $I'_{N_{\lambda,\mu}^{-}}(\omega_{1,n},\omega_{2,n})\rightarrow 0$ contains a convergent subsequence for all $\lambda,\mu\in (0,\Lambda_{*})$.

Proof

By hypothesis there exists a sequence $\{\theta_{n}\}\subset\mathbb {R}$ such that

$$ I'_{\lambda,\mu}(\omega_{1,n},\omega_{2,n})= \theta_{n}\psi '_{\lambda,\mu}(\omega_{1,n}, \omega_{2,n})+o(1). $$

Recall that $(\omega_{1,n},\omega_{2,n})\in N_{\lambda,\mu}^{-}$ and so

$$ \psi'_{\lambda,\mu}(\omega_{1,n},\omega_{2,n}) (\omega _{1,n},\omega_{2,n})< 0. $$

If $\psi'_{\lambda,\mu}(\omega_{1,n},\omega_{2,n})(\omega _{1,n},\omega_{2,n})\rightarrow0$, we see from (2.10) and (2.11) that there are two positive numbers $C_{1}$, $C_{2}$ independent of $(\omega_{1,n},\omega _{2,n})$ and λ, μ such that

$$\begin{aligned}& \bigl\Vert (\omega_{1,n},\omega_{2,n})\bigr\Vert _{H}^{2}\leq C_{1}\bigl\Vert ( \omega_{1,n},\omega _{2,n})\bigr\Vert _{H}^{2_{s}^{*}}+o(1) \quad \text{and} \\& \bigl\Vert (\omega_{1,n},\omega_{2,n}) \bigr\Vert _{H}^{2}\leq(\lambda+\mu )C_{2}\bigl\Vert (\omega_{1,n},\omega_{2,n})\bigr\Vert _{H}^{q}+o(1) \end{aligned}$$

or

$$\begin{aligned}& \bigl\Vert (\omega_{1,n},\omega_{2,n})\bigr\Vert _{H}^{2}\geq C_{1}^{-\frac{2}{2_{s}^{*}-2}}+o(1) \quad \text{and} \\& \bigl\Vert (\omega_{1,n},\omega_{2,n})\bigr\Vert _{H}^{2}\leq(\lambda+\mu )^{\frac{2}{2-q}}c_{2}^{\frac{2}{2-q}}+o(1). \end{aligned}$$

If $\lambda,\mu>0$ is sufficiently small, this is impossible. Thus we may assume that $\psi'_{\lambda,\mu}(\omega_{1,n}, \omega _{2,n})(\omega_{1,n},\omega_{2,n})\rightarrow l<0$ as $n\rightarrow \infty$. Since $I'_{\lambda,\mu}(\omega_{1,n},\omega_{2,n})(\omega _{1,n},\omega_{2,n})=0$, we conclude that $\theta_{n}\rightarrow0$ and, consequently, $I'_{\lambda,\mu}(\omega_{1,n},\omega _{2,n})\rightarrow0$. Using this information we have

$$ I_{\lambda,\mu}(\omega_{1,n},\omega_{2,n})\rightarrow c\in(- \infty ,c_{\lambda,\mu})\quad \text{and}\quad I'_{\lambda,\mu}( \omega_{1,n},\omega _{2,n})\rightarrow0, $$

so by Lemma 2.7 the proof is complete. □

Lemma 4.3

If $\lambda,\mu,\sigma\in(0,\Lambda_{*})$, then

$$ \operatorname{cat}\bigl(N_{\lambda,\mu}^{-}(c_{\lambda,\mu}-\sigma)\bigr)\geq \operatorname{cat}(\Omega). $$

Proof

Suppose that

$$ N_{\lambda,\mu}^{-}(c_{\lambda,\mu}-\sigma)=A_{1}\cup\cdots \cup A_{n}, $$

where $A_{j}$, $j=1,\ldots,n$, is closed and contractible in $N_{\lambda ,\mu}^{-}(c_{\lambda,\mu}-\sigma)$, i.e., there exists $h_{j}\in C([0,1]\times A_{j},N_{\lambda,\mu}^{-}(c_{\lambda,\mu}-\sigma ))$ such that

$$ h_{j}(0,z)=z \quad \text{and}\quad h_{j}(1,z)=\omega \quad \text{for all } z\in A_{j}, $$

where $\omega\in A_{j}$ is fixed. Consider $B_{j}:=\gamma^{-1}(A_{j})$, $1\leq j\leq n$. The sets $B_{j}$ are closed and

$$ \Omega_{r}^{-}=B_{1}\cup\cdots\cup B_{n}. $$

Noting Lemma 3.5, we define the deformation $g_{j}:[0,1]\times B_{j}\rightarrow\Omega_{r}^{+}$ by setting

$$ g_{j}(t,y):=H_{\lambda,\mu}\bigl(t,h_{j}\bigl(t,\gamma(y) \bigr)\bigr) $$

for $\lambda,\mu,\sigma\in(0,\Lambda_{*})$. Note that

$$ g_{j}(0,y):=H_{\lambda,\mu}\bigl(0,h_{j}\bigl(0,\gamma(y) \bigr)\bigr)=y \quad \text{for all } y\in B_{j} $$

and

$$ g_{j}(1,y):=H_{\lambda,\mu}\bigl(1,h_{j}\bigl(1,\gamma(y) \bigr)\bigr)=\Phi_{\lambda,\mu }(\omega)\in\Omega_{r}^{+}. $$

Thus the sets $B_{j}$ are contractible in $\Omega_{r}^{+}$. It follows that

$$ \operatorname{cat}(\Omega)=\operatorname{cat}_{\Omega_{r}^{+}}\bigl( \Omega_{r}^{-}\bigr)\leq n. $$

 □

Now, we can give the proof of Theorem 1.1.

Proof of Theorem 1.1

Applying Lemmas 2.7 and 4.2, $I_{N_{\lambda,\mu}^{-}}$ satisfies $(\mathit{PS})_{c}$ condition for all $c\in(-\infty,c_{\lambda,\mu})$. Then, by Lemmas 3.2 and 4.3, $I_{N_{\lambda,\mu}^{-}}$ contains at least $\operatorname{cat}(\Omega)$ critical points in $N_{\lambda,\mu}^{-}(c_{\lambda,\mu }-\sigma)$. Hence, we deduce from Lemma 4.1 that $I_{\lambda,\mu}$ has at least $\operatorname{cat}(\Omega)$ critical points in $N_{\lambda,\mu}^{-}$. Moreover, by Lemma 2.6 and $N_{\lambda,\mu}^{+}\cap N_{\lambda,\mu }^{-}=\emptyset$, $I_{\lambda,\mu}$ has at least $\operatorname{cat}(\Omega)+1$ critical points in H. If we change the definition of $J_{\lambda,\mu }$ as follows:

$$\begin{aligned} J_{\lambda,\mu}(u,v) :=&\frac{1}{2} \int_{\Omega}\bigl(\bigl\vert (-\Delta)^{\frac{s}{4}}u\bigr\vert ^{2}+\bigl\vert (-\Delta)^{\frac{s}{4}}v\bigr\vert ^{2}\bigr)\,dx \\ &{}-\frac {1}{q} \int_{\Omega}\bigl(\lambda u_{+}^{q}+\mu v_{+}^{q} \bigr)\,dx-\frac {1}{2_{s}^{*}} \int_{\Omega}u_{+}^{\alpha}v_{+}^{\beta}\,dx, \end{aligned}$$

(4.2)

where $u_{+}=\max\{u,0\}$ and $v_{+}=\max\{v,0\}$. Then all the steps of our paper for (4.2). Thus we see that $J_{\lambda,\mu}$ has at least $\operatorname{cat}(\Omega)+1$ non-negative critical points. By the maximum principle [26], we complete the proof. □