Multiple solutions to the Kirchhoff fractional equation involving Hardy–Littlewood–Sobolev critical exponent

Abstract

In this paper, we study a fractional Kirchhoff type equation with Hardy–Littlewood–Sobolev critical exponent. By using variational methods, we obtain the existence of mountain-pass type solution and negative energy solutions. Also, we prove some further properties of solutions.

1 Introduction

In this paper, we study the following fractional Kirchhoff type equation with Hardy–Littlewood–Sobolev critical exponent:

$$\begin{aligned} \textstyle\begin{cases} (a+b \int _{\varOmega }\int _{\varOmega } \frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{N+2 \alpha }}\,\mathrm{d} x \,\mathrm{d}y )(- \Delta )^{\alpha }u \\ \quad = (\int _{\varOmega } \frac{\beta F(u(y))+ \vert u(y) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d}y ) \times (\beta f(u) +2_{\mu }^{*} \vert u \vert ^{2_{ \mu }^{*}-2} u ) + \gamma \vert u \vert ^{q-2}u \quad \text{in } \varOmega , \\ u \in H_{0}^{\alpha }(\varOmega ), \end{cases}\displaystyle \end{aligned}$$

(1.1)

where $\varOmega \subset \mathbb{R}^{N} (N \ge 3)$ is a smooth bounded domain, a, $b>0$ are constants, $\alpha \in (0,1)$, $(-\Delta )^{ \alpha }$ is the fractional Laplace operator, $\mu \in (0,N)$, $2_{\mu }^{*}=\frac{2N-\mu }{N-2 \alpha }$ is the critical exponent of the Hardy–Littlewood–Sobolev inequality, F is the primitive function of f, $q \in (1,2)$, β, $\gamma >0$ are parameters.

The investigation of (1.1) is motivated by the following fractional Kirchhoff type equation:

$$\begin{aligned} \textstyle\begin{cases} (a+b \int _{\varOmega }\int _{\varOmega } \frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{N+2 \alpha }}\,\mathrm{d} x \,\mathrm{d}y )(- \Delta )^{\alpha }u= h(u) \quad \text{in } \varOmega , \\ u \in H_{0}^{\alpha }(\varOmega ), \end{cases}\displaystyle \end{aligned}$$

(1.2)

where h is a nonlinearity with subcritical growth, or involving the critical exponent. When $b=0$, problem (1.2) reduces to the standard fractional equation. The fractional equation appears in various areas such as plasma physics, optimization, finance, free boundary obstacle problems, population dynamics, and minimal surfaces. For more background, we refer to [4] and the references therein. In recent years, many papers have focused on fractional problems on bounded or unbounded domains.

The Kirchhoff equation occurs in various branches of mathematical physics. For example, it can be used to model suspension bridges. Also, it appears in other fields like biological systems, such as population density. Because of the presence of the nonlocal term, the problem is not a pointwise identity, which causes additional mathematical difficulties. In [12], the authors established a stationary Dirichlet problem of Kirchhoff type and proved the existence and asymptotic behavior to solutions. In [3], the authors extended the results in [12] to a more general case. In [11], the author obtained infinitely many solutions to a critical Kirchhoff type fractional problem. There are also papers on problems in the whole space. In [22], the authors obtained the existence and multiplicity of solutions to a fractional Kirchhoff type eigenvalue problem. In [23], the authors studied a nonhomogeneous fractional p-Laplacian equation of Schrödinger–Kirchhoff type. In [17], the authors considered ground states to a fractional Kirchhoff type problem with Sobolev critical exponent:

$$ \biggl(a+b \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{\alpha }{2}}u \bigr\vert ^{2} \,\mathrm{d}x \biggr) (-\Delta )^{\alpha }u + V(x) u = f(u) \quad \text{in } \mathbb{R}^{N}, $$

(1.3)

where $N=3$ with $\alpha \in (\frac{3}{4},1)$. In [29], we continued the studies in [17] and considered the equation

$$ \biggl(1+b \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{3+2 \alpha }} \,\mathrm{d} x \,\mathrm{d} y \biggr) (-\Delta )^{\alpha }u + u = \beta f(u)+u^{2_{\alpha }^{*}-1} \quad \text{in } \mathbb{R}^{3}, $$

(1.4)

where $2_{\alpha }^{*}=\frac{6}{3-2\alpha }$ is the Sobolev critical exponent. Under some conditions on b, β, f, we obtained the existence of ground state solutions when $\alpha =\frac{3}{4}$ and the non-existence of nontrivial solutions when $\alpha \in (0,\frac{3}{4}]$. All the critical problems mentioned above contain only the Sobolev critical exponent. Also, when we consider the problem on a bounded domain, we infer from [6] that the results may be quite different. Thus, it is natural to ask what happens when we study the fractional Kirchhoff type equation with Hardy–Littlewood–Sobolev critical exponent on a bounded domain?

The problem involving Hardy–Littlewood–Sobolev critical exponent is closely related to the Choquard type equation, which has been well studied recently. The Choquard equation can be used to describe the quantum mechanics of a polaron at rest. Also, it is known as the stationary Hartree equation, or the Schrödinger–Newton equation. In [15], Lieb first proved the existence and uniqueness of radial ground state solutions to the following equation:

$$\begin{aligned} -\Delta u + u = \biggl( \int _{\mathbb{R}^{3}} \frac{ \vert u(y) \vert ^{p}}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d}y \biggr) \vert u \vert ^{p-2} u \quad \text{in } \mathbb{R}^{3}. \end{aligned}$$

(1.5)

Later, Lions [16] obtained the existence of infinitely many radial solutions. The authors in [18, 19] studied the existence, qualitative properties, and decay asymptotics of ground state solutions to a more general Choquard equation. For other related results, we refer the readers to [1, 7, 20, 21] for the subcritical case. There are also papers studying Choquard equations involving the Hardy–Littlewood–Sobolev critical exponent. In [13], the authors proved the existence and non-existence of solutions to a Brezis–Nirenberg type Choquard equation. In [27], the authors studied multiple solutions for a nonhomogeneous Choquard equation with Dirichlet boundary condition. In [30], we obtained multiplicity and concentration behavior of positive solutions to a singularly perturbed Choquard problem with critical growth.

In this paper, we study multiplicity of solutions to the fractional Dirichlet problem (1.1). By using the Ekeland variational principle and the mountain pass theorem, we obtain nontrivial solutions to (1.1) with positive or negative energy in a certain range of parameters. Moreover, we show some further properties of the set of solutions. Our results are new even in the case $\gamma =0$. Recall that in [25], Servadei and Valdinoci first used the mountain pass theorem to solve the fractional problem. In this paper, since problem (1.1) includes the Kirchhoff type nonlocal term and the nonlocal critical term, it is not easy to check the geometric structure of the functional associated with the equation, the boundedness and convergence of the corresponding Palais–Smale sequence. Also, we have to distinguish between different solutions. Now we state the results. We first consider the case $\mu \in (0,4\alpha )$. For this purpose, we assume f satisfies the following conditions:

$(f_{1})$ :

$f \in C(\mathbb{R}, \mathbb{R})$ and $\lim_{u \rightarrow 0}\frac{f(u)}{u^{\frac{N-\mu }{N}}}= \lim_{u \rightarrow \infty }\frac{f(u)}{|u|^{2_{\mu }^{*}-2}u}=0$, where $2_{\mu }^{*}=\frac{2N-\mu }{N-2 \alpha }$.

$(f_{2})$ :

$F(u)=\int _{0}^{u} f(s) \,\mathrm{d}s \ge 0$ for $u \in \mathbb{R}$. Moreover, there exists $\xi >0$ such that $F(\xi )=\int _{0}^{\xi }f(s) \,\mathrm{d}s>0$.

$(f_{3})$ :

$\frac{1}{2}f(u)u - F(u) \ge 0$ for $u \in \mathbb{R}$, where $F(u)=\int _{0}^{u}f(s) \,\mathrm{d}s$.

Theorem 1.1

Let $\mu \in (0,4 \alpha )$, a, b, $\beta >0$. When $N \ge 4$, or $N=3$ with $\alpha \in (0,\frac{3}{4}]$, we assume $(f_{1})$–$(f_{3})$; when $N=3$ with $\alpha \in (\frac{3}{4},1)$, we assume $(f_{1})$–$(f_{3})$ and $\lim_{u \rightarrow +\infty }\frac{F(u)}{u^{\frac{4\alpha -\mu }{3-2 \alpha }}}=+\infty$. Then there exists $\gamma _{1}>0$ such that, for $\gamma \in (0,\gamma _{1})$, problem (1.1) has a negative energy solution $u_{1,\gamma }$ and a mountain-pass type solution $u_{2, \gamma }$ with positive energy. Moreover,

• (i) $u_{1,\gamma } \rightarrow 0$ in $H_{0}^{\alpha }(\varOmega )$ as $\gamma \rightarrow 0$;

• (ii) $u_{2,\gamma } \rightarrow u_{0}$ in $H_{0}^{\alpha }( \varOmega )$ as $\gamma \rightarrow 0$, where $u_{0}$ is the nontrivial solution of (1.1) with $\gamma =0$.

Remark 1.1

A typical example satisfying $(f_{1})$–$(f_{3})$ is the function $f(u)=|u|^{q-2}u$, where $q \in (2,2_{\mu }^{*})$, $u \in \mathbb{R}$.

Now we consider the case $\mu \ge 4 \alpha$. Define the best fractional Sobolev constant:

$$\begin{aligned} S_{\alpha ,\mu }:=\inf_{D^{\alpha ,2}(\mathbb{R}^{N}) \setminus \{0\}} \frac{ \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}}\frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{N+2 \alpha }} \,\mathrm{d} x \,\mathrm{d} y}{ (\int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert u(x) \vert ^{2_{\mu }^{*}} \vert u(y) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d}x\,\mathrm{d}y )^{\frac{1}{2_{\mu }^{*}}}}, \end{aligned}$$

(1.6)

where $2_{\mu }^{*}=\frac{2N-\mu }{N-2 \alpha }$ is the critical exponent for the Hardy–Littlewood–Sobolev inequality.

Theorem 1.2

Let $N \ge 3$, $\mu =4 \alpha$, $a>0$, $b>\frac{2}{S_{\alpha ,\mu } ^{2}}$, $\beta >0$. Assume that $(f_{1})$–$(f_{2})$. Then there exist $\beta _{0}$, $\gamma _{2}>0$ such that, for $\beta >\beta _{0}$ and $\gamma \in (0,\gamma _{2})$, problem (1.1) has a mountain-pass type solution $v_{0,\gamma }$ with positive energy and two negative energy solutions $v_{1,\gamma }$, $v_{2,\gamma }$. Moreover,

• (i) $v_{0,\gamma } \rightarrow v_{0}$ in $H_{0}^{\alpha }( \varOmega )$ and $v_{1,\gamma } \rightarrow v_{1}$ in $H_{0}^{\alpha }( \varOmega )$ as $\gamma \rightarrow 0$, where $v_{0} \ne v_{1}$ are nontrivial solutions of (1.1) with $\gamma =0$;

• (ii) $v_{2,\gamma } \rightarrow 0$ in $H_{0}^{\alpha }(\varOmega )$ as $\gamma \rightarrow 0$.

Let

$$\begin{aligned} a(b)=\frac{\mu -4\alpha }{N-2\alpha } \biggl(\frac{2_{\mu }^{*}}{S_{ \alpha ,\mu }^{2_{\mu }^{*}}} \biggr)^{\frac{1}{2-2_{\mu }^{*}}} \biggl(\frac{2_{ \mu }^{*}-1}{b} \biggr)^{\frac{N-\mu +2\alpha }{\mu -4\alpha }}. \end{aligned}$$

(1.7)

Theorem 1.3

Let $N \ge 3$, $\mu > 4 \alpha$, $b>0$, $a>a(b)$, $\beta >0$. Assume that $(f_{1})$. Moreover, $F(u)=\int _{0}^{u} f(s) \,\mathrm{d}s \ge 0$ for $u \in \mathbb{R}$. Then there exist $b_{0}$, $\gamma _{3}>0$ such that, for $b \in (0,b_{0})$ and $\gamma \in (0,\gamma _{3})$, problem (1.1) has a mountain-pass type solution $w_{0,\gamma }$ with positive energy and two negative energy solutions $w_{1,\gamma }$, $w_{2,\gamma }$. Moreover,

• (i) $w_{0,\gamma } \rightarrow w_{0}$ in $H_{0}^{\alpha }( \varOmega )$ and $w_{1,\gamma } \rightarrow w_{1}$ in $H_{0}^{\alpha }( \varOmega )$ as $\gamma \rightarrow 0$, where $w_{0} \ne w_{1}$ are nontrivial solutions of (1.1) with $\gamma =0$;

• (ii) $w_{2,\gamma } \rightarrow 0$ in $H_{0}^{\alpha }(\varOmega )$ as $\gamma \rightarrow 0$.

2 Preliminary lemmas

In this section, we give some definitions and lemmas. Let $\alpha \in (0,1)$. Define $H_{0}^{\alpha }(\varOmega )=\{u \in H ^{\alpha }(\mathbb{R}^{N}): u(x)=0 \mbox{ a.e. } x \in \mathbb{R}^{N} \setminus \varOmega \}$. For $u \in H_{0}^{\alpha }(\varOmega )$, define the norm $\|u\|_{\alpha }= ( \int _{\mathbb{R}^{N}} \int _{\mathbb{R} ^{N}}\frac{|u(x)-u(y)|^{2}}{|x-y|^{N+2\alpha }} \,\mathrm{d} x \,\mathrm{d} y )^{\frac{1}{2}}$. By [10], we get $\|u\|_{\alpha }= (\int _{\mathbb{R}^{3}}|(-\Delta )^{\frac{ \alpha }{2}}u|^{2} \,\mathrm{d}x )^{\frac{1}{2}}$. Then $(H_{0}^{\alpha }(\varOmega ), {\|\cdot\|}_{\alpha } )$ is the fractional Hilbert space. Define $\|u\|_{t}= (\int _{\varOmega }|u|^{t} \,\mathrm{d}x )^{\frac{1}{t}}$, where $t \ge 1$. Let $2_{\alpha } ^{*}=\frac{2N}{N-2\alpha }$ be the fractional Sobolev critical exponent. By [10, 24], the embedding $H^{\alpha }(\mathbb{R}^{N}) \hookrightarrow L^{t}(\mathbb{R}^{N})$ is continuous for $t \in [2,2_{ \alpha }^{*}]$, and is locally compact for $t \in [2,2_{\alpha }^{*})$. So the embedding $H^{\alpha }(\mathbb{R}^{N}) \hookrightarrow L^{2_{\alpha }^{*}}(\mathbb{R}^{N})$ is not compact. In this case, we define the best fractional Sobolev constant:

$$\begin{aligned} S_{\alpha }:=\inf_{u \in D^{\alpha ,2}(\mathbb{R}^{N}) \setminus \{0 \}} \frac{\int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}}\frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{N+2 \alpha }} \,\mathrm{d} x \,\mathrm{d} y}{ (\int _{\mathbb{R}^{N}} \vert u(x) \vert ^{2_{ \alpha }^{*}} \,\mathrm{d}x )^{\frac{2}{2^{*}_{\alpha }}}}. \end{aligned}$$

(2.1)

By [8, 26], we know that $S_{\alpha }$ can be attained by

$$\begin{aligned} U_{\varepsilon }(x)=\varepsilon ^{-\frac{N-2\alpha }{2}} \frac{\kappa _{0}}{ (\mu ^{2}+ \vert \frac{x}{\varepsilon S_{\alpha }^{\frac{1}{2 \alpha }}} \vert ^{2} )^{\frac{N-2\alpha }{2}}}, \end{aligned}$$

(2.2)

where $\varepsilon >0$, $\kappa _{0} \in \mathbb{R}\setminus \{0\}$, $\mu >0$. Moreover,

$$\begin{aligned} \Vert U_{\varepsilon } \Vert _{\alpha }^{2}= \int _{\mathbb{R}^{N}} \vert U_{\varepsilon } \vert ^{2_{\alpha }^{*}} \,\mathrm{d}x=S_{\alpha }^{\frac{N}{2\alpha }}. \end{aligned}$$

(2.3)

Choose $r_{0}>0$ such that $B_{2 r_{0}}(0) \subset \varOmega$. Define $u_{\varepsilon }(x) = \psi (x) U_{\varepsilon }(x)$, where $\psi \in C_{0}^{\infty } (B_{2r_{0}}(0) )$ such that $\psi (x) = 1$ for $|x|< r_{0}$, $0 \leq \psi \leq 1$ and $|\nabla \psi | \le 2$. By [26], we get

$$\begin{aligned} \Vert u_{\varepsilon } \Vert _{\alpha }^{2} \le S_{\alpha }^{\frac{N}{2\alpha }} + O\bigl(\varepsilon ^{N-2\alpha } \bigr). \end{aligned}$$

(2.4)

We introduce the following Hardy–Littlewood–Sobolev inequality, which leads to a new type of critical problem.

Lemma 2.1

(Hardy–Littlewood–Sobolev inequality [14])

Let s, $t>1$ and $\mu \in (0,N)$ with $\frac{1}{s}+\frac{1}{t}+\frac{\mu }{N}=2$. Let $f \in L^{s}(\mathbb{R}^{N})$ and $h \in L^{t}(\mathbb{R}^{N})$. Then there exists a constant $C(N,s,t,\mu )$ independent of f, h such that

$$\begin{aligned} \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}}\frac{f(x)h(y)}{ \vert x-y \vert ^{ \mu }} \,\mathrm{d} x \,\mathrm{d} y \le C(N,s,t,\mu ) \Vert f \Vert _{s} \Vert h \Vert _{t}. \end{aligned}$$

If $s=t=\frac{2N}{2N-\mu }$, then $C(N,s,t,\mu )=C(N,\mu )= \pi ^{\frac{\mu }{2}}\frac{\varGamma (\frac{N}{2}-\frac{\mu }{2})}{\varGamma (N-\frac{\mu }{2})} \{\frac{\varGamma (\frac{N}{2})}{\varGamma (N)} \} ^{-1+\frac{\mu }{N}}$.

By Lemma 2.1, we get, for all $u \in D^{\alpha ,2}(\mathbb{R}^{N})$,

$$\begin{aligned} \biggl( \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{ \vert u(x) \vert ^{2_{\mu }^{*}} \vert u(y) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }}\,\mathrm{d} x \,\mathrm{d} y \biggr)^{\frac{1}{2_{\mu }^{*}}} \le \bigl[C(N,\mu )\bigr]^{\frac{1}{2_{\mu }^{*}}} \Vert u \Vert _{2_{\alpha }^{*}}^{2}. \end{aligned}$$

(2.5)

Here $2_{\mu }^{*}=\frac{2N-\mu }{N-2\alpha }$ is called the critical exponent for the Hardy–Littlewood–Sobolev inequality. In order to deal with the term $(\int _{\varOmega } \frac{|u(y)|^{2_{\mu }^{*}}}{|x-y|^{\mu }}\,\mathrm{d}y ) |u|^{2_{\mu }^{*}-2} u$, we define the constant $S_{\alpha ,\mu }$ in (1.6). Since (2.5) holds, by the definitions of $S_{\alpha ,\mu }$ and $S_{\alpha }$, we have the following result. The proof can be found in [9].

Lemma 2.2

$S_{\alpha ,\mu }=\frac{S_{\alpha }}{[C(N,\mu )]^{\frac{N-2\alpha }{2N-\mu }}}$. Moreover, the constant $S_{\alpha ,\mu }$ is achieved if and only if

$$\begin{aligned} u(x)=C \biggl(\frac{c}{c^{2}+ \vert x-d \vert ^{2}} \biggr)^{\frac{N-2\alpha }{2}}, \end{aligned}$$

where $C>0$, $d \in \mathbb{R}^{N}$, and $c >0$.

We establish the following Lemmas 2.3–2.5, which is crucial for estimating upper boundedness for the functional of (1.1).

Lemma 2.3

Let $\varepsilon \in (0, \frac{r_{0}}{\mu S_{\alpha }^{\frac{1}{2 \alpha }}} )$. Then there exists $\sigma >0$ such that $\int _{\varOmega }|u_{\varepsilon }|^{q} \,\mathrm{d} x \ge \sigma \varepsilon ^{\frac{(N-2\alpha )q}{2}}$.

Proof

By $\varepsilon \in (0, \frac{r_{0}}{\mu S_{\alpha }^{\frac{1}{2 \alpha }}} )$, we get $\mu ^{2} \varepsilon ^{2} \le \frac{r_{0} ^{2}}{ S_{\alpha }^{\frac{1}{\alpha }}}$. Then

$$\begin{aligned} \int _{\varOmega } \vert u_{\varepsilon } \vert ^{q} \,\mathrm{d} x &\ge \int _{B_{r_{0}}(0)} \vert U _{\varepsilon } \vert ^{q} \,\mathrm{d} x = \int _{B_{r_{0}}(0)} \frac{\kappa _{0}^{q} \varepsilon ^{\frac{(N-2\alpha )q}{2}}}{ (\mu ^{2} \varepsilon ^{2}+ \vert \frac{x}{ S_{\alpha }^{\frac{1}{2\alpha }}} \vert ^{2} )^{\frac{(N-2\alpha )q}{2}}} \,\mathrm{d} x \\ &\ge \frac{\kappa _{0}^{q} \varepsilon ^{\frac{(N-2\alpha )q}{2}}}{ (\frac{2 r_{0}^{2}}{ S_{\alpha }^{\frac{1}{\alpha }}} ) ^{\frac{(N-2\alpha )q}{2}}} \int _{B_{r_{0}}(0)}\,\mathrm{d}x. \end{aligned}$$

So Lemma 2.3 holds. □

The relationship between $S_{\alpha ,\mu }$ and $S_{\alpha }$ is crucial for proving the following estimate.

Lemma 2.4

$$\begin{aligned} \int _{\varOmega } \int _{\varOmega } \frac{ \vert u_{\varepsilon }(y) \vert ^{2_{\mu } ^{*}} \vert u_{\varepsilon }(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }}\,\mathrm{d}x \,\mathrm{d}y \ge \bigl[C(N,\mu )\bigr]^{\frac{N}{2 \alpha }}S_{\alpha ,\mu }^{\frac{2N- \mu }{2 \alpha }}-O \bigl(\varepsilon ^{\frac{2N-\mu }{2}}\bigr). \end{aligned}$$

(2.6)

Proof

By a direct calculation,

$$\begin{aligned} & \int _{\varOmega } \int _{\varOmega } \frac{ \vert u_{\varepsilon }(y) \vert ^{2_{\mu } ^{*}} \vert u_{\varepsilon }(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }}\,\mathrm{d}x \,\mathrm{d}y \\ &\quad \ge \int _{B_{r_{0}}(0)} \int _{B_{r_{0}}(0)} \frac{ \vert U_{ \varepsilon }(y) \vert ^{2_{\mu }^{*}} \vert U_{\varepsilon }(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d}x \,\mathrm{d}y \\ &\quad = \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert U_{\varepsilon }(y) \vert ^{2_{ \mu }^{*}} \vert U_{\varepsilon }(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }} \,\mathrm{d}x \, \mathrm{d}y- 2 \int _{\mathbb{R}^{N} \setminus B_{r_{0}}(0)} \int _{B_{r_{0}}(0)} \frac{ \vert U_{\varepsilon }(y) \vert ^{2_{\mu }^{*}} \vert U_{ \varepsilon }(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }}\,\mathrm{d}x \,\mathrm{d}y \\ & \qquad {}- \int _{\mathbb{R}^{N} \setminus B_{r_{0}}(0)} \int _{\mathbb{R}^{N} \setminus B_{r_{0}}(0)} \frac{ \vert U_{\varepsilon }(y) \vert ^{2_{ \mu }^{*}} \vert U_{\varepsilon }(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }} \,\mathrm{d}x \, \mathrm{d}y. \end{aligned}$$

(2.7)

By Lemma 2.2, we know that $S_{\alpha ,\mu }$ is attained by $U_{\varepsilon }$. Together with $\|U_{\varepsilon }\|_{\alpha }^{2}=S_{\alpha }^{\frac{N}{2\alpha }}$ and $S_{\alpha ,\mu }=\frac{S_{\alpha }}{[C(N,\mu )]^{\frac{N-2\alpha }{2N-\mu }}}$, we get

$$\begin{aligned} \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert U_{\varepsilon }(y) \vert ^{2_{\mu }^{*}} \vert U_{\varepsilon }(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }}\,\mathrm{d}x \,\mathrm{d}y= \bigl[C(N,\mu )\bigr]^{\frac{N}{2 \alpha }}S_{\alpha ,\mu }^{\frac{2N-\mu }{2 \alpha }}. \end{aligned}$$

(2.8)

By Lemma 2.1,

$$\begin{aligned} & \int _{\mathbb{R}^{N} \setminus B_{r_{0}}(0)} \int _{B_{r_{0}}(0)} \frac{ \vert U _{\varepsilon }(y) \vert ^{2_{\mu }^{*}} \vert U_{\varepsilon }(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d}x \,\mathrm{d}y \\ &\quad \le \int _{\mathbb{R}^{N} \setminus B_{r_{0}}(0)} \int _{B_{r_{0}}(0)} \frac{C \varepsilon ^{2N-\mu }}{(\varepsilon ^{2}+ \vert x \vert ^{2})^{\frac{2N- \mu }{2}}(\varepsilon ^{2}+ \vert y \vert ^{2})^{\frac{2N-\mu }{2}} \vert x-y \vert ^{\mu }} \,\mathrm{d}x \,\mathrm{d}y \\ &\quad \le C \varepsilon ^{2N-\mu } \biggl( \int _{\mathbb{R}^{N} \setminus B _{r_{0}}(0)}\frac{1}{(\varepsilon ^{2}+ \vert y \vert ^{2})^{N}}\,\mathrm{d}y \biggr) ^{\frac{2N-\mu }{2N}} \biggl( \int _{B_{r_{0}}(0)}\frac{1}{(\varepsilon ^{2}+ \vert x \vert ^{2})^{N}}\,\mathrm{d}x \biggr)^{\frac{2N-\mu }{2N}} \\ &\quad =O\bigl(\varepsilon ^{\frac{2N-\mu }{2}}\bigr). \end{aligned}$$

(2.9)

Also,

$$\begin{aligned} & \int _{\mathbb{R}^{N} \setminus B_{r_{0}}(0)} \int _{\mathbb{R}^{N} \setminus B_{r_{0}}(0)} \frac{ \vert U_{\varepsilon }(y) \vert ^{2_{ \mu }^{*}} \vert U_{\varepsilon }(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }} \,\mathrm{d}x \,\mathrm{d}y \\ &\quad \le \int _{\mathbb{R}^{N} \setminus B_{r_{0}}(0)} \int _{\mathbb{R}^{N} \setminus B_{r_{0}}(0)} \frac{C \varepsilon ^{2N- \mu }}{(\varepsilon ^{2}+ \vert x \vert ^{2})^{\frac{2N-\mu }{2}}(\varepsilon ^{2}+ \vert y \vert ^{2})^{\frac{2N- \mu }{2}} \vert x-y \vert ^{\mu }}\,\mathrm{d}x \,\mathrm{d}y \\ &\quad \le C \varepsilon ^{2N-\mu } \biggl( \int _{\mathbb{R}^{N} \setminus B _{r_{0}}(0)}\frac{1}{(\varepsilon ^{2}+ \vert y \vert ^{2})^{N}}\,\mathrm{d}y \biggr) ^{\frac{2N-\mu }{2N}} \biggl( \int _{\mathbb{R}^{N} \setminus B_{r_{0}}(0)}\frac{1}{( \varepsilon ^{2}+ \vert x \vert ^{2})^{N}}\,\mathrm{d}x \biggr)^{\frac{2N-\mu }{2N}} \\ &\quad =O\bigl(\varepsilon ^{2N-\mu }\bigr). \end{aligned}$$

(2.10)

By (2.7)–(2.10), we get (2.6). □

Lemma 2.5

Let $\beta >0$, $\mu \in (0,4 \alpha )$, and $t>0$. When $N \ge 4$, or $N=3$ with $\alpha \in (0,\frac{3}{4}]$, we assume $(f_{1})$–$(f_{3})$; when $N=3$ with $\alpha \in (\frac{3}{4},1)$, we assume $(f_{1})$–$(f_{2})$ and $\lim_{u \rightarrow +\infty }\frac{F(u)}{u^{\frac{4\alpha -\mu }{3-2 \alpha }}}=+\infty$. Then there exists $C_{0}>0$ such that, for all $L>0$, there exists $\varepsilon (t,L)>0$ such that, for $\varepsilon \in (0,\varepsilon (t,L))$,

$$\begin{aligned} \int _{\varOmega } \int _{\varOmega }\frac{\beta F(tu_{\varepsilon }(y)) \vert tu _{\varepsilon }(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }}\,\mathrm{d}x \,\mathrm{d}y \ge C_{0} t^{\frac{2(N+2\alpha -\mu )}{N-2\alpha }} L \varepsilon ^{N-2\alpha }. \end{aligned}$$

Proof

When $N \ge 4$, or $N=3$ with $\alpha \in (0,\frac{3}{4}]$, by $(f_{3})$, we get $\frac{F(u)}{u^{2}}$ is increasing for $u>0$. Since $\frac{4 \alpha -\mu }{N-2 \alpha }<2$ when $N \ge 4$, or $N=3$ with $\alpha \in (0,\frac{3}{4}]$, by $(f_{2})$, we get $\lim_{u \rightarrow +\infty }\frac{F(u)}{u^{\frac{4 \alpha -\mu }{N-2 \alpha }}}=+\infty$. So, for all $L>0$, there exists $R_{L}>0$ such that $F(u) \ge L |u|^{\frac{4\alpha -\mu }{N-2\alpha }}$ for $u \ge R_{L}$. Let $\varepsilon (t,L)=\min \{1,\frac{r_{0}}{ \mu S_{\alpha }^{\frac{1}{2\alpha }}},\frac{1}{2 \mu ^{2}} (\frac{t \kappa _{0}}{R_{L}} )^{\frac{2}{N-2\alpha }} \}$ and $\varepsilon \in (0,\varepsilon (t,L))$. Since $u_{\varepsilon }(x) \ge \frac{\kappa _{0} \varepsilon ^{\frac{2\alpha -N}{2}}}{(2 \mu ^{2})^{\frac{N-2 \alpha }{2}}}$ for $|x| \le \mu S_{\alpha }^{\frac{1}{2\alpha }} \varepsilon$, we obtain that

$$\begin{aligned} F(tu_{\varepsilon }) \ge \frac{L t^{\frac{4\alpha -\mu }{N-2\alpha }} \kappa _{0}^{\frac{4\alpha -\mu }{N-2\alpha }} \varepsilon ^{\frac{\mu -4\alpha }{2}}}{(2 \mu ^{2})^{\frac{4\alpha - \mu }{2}}}, \quad \vert x \vert \le \mu S_{\alpha }^{\frac{1}{2\alpha }}\varepsilon . \end{aligned}$$

Then, by $(f_{2})$, we derive that there exist $C_{0}'$, $C_{0}>0$ such that

$$\begin{aligned} & \int _{\varOmega } \int _{\varOmega }\frac{\beta F(tu_{\varepsilon }(y)) \vert tu _{\varepsilon }(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }}\,\mathrm{d}x \,\mathrm{d}y \\ &\quad \ge \int _{B_{\mu S_{\alpha }^{\frac{1}{2\alpha }}\varepsilon }(0)} \int _{B_{\mu S_{\alpha }^{\frac{1}{2\alpha }}\varepsilon }(0)}\frac{C _{0}' F(tu_{\varepsilon }(y)) \vert tu_{\varepsilon }(x) \vert ^{2_{\mu }^{*}}}{ \vert x \vert ^{ \mu }+ \vert y \vert ^{\mu }}\,\mathrm{d}x \,\mathrm{d} y \\ &\quad \ge \frac{C_{0}'}{2 (\mu S_{\alpha }^{\frac{1}{2\alpha }} \varepsilon )^{\mu }} \int _{B_{\mu S_{\alpha }^{\frac{1}{2\alpha }}\varepsilon }(0)} \int _{B_{\mu S_{\alpha }^{\frac{1}{2\alpha }}\varepsilon }(0)} F\bigl(tu _{\varepsilon }(y)\bigr) \bigl\vert tu_{\varepsilon }(x) \bigr\vert ^{2_{\mu }^{*}}\,\mathrm{d}x \, \mathrm{d} y \\ &\quad \ge C_{0} t^{\frac{2(N+2\alpha -\mu )}{N-2\alpha }} L \varepsilon ^{N-2\alpha }. \end{aligned}$$

So Lemma 2.5 holds. □

Let $H(u)=\beta F(u)+|u|^{2_{\mu }^{*}}$ and $h(u)=\frac{\partial H(u)}{ \partial u}$. Define the functional on $H_{0}^{\alpha }(\varOmega )$ by

$$\begin{aligned} I_{\gamma }(u) = \frac{a}{2} \Vert u \Vert _{\alpha }^{2}+\frac{b}{4} \Vert u \Vert _{ \alpha }^{4} -\frac{1}{2} \int _{\varOmega } \int _{\varOmega } \frac{H(u(y)) H(u(x))}{ \vert x-y \vert ^{ \mu }} \,\mathrm{d} x \,\mathrm{d} y - \frac{\gamma }{q} \int _{\varOmega } \vert u \vert ^{q} \, \mathrm{d}x. \end{aligned}$$

(2.11)

Then $I_{\gamma }: H_{0}^{\alpha }(\varOmega ) \mapsto \mathbb{R}$ is of class $C^{1}$ and critical points of $I_{\gamma }$ are solutions of (1.1).

Similar to the well-known Brezis–Lieb lemma in [28], we have the following Brezis–Lieb splitting.

Lemma 2.6

Assume that $(f_{1})$. If $u_{n} \rightharpoonup u$ weakly in $H_{0}^{\alpha }(\varOmega )$, let $v_{n}=u_{n}-u$, we have

$$\begin{aligned} & \int _{\varOmega } \int _{\varOmega } \frac{H(u_{n}(y))H(u_{n}(x))}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d} x \,\mathrm{d} y- \int _{\varOmega } \int _{\varOmega } \frac{H(u(y)) H(u(x))}{ \vert x-y \vert ^{\mu }}\,\mathrm{d} x \,\mathrm{d} y \\ &\quad = \int _{\varOmega } \int _{\varOmega } \frac{ \vert v_{n}(y) \vert ^{2_{\mu }^{*}} \vert v _{n}(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }}\,\mathrm{d} x \,\mathrm{d} y+o_{n}(1), \end{aligned}$$

(2.12)

and

$$\begin{aligned} & \int _{\varOmega } \int _{\varOmega }\frac{H(u_{n}(y)) h(u_{n}(x))u_{n}(x)}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d} x \,\mathrm{d} y- \int _{\varOmega } \int _{\varOmega }\frac{H(u(y)) h(u(x))u(x)}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d} x \,\mathrm{d} y \\ &\quad = 2_{\mu }^{*} \int _{\varOmega } \int _{\varOmega }\frac{ \vert v_{n}(y) \vert ^{2_{ \mu }^{*}} \vert v_{n}(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }}+o_{n}(1). \end{aligned}$$

(2.13)

Proof

By Lemmas 2.2 and 2.4 in [5], we can prove that

$$\begin{aligned} & \int _{\varOmega } \int _{\varOmega } \frac{H(u_{n}(y)) H(u_{n}(x))}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d} x \,\mathrm{d} y- \int _{\varOmega } \int _{\varOmega } \frac{H(u(y)) H(u(x))}{ \vert x-y \vert ^{\mu }}\,\mathrm{d} x \,\mathrm{d} y \\ &\quad = \int _{\varOmega } \int _{\varOmega } \frac{H(v_{n}(y)) H(v_{n}(x))}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d} x \,\mathrm{d} y+o_{n}(1). \end{aligned}$$

(2.14)

Also,

$$\begin{aligned} & \int _{\varOmega } \int _{\varOmega }\frac{H(v_{n}(y)) h(v_{n}(x))v_{n}(x)}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d} x \,\mathrm{d} y+o_{n}(1) \\ &\quad = \int _{\varOmega } \int _{\varOmega }\frac{H(u_{n}(y)) h(u_{n}(x))u_{n}(x)}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d} x \,\mathrm{d} y- \int _{\varOmega } \int _{\varOmega }\frac{H(u(y)) h(u(x))u(x)}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d} x \,\mathrm{d} y. \end{aligned}$$

(2.15)

By Lemma 2.1,

$$\begin{aligned} & \int _{\varOmega } \int _{\varOmega } \frac{ \vert F(v_{n}(y)) \vert \vert F(v_{n}(x)) \vert }{ \vert x-y \vert ^{ \mu }}\,\mathrm{d} x \,\mathrm{d} y \le C(N,\mu ) \bigl\Vert F(v_{n}) \bigr\Vert _{\frac{2N}{2N- \mu }}^{2}, \\ & \int _{\varOmega } \int _{\varOmega } \frac{ \vert F(v_{n}(y)) \vert \vert v_{n}(x) \vert ^{2_{ \mu }^{*}}}{ \vert x-y \vert ^{\mu }}\,\mathrm{d} x \,\mathrm{d} y \le C(N,\mu ) \bigl\Vert F(v _{n}) \bigr\Vert _{\frac{2N}{2N-\mu }}\||v_{n}|^{2_{\mu }^{*}}\|_{\frac{2N}{2N- \mu }}, \\ & \int _{\varOmega } \int _{\varOmega } \frac{ \vert F(v_{n}(y)) \vert \vert f(v_{n}(x)) \vert \vert v _{n}(x) \vert }{ \vert x-y \vert ^{\mu }}\,\mathrm{d} x \,\mathrm{d} y \le C(N,\mu ) \bigl\Vert F(v _{n}) \bigr\Vert _{\frac{2N}{2N-\mu }} \bigl\Vert f(v_{n})v_{n} \bigr\Vert _{\frac{2N}{2N-\mu }}, \\ & \int _{\varOmega } \int _{\varOmega } \frac{ \vert v_{n}(y) \vert ^{2_{\mu }^{*}} \vert f(v _{n}(x))v_{n}(x) \vert }{ \vert x-y \vert ^{\mu }}\,\mathrm{d} x \,\mathrm{d} y \le C(N, \mu )\||v_{n}|^{2_{\mu }^{*}}\|_{\frac{2N}{2N-\mu }} \bigl\Vert f(v_{n})v_{n} \bigr\Vert _{\frac{2N}{2N-\mu }}. \end{aligned}$$

(2.16)

Since $\lim_{n \rightarrow \infty } \|v_{n}\|_{t} = 0$ for all $t \in (2,2_{\alpha }^{*})$, by $(f_{1})$, we get $\|F(v_{n})\|_{\frac{2N}{2N- \mu }}=\|f(v_{n})v_{n}\|_{\frac{2N}{2N-\mu }}=o_{n}(1)$. Together with (2.14)–(2.16), we get the results. □

3 The case $\mu < 4\alpha$

In this section, we study (1.1) for the case $\mu < 4\alpha$ and prove Theorem 1.1. Since $\mu <4\alpha$, we get $2_{\mu }^{*} =\frac{2N-\mu }{N-2\alpha } >2$. So $2 2_{\mu }^{*} >4$. Let

$$\begin{aligned} h(t)=\frac{a S_{\alpha }^{\frac{N}{2\alpha }} }{2} t^{2}+ \frac{b S _{\alpha }^{\frac{N}{\alpha }} }{4}t^{4} -\frac{[C(N,\mu )]^{\frac{N}{2 \alpha }}S_{\alpha ,\mu }^{\frac{2N-\mu }{2\alpha }}}{2}t^{2 2_{ \mu }^{*}}, \end{aligned}$$

where $t>0$. By the structure of h, there exists $T \in (0,+\infty )$ such that $h(T)=\sup_{t \ge 0} h(t)$ and $h'(T)=0$. Moreover, $h'(t)>0$ for $t \in (0,T)$ and $h'(t)<0$ for $t \in (T,+\infty )$. Let $\lambda _{1}=\inf_{u \in H_{0}^{\alpha }(\varOmega ) \setminus \{0\}}\frac{ \|u\|_{\alpha }^{2}}{\int _{\varOmega }|u|^{2} \,\mathrm{d}x}$. Then

$$\begin{aligned} \lambda _{1} \int _{\varOmega } \vert u \vert ^{2} \,\mathrm{d}x \le \Vert u \Vert _{\alpha }^{2}, \quad \forall u \in H_{0}^{\alpha }(\varOmega ). \end{aligned}$$

(3.1)

By the Sobolev embedding theorem, there exists $S_{q}>0$ such that

$$\begin{aligned} \biggl( \int _{\varOmega } \vert u \vert ^{q}\,\mathrm{d} x \biggr)^{\frac{2}{q}} \le \frac{ \Vert u \Vert _{\alpha }^{2}}{S_{q}}, \quad \forall u \in H_{0}^{\alpha }(\varOmega ). \end{aligned}$$

(3.2)

Let $\eta _{0}=\frac{a(2-q)}{4q} (\frac{4-q}{2a} )^{ \frac{2}{2-q}}\frac{1}{S_{q}^{\frac{q}{2-q}}}$. We establish the following local compactness result for $I_{\gamma }$, which plays an important role in applying the critical point theorems.

Lemma 3.1

Let $\beta >0$, $\gamma \ge 0$. Assume that $(f_{1})$ and $(f_{3})$. If $\{u_{n}\} \subset H_{0}^{\alpha }(\varOmega )$ is a sequence such that $I_{\gamma }(u_{n}) \rightarrow c \in (0, \sup_{t \ge 0} h(t)- \eta _{0} \gamma ^{\frac{2}{2-q}} )$ and $I_{\gamma }'(u_{n}) \rightarrow 0$, then $\{u_{n}\}$ converges strongly in $H_{0}^{\alpha }(\varOmega )$ up to a subsequence.

Proof

By $I_{\gamma }(u_{n}) \rightarrow c$, $I_{\gamma }'(u_{n})\rightarrow 0$, $(f_{3})$, and (3.2),

$$\begin{aligned} & c_{\gamma }+o_{n}(1)+o_{n}(1) \Vert u_{n} \Vert _{\alpha } \\ &\quad = I_{\gamma }(u_{n}) -\frac{1}{4} \bigl(I_{\gamma }'(u_{n}),u_{n} \bigr) \\ &\quad = \frac{a}{4} \Vert u_{n} \Vert _{\alpha }^{2} +\frac{1}{2} \int _{\varOmega } \int _{\varOmega }\frac{H(u_{n}(y)) (\frac{1}{2}h(u_{n}(x))u_{n}(x)-H(u _{n}(x)))}{ \vert x-y \vert ^{\mu }}\,\mathrm{d} x \,\mathrm{d} y \\ &\qquad{} - \gamma \biggl(\frac{1}{q}-\frac{1}{4} \biggr) \int _{\varOmega } \vert u _{n} \vert ^{q}\,\mathrm{d}x \ge \frac{a}{4} \Vert u_{n} \Vert _{\alpha }^{2}- \gamma \biggl( \frac{1}{q}-\frac{1}{4} \biggr)\frac{ \Vert u_{n} \Vert _{\alpha }^{q}}{S _{q}^{\frac{q}{2}}}. \end{aligned}$$

(3.3)

So $\|u_{n}\|_{\alpha }$ is bounded. Assume that $u_{n} \rightharpoonup u$ weakly in $H_{0}^{\alpha }(\varOmega )$. Let $A= \lim_{n \rightarrow \infty } \|u_{n}\|_{\alpha }^{2}$. Define the functionals $\hat{I}_{\gamma }$, $\tilde{I}_{\gamma }$, Ĵ, J̃ on $H_{0}^{\alpha }(\varOmega )$ by

$$\begin{aligned} &\hat{I}_{\gamma }(u)=\frac{a}{2} \Vert u \Vert _{\alpha }^{2}+\frac{bA}{4} \Vert u \Vert _{\alpha }^{2}-\frac{\gamma }{q} \int _{\varOmega } \vert u \vert ^{q}\,\mathrm{d}x- \frac{1}{2} \int _{\varOmega } \int _{\varOmega }\frac{H(u(y)) H(u(x))}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d} x \,\mathrm{d} y, \\ &\tilde{I}_{\gamma }(u) = \frac{a}{2} \Vert u \Vert _{\alpha }^{2}+ \frac{bA}{2} \Vert u \Vert _{\alpha }^{2}-\frac{\gamma }{q} \int _{\varOmega } \vert u \vert ^{q} \, \mathrm{d}x-\frac{1}{2} \int _{\varOmega } \int _{\varOmega }\frac{H(u(y)) H(u(x))}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d} x \,\mathrm{d} y, \\ & \hat{J}(u)= \frac{a}{2} \Vert u \Vert _{\alpha }^{2}+ \frac{bA}{4} \Vert u \Vert _{ \alpha }^{2}- \frac{1}{2} \int _{\varOmega } \int _{\varOmega }\frac{ \vert u(y) \vert ^{2_{ \mu }^{*}} \vert u(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }}\,\mathrm{d} x \,\mathrm{d} y, \\ & \tilde{J}(u) = \frac{a}{2} \Vert u \Vert _{\alpha }^{2}+ \frac{bA}{2} \Vert u \Vert _{\alpha }^{2}- \frac{1}{2} \int _{\varOmega } \int _{\varOmega }\frac{ \vert u(y) \vert ^{2_{ \mu }^{*}} \vert u(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }}\,\mathrm{d} x \,\mathrm{d} y. \end{aligned}$$

By $I_{\gamma }(u_{n}) \rightarrow c$, $I_{\gamma }'(u_{n})\rightarrow 0$, we get $\hat{I}_{\gamma }(u_{n}) \rightarrow c$, $\tilde{I}_{ \gamma }'(u_{n})\rightarrow 0$. Then $\tilde{I}_{\gamma }'(u)=0$. Let $v_{n}=u_{n}-u$. By Lemma 2.6,

$$\begin{aligned} c - \hat{I}_{\gamma }(u)=\hat{I}_{\gamma }(u_{n})- \hat{I}_{\gamma }(u) +o_{n}(1) =\hat{J}(v_{n})+o_{n}(1). \end{aligned}$$

(3.4)

Also,

$$\begin{aligned} o_{n}(1)=\bigl(\tilde{I}_{\gamma }'(u_{n}),u_{n} \bigr)-\bigl(\tilde{I}_{\gamma }'(u),u\bigr)=\bigl( \tilde{J}'(v_{n}),v_{n} \bigr)+o_{n}(1). \end{aligned}$$

(3.5)

By (2.1), (2.5), and (3.5),

$$\begin{aligned} a \lim_{n \rightarrow \infty } \Vert v_{n} \Vert _{\alpha }^{2}+b \lim_{n \rightarrow \infty } \Vert v_{n} \Vert _{\alpha }^{4} \le \frac{2_{ \mu }^{*} C(N,\mu )}{S_{\alpha }^{\frac{2N-\mu }{N-2\alpha }}} \lim_{n \rightarrow \infty } \Vert v_{n} \Vert _{\alpha }^{2 2_{\mu }^{*}}. \end{aligned}$$

(3.6)

Assume that $\lim_{n \rightarrow \infty } \|v_{n}\|_{\alpha }^{2}= l$. If $l>0$, by (3.6), we get

$$\begin{aligned} a l +b l^{2} \le \frac{2_{\mu }^{*} C(N,\mu )}{S_{\alpha }^{\frac{2N- \mu }{N-2\alpha }}} l^{2_{\mu }^{*}}. \end{aligned}$$

(3.7)

By (3.7) and Lemma 2.2, we have $h' (\frac{l^{ \frac{1}{2}}}{S_{\alpha }^{\frac{N}{4\alpha }}} ) \le 0$. Then $\frac{l^{\frac{1}{2}}}{S_{\alpha }^{\frac{N}{4\alpha }}} \ge T$. So, by (3.4)–(3.5) and Lemma 2.2,

$$\begin{aligned} c&=\lim_{n \rightarrow \infty } \biggl( \hat{J}(v_{n})-\frac{1}{2 2_{ \mu }^{*}}\bigl(\tilde{J}'(v_{n}),v_{n} \bigr) \biggr)+\hat{I}_{\gamma }(u) \\ &\ge \frac{a(N+2\alpha -\mu )}{2(2N-\mu )}\lim_{n \rightarrow \infty } \Vert v_{n} \Vert _{\alpha }^{2} + \frac{b(4\alpha -\mu )}{4(2N-\mu )} \lim_{n \rightarrow \infty } \Vert v_{n} \Vert _{\alpha }^{4}+\hat{I}_{\gamma }(u) \\ &\ge\frac{a(N+2\alpha -\mu )}{2(2N-\mu )} S_{\alpha }^{\frac{N}{2 \alpha }}T^{2}+ \frac{b(4\alpha -\mu )}{4(2N-\mu )}S_{\alpha }^{\frac{N}{ \alpha }}T^{4}+ \hat{I}_{\gamma }(u) \\ &= h(T)-\frac{1}{2 2_{\mu }^{*}}\bigl(h'(T),T\bigr)+ \hat{I}_{\gamma }(u)= \sup_{t \ge 0}h(t)+ \hat{I}_{\gamma }(u). \end{aligned}$$

(3.8)

Since $\tilde{I}_{\gamma }'(u)=0$, by $(f_{3})$,

$$\begin{aligned} \hat{I}_{\gamma }(u) &= \hat{I}_{\gamma }(u)- \frac{1}{4}\bigl(\tilde{I}_{ \gamma }'(u),u\bigr) \ge \frac{a}{4} \Vert u \Vert _{\alpha }^{2}- \gamma \biggl(\frac{1}{q}- \frac{1}{4} \biggr) \frac{ \Vert u \Vert _{\alpha }^{q}}{S_{q}^{\frac{q}{2}}} \\ &\ge \inf_{t \ge 0} \biggl[\frac{a}{4} t^{2} - \gamma \biggl(\frac{1}{q}- \frac{1}{4} \biggr) \frac{1}{S_{q}^{\frac{q}{2}}}t^{q} \biggr] =-\eta _{0} \gamma ^{\frac{2}{2-q}}. \end{aligned}$$

(3.9)

By (3.8)–(3.9), we get a contradiction. So $l=0$. By (2.1) and (2.5), we have $\lim_{n \rightarrow \infty } \int _{\varOmega } \int _{\varOmega } \frac{|v _{n}(y)|^{2_{\mu }^{*}} |v_{n}(x)|^{2_{\mu }^{*}}}{|x-y|^{\mu }} \,\mathrm{d} x \,\mathrm{d} y = 0$. Then, by (3.5), we get $u_{n} \rightarrow u_{\gamma }$ in $H_{0}^{\alpha }(\varOmega )$. □

From Lemma 3.1, we know that it is crucial to prove the estimate of upper boundedness for $I_{\gamma }$. Now we obtain the following result.

Lemma 3.2

Let $\beta >0$. Assume that $(f_{1})$–$(f_{3})$. Then there exists $\gamma _{1}'>0$ such that $\sup_{t \ge 0}I_{\gamma }(t u_{\varepsilon }) < \sup_{t \ge 0} h(t)-\eta _{0} \gamma ^{\frac{2}{2-q}}$ for $\gamma \in [0,\gamma _{1}')$.

Proof

By (2.4) and Lemma 2.4, there exists $\varepsilon _{1} \in (0,1)$ such that $\|u_{\varepsilon }\|_{\alpha }^{2} \le \frac{3 S_{\alpha } ^{\frac{N}{2\alpha }}}{2}$ and $\int _{\varOmega } \int _{\varOmega } \frac{|u _{\varepsilon }(y)|^{2_{\mu }^{*}}|u_{\varepsilon }(x)|^{2_{\mu }^{*}}}{|x-y|^{ \mu }} \,\mathrm{d} x \,\mathrm{d} y \ge \frac{[C(N,\mu )]^{\frac{N}{2 \alpha }}S_{\alpha ,\mu }^{\frac{2N-\mu }{2\alpha }}}{2}$ for $\varepsilon \in (0,\varepsilon _{1})$. Let $\varepsilon \in (0, \varepsilon _{1})$. Then, by $(f_{2})$, there exist small $t_{1} \in (0,1)$ and large $t_{2}>1$ independent of ε and γ such that $\sup_{t \in [0, t_{1}]\cup [t_{2},+\infty )}I _{\gamma } (t u_{\varepsilon } ) \le \frac{1}{2}\sup_{t \ge 0}h(t)$. Let

$$\begin{aligned} y(t)=\frac{at^{2}}{2} \Vert u_{\varepsilon } \Vert _{\alpha }^{2} + \frac{b t ^{4}}{4} \Vert u_{\varepsilon } \Vert _{\alpha }^{4}- \frac{t^{2 2_{\mu }^{*}}}{2} \int _{\varOmega } \int _{\varOmega } \frac{ \vert u _{\varepsilon }(y) \vert ^{2_{\mu }^{*}} \vert u_{\varepsilon }(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{ \mu }} \,\mathrm{d} x \,\mathrm{d} y. \end{aligned}$$

By (2.4) and Lemma 2.4, there exists $\varepsilon _{2} \in (0, \varepsilon _{1})$ such that, for $\varepsilon \in (0,\varepsilon _{2})$,

$$\begin{aligned} \max_{t \in [t_{1},t_{2}]}y(t) \le \sup _{t \ge 0} h(t)+C \varepsilon ^{N-2\alpha } + C \varepsilon ^{\frac{2N-\mu }{2}}. \end{aligned}$$

(3.10)

By (3.10), Lemmas 2.3 and 2.5, there exists $\varepsilon _{3} \in (0,\varepsilon _{2})$ such that, for $\varepsilon \in (0,\varepsilon _{3})$,

$$\begin{aligned} \max_{t \in [t_{1},t_{2}]}I_{\gamma }(t u_{\varepsilon }) < \sup_{t \ge 0} h(t) - \frac{\gamma \sigma t_{1}^{q} \varepsilon ^{\frac{(N-2\alpha )q}{2}}}{q}. \end{aligned}$$

(3.11)

So $\sup_{t \ge 0}I_{\gamma }(t u_{\varepsilon }) < \sup_{t \ge 0} h(t)$ for $\gamma =0$. For $\gamma >0$, we let $\varepsilon = \gamma ^{\frac{1}{(N-2\alpha )(2-q)}}$. Then $\max_{t \in [t_{1},t_{2}]}I_{\gamma }(t u_{\varepsilon }) < \sup_{t \ge 0} h(t) - \frac{\sigma t_{1}^{q} \gamma ^{\frac{4-q}{2(2-q)}}}{q}$. Thus, there exists $\gamma _{1}'' >0$ independent of t such that $\max_{t \in [t_{1},t_{2}]}I_{\gamma }(t u_{\varepsilon }) < \sup_{t \ge 0}h(t) - \eta _{0}\gamma ^{\frac{2}{2-q}}$ for $\gamma \in (0,\gamma _{1}'')$. Recall that $\sup_{t \in [0, t_{1}]\cup [t_{2},+\infty )}I_{\gamma } (t u_{\varepsilon } ) \le \frac{1}{2}\sup_{t \ge 0}h(t)$. Then there exists $\gamma _{1}' \in (0,\gamma _{1}'')$ such that $\sup_{t \ge 0}I_{\gamma }(t u_{\varepsilon }) < \sup_{t \ge 0} h(t)- \eta _{0} \gamma ^{\frac{2}{2-q}}$ for $\gamma \in (0,\gamma _{1}')$. □

Since Lemmas 3.1–3.2 hold, by using the Ekeland variational principle and the mountain pass theorem, we prove that (1.1) has two different nontrivial solutions. Moreover, we obtain some further properties of solutions.

Proof of Theorem 1.1

By $(f_{1})$, Lemma 2.1, (2.1), and (3.1), there exists $C_{1}>0$ such that

$$\begin{aligned} & \int _{\varOmega } \int _{\varOmega } \frac{H(u(y)) H(u(x))}{ \vert x-y \vert ^{\mu }} \,\mathrm{d} x \,\mathrm{d} y \\ &\quad \le C_{1} \biggl( \int _{\varOmega } \vert u \vert ^{2} \,\mathrm{d}x \biggr)^{\frac{2N- \mu }{N}} + C_{1} \biggl( \int _{\varOmega } \vert u \vert ^{2_{\alpha }^{*}} \,\mathrm{d}x \biggr)^{\frac{2N-\mu }{N}} \\ &\quad \le \frac{C_{1}}{\lambda _{1}^{\frac{2N-\mu }{N}}} \Vert u \Vert _{\alpha } ^{\frac{2(2N-\mu )}{N}} +\frac{C_{1}}{S_{\alpha }^{2_{\mu }^{*}}} \Vert u \Vert _{\alpha }^{\frac{2(2N-\mu )}{N-2\alpha }}. \end{aligned}$$

(3.12)

By (3.2) and (3.12),

$$\begin{aligned} I_{\gamma }(u) &\ge \frac{a}{2} \Vert u \Vert _{\alpha }^{2} -\frac{C_{1}}{ \lambda _{1}^{\frac{2N-\mu }{N}}} \Vert u \Vert _{\alpha }^{\frac{2(2N-\mu )}{N}} -\frac{C_{1}}{S_{\alpha }^{2_{\mu }^{*}}} \Vert u \Vert _{\alpha }^{\frac{2(2N- \mu )}{N-2\alpha }}-\frac{\gamma }{q S_{q}^{\frac{q}{2}}} \Vert u \Vert _{ \alpha }^{q}. \end{aligned}$$

(3.13)

Let

$$\begin{aligned} L_{0}= \min \biggl\{ \biggl(\frac{a}{8C_{1}} \biggr)^{\frac{N}{2(N- \mu )}}\lambda _{1}^{\frac{2N-\mu }{2(N-\mu )}}, \biggl( \frac{a}{8C _{1}} \biggr)^{\frac{N-2\alpha }{2(N-\mu +2\alpha )}}S_{\alpha }^{\frac{2N- \mu }{2(N-\mu +2\alpha )}} \biggr\} . \end{aligned}$$

Let $\gamma \in (0, \frac{aqS_{q}^{\frac{q}{2}}}{8} (\frac{L _{0}}{2} )^{2-q} )$. Choose $\rho _{0} \in ( \frac{L_{0}}{2},L_{0})$. Then, by (3.13), we get $I_{\gamma }(u) \ge \frac{a}{8}\|u\|_{\alpha }^{2} \ge \frac{a}{8}\rho _{0}^{2}$ for $\|u\|_{\alpha }=\rho _{0}$. Let $\gamma \in (0,\min \{\gamma _{1}', \frac{aqS_{q}^{\frac{q}{2}}}{8} (\frac{L_{0}}{2} ) ^{2-q} \} )$ with $\gamma _{1}'$ given in Lemma 3.2. Choose $u_{0} \in H_{0}^{\alpha }(\varOmega ) \setminus \{0\}$. By $(f_{2})$, we have $I_{\gamma }(tu_{0}) \le \frac{at^{2}}{2} \|u_{0}\|_{\alpha } ^{2}+\frac{bt^{4}}{4}\|u_{0}\|_{\alpha }^{4} - \frac{\gamma t^{q}}{q} \int _{\varOmega }|u_{0}|^{q} \,\mathrm{d}x$. Then $I_{\gamma }(t u_{0})<0$ for $t>0$ sufficiently small. So $\inf_{\|u\|_{\alpha }\le \rho _{0}}I_{\gamma }(u)<0$. By the Ekeland variational principle, we derive that there exists a sequence $\{u_{n}\} \subset H_{0}^{\alpha }(\varOmega )$ such that $I_{\gamma }(u _{n}) \rightarrow \inf_{\|u\|_{\alpha }\le \rho _{0}}I_{\gamma }(u)<0$ and $I_{\gamma }'(u_{n}) \rightarrow 0$. By Lemma 3.1, there exists $u_{1,\gamma } \in H_{0}^{\alpha }(\varOmega )$ such that $u_{n} \rightarrow u_{1,\gamma }$ in $H_{0}^{\alpha }(\varOmega )$. Then $I_{\gamma }(u_{1, \gamma })<0$, $I_{\gamma }'(u_{1,\gamma })=0$. We note that

$$\begin{aligned} 0 & > I_{\gamma }(u_{1,\gamma }) -\frac{1}{4} \bigl(I_{\gamma }'(u_{1, \gamma }),u_{1,\gamma } \bigr) \\ & \ge \frac{a}{4} \Vert u_{1,\gamma } \Vert _{\alpha }^{2} - \gamma \biggl(\frac{1}{q}- \frac{1}{4} \biggr) \int _{\varOmega } \vert u_{1,\gamma } \vert ^{q}\,\mathrm{d}x \ge \frac{a}{4} \Vert u_{1,\gamma } \Vert _{\alpha }^{2}- \gamma \biggl( \frac{1}{q}- \frac{1}{4} \biggr)\frac{ \Vert u_{1,\gamma } \Vert _{\alpha }^{q}}{S_{q}^{ \frac{q}{2}}}, \end{aligned}$$

from which we derive that $\|u_{1,\gamma }\|_{\alpha }\rightarrow 0$ as $\gamma \rightarrow 0$.

By $(f_{2})$, we get $\lim_{t \rightarrow +\infty }I_{\gamma }(t u _{0})\rightarrow -\infty$. Also, $I_{\gamma }(0)=0$. By the mountain pass theorem in [2], there exists a sequence $\{u_{n}\} \subset H_{0}^{\alpha }(\varOmega )$ such that $I_{\gamma }(u_{n}) \rightarrow c_{\gamma }$ and $I_{\gamma }'(u_{n}) \rightarrow 0$, where $c_{\gamma }=\inf_{g \in G_{\gamma }} \max_{0 \le t \le 1}I_{\gamma }(g(t))$ with $G_{\gamma }= \{g \in C ([0,1],H_{0} ^{\alpha }(\varOmega )): g(0) = 0, I_{\gamma }(g(1)) <0\}$. Recall that $c_{\gamma }\ge \frac{a}{8}\rho _{0}^{2}$. By the definition of $c_{\gamma }$ and Lemma 3.2, we get $c_{\gamma }\le \sup_{t \ge 0}I _{\gamma }(t u_{\varepsilon }) < \sup_{t \ge 0} h(t)-\eta _{0} \gamma ^{\frac{2}{2-q}}$. By Lemma 3.1, there exists $u_{2,\gamma } \in H_{0}^{\alpha }(\varOmega )$ such that $u_{n} \rightarrow u_{2, \gamma }$ in $H_{0}^{\alpha }(\varOmega )$. So $I_{\gamma }(u_{2,\gamma })=c_{\gamma }\ge \frac{a}{8}\rho _{0}^{2}$ and $I_{\gamma }'(u_{2, \gamma })=0$.

Let $\gamma \ge 0$. For all $g \in G_{0}$, we have $g \in G_{\gamma }$. Then $c_{\gamma } \le \max_{t \in [0,1]}I_{\gamma }(g(t)) \le \max_{t \in [0,1]}I_{0}(g(t))$ for all $g \in G_{0}$, from which we derive that $c_{\gamma } \le c_{0}$. Then $I_{\gamma }(u_{2,\gamma })=c _{\gamma }\in [\frac{a}{8}\rho _{0}^{2},c_{0} ]$, $I_{\gamma }'(u_{2,\gamma })=0$. By (3.3), we know that $\|u_{2,\gamma }\|_{\alpha }$ is bounded. Then $\lim_{\gamma \rightarrow 0}I_{0}(u_{2,\gamma })= \lim_{\gamma \rightarrow 0}I_{\gamma }(u_{2,\gamma }) \in [\frac{a}{8} \rho _{0}^{2},c_{0} ]$, $\lim_{\gamma \rightarrow 0}I_{0}'(u_{2, \gamma })=0$. By Lemma 3.2, we have $c_{0} < \sup_{t \ge 0}h(t)$. Then, by Lemma 3.1, we get $u_{2,\gamma } \rightarrow u_{0}$ as $\gamma \rightarrow 0$, $I_{0}(u_{0}) \in [\frac{a}{8}\rho _{0} ^{2},c_{0} ]$ and $I_{0}'(u_{0})=0$. □

4 The case $\mu =4\alpha$ and $b>\frac{2}{S_{\alpha ,\mu }^{2}}$

In this section, we study (1.1) for the case $\mu = 4\alpha$ and prove Theorem 1.2. Since $\mu =4\alpha$, we have $2_{\mu }^{*}=\frac{2N-\mu }{N-2\alpha }=2$. By (1.6),

$$\begin{aligned} b \Vert u \Vert _{\alpha }^{4}-2 \int _{\varOmega } \int _{\varOmega } \frac{ \vert u(y) \vert ^{2} \vert u(x) \vert ^{2}}{ \vert x-y \vert ^{4 \alpha }} \,\mathrm{d} x \,\mathrm{d} y \ge \biggl(b-\frac{2}{S_{\alpha , \mu }^{2}} \biggr) \Vert u \Vert _{\alpha }^{4}. \end{aligned}$$

(4.1)

We first establish the following compactness result for $I_{\gamma }$.

Lemma 4.1

Let $\beta >0$, $\gamma \ge 0$. Assume that $(f_{1})$. If $\{u_{n}\} \subset H_{0}^{\alpha }(\varOmega )$ is a sequence such that $I_{\gamma }(u_{n}) \rightarrow c$ and $I_{\gamma }'(u_{n}) \rightarrow 0$, then $\{u_{n}\}$ converges strongly in $H_{0}^{\alpha }(\varOmega )$ up to a subsequence.

Proof

By $(f_{1})$, for all $\delta >0$, there exists $C_{\delta }>0$ such that $|F(u)| \le \delta |u|^{2_{\mu }^{*}}+ C_{\delta }|u|^{\frac{2N- \mu }{N}}$ for $u \in \mathbb{R}$. Then by (2.1), (3.1), and Lemma 2.1, there exists $C_{2}>0$ such that

$$\begin{aligned} & \biggl\vert \int _{\varOmega } \int _{\varOmega } \frac{\beta ^{2} F(u(y)) F(u(x)) + 2\beta F(u(y)) \vert u(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }} \,\mathrm{d} x \,\mathrm{d} y \biggr\vert \\ &\quad \le \beta ^{2} C(N,\mu ) \bigl\Vert F(u) \bigr\Vert _{\frac{2N}{2N-\mu }}^{2} + 2\beta C(N, \mu ) \bigl\Vert F(u) \bigr\Vert _{\frac{2N}{2N-\mu }}\||u|^{2_{\mu }^{*}}\|_{\frac{2N}{2N- \mu }} \\ &\quad \le C_{2} \beta ^{2} \biggl[\delta ^{2} \biggl( \int _{\varOmega } \vert u \vert ^{2_{ \alpha }^{*}} \,\mathrm{d}x \biggr)^{\frac{2N-\mu }{N}}+C_{\delta }^{2} \biggl( \int _{\varOmega } \vert u \vert ^{2} \,\mathrm{d}x \biggr)^{\frac{2N-\mu }{N}} \biggr] \\ &\qquad {}+ C_{2} \beta \biggl[\delta \biggl( \int _{\varOmega } \vert u \vert ^{2_{ \alpha }^{*}} \,\mathrm{d}x \biggr)^{\frac{2N-\mu }{N}}+C_{\delta } \biggl( \int _{\varOmega } \vert u \vert ^{2} \,\mathrm{d}x \biggr)^{\frac{2N-\mu }{2N}} \biggl( \int _{\varOmega } \vert u \vert ^{2_{\alpha }^{*}} \,\mathrm{d}x \biggr)^{\frac{2N- \mu }{2N}} \biggr] \\ &\quad \le C_{2} \beta ^{2} \biggl(\frac{\delta ^{2}}{S_{\alpha }^{\frac{2N- \mu }{N-2\alpha }}} \Vert u \Vert _{\alpha }^{\frac{2(2N-\mu )}{N-2\alpha }} +\frac{C _{\delta }^{2}}{\lambda _{1}^{\frac{2N-\mu }{N}}} \Vert u \Vert _{\alpha }^{\frac{2(2N- \mu )}{N}} \biggr) \\ & \qquad{} + C_{2} \beta \biggl[\frac{\delta }{S_{\alpha }^{\frac{2N- \mu }{N-2\alpha }}} \Vert u \Vert _{\alpha }^{\frac{2(2N-\mu )}{N-2\alpha }} +\frac{C _{\delta }}{S_{\alpha }^{\frac{2N-\mu }{2(N-2\alpha )}} \lambda _{1} ^{\frac{2N-\mu }{2N}}} \Vert u \Vert _{\alpha }^{\frac{2N-\mu }{N-2\alpha }+\frac{2N- \mu }{N}} \biggr]. \end{aligned}$$

(4.2)

Since $\mu =4\alpha$, by (3.2) and (4.1)–(4.2),

$$\begin{aligned} I_{\gamma }(u) &\ge\frac{a}{2} \Vert u \Vert _{\alpha }^{2}+\frac{1}{4} \biggl(b- \frac{2}{S _{\alpha ,\mu }^{2}} \biggr) \Vert u \Vert _{\alpha }^{4} - C_{2} \beta ^{2} \biggl(\frac{\delta ^{2}}{S_{\alpha }^{2}} \Vert u \Vert _{\alpha }^{4} +\frac{C _{\delta }^{2}}{\lambda _{1}^{\frac{2(N-2\alpha )}{N}}} \Vert u \Vert _{\alpha } ^{\frac{4(N-2\alpha )}{N}} \biggr) \\ &\quad {}- C_{2} \beta \biggl[\frac{\delta }{S_{\alpha }^{2}} \Vert u \Vert _{\alpha } ^{4} + \frac{C_{\delta }}{S_{\alpha }\lambda _{1}^{\frac{N-2\alpha }{N}}} \Vert u \Vert _{\alpha }^{2+\frac{2(N-2\alpha )}{N}} \biggr]-\frac{\gamma }{q S _{q}^{\frac{q}{2}}} \Vert u \Vert _{\alpha }^{q}. \end{aligned}$$

(4.3)

By choosing $\delta >0$ sufficiently small, we get $I_{\gamma }(u) \rightarrow +\infty$ as $\|u\|_{\alpha }\rightarrow \infty$. Since $I_{\gamma }(u_{n}) \rightarrow c$, we obtain that $\|u_{n}\|_{\alpha }$ is bounded. Assume that $u_{n} \rightharpoonup u$ weakly in $H_{0}^{\alpha }(\varOmega )$. Let $A= \lim_{n \rightarrow \infty } \|u_{n}\|_{\alpha }^{2}$. Define the functionals $\tilde{I}_{\gamma }$, J̃ on $H_{0}^{\alpha }( \varOmega )$ by

$$\begin{aligned} &\tilde{I}_{\gamma }(u) = \frac{a}{2} \Vert u \Vert _{\alpha }^{2}+ \frac{bA}{2} \Vert u \Vert _{\alpha }^{2}-\frac{\gamma }{q} \int _{\varOmega } \vert u \vert ^{q} \, \mathrm{d}x-\frac{1}{2} \int _{\varOmega } \int _{\varOmega }\frac{H(u(y)) H(u(x))}{ \vert x-y \vert ^{4 \alpha }}\,\mathrm{d} x \,\mathrm{d} y, \\ & \tilde{J}(u) = \frac{a}{2} \Vert u \Vert _{\alpha }^{2}+ \frac{bA}{2} \Vert u \Vert _{\alpha }^{2}- \frac{1}{2} \int _{\varOmega } \int _{\varOmega }\frac{ \vert u(y) \vert ^{2} \vert u(x) \vert ^{2}}{ \vert x-y \vert ^{4 \alpha }}\,\mathrm{d} x \,\mathrm{d} y. \end{aligned}$$

By $I_{\gamma }'(u_{n})\rightarrow 0$, we get $\tilde{I}_{\gamma }'(u _{n})\rightarrow 0$. Then $\tilde{I}_{\gamma }'(u)=0$. Let $v_{n}=u _{n}-u$. By (4.1) and Lemma 2.6,

$$\begin{aligned} o_{n}(1) &= \bigl(\tilde{J}'(v_{n}),v_{n} \bigr) \ge a \Vert v_{n} \Vert _{\alpha }^{2} +b \Vert v_{n} \Vert _{\alpha }^{4} -2 \int _{\varOmega } \int _{ \varOmega } \frac{ \vert v_{n}(y) \vert ^{2} \vert v_{n}(x) \vert ^{2}}{ \vert x-y \vert ^{4\alpha }} \,\mathrm{d} x \,\mathrm{d} y \\ &\ge a \Vert v_{n} \Vert _{\alpha }^{2} + \biggl(b-\frac{2}{S_{\alpha ,\mu } ^{2}} \biggr) \Vert v_{n} \Vert _{\alpha }^{4}. \end{aligned}$$

(4.4)

So $u_{n} \rightarrow u$ in $H_{0}^{\alpha }(\varOmega )$. □

Now we use Lemma 4.1 to prove Theorem 1.2.

Proof of Theorem 1.2

Choose $r>0$ sufficiently small such that $B_{2r}(0) \subset \varOmega$. Define $w_{r} \in C_{0}^{\infty }(B_{2r}(0)) \setminus \{0\}$ such that $w_{r}(x)=\xi$ for $|x| \le r$, $w_{r}(x) \ge 0$ for $|x| \le 2r$, and $w_{r}(x)=0$ for $|x| \ge 2r$. Then $w_{r} \in H_{0}^{\alpha }(\varOmega )$. Moreover, we have $F(w_{r})=F(\xi )>0$ for $|x| \le r$. By $(f_{2})$,

$$\begin{aligned} I_{\gamma }(w_{r}) \le I_{0}(w_{r}) \le \frac{a}{2} \Vert w_{r} \Vert _{\alpha }^{2}+ \frac{b}{4} \Vert w_{r} \Vert _{\alpha }^{4} - \int _{B_{r}(0)} \int _{B_{r}(0)} \frac{\beta F(w_{r}(y)) \vert w_{r}(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{ \mu }} \,\mathrm{d} x \,\mathrm{d} y. \end{aligned}$$

(4.5)

Then there exists $\beta _{0}>0$ such that $I_{\gamma }(w_{r})<0$ for $\beta >\beta _{0}$. Let $\beta >\beta _{0}$. By choosing $\delta >0$ sufficiently small in (4.3), we derive that

$$\begin{aligned} I_{\gamma }(u) \ge \frac{a}{2} \Vert u \Vert _{\alpha }^{2} - \frac{C_{2} C_{ \delta }^{2} \beta ^{2}}{\lambda _{1}^{\frac{2(N-2 \alpha )}{N}}} \Vert u \Vert _{\alpha }^{\frac{4(N-2\alpha )}{N}}- \frac{C_{2} C_{\delta }\beta }{S _{\alpha }\lambda _{1}^{\frac{N-2\alpha }{N}}} \Vert u \Vert _{\alpha }^{2+\frac{2(N-2 \alpha )}{N}}-\frac{\gamma }{q S_{q}^{\frac{q}{2}}} \Vert u \Vert _{\alpha } ^{q}. \end{aligned}$$

(4.6)

Let

$$\begin{aligned} L_{1}= \min \biggl\{ \Vert w_{r} \Vert _{\alpha }, \biggl(\frac{a \lambda _{1} ^{\frac{2(N-2 \alpha )}{N}}}{8C_{2} C_{\delta }^{2} \beta ^{2}} \biggr) ^{\frac{N}{2(N-4\alpha )}}, \biggl(\frac{a S_{\alpha }\lambda _{1}^{\frac{2(N-2 \alpha )}{2N}}}{8C_{2} C_{\delta }\beta } \biggr)^{\frac{N}{2(N-2\alpha )}} \biggr\} . \end{aligned}$$

Let $\gamma \in (0,\frac{aqS_{q}^{\frac{q}{2}}}{8} (\frac{L _{1}}{2} )^{2-q} )$. Choose $\varrho \in (\frac{L_{1}}{2},L _{1})$. By (4.6), we get $I_{\gamma }(u) \ge \frac{a}{8}\|u\| _{\alpha }^{2} = \frac{a}{8}\varrho ^{2}$ for $\|u\|_{\alpha }=\varrho$. Also, $I_{\gamma }(0)=0$. By the mountain pass theorem in [2], there exists a sequence $\{u_{n}\} \subset H_{0}^{\alpha }( \varOmega )$ such that $I_{\gamma }(u_{n}) \rightarrow c_{\gamma }'$ and $I_{\gamma }'(u_{n}) \rightarrow 0$, where $c_{\gamma }'= \inf_{g \in G_{\gamma }} \max_{0 \le t \le 1}I_{\gamma }(g(t))$ with $G_{\gamma }= \{g \in C ([0,1],H_{0}^{\alpha }(\varOmega )): g(0) = 0, I _{\gamma }(g(1)) <0\}$. Moreover, $c_{\gamma }' \ge \frac{a}{8}\varrho ^{2}$. By Lemma 4.1, there exists $v_{0,\gamma } \in H_{0}^{\alpha }(\varOmega )$ such that $u_{n} \rightarrow v_{0,\gamma }$ in $H_{0}^{ \alpha }(\varOmega )$. Then $I_{\gamma }(v_{0,\gamma })=c_{\gamma }' \ge \frac{a}{8}\varrho ^{2}$ and $I_{\gamma }'(v_{0,\gamma })=0$. By the proof of Theorem 1.1, we get $c_{\gamma }' \le c_{0}'$. Similar to (4.2), we can derive from $(f_{1})$ that, for all $\delta >0$, there exists $C_{\delta }>0$ such that, for all $u \in H_{0}^{\alpha }(\varOmega )$,

$$\begin{aligned} & \biggl\vert \int _{\varOmega } \int _{\varOmega } \frac{\beta ^{2} F(u(y)) f(u(x))u(x) + \beta (2_{\mu }^{*} F(u(y))+f(u(y))u(y) ) \vert u(x) \vert ^{2_{ \mu }^{*}}}{ \vert x-y \vert ^{\mu }} \,\mathrm{d} x \,\mathrm{d} y \biggr\vert \\ &\quad \le C \beta ^{2} \bigl(\delta ^{2} \Vert u \Vert _{\alpha }^{\frac{2(2N-\mu )}{N-2 \alpha }} +C_{\delta }^{2} \Vert u \Vert _{\alpha }^{\frac{2(2N-\mu )}{N}} \bigr)+ C \beta \bigl(\delta \Vert u \Vert _{\alpha }^{\frac{2(2N-\mu )}{N-2\alpha }} +C _{\delta } \Vert u \Vert _{\alpha }^{\frac{2N-\mu }{N-2\alpha }+ \frac{2N-\mu }{N}} \bigr). \end{aligned}$$

(4.7)

Since $I_{\gamma }'(v_{0,\gamma })=0$, by (3.2), (4.1), and (4.7),

$$\begin{aligned} &a \Vert v_{0,\gamma } \Vert _{\alpha }^{2}+ \biggl(b-\frac{2}{S_{\alpha , \mu }^{2}} \biggr) \Vert v_{0,\gamma } \Vert _{\alpha }^{4}-\frac{\gamma }{S _{q}^{\frac{q}{2}}} \Vert v_{0,\gamma } \Vert _{\alpha }^{q} \\ &\quad \le C \beta ^{2} \bigl(\delta ^{2} \Vert v_{0,\gamma } \Vert _{\alpha }^{4} +C _{\delta }^{2} \Vert v_{0,\gamma } \Vert _{\alpha }^{\frac{4(N-2\alpha )}{N}} \bigr) + C \beta \bigl(\delta \Vert v_{0,\gamma } \Vert _{\alpha }^{4} +C_{\delta } \Vert v_{0,\gamma } \Vert _{\alpha }^{2+\frac{2(N-2\alpha )}{N}} \bigr). \end{aligned}$$

(4.8)

By choosing $\delta >0$ sufficiently small, we derive that $\|v_{0,\gamma }\|_{\alpha }$ is bounded. Then we have $\lim_{\gamma \rightarrow 0}I_{0}(v_{0,\gamma })\in [\frac{a}{8} \varrho ^{2},c_{0}' ]$ and $\lim_{\gamma \rightarrow 0}I_{0}'(v _{0,\gamma })=0$. By Lemma 4.1, we get $v_{0,\gamma } \rightarrow v _{0}$ as $\gamma \rightarrow 0$, $I_{0}(v_{0}) \in [\frac{a}{8} \varrho ^{2},c_{0}' ]$ and $I_{0}'(v_{0})=0$.

Recall that $I_{\gamma }(w_{r}) \le I_{0}(w_{r})<0$ with $\|w_{r}\| _{\alpha }> \varrho$ and $I_{\gamma }(u) \rightarrow +\infty$ as $\|u\|_{\alpha }\rightarrow \infty$. Then there exists $R>0$ independent of γ such that $\varrho < \|w_{r}\|_{\alpha }<R$ and $I_{\gamma }(u)>0$ for $\|u\|_{\alpha }=R$. Let $m_{1,\gamma }= \inf_{\varrho \le \|u\|_{\alpha }\le R}I_{\gamma }(u)$. Then $m_{1,\gamma } \le I_{\gamma }(w_{r}) \le I_{0}(w_{r})<0$. By the Ekeland variational principle, there exists a sequence $\{u_{n}\} \subset H_{0}^{1}(\varOmega )$ such that $\varrho < \|u_{n}\|_{\alpha }<R$, $I_{\gamma }(u_{n}) \rightarrow m_{1,\gamma }$, and $I_{\gamma }'(u _{n}) \rightarrow 0$. By Lemma 4.1, there exists $v_{1,\gamma } \in H_{0}^{1}(\varOmega )$ such that $u_{n} \rightarrow v_{1,\gamma }$ in $H_{0}^{1}(\varOmega )$. Then $\varrho < \|v_{1,\gamma }\|_{\alpha }< R$, $I_{\gamma }(v_{1,\gamma })=m_{1,\gamma } \le I_{0}(w_{r})<0$, and $I_{\gamma }'(v_{1,\gamma })=0$.

By $(f_{2})$, we get

$$\begin{aligned} I_{\gamma }(t w_{r}) \le \frac{a t^{2}}{2} \Vert w_{r} \Vert _{\alpha }^{2}+ \frac{b t^{4}}{4} \Vert w_{r} \Vert _{\alpha }^{4} -\frac{ \gamma t^{q}}{q} \int _{\varOmega } \vert w_{r} \vert ^{q} \,\mathrm{d} x. \end{aligned}$$

Then there exists small $t_{r}>0$ such that $\|t_{r} w_{r}\|_{\alpha }<\varrho$ and $I_{\gamma }(t_{r} w_{r})<0$. Let $m_{2,\gamma }= [4] \inf_{\|u\|_{\alpha }\le \varrho }I_{\gamma }(u)$. Then $m_{2,\gamma }<0$. By the Ekeland variational principle, we derive that there exists a sequence $\{u_{n}\} \subset H_{0}^{1}(\varOmega )$ such that $\|u_{n}\|_{\alpha }< \varrho$, $I_{\gamma }(u_{n}) \rightarrow m _{2,\gamma }$, and $I_{\gamma }'(u_{n}) \rightarrow 0$. By Lemma 4.1, there exists $v_{2,\gamma } \in H_{0}^{1}(\varOmega )$ such that $u_{n} \rightarrow v_{2,\gamma }$ in $H_{0}^{1}(\varOmega )$. Then $\|v_{2,\gamma }\|_{\alpha }< \varrho$, $I_{\gamma }(v_{2,\gamma })=m _{2,\gamma }<0$, and $I_{\gamma }'(v_{2,\gamma })=0$.

Assume that there exists $v_{\gamma }\in H_{0}^{\alpha }(\varOmega )$ such that $\|v_{\gamma }\|_{\alpha }$ is bounded, $I_{\gamma }(v_{\gamma })<0$ and $I_{\gamma }'(v_{\gamma })=0$. Then $I_{0}(v_{\gamma }) \le o_{\gamma }(1)$ and $I_{0}'(v_{\gamma })=o _{\gamma }(1)$. By Lemma 4.1, we get $v_{\gamma }\rightarrow v$ in $H_{0}^{\alpha }(\varOmega )$ as $\gamma \rightarrow 0$. Thus, by $\varrho < \|v_{1,\gamma }\|_{\alpha }< R$, $I_{\gamma }(v_{1,\gamma }) \le I_{0}(w_{r})<0$, and $I_{\gamma }'(v_{1,\gamma })=0$, we obtain that $v_{1,\gamma } \rightarrow v_{1}$ in $H_{0}^{\alpha }(\varOmega )$ as $\gamma \rightarrow 0$, $I_{0}(v_{1})<0$ and $I_{0}'(v_{1})=0$. Also, by $\|v_{2,\gamma }\|_{\alpha }< \varrho$, $I_{\gamma }(v_{2,\gamma })=m_{2,\gamma }<0$, and $I_{\gamma }'(v_{2,\gamma })=0$, we obtain that $v_{2,\gamma } \rightarrow v_{2}$ in $H_{0}^{\alpha }(\varOmega )$ as $\gamma \rightarrow 0$ with $\|v_{2}\|_{\alpha }\le \varrho$, $I_{0}(v_{2}) \le 0$, and $I_{0}'(v_{2})=0$. By (4.6), we get $I_{0}(u) \ge \frac{a}{4}\|u\|_{\alpha }^{2} -\frac{\gamma }{q S_{q}^{\frac{q}{2}}}\|u\|_{\alpha }^{q}$ for $\|u\|_{\alpha }\le \varrho$. Then $v_{2}=0$. □

5 The case $\mu > 4 \alpha$

In this section, we study (1.1) for the case $\mu > 4\alpha$ and prove Theorem 1.3. Since $\mu >4\alpha$, we have $2_{\mu }^{*}=\frac{2N-\mu }{N-2\alpha }<2$. Then $2 2_{\mu }^{*}<4$. We first establish the following compactness result for $I_{\gamma }$.

Lemma 5.1

Let $b>0$, $a>a(b)$, $\gamma \ge 0$ with $a(b)$ given in (1.7). Assume that $(f_{1})$. If $\{u_{n}\} \subset H_{0}^{\alpha }(\varOmega )$ is a sequence such that $I_{\gamma }(u_{n}) \rightarrow c$ and $I_{\gamma }'(u_{n}) \rightarrow 0$, then $\{u_{n}\}$ converges strongly in $H_{0}^{\alpha }(\varOmega )$ up to a subsequence.

Proof

In this case, we know that (3.12) holds. By (3.2) and (3.12),

$$\begin{aligned} I_{\gamma }(u) &\ge \frac{a}{2} \Vert u \Vert _{\alpha }^{2}+\frac{b}{4} \Vert u \Vert _{\alpha }^{4} -\frac{C_{1}}{\lambda _{1}^{\frac{2N-\mu }{N}}} \Vert u \Vert _{\alpha }^{\frac{2(2N-\mu )}{N}} -\frac{C_{1}}{S_{\alpha }^{2_{ \mu }^{*}}} \Vert u \Vert _{\alpha }^{22_{\mu }^{*}}-\frac{\gamma }{q S_{q}^{ \frac{q}{2}}} \Vert u \Vert _{\alpha }^{q}. \end{aligned}$$

(5.1)

By (5.1), we get $I_{\gamma }(u) \rightarrow +\infty$ as $\|u\|_{\alpha }\rightarrow \infty$. Then $\|u_{n}\|_{\alpha }$ is bounded. We assume that $u_{n} \rightharpoonup u$ weakly in $H_{0}^{\alpha }(\varOmega )$. Let $A=\lim_{n \rightarrow \infty } \|u _{n}\|_{\alpha }^{2}$. Define the functionals $\tilde{I}_{\gamma }$, J̃ on $H_{0}^{\alpha }(\varOmega )$ by

$$\begin{aligned} &\tilde{I}_{\gamma }(u) = \frac{a}{2} \Vert u \Vert _{\alpha }^{2}+ \frac{bA}{2} \Vert u \Vert _{\alpha }^{2}-\frac{\gamma }{q} \int _{\varOmega } \vert u \vert ^{q} \, \mathrm{d}x-\frac{1}{2} \int _{\varOmega } \int _{\varOmega }\frac{H(u(y)) H(u(x))}{ \vert x-y \vert ^{ \mu }}\,\mathrm{d} x \,\mathrm{d} y, \\ & \tilde{J}(u) = \frac{a}{2} \Vert u \Vert _{\alpha }^{2}+ \frac{bA}{2} \Vert u \Vert _{\alpha }^{2}- \frac{1}{2} \int _{\varOmega } \int _{\varOmega }\frac{ \vert u(y) \vert ^{2_{ \mu }^{*}} \vert u(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }}\,\mathrm{d} x \,\mathrm{d} y. \end{aligned}$$

By $I_{\gamma }'(u_{n})\rightarrow 0$, we get $\tilde{I}_{\gamma }'(u _{n})\rightarrow 0$. Then $\tilde{I}_{\gamma }'(u)=0$. Together with Lemma 2.6, we have $(\tilde{J}'(v_{n}),v_{n} ) =o_{n}(1)$. Here $v_{n}=u_{n}-u$. Then, by (1.6),

$$\begin{aligned} a \lim_{n \rightarrow \infty } \Vert v_{n} \Vert _{\alpha }^{2}+b \lim_{n \rightarrow \infty } \Vert v_{n} \Vert _{\alpha }^{4} \le \frac{2_{ \mu }^{*}}{S_{\alpha ,\mu }^{\frac{2N-\mu }{N-2\alpha }}} \lim_{n \rightarrow \infty } \Vert v_{n} \Vert _{\alpha }^{2 2_{\mu }^{*}}. \end{aligned}$$

(5.2)

By (5.2) and Young’s inequality,

$$\begin{aligned} a\lim_{n \rightarrow \infty } \Vert v_{n} \Vert _{\alpha }^{2}+b \lim_{n \rightarrow \infty } \Vert v_{n} \Vert _{\alpha }^{4} \le a(b) \lim _{n \rightarrow \infty } \Vert v_{n} \Vert _{\alpha }^{2} + b \lim_{n \rightarrow \infty } \Vert v_{n} \Vert _{\alpha }^{4}. \end{aligned}$$

(5.3)

Then $u_{n} \rightarrow u$ in $H_{0}^{\alpha }(\varOmega )$. □

Now we use Lemma 5.1 to prove Theorem 1.3.

Proof of Theorem 1.3

Let $w_{0} \in H _{0}^{\alpha }(\varOmega ) \setminus \{0\}$. By $(f_{2})$,

$$\begin{aligned} I_{\gamma }(t w_{0}) \le \frac{a t^{2}}{2} \Vert w_{0} \Vert _{\alpha }^{2}+ \frac{b t^{4}}{4} \Vert w_{0} \Vert _{\alpha }^{4} -\frac{t ^{2 2_{\mu }^{*}}}{2} \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert w _{0}(y) \vert ^{2_{\mu }^{*}} \vert w_{0}(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }} \,\mathrm{d} x \,\mathrm{d} y. \end{aligned}$$

Obviously, there exists large $t_{0}>0$ such that

$$\begin{aligned} \frac{a t_{0}^{2}}{2} \Vert w_{0} \Vert _{\alpha }^{2} -\frac{t_{0}^{2 2_{ \mu }^{*}}}{2} \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert w _{0}(y) \vert ^{2_{\mu }^{*}} \vert w_{0}(x) \vert ^{2_{\mu }^{*}}}{ \vert x-y \vert ^{\mu }} \,\mathrm{d} x \,\mathrm{d} y< 0. \end{aligned}$$

Choose $b_{0}>0$ sufficiently small such that $I_{\gamma }(t_{0} w_{0})<0$ for $b \in (0,b_{0})$. Let $b \in (0,b _{0})$ and $a>a(b)$. Recall that (5.1) holds. Let

$$\begin{aligned} L_{2}= \min \biggl\{ \Vert t_{0} w_{0} \Vert _{\alpha }, \biggl(\frac{a \lambda _{1}^{\frac{2N-\mu }{N}}}{8C_{1}} \biggr)^{\frac{N}{2(N-\mu )}}, \biggl(\frac{a S_{\alpha }^{2_{\mu }^{*}}}{8C_{1}} \biggr)^{\frac{N-2\alpha }{2(N- \mu +2\alpha )}} \biggr\} . \end{aligned}$$

Let $\gamma \in (0,\frac{aqS_{q}^{\frac{q}{2}}}{8} (\frac{L _{2}}{2} )^{2-q} )$. Choose $\varrho \in (\frac{L_{2}}{2},L _{2})$. By (5.1), we get $I_{\gamma }(u) \ge \frac{a}{8}\|u\| _{\alpha }^{2} = \frac{a}{8}\varrho ^{2}$ for $\|u\|_{\alpha }=\varrho$. Also, $I_{\gamma }(0)=0$. By the mountain pass theorem in [2], there exists a sequence $\{u_{n}\} \subset H_{0}^{\alpha }( \varOmega )$ such that $I_{\gamma }(u_{n}) \rightarrow c_{\gamma }'' >0$ and $I_{\gamma }'(u_{n}) \rightarrow 0$. Then, by Lemma 5.1, there exists $w_{0,\gamma } \in H_{0}^{\alpha }(\varOmega )$ such that $u_{n} \rightarrow w_{0,\gamma }$ in $H_{0}^{\alpha }(\varOmega )$, $I_{\gamma }(w_{0,\gamma })=c_{\gamma }''>0$, and $I_{\gamma }'(w_{0, \gamma })=0$. Similar to the proof of Theorem 1.2, we can derive that $w_{0,\gamma } \rightarrow w_{0}$ in $H_{0}^{\alpha }(\varOmega )$ as $\gamma \rightarrow 0$, where $w_{0}$ is a nontrivial solution of (1.1) with $\gamma =0$ and $I_{0}(w_{0})>0$. Recall that $I_{\gamma }(u) \rightarrow +\infty$ as $\|u\|_{\alpha }\rightarrow \infty$. Then there exists $R>0$ such that $\varrho < \|t_{0} w_{0} \|_{\alpha }<R$ and $I_{\gamma }(u)>0$ for $\|u\|_{\alpha }=R$. We note that $I_{\gamma }(u) \ge \frac{a}{8}\varrho ^{2}$ for $\|u\|_{\alpha }= \varrho$, $I_{\gamma }(t_{0} w_{0})<0$, and $I_{\gamma }(t w_{0})<0$ for $t>0$ sufficiently small. Similar to the proof of Theorem 1.2, we can derive that $\inf_{\varrho \le \|u\|_{\alpha } \le R}I_{\gamma }(u)$ is attained by a function $w_{1,\gamma }$. Moreover, $I_{\gamma }(w_{1,\gamma })<0$, $I_{\gamma }'(w_{1,\gamma })=0$, and $w_{1,\gamma } \rightarrow w_{1}$ in $H_{0}^{\alpha }(\varOmega )$ as $\gamma \rightarrow 0$, where $w_{1}$ is a nontrivial solution of (1.1) with $\gamma =0$ and $I_{0}(w_{1})<0$. Also, $\inf_{\|u\|_{\alpha }\le \varrho }I_{\gamma }(u)$ is attained by a function $w_{2,\gamma }$. Moreover, $I_{\gamma }(w_{2,\gamma })<0$, $I_{\gamma }'(w_{2,\gamma })=0$, and $w_{2, \gamma } \rightarrow 0$ in $H_{0}^{\alpha }(\varOmega )$ as $\gamma \rightarrow 0$. □