Since we want to consider the existence of minimizers for problem (6) with \(p=\frac{8s}{N}\), we first introduce the following Gagliardo–Nirenber inequality [6]:
$$ \int _{\mathbb{R}^{N}} \vert u \vert ^{2+\frac{8s}{N}}\,dx\leq \frac{N+4s}{2N \vert U(x) \vert _{2}^{\frac{8s}{N}}} \biggl( \int _{\mathbb{R}^{N}} \vert u \vert ^{2}\,dx \biggr)^{\frac{4s-N}{N}} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2}. $$
(12)
Here the function \(U(x)\) is the unique ground state of the equation
$$ (-\Delta )^{s}u+\frac{4s-N}{2N}u= \vert u \vert ^{\frac{8s}{N}}u,\quad x\in \mathbb{R}^{N}. $$
(13)
Using the Pohozaev identity and equation (13) [6, 10], we can get that
$$ \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx= \int _{ \mathbb{R}^{N}} \vert u \vert ^{2}\,dx= \frac{2N}{N+4s} \int _{\mathbb{R}^{N}} \vert u \vert ^{2+ \frac{8s}{N}}\,dx. $$
(14)
Lemma 2.1
Assume that\(V(x)\geq 0\). Then, for any\(c\in (0,c^{*})\), we have\(e(c)\geq 0\).
Proof
For any \(u\in S_{c}\), using the Gagliardo–Nirenberg inequality (12), we get that
$$\begin{aligned} I_{p}(u)&=\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx+ \frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2}+\frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u \vert ^{ \gamma +2}\,dx \\ &\quad {}-\frac{N}{2N+8s} \int _{\mathbb{R}^{N}} \vert u \vert ^{2+\frac{8s}{N}}\,dx \\ &\geq \frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2}- \frac{c^{\frac{8s-2N}{N}}}{4 \vert U(x) \vert _{2}^{\frac{8s}{N}}} \biggl( \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad {}+ \frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u \vert ^{\gamma +2}\,dx \\ &\geq \frac{b}{4} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{ \frac{8s-2N}{N}} \biggr) \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2}+\frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u \vert ^{ \gamma +2}\,dx, \end{aligned}$$
(15)
which, together with \(V(x)\geq 0\), implies that \(I_{p}(u)>0\). Hence we have
□
Lemma 2.2
Let\(\gamma \in (\frac{4s}{N},\frac{8s}{N})\), and let\(|V|_{\infty }c^{\frac{8s-\gamma N+\gamma (4s-N)}{4s}}\)be small enough such that
$$ \begin{aligned} &\frac{ \vert V \vert _{\infty }}{\gamma +2} \biggl( \frac{N+4s}{2N \vert U \vert _{2}^{\frac{8s}{N}}} \biggr)^{\frac{\gamma N}{8s}}c^{ \frac{8s-\gamma N+\gamma (4s-N)}{4s}} \\ &\quad \leq \biggl( \frac{2as}{8s-\gamma N} \biggr)^{\frac{8s-\gamma N}{4s}} \biggl( \frac{bs}{\gamma N-4s} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{ \frac{8s-2N}{N}} \biggr) \biggr)^{\frac{\gamma N-4s}{4s}}. \end{aligned} $$
Then\(e(c)\geq 0\).
Proof
For any \(u\in S_{c}\), using the Hölder and Gagliardo–Nirenberg inequalities, we have
$$\begin{aligned} \int _{\mathbb{R}^{N}} \vert u \vert ^{\gamma +2}\,dx&\leq \biggl( \int _{\mathbb{R}^{N}} \vert u \vert ^{2}\,dx \biggr)^{\frac{8s-\gamma N}{8s}} \biggl( \int _{\mathbb{R}^{N}} \vert u \vert ^{2+ \frac{8s}{N}}\,dx \biggr)^{\frac{\gamma N}{8s}} \\ &\leq \biggl(\frac{N+4s}{2N \vert U \vert _{2}^{\frac{8s}{N}}} \biggr)^{ \frac{\gamma N}{8s}}c^{\frac{8s-\gamma N+\gamma (4s-N)}{4s}} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{ \frac{\gamma N}{4s}}, \end{aligned}$$
which, combined with (15), indicates that
$$\begin{aligned} I_{p}(u)&=\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx+ \frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2}+\frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u \vert ^{ \gamma +2}\,dx \\ &\quad {}-\frac{N}{2N+8s} \int _{\mathbb{R}^{N}} \vert u \vert ^{2+\frac{8s}{N}}\,dx \\ &\geq \frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx+ \frac{b}{4} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{\frac{8s-2N}{N}} \biggr) \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad {}-\frac{ \vert V \vert _{\infty }}{\gamma +2} \biggl( \frac{N+4s}{2N \vert U \vert _{2}^{\frac{8s}{N}}} \biggr)^{\frac{\gamma N}{8s}}c^{ \frac{8s-\gamma N+\gamma (4s-N)}{4s}} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{\frac{\gamma N}{4s}}. \end{aligned}$$
(16)
Let \(\delta =\frac{8s-\gamma N}{4s}\) and \(\beta =1-\delta =\frac{\gamma N-4s}{4s}\). Using the Young inequality, we have
$$\begin{aligned} &\frac{a}{2}t+\frac{b}{4} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{ \frac{8s-2N}{N}} \biggr)t^{2}\\ &\quad \geq \biggl(\frac{a}{2\delta } \biggr)^{\delta } \biggl( \frac{b (1- (\frac{c}{c^{*}} )^{\frac{8s-2N}{N}} )}{4\beta } \biggr)^{\beta }t^{\delta +2\beta } \\ &\quad = \biggl(\frac{2as}{8s-\gamma N} \biggr)^{\frac{8s-\gamma N}{4s}} \biggl(\frac{bs}{\gamma N-4s} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{ \frac{8s-2N}{N}} \biggr) \biggr)^{\frac{\gamma N-4s}{4s}} t^{ \frac{\gamma N}{4s}}. \end{aligned}$$
Thus from (16) it follows that
$$\begin{aligned} I_{p}(u)&\geq \biggl[ \biggl(\frac{2as}{8s-\gamma N} \biggr)^{ \frac{8s-\gamma N}{4s}} \biggl(\frac{bs}{\gamma N-4s} \biggl(1- \biggl( \frac{c}{c^{*}} \biggr)^{\frac{8s-2N}{N}} \biggr) \biggr)^{ \frac{\gamma N-4s}{4s}} \\ &\quad{}-\frac{ \vert V \vert _{\infty }}{\gamma +2} \biggl( \frac{N+4s}{2N \vert U \vert _{2}^{\frac{8s}{N}}} \biggr)^{\frac{\gamma N}{8s}}c^{ \frac{8s-\gamma N+\gamma (4s-N)}{4s}} \biggr] \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{\frac{\gamma N}{4s}}. \end{aligned}$$
(17)
If we choose \(|V|_{\infty }c^{\frac{8s-\gamma N+\gamma (4s-N)}{4s}}\) small enough such that
$$ \begin{aligned} &\frac{ \vert V \vert _{\infty }}{\gamma +2} \biggl( \frac{N+4s}{2N \vert U \vert _{2}^{\frac{8s}{N}}} \biggr)^{\frac{\gamma N}{8s}}c^{ \frac{8s-\gamma N+\gamma (4s-N)}{4s}} \\ &\quad \leq \biggl( \frac{2as}{8s-\gamma N} \biggr)^{\frac{8s-\gamma N}{4s}} \biggl( \frac{bs}{\gamma N-4s} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{ \frac{8s-2N}{N}} \biggr) \biggr)^{\frac{\gamma N-4s}{4s}}, \end{aligned} $$
then (17) indicates that
□
Lemma 2.3
If\(c>c^{*}\), then\(e(c)<-\infty \).
Proof
Set
$$ u_{\lambda }(x)=\frac{c\lambda ^{\frac{N}{2}}}{ \vert U \vert _{2}}U(\lambda x). $$
Then using (14), we have
$$\begin{aligned}& \int _{\mathbb{R}^{N}}u_{\lambda }^{2}(x) \,dx=c^{2}, \\& \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert u_{\lambda }^{2}(x)-u_{\lambda }^{2}(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx \,dy= \frac{c^{2}\lambda ^{2s}}{ \vert U \vert _{2}^{2}} \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}} U \bigr\vert ^{2}\,dx=c^{2}\lambda ^{2s}, \\& \int _{\mathbb{R}^{N}} \bigl\vert u_{\lambda }(x) \bigr\vert ^{2+\frac{8s}{N}}\,dx= \frac{(N+4s)c^{2+\frac{8s}{N}}\lambda ^{4s}}{2N \vert U \vert _{2}^{\frac{8s}{N}}}, \\& \int _{\mathbb{R}^{N}}V(x) \bigl\vert u_{\lambda }(x) \bigr\vert ^{2+\gamma }\,dx= \frac{c^{2+\gamma }\lambda ^{\frac{N\gamma }{2}}}{ \vert U \vert _{2}^{2+\gamma }} \int _{\mathbb{R}^{N}}V \biggl(\frac{x}{\lambda } \biggr) \bigl\vert U(x) \bigr\vert ^{2+ \gamma }\,dx. \end{aligned}$$
Hence we can deduce that \(u_{\lambda }\in S_{c}\) and
$$\begin{aligned} I_{p}(u_{\lambda })&=\frac{a}{2}c^{2}\lambda ^{2s}+\frac{b}{4}c^{4} \lambda ^{4s}+ \frac{c^{\gamma +2}}{(2+\gamma ) \vert U \vert _{2}^{2+\gamma }} \lambda ^{\frac{N\gamma }{2}} \int _{\mathbb{R}^{N}}V \biggl( \frac{x}{\lambda } \biggr) \vert U \vert ^{2+\gamma }\,dx- \frac{c^{2+\frac{8s}{N}}}{4 \vert U \vert _{2}^{\frac{8s}{N}}}\lambda ^{4s} \\ &=\frac{a}{2}c^{2}\lambda ^{2s}+\frac{b}{4}c^{4} \lambda ^{4s} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{\frac{8s-2N}{N}} \biggr) \\ &\quad {}+ \frac{c^{\gamma +2}}{(2+\gamma ) \vert U \vert _{2}^{2+\gamma }}\lambda ^{ \frac{N\gamma }{2}} \int _{\mathbb{R}^{N}}V \biggl(\frac{x}{\lambda } \biggr) \vert U \vert ^{2+\gamma }\,dx. \end{aligned}$$
(18)
From \(\gamma <\frac{8s}{N}\) we get that \(\frac{N\gamma }{2}<4s\). Then (18) indicates that \(I_{p}(u_{\lambda })\rightarrow -\infty \) as \(\lambda \rightarrow \infty \), and the lemma is proved. □
Lemma 2.4
For any\(c>0\), we have\(e(c)\leq 0\).
Proof
For any \(u\in S_{c}\) and constant \(\lambda >0\), let \(u_{\lambda }(x)=\lambda ^{\frac{N}{2}}u(\lambda x)\). Then \(u_{\lambda }\in S_{c}\), and from (7) we have
$$\begin{aligned} I_{p}(u_{\lambda })&=\frac{a}{2}\lambda ^{2s} \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}}u \bigr\vert ^{2}\,dx+\frac{b}{4}\lambda ^{4s} \biggl( \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}}u \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad{}+\frac{1}{2+\gamma }\lambda ^{\frac{N\gamma }{2}} \int _{\mathbb{R}^{N}}V \biggl(\frac{x}{\lambda } \biggr) \bigl\vert u(x) \bigr\vert ^{2+\gamma }\,dx- \frac{1}{p+2} \lambda ^{4s} \int _{\mathbb{R}^{N}} \bigl\vert u(x) \bigr\vert ^{2+\frac{8s}{N}}\,dx. \end{aligned}$$
(19)
Hence \(I_{p}(u_{\lambda })\rightarrow 0\) as \(\lambda \rightarrow 0\), which indicates that \(e(c)\leq 0\). □
Lemma 2.5
Assume that the function\(V(x)\)satisfies condition (10), \(\frac{N\gamma }{2}+\alpha <4s\), andais small enough. Then\(e(c)<0\).
Proof
For fixed \(|x_{0}|=2\), assume that \(\varphi (x)\in C_{c}^{\infty }(\mathbb{R}^{N})\) is such that \(\operatorname{supp} \varphi \in B_{1}(x_{0})\) and \(\int _{\mathbb{R}^{N}}\varphi ^{2}(x)\,dx=c^{2}\). For constant \(\lambda >0\), take
$$ \varphi _{\lambda }(x)=\lambda ^{\frac{N}{2}}\varphi (\lambda x). $$
Then
$$\begin{aligned}& \int _{\mathbb{R}^{N}}\varphi _{\lambda }^{2}(x)\,dx= \int _{\mathbb{R}^{N}} \varphi ^{2}(x)\,dx=c^{2}, \end{aligned}$$
(20)
$$\begin{aligned}& \begin{aligned}[b] \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert \varphi _{\lambda }^{2}(x)-\varphi _{\lambda }^{2}(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx \,dy&= \lambda ^{2s} \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert \varphi ^{2}(x)-\varphi ^{2}(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx \,dy\\ &= \lambda ^{2s} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} \varphi \bigr\vert ^{2}\,dx, \end{aligned} \end{aligned}$$
(21)
$$\begin{aligned}& \int _{\mathbb{R}^{N}} \bigl\vert \varphi _{\lambda }(x) \bigr\vert ^{2+\frac{8s}{N}}\,dx= \lambda ^{4s} \int _{\mathbb{R}^{N}} \bigl\vert \varphi (x) \bigr\vert ^{2+\frac{8s}{N}}\,dx, \end{aligned}$$
(22)
and
$$ \int _{\mathbb{R}^{N}}V(x) \bigl\vert \varphi _{\lambda }(x) \bigr\vert ^{2+\gamma }\,dx= \lambda ^{\frac{N\gamma }{2}} \int _{\mathbb{R}^{N}}V\biggl(\frac{x}{\lambda }\biggr) \bigl\vert \varphi (x) \bigr\vert ^{2+\gamma }\,dx\leq -C\lambda ^{\frac{N\gamma }{2}+\alpha } $$
(23)
as \(\lambda \rightarrow 0\).
From (20) we know that \(\varphi _{\lambda }\in S_{c}\). Then (21)–(23) indicate that
$$\begin{aligned} \begin{aligned}[b] I_{p}(\varphi _{\lambda })&\leq \frac{a\lambda ^{2s}}{2} \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} \varphi \bigr\vert ^{2}\,dx+ \frac{b\lambda ^{4s}}{4} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} \varphi \bigr\vert ^{2}\,dx\\ &\quad {}-C\lambda ^{\frac{N\gamma }{2}+\alpha }- \frac{N\lambda ^{4s}}{2N+8s} \int _{\mathbb{R}^{N}} \bigl\vert \varphi (x) \bigr\vert ^{2+ \frac{8s}{N}}\,dx. \end{aligned} \end{aligned}$$
(24)
For \(2\leq \gamma <\frac{8s}{N}\), we have \(2s< N\leq \frac{N\gamma }{2}<4s\). If \(\frac{N\gamma }{2}+\alpha <4s\), then there is a small \(\lambda _{0}>0\) such that
$$ \frac{b\lambda _{0}^{4s}}{4} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} \varphi \bigr\vert ^{2}\,dx-C\lambda _{0}^{\frac{N\gamma }{2}+\alpha }- \frac{N\lambda _{0}^{4s}}{2N+8s} \int _{\mathbb{R}^{N}} \bigl\vert \varphi (x) \bigr\vert ^{2+ \frac{8s}{N}}\,dx< 0. $$
Moreover, if
$$ a< \frac{-\frac{b\lambda _{0}^{2s}}{2}\int _{\mathbb{R}^{N}} \vert (-\Delta )^{\frac{s}{2}} \varphi \vert ^{2}\,dx+2C\lambda _{0}^{\frac{N\gamma }{2}+\alpha -2s}+\frac{N\lambda _{0}^{2s}}{N+4s}\int _{\mathbb{R}^{N}} \vert \varphi (x) \vert ^{2+\frac{8s}{N}}\,dx}{\int _{\mathbb{R}^{N}} \vert (-\Delta )^{\frac{s}{2}} \varphi \vert ^{2}\,dx}, $$
then from (24) we can deduce that
$$ e(c)\leq \inf I_{p}(\varphi _{\lambda })< 0. $$
□
Lemma 2.6
For any\(c\in (0,c^{*})\)and any\(d\in (0,c)\), if\(e(c)<0\), then
$$ e(c)< e(d)+e\bigl(\sqrt{c^{2}-d^{2}}\bigr). $$
Proof
Let \(\{u_{n}\}\) be any minimizing sequence. Then
$$\begin{aligned} \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{\gamma +2} \,dx&= \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{( \gamma +2)\theta } \vert u_{n} \vert ^{(\gamma +2)(1-\theta )}\,dx \\ &\leq \biggl( \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2}\,dx \biggr)^{ \frac{(2+\gamma )\theta }{2}} \biggl( \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2^{*}(s)}\,dx \biggr)^{\frac{(2+\gamma )(1-\theta )}{2^{*}(s)}} \\ &\leq C \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{\frac{\gamma N}{4s}}, \end{aligned}$$
(25)
where \(\theta =\frac{2s(2+\gamma )-\gamma N}{2(2+\gamma )s}\).
Using (12) and (25), we have
$$\begin{aligned} I_{p}(u_{n})&=\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx+\frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad {}+ \frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u_{n} \vert ^{\gamma +2} \,dx-\frac{N}{2N+8s} \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2+\frac{8s}{N}}\,dx \\ &\geq \frac{b}{4} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{ \frac{8s-2N}{N}} \biggr) \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad {}-C \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{\frac{\gamma N}{4s}}, \end{aligned}$$
(26)
where \(c^{*}= (b|U(x)|_{2}^{\frac{8s}{N}} )^{\frac{N}{8s-2N}}\). Since \(\gamma <\frac{8s}{N}\), we have that \(\frac{\gamma N}{4s}<2\). Since \(\{u_{n}\}\) is a minimizing sequence and \(c< c^{*}\), we have \(e(c)=\lim_{n\rightarrow \infty }I_{p}(u_{n})\), and the sequence \(\{u_{n}\}\) is bounded in the space \(H^{s}(\mathbb{R}^{N})\). Moreover, from (26) we can deduce that \(0>e(c)>-\infty \) and
$$\begin{aligned} &\lim_{n\rightarrow \infty }\frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u_{n} \vert ^{ \gamma +2} \,dx \\ &\quad \leq e(c)-\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx- \frac{b}{4} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{\frac{8s-2N}{N}} \biggr) \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad < 0. \end{aligned}$$
(27)
For \(\lambda >1\), defining \(\bar{u}_{n}=\lambda u_{n}\), we have
$$\begin{aligned}& \int _{\mathbb{R}^{N}}\bar{u}_{n}^{2}\,dx=\lambda ^{2} \int _{\mathbb{R}^{N}}u_{n}^{2}\,dx= \lambda ^{2}c^{2},\qquad \int _{\mathbb{R}^{N}}V(x)\bar{u}_{n}^{ \gamma +2}\,dx=\lambda ^{\gamma +2} \int _{\mathbb{R}^{N}}V(x)u_{n}^{ \gamma +2}\,dx, \\& \begin{aligned} \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert \bar{u}_{n}^{2}(x)-\bar{u}_{n}^{2}(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx \,dy&= \lambda ^{2} \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert u_{n}^{2}(x)-u_{n}^{2}(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx \,dy\\ &=\lambda ^{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx, \end{aligned} \\& \int _{\mathbb{R}^{N}} \vert \bar{u}_{n} \vert ^{2+\frac{8s}{N}}\,dx=\lambda ^{2+ \frac{8s}{N}} \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2+\frac{8s}{N}} \,dx. \end{aligned}$$
Then
$$\begin{aligned} I_{p}(\bar{u}_{n})&=\frac{a\lambda ^{2}}{2} \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx+\frac{b\lambda ^{4}}{4} \biggl( \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad {}+ \frac{\lambda ^{2}}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u_{n} \vert ^{ \gamma +2} \,dx-\frac{N\lambda ^{2+\frac{8s}{N}}}{2N+8s} \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2+ \frac{8s}{N}}\,dx \\ &\geq \lambda ^{4}I_{p}(u_{n})+\bigl(\lambda ^{2}-\lambda ^{4}\bigr) \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \\ &\quad {}+ \bigl(\lambda ^{4}- \lambda ^{2+\frac{8s}{N}} \bigr)\frac{N}{2N+8s} \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2+ \frac{8s}{N}}\,dx \\ &\quad {}+ \bigl(\lambda ^{\gamma +2}-\lambda ^{4} \bigr) \frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u_{n} \vert ^{\gamma +2} \,dx, \end{aligned}$$
(28)
which, together with \(\lambda >1\), \(\gamma \geq 2\), and (27), indicates that
$$ e(\lambda c)\leq \lim_{n\rightarrow \infty } I_{p}( \bar{u}_{n})\leq \lambda ^{4} \lim_{n\rightarrow \infty } I_{p}(u_{n})=\lambda ^{4}e(c). $$
(29)
Since \(e(c)<0\), this means that
$$ e(\lambda c)< \lambda e(c). $$
Then for any \(d\in [0,c)\), we have
$$ e(c)< e(d)+e\bigl(\sqrt{c^{2}-d^{2}}\bigr). $$
□