In this section, to overcome the lack of compactness, we need to consider the functional \(\varphi_{\lambda}\) in the functions space E. We shall prove that \(\varphi_{\lambda}\) satisfies the conditions of Theorem 3 in the next several lemmas.
Lemma 3
Let
\(\mathcal{T}_{\lambda}\)
be the set of paths defined in Theorem
3. Then
\(\mathcal{T}_{\lambda}\neq\emptyset\)
for
\(\lambda\in I=[\delta,1]\), where
\(\delta\in(0,1)\)
is a positive constant.
Proof
We choose \(\eta\in C_{0}^{\infty}(\mathbb{R}^{N})\) with \(\|\eta\|_{E}=1\) and \(\sup\eta\subset B_{r}(0)\) for some \(r>0\). From \(H(f)\)(4), we know that for any \(c_{3}>0\) with \(c_{3}\delta\int_{B_{r}(0)}\eta^{2}\,dx> 1/2\), there exists \(c_{4}>0\) such that
$$ F(u)\geq c_{3}|u|^{2}-c_{4},\quad u\in\mathbb{R}_{+}. $$
(3)
Then we have
$$\begin{aligned} \varphi_{\lambda}(t\eta) =&\frac{t^{2}}{2} \int_{\mathbb{R}^{N}} \bigl(\bigl\vert (-\Delta )^{\frac{\alpha}{2}}\eta\bigr\vert ^{2} +V(x)\eta^{2} \bigr)\,dx-\lambda \int_{\mathbb{R}^{N}}F(t\eta) \,dx \\ =& \frac{t^{2}}{2}-\lambda \int_{\mathbb{R}^{N}}F(t\eta) \,dx \\ \leq& \frac{t^{2}}{2}-\delta c_{3}t^{2} \int_{B_{r}(0)}\eta^{2}\,dx+c_{4}\bigl\vert B_{r}(0)\bigr\vert \\ =& t^{2} \biggl(\frac{1}{2}-\delta c_{3} \int_{B_{r}(0)}\eta^{2}\,dx \biggr)+c_{4}\bigl\vert B_{r}(0)\bigr\vert . \end{aligned}$$
Therefore, we can choose \(t>0\) large such that \(\varphi_{\lambda}(t\eta)<0\). The proof is completed. □
Lemma 4
Let
\(c_{\lambda}\)
be the set of paths defined in Theorem
3. Then there exists a constant
\(\overline{c}>0\)
such that
\(c_{\lambda}\geq\overline{c}\)
for
\(\lambda\in[\delta,1]\).
Proof
By \(H(f)\)(2) and \(H(f)\)(3), for any \(\varepsilon\in(0, 1/2)\), there exists \(c_{\varepsilon}>0\) such that
$$ \bigl\vert F(u)\bigr\vert \leq\frac{\varepsilon V_{0}}{2}|u|^{2}+c_{\varepsilon}|u|^{p}, \quad u\in\mathbb{R}_{+}. $$
(4)
Furthermore, by Theorem 2, we have \(E\hookrightarrow L^{p}(\mathbb{R}^{N})\) compactly. Then there exists \(c_{5}>0\) such that \(|u|_{p}\leq c_{5}\|u\|_{E}\). Hence, for any \(u\in E\) and \(\lambda\in [\delta,1]\), using (4) it follows that
$$\begin{aligned} \varphi_{\lambda}(u) =&\frac{1}{2} \int_{\mathbb{R}^{N}} \bigl(\bigl\vert (-\Delta)^{\frac {\alpha}{2}}u\bigr\vert ^{2} +V(x)u^{2} \bigr)\,dx-\lambda \int_{\mathbb{R}^{N}}F(u) \,dx \\ \geq& \frac{1}{2}\|u\|_{E}^{2}- \int_{\mathbb{R}^{N}} \biggl( \frac{\varepsilon V_{0}}{2}|u|^{2}+c_{\varepsilon}|u|^{p} \biggr)\,dx \\ \geq& \frac{1}{2}\|u\|_{E}^{2}- \frac{\varepsilon}{2} \int_{\mathbb{R}^{N}} V(x)|u|^{2}\,dx-c_{\varepsilon}\int_{\mathbb{R}^{N}}|u|^{p}\,dx \\ \geq& \frac{1}{2}\|u\|_{E}^{2}- \frac{\varepsilon}{2} \int_{\mathbb{R}^{N}} \bigl( \bigl\vert (-\Delta)^{\frac{\alpha}{2}}u\bigr\vert ^{2}+V(x)|u|^{2} \bigr)\,dx-c_{\varepsilon}\int _{\mathbb{R}^{N}}|u|^{p}\,dx \\ \geq& \frac{1}{4}\|u\|_{E}^{2}-c_{\varepsilon}|u|_{p}^{p} \\ \geq& \frac{1}{4}\|u\|_{E}^{2}-c_{\varepsilon}c_{5}^{p}\|u\|_{E}^{p}, \end{aligned}$$
which implies that there exists \(\rho>0\) such that \(\varphi_{\lambda}(u)>0\) for every \(u\in E\) and \(\|u\|_{E}\in(0,\rho]\). In particular, for \(\|u\|_{E}=\rho\), we have \(\varphi_{\lambda}(u)\geq \overline{c}>0\). Fix \(\lambda\in[\delta,1]\) and \(\gamma\in\mathcal {T}_{\lambda}\). By the definition of \(\mathcal{T}_{\lambda}\) we can see that \(\|\gamma(1)\|_{E}>\rho\). By continuity, we deduce that there exists \(t_{\gamma}\in(0,1)\) such that \(\|\gamma(t_{\gamma})\|_{E}=\rho\). Thus, for any \(\lambda\in[\delta,1]\),
$$c_{\lambda}\geq\inf_{\gamma\in\mathcal{T}_{\lambda}}\varphi_{\lambda}\bigl( \gamma(t_{\gamma})\bigr)\geq\overline{c}>0. $$
Therefore, we complete the proof. □
Next we prove that the functional \(\varphi_{\lambda}\) can achieve the critical value at \(c_{\lambda}\) for any \(\lambda\in[\delta,1]\).
Lemma 5
For any
\(\lambda\in[\delta,1]\), each bounded
\((PS)\)
sequence of the functional
\(\varphi_{\lambda}\)
admits a convergent subsequence.
Proof
Let \(\lambda\in[\delta,1]\). Suppose that \(\{u_{n}\}\subset E\) is a \((PS)\) sequence for \(\varphi_{\lambda}\), that is, \(\{u_{n}\}\) and \(\varphi_{\lambda}(u_{n})\) are bounded, \(\varphi'_{\lambda}(u_{n})\rightarrow0\) in \(E'\), where \(E'\) is the dual space of E. Then there exists \(u\in E\) such that \(u_{n} \rightharpoonup u\) in E. Thus, Theorem 2 implies that
$$\begin{aligned}& u_{n}\rightarrow u\quad \mbox{in } L^{p}\bigl( \mathbb{R}^{N}\bigr), \\& u_{n}\rightarrow u \quad \mbox{a.e. in } \mathbb{R}^{N}. \end{aligned}$$
By virtue of hypothesis \(H(f)\)(2) and \(H(f)\)(3), for any \(\varepsilon\in(0, 1/2)\), there exists \(c_{\varepsilon}>0\) such that
$$ \bigl\vert f(u)\bigr\vert \leq \varepsilon |u|+c_{\varepsilon}|u|^{p-1},\quad u\in\mathbb{R}_{+}. $$
(5)
So it follows from (5) that
$$\begin{aligned} \biggl\vert \int_{\mathbb{R}^{N}}f(u_{n}) (u_{n}-u) \,dx\biggr\vert \leq& \int_{\mathbb {R}^{N}}\bigl\vert f(u_{n})\bigr\vert | u_{n}-u | \,dx \\ \leq& \int_{\mathbb{R}^{N}}\bigl(\varepsilon |u_{n}|+c_{\varepsilon}|u_{n}|^{p-1} \bigr)| u_{n}-u | \,dx \\ \leq& \varepsilon |u_{n}|_{2}| u_{n}-u |_{2} + c_{\varepsilon}\bigl\vert \vert u_{n}\vert ^{p-1}\bigr\vert _{p'}| u_{n}-u |_{p} \\ \leq& \varepsilon |u_{n}|_{2}| u_{n}-u |_{2} + c_{\varepsilon}|u_{n}|^{p-1}_{p}| u_{n}-u |_{p} \\ \leq& \varepsilon c_{5} \|u_{n}\|_{E}| u_{n}-u |_{2} + c_{\varepsilon}c_{5}^{p-1} \|u_{n}\|^{p-1}_{E}| u_{n}-u |_{p}, \end{aligned}$$
which implies that
$$\int_{\mathbb{R}^{N}}f(u_{n}) (u_{n}-u) \,dx \rightarrow0\quad \mbox{as } n\rightarrow\infty. $$
Thus
$$\begin{aligned}& \bigl\langle \varphi_{\lambda}'(u_{n}),u_{n}-u \bigr\rangle \\& \quad = \int_{\mathbb{R}^{N}} \bigl((-\Delta)^{\frac{\alpha}{2}}u_{n}(-\Delta )^{\frac{\alpha}{2}}(u_{n}-u) +V(x)u_{n}(u_{n}-u) \bigr)\,dx \\& \qquad {}-\lambda \int_{\mathbb{R}^{N}}f(u_{n}) (u_{n}-u) \,dx \\& \quad = \int_{\mathbb{R}^{N}} \bigl((-\Delta)^{\frac{\alpha}{2}}u_{n}(-\Delta )^{\frac{\alpha}{2}}(u_{n}-u) +V(x)u_{n}(u_{n}-u) \bigr)\,dx+o(1) \end{aligned}$$
and
$$\int_{\mathbb{R}^{N}} \bigl((-\Delta)^{\frac{\alpha}{2}}u_{n}(- \Delta)^{\frac {\alpha}{2}}(u_{n}-u)+V(x)u_{n}(u_{n}-u) \bigr)\,dx \rightarrow0 \quad \text{as } n\rightarrow\infty. $$
Therefore we conclude that \(\|u_{n}\|_{E}^{2}\rightarrow\|u\|_{E}^{2}\). This together with \(u_{n}\rightharpoonup u\) in E shows that \(u_{n}\rightarrow u\) in E. The proof is completed. □
Now we are in the position to show that the modified functional \(\varphi_{\lambda}\) has a nontrivial critical point.
Lemma 6
For almost every
\(\lambda\in[\delta,1]\), there exists
\(u_{\lambda}\in E\backslash\{0\}\)
such that
\(\varphi'_{\lambda}(u_{\lambda})=0\)
and
\(\varphi_{\lambda}(u_{\lambda})=c_{\lambda}\).
Proof
By virtue of Theorem 3, for almost every \(\lambda\in[\delta,1]\), there exists a bounded sequence \(\{u_{\lambda}^{n}\}\subset E\) such that
$$\varphi_{\lambda}\bigl(u_{\lambda}^{n}\bigr)\rightarrow c_{\lambda}\quad \mbox{and} \quad \varphi'_{\lambda}\bigl(u_{\lambda}^{n}\bigr)\rightarrow0 \quad \text{as } n \rightarrow\infty. $$
According to Lemma 5, we may assume that there exists \(u_{\lambda}\in E\) such that \(u_{\lambda}^{n}\rightarrow u_{\lambda}\) in E. Then it follows that \(\varphi_{\lambda}(u_{\lambda})= c_{\lambda}\) and \(\varphi'_{\lambda}(u_{\lambda})=0\) and \(u_{\lambda}\neq0\) from Lemma 4. □
From Lemma 6 we know that there exist a sequence \(\lambda_{n}\in[\delta,1]\) with \(\lambda_{n}\rightarrow1^{-}\) and an associated sequence \(\{u_{n}\}\subset E\) such that
$$ \varphi_{\lambda_{n}}(u_{n})=c_{\lambda_{n}} \quad \mbox{and}\quad \varphi'_{\lambda_{n}}(u_{n})=0. $$
(6)
Next, we will show that the sequence \(\{u_{n}\}\) is bounded, which is a key ingredient in this paper.
Lemma 7
Let
\(u_{n}\)
be a critical point of
\(\varphi_{\lambda_{n}}\)
at the level
\(c_{\lambda_{n}}\)
as defined in (6). Then there exists a constant
\(c_{6}>0\)
such that
\(\|u_{n}\|_{E}\leq c_{6}\)
for all
n.
Proof
In view of Lemma 2 and (6), we see that \(u_{n}\) satisfies the following Pohozaev identity:
$$\begin{aligned}& \frac{N-2\alpha}{2} \int_{\mathbb{R}^{N}}\bigl\vert (-\Delta)^{\frac{\alpha }{2}}u_{n} \bigr\vert ^{2}\,dx+\frac{N}{2} \int_{\mathbb{R}^{N}}V(x)|u_{n}|^{2}\,dx + \frac{1}{2} \int_{\mathbb{R}^{N}}\bigl\langle \nabla V(x),x\bigr\rangle |u_{n}|^{2} \,dx \\& \quad =\lambda_{n}N \int_{\mathbb{R}^{N}}F(u_{n})\,dx. \end{aligned}$$
(7)
Recall that \(\varphi_{\lambda_{n}}(u_{n})=c_{\lambda_{n}}\). So we have
$$ \frac{N}{2} \int_{\mathbb{R}^{N}} \bigl(\bigl|(-\Delta)^{\frac{\alpha}{2}}u_{n}\bigr|^{2} +V(x)u_{n}^{2} \bigr)\,dx-\lambda_{n}N \int_{\mathbb{R}^{N}}F(u_{n}) \,dx=c_{\lambda_{n}}N. $$
(8)
Then from (7), (8), and hypothesis \(H(f)\)(2), it follows that
$$\begin{aligned} \alpha \int_{\mathbb{R}^{N}}\bigl\vert (-\Delta)^{\frac{\alpha}{2}}u_{n} \bigr\vert ^{2}\,dx =&c_{\lambda _{n}}N+\frac{1}{2} \int_{\mathbb{R}^{N}} \bigl\langle \nabla V(x),x\bigr\rangle \vert u_{n}\vert ^{2}\,dx \\ \leq& c_{\lambda_{n}}N +\frac{1}{2}\bigl\vert \bigl\langle \nabla V(x),x\bigr\rangle \bigr\vert _{L^{\frac{N}{2\alpha}}(\mathbb{R}^{N})} \vert u_{n}\vert _{2_{\alpha}^{*}}^{2} \\ \leq& c_{\lambda_{n}}N +\frac{1}{2S_{\alpha}}\bigl\vert \bigl\langle \nabla V(x),x\bigr\rangle \bigr\vert _{L^{\frac{N}{2\alpha}}(\mathbb{R}^{N})} \int_{\mathbb{R}^{N}}\bigl\vert (-\Delta)^{\frac{\alpha}{2}}u_{n} \bigr\vert ^{2}\,dx. \end{aligned}$$
(9)
Set \(M_{0}=\alpha- \frac{1}{2S_{\alpha}}|\langle\nabla V(x),x\rangle|_{L^{\frac{N}{2\alpha}}(\mathbb{R}^{N})}\). Then
$$ M_{0} \int_{\mathbb{R}^{N}}\bigl\vert (-\Delta)^{\frac{\alpha}{2}}u_{n} \bigr\vert ^{2}\,dx\leq c_{\lambda_{n}}N. $$
(10)
We estimate the right-hand side of (9). By the min-max definition of the mountain pass level \(c_{\lambda_{n}}\), Lemma 3 and (3), we have
$$\begin{aligned} c_{\lambda_{n}} \leq&\max_{t} \varphi_{\lambda_{n}}(t\eta) \\ \leq& \max_{t} \biggl\{ \frac{t^{2}}{2}- \lambda_{n} \int_{\mathbb {R}^{N}}F(t\eta) \,dx \biggr\} \\ \leq& \max_{t} \biggl\{ t^{2} \biggl( \frac{1}{2}-\delta c_{3} \int_{B_{r}(0)}\eta^{2}\,dx \biggr)+c_{4}\bigl\vert B_{r}(0)\bigr\vert \biggr\} \\ \leq& c_{4}\bigl\vert B_{r}(0)\bigr\vert . \end{aligned}$$
(11)
From (10) and (11), we conclude that
$$ M_{0} \int_{\mathbb{R}^{N}}\bigl\vert (-\Delta)^{\frac{\alpha}{2}}u_{n} \bigr\vert ^{2}\,dx\leq c_{4}N\bigl\vert B_{r}(0)\bigr\vert . $$
(12)
Note that \(E\subset\dot{H}^{\alpha}(\mathbb{R}^{N})\) and the embedding \(\dot{H}^{\alpha}(\mathbb{R}^{N})\hookrightarrow L^{2_{\alpha}^{*}}(\mathbb{R}^{N})\) is continuous. Then we have
$$|u|^{2}_{2_{\alpha}^{*}}\leq\frac{1}{S_{\alpha}} \int_{\mathbb{R}^{N}}\bigl\vert (-\Delta)^{\frac{\alpha}{2}}u\bigr\vert ^{2}\,dx, \quad \forall u\in E. $$
Recall from (6)
$$ \int_{\mathbb{R}^{N}} \bigl(\bigl\vert (-\Delta)^{\frac{\alpha}{2}}u_{n} \bigr\vert ^{2} +V(x)u_{n}^{2} \bigr)\,dx=N \lambda_{n} \int_{\mathbb{R}^{N}}f(u_{n})u_{n}\, dx. $$
(13)
Using (5) in (13), we obtain
$$\begin{aligned}& \int_{\mathbb{R}^{N}} \bigl(\bigl\vert (-\Delta)^{\frac{\alpha}{2}}u_{n} \bigr\vert ^{2} +V(x)u_{n}^{2} \bigr)\,dx \\& \quad \leq N \int_{\mathbb{R}^{N}} \bigl(\varepsilon \vert u_{n}\vert ^{2}+c_{\varepsilon} \vert u_{n}\vert ^{p} \bigr)\,dx \\& \quad \leq N \int_{\mathbb{R}^{N}} \biggl(\varepsilon \frac{V(x)}{V_{0}}\vert u_{n}\vert ^{2}+c_{\varepsilon} \vert u_{n} \vert ^{2_{\alpha}^{*}} \biggr)\,dx \\& \quad \leq \frac{N\varepsilon}{V_{0}} \int_{\mathbb{R}^{N}} \bigl(\bigl\vert (-\Delta)^{\frac{\alpha}{2}}u_{n} \bigr\vert ^{2} + V(x)\vert u_{n}\vert ^{2} \bigr)\,dx + c_{\varepsilon}N \int_{\mathbb{R}^{N}} \vert u_{n}\vert ^{2_{\alpha}^{*}} \,dx \\& \quad \leq \frac{N\varepsilon}{V_{0}} \int_{\mathbb{R}^{N}} \bigl(\bigl\vert (-\Delta)^{\frac{\alpha}{2}}u_{n} \bigr\vert ^{2} + V(x)\vert u_{n}\vert ^{2} \bigr)\,dx \\& \qquad {}+ c_{\varepsilon}N \biggl[\frac{1}{S_{\alpha}} \int_{\mathbb{R}^{N}}\bigl\vert (-\Delta)^{\frac{\alpha}{2}}u_{n} \bigr\vert ^{2} \,dx \biggr]^{\frac{N}{N-2\alpha}}, \end{aligned}$$
(14)
which implies that
$$\begin{aligned}& \biggl(1-\frac{N\varepsilon}{V_{0}} \biggr) \int_{\mathbb{R}^{N}} \bigl(\bigl\vert (-\Delta)^{\frac{\alpha}{2}}u_{n} \bigr\vert ^{2} +V(x)u_{n}^{2} \bigr)\,dx \\& \quad \leq c_{\varepsilon}N \biggl[\frac{1}{S_{\alpha}} \int_{\mathbb{R}^{N}}\bigl\vert (-\Delta)^{\frac{\alpha}{2}}u_{n} \bigr\vert ^{2} \,dx \biggr]^{\frac{N}{N-2\alpha}} \\& \quad \leq c_{\varepsilon}N \biggl[\frac{1}{S_{\alpha}} \frac{c_{4}N\vert B_{r}(0)\vert }{\alpha} \biggr]^{\frac{N}{N-2\alpha}}. \end{aligned}$$
It follows that
$$\|u_{n}\|_{E}^{2}\leq c_{\varepsilon}N \biggl[ \frac{1}{S_{\alpha}} \frac{c_{4}N|B_{r}(0)|}{M_{0}} \biggr]^{\frac{N}{N-2\alpha}}\frac{V_{0}}{(V_{0}- N\varepsilon )}. $$
Then the conclusion holds. □
Finally, we are ready to prove our main theorem.
Proof of Theorem 1
Let \(u_{n}\) be a critical point for \(\varphi_{\lambda_{n}}\) at the level \(c_{\lambda_{n}}\). Then from Lemma 7 we may assume that \(\|u_{n}\|_{E}\leq c_{6}\).
Note that \(\lambda_{n}\rightarrow1\), we can show that \(\{u_{n}\}\) is a \((PS)\) sequence of φ. Indeed, the boundedness of \(\{u_{n}\}\) implies that \(\{\varphi(u_{n})\}\) is bounded. Also
$$ \bigl\langle \varphi'(u_{n}),v\bigr\rangle =\bigl\langle \varphi_{\lambda_{n}}'(u_{n}),v\bigr\rangle +( \lambda_{n}-1) \int_{\mathbb{R}^{N}}f(u_{n})v \,dx, \quad \forall u\in E. $$
Hence, \(\varphi'(u_{n})\rightarrow0\), and consequently \(\{u_{n}\}\) is a bounded \((PS)\) sequence of φ. By Lemma 5, \(\{u_{n}\}\) has a convergent subsequence, hence without loss of generality we may assume that \(u_{n}\rightarrow u\). Therefore, \(\varphi'(u)= 0\). By virtue of Lemma 4, we have
$$ \varphi(u) =\lim_{n\rightarrow\infty} \varphi(u_{n}) =\lim _{n\rightarrow\infty} \varphi_{\lambda_{n}}(u_{n})\geq \overline{c}>0, $$
and u is a positive solution by the condition \(H(f)\)(1). The proof is completed. □