In this section, for the notation in Lemma 5, the space \(X=E\), and related functional on E is \(I=\varphi\). Recall that a sequence \(\{u_{n}\}\subset E\) is called a Palais-Smale sequence of φ the level c, a \((\mathit{PS})_{c}\) sequence for short, if \(\varphi (u_{n})\rightarrow c\) and \(\varphi'(u_{n})\rightarrow0\). A sequence \(\{u_{n}\}\subset E\) is called a Cerami sequence ϕ at the level c, a \((C)_{c}\) sequence for short, if
$$\varphi(u_{n})\rightarrow c\quad \mbox{and} \quad \bigl(1+\Vert u_{n}\Vert \bigr)\varphi '(u_{n}) \rightarrow0. $$
We deduce from (H
f
)(iii) that
$$f(x,t)t-2F(x,t)\geq0, \quad \forall(x,t)\in\mathbb{R}^{N}\times \mathbb{R}. $$
Let \(t>0\). For \(x\in\mathbb{R}^{N}\), by direct computations we have
$$ \frac{\partial}{\partial t}\biggl(\frac{F(x,t)}{t^{2}}\biggr)= \frac {t^{2}f(x,t)-2tF(x,t)}{t^{4}}\geq0. $$
(5)
Taking into account hypothesis (H
f
)(ii) we deduce
$$ \lim_{t\rightarrow0^{+}} \frac{F(x,t)}{t^{2}}=0. $$
(6)
So it follows from (5) and (6) that \(F(x,t)\geq0\) for all \(x\in\mathbb{R}^{N}\) and \(t\geq0\). Arguing similarly for the case \(t\leq0\), eventually we obtain
$$ F(x,t)\geq0, \quad \forall(x,t)\in\mathbb{R}^{N}\times \mathbb{R}. $$
(7)
Lemma 6
There exists
\(r>0\)
and
\(\eta_{0}\in E\)
such that
\(\|\eta_{0}\|>r\)
and
$$\inf_{\|u\|=r}\varphi(u)>\varphi(0)=0\geq\varphi( \eta_{0}). $$
Proof
First of all, from (H
f
)(i) and (H
f
)(ii) it follows that, for all given \(\varepsilon>0\), there exists \(c_{\varepsilon}>0\) such that
$$F(x,t)\leq\varepsilon|t|^{2}+c_{\varepsilon}|t|^{2_{\alpha}^{*}},\quad \forall(x,t)\in\mathbb{R}^{N}\times\mathbb{R}. $$
Consequently, using Lemma 2, we have
$$\begin{aligned} \varphi(u) =&-\frac{1}{2} \int_{\vert \theta \vert =1}\bigl(D_{\theta}^{\alpha}u, D_{-\theta}^{\alpha}u\bigr)M(d\theta)+\frac{1}{2} \int_{\mathbb {R}^{N}}V(x)u^{2}\,dx- \int_{\mathbb{R}^{N}}F(x,u)\,dx \\ =&-\frac{\cos(\pi\alpha)}{2} \int_{\vert \theta \vert =1}\bigl\vert D_{\theta}^{\alpha}u\bigr\vert _{2}^{2}M(d\theta)+\frac{1}{2} \int_{\mathbb{R}^{N}}V(x)u^{2}\,dx- \int _{\mathbb{R}^{N}}F(x,u)\,dx \\ \geq&-\frac{\cos(\pi\alpha)}{2} \biggl[ \int_{\vert \theta \vert =1}\bigl\vert D_{\theta}^{\alpha}u\bigr\vert _{2}^{2}M(d\theta)+ \int_{\mathbb{R}^{N}}V(x)u^{2}\,dx \biggr]-\varepsilon \int_{\mathbb{R}^{N}}\vert u\vert ^{2}\,dx \\ &{}-c_{\varepsilon}\int _{\mathbb{R}^{N}}\vert u\vert ^{2_{\alpha}^{*}}\,dx. \end{aligned}$$
Since \(E\hookrightarrow L^{p}(\mathbb{R}^{N})\), \(2\leq p\leq2_{\alpha}^{*}\), we know there exists a constant \(C>0\) such that \(|u|_{p}\leq C\|u\|\). So
$$ \varphi(u) \geq-\frac{\cos(\pi\alpha)}{2}\|u\|^{2}-\varepsilon C^{2}\|u\|^{2} -c_{\varepsilon}C^{2_{\alpha}^{*}}\|u \|^{2_{\alpha}^{*}}. $$
By choosing \(\varepsilon>0\) such that \(-\frac{\cos(\pi\alpha )}{2}-\varepsilon C^{2}\geq-\frac{\cos(\pi\alpha)}{4}\), we obtain
$$ \varphi(u) \geq-\frac{\cos(\pi\alpha)}{4}\|u\|^{2}-c_{\varepsilon}C^{2_{\alpha}^{*}}\|u\|^{2_{\alpha}^{*}}, \quad \forall u\in E. $$
That is, there exist \(r>0\) and \(\rho>0\), such that
$$ \varphi(u) \geq \rho>0,\qquad \|u\|=r. $$
Using (1), it is easy to see that for any \(u\neq0\), we have \(\varphi(tu)\rightarrow-\infty\) as \(t\rightarrow+\infty\). Hence, there is a point \(\eta_{0}\in E\backslash\overline{B}_{r}\) such that \(\varphi(\eta_{0})\leq0\). □
By Lemma 6 we see that φ has a mountain pass geometry: that is, setting
$$\Gamma=\bigl\{ \gamma\in C\bigl([0,1],E\bigr):\gamma(0)=0,\varphi\bigl(\gamma(1) \bigr)< 0\bigr\} , $$
we have \(\Gamma\neq\emptyset\). Moreover, it is easy to see that
$$ c=\inf_{\gamma\in\Gamma}\max_{t\in [0,1]}\varphi \bigl(\gamma(t)\bigr)>0. $$
(8)
Take \(F =E\) in Lemma 5. Equation (8) implies that \(\varphi ^{c}\) separates \(\gamma(0)=0\) and \(\gamma(1)=\eta_{0}\), and there exists a \((C)_{c}\) sequence \(\{u_{n}\}\) for φ.
Claim
The sequence
\(\{u_{n}\}\)
is bounded.
If \(\{u_{n}\}\) is unbounded, up to a subsequence we may assume that
$$\varphi(u_{n})\rightarrow c, \qquad \Vert u_{n}\Vert \rightarrow+\infty, \qquad \bigl\Vert \varphi'(u_{n}) \bigr\Vert \Vert u_{n}\Vert \rightarrow0. $$
In particular,
$$ \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \biggl(\frac {1}{2}f(x,u_{n})u_{n}-F(x,u_{n}) \biggr)\, dx= \lim_{n\rightarrow\infty} \biggl(\varphi(u_{n})- \frac {1}{2}\varphi'(u_{n})u_{n} \biggr)=c. $$
(9)
Let \(w_{n}=\frac{u_{n}}{\|u_{n}\|}\), then \(\{w_{n}\}\) is bounded in E. We claim that
$$ \lim_{n\rightarrow\infty}\sup_{y\in\mathbb {R}^{N}} \int_{B_{2}(y)}|w_{n}|^{2}\, dx=0. $$
(10)
Otherwise, for some \(\delta>0\), up to a subsequence we have
$$ \sup_{y\in\mathbb{R}^{N}} \int_{B_{2}(y)}|w_{n}|^{2}\, dx\geq\delta>0. $$
So we can choose \(\{y_{n}\} \subset\mathbb{R}^{N}\) such that
$$ \sup_{y\in\mathbb{R}^{N}} \int_{B_{2}(y_{n})}|w_{n}|^{2}\, dx\geq \frac {\delta}{2}. $$
It is easy to see that the number of points in \(Z^{N}\cap B_{2}(y_{n})\) is less than \(4^{N}\); there exists \(z_{n}\in B_{2}(y_{n})\) such that
$$ \int_{B_{2}(z_{n})}|w_{n}|^{2}\, dx\geq \frac{\delta}{2^{2N+1}}. $$
(11)
Let \(\overline{w}_{n}(x) = w_{n}(x+z_{n})\). Then \(\{\overline{w}_{n}\}\) is also bounded in E. Passing to a subsequence we have
$$\overline{w}_{n}\rightarrow\overline{w} \quad \mbox{in } L^{2}_{\mathrm{loc}}\bigl(\mathbb{R}^{N}\bigr) \quad \mbox{and} \quad \overline{w}_{n}(x)\rightarrow \overline{w}(x) \quad \mbox{a.e. } x\in\mathbb{R}^{N}. $$
Remark that
$$ \int_{B_{2}(0)}|\overline{w}_{n}|^{2}\, dx= \int_{B_{2}(z_{n})}|w_{n}|^{2}\, dx $$
(12)
and so \(\overline{w}\neq0\). Let \(\overline{u}_{n}=\|u_{n}\|\overline {w}_{n}\). It is easy to see that \(\overline{u}_{n}(x)\rightarrow+\infty\) as \(n\rightarrow+\infty\). Using (1) we have
$$ \lim_{n\rightarrow+\infty}\frac{F(x,\overline {u}_{n}(x))}{|\overline{u}_{n}(x)|^{2}}\bigl\vert \overline{w}_{n}(x)\bigr\vert ^{2}=+\infty. $$
(13)
Recall that \(f(x, u)\) is 1-periodic with respect to x. Using this fact, we obtain
$$ \int_{\mathbb{R}^{N}}F(x,u_{n})\, dx= \int_{\mathbb{R}^{N}}F(x,\overline{u}_{n})\, dx. $$
Since the set \(\Omega_{0}= \{x\in\mathbb{R}^{N}:\overline{w}(x)\neq0\} \) has positive Lebesgue measure, using (13) we have
$$\begin{aligned} \frac{1}{2} =& \frac{1}{2\|u_{n}\|^{2}} \int_{\mathbb{R}^{N}} \int_{|\theta |=1} \int_{\mathbb{R}^{N}}\bigl\vert D_{\theta}^{\alpha}u_{n}\bigr\vert ^{2}\, dxM(d\theta)+ \int _{\mathbb{R}^{N}}V(x)|u_{n}|^{2}\,dx \\ =& \frac{1}{\|u_{n}\|^{2}} \biggl(\varphi(u_{n})+ \int_{\mathbb{R}^{N}} F(x,u_{n})\,dx \biggr) \\ \geq& \frac{1}{\|u_{n}\|^{2}} \int_{\mathbb{R}^{N}} F(x,u_{n})\,dx-1 \\ =&\frac{1}{\|u_{n}\|^{2}} \int_{\mathbb{R}^{N}} F(x,\overline{u}_{n})\,dx-1 \\ \geq& \int_{\overline{w}\neq0} \frac{F(x,\overline {u}_{n}(x))}{|\overline{u}_{n}(x)| ^{2}}\bigl\vert \overline{w}_{n}(x) \bigr\vert ^{2}\,dx-1\rightarrow+\infty. \end{aligned}$$
This is impossible. Therefore we have proved (10). Hence, using Lemma 4 and (10), we obtain
$$ w_{n}\rightarrow0 \quad \mbox{in } L^{s}\bigl( \mathbb{R}^{N}\bigr), s\in\bigl(2,2_{\alpha}^{*}\bigr). $$
(14)
Next, we shall derive a contradiction as follows. Given a real number \(R >0\), by (H
f
)(i) and (H
f
)(ii), for any \(\varepsilon>0\), there exists \(c_{\varepsilon}>0\) such that
$$ F(x,Rt)\leq\varepsilon\bigl(|t|^{2}+|t|^{2_{\alpha}^{*}} \bigr)+c_{\varepsilon}|t|^{s}. $$
(15)
Note that \(\|w_{m}\|=1\). So from (14), (15), and the continuity of the embedding \(E\hookrightarrow L^{p}(\mathbb{R}^{N})\) (since \(p\in[2,2_{\alpha}^{*}]\)), we obtain
$$\begin{aligned} \limsup_{n\rightarrow\infty} \int_{\mathbb {R}^{N}}F(x,Rw_{n})\, dx \leq&\limsup _{n\rightarrow\infty} \bigl[\varepsilon C^{2}\|w_{n} \|^{2}+\varepsilon C^{2_{\alpha}^{*}}\|w_{n} \|^{2_{\alpha}^{*}}+c_{\varepsilon}|w_{n}|_{s}^{s} \bigr] \\ \leq&\varepsilon\bigl(C^{2}+C^{2_{\alpha}^{*}}\bigr). \end{aligned}$$
Now let \(\varepsilon\rightarrow0\), we obtain
$$ \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}}F(x,Rw_{n})\, dx=0. $$
(16)
Let \(t_{n}\in[0, 1]\) such that \(\varphi(t_{n}u_{n})=\max_{t\in [0,1]}\varphi(tu_{n})\). Given \(m>0\). Since for n large enough we have \((-\frac{4m}{\cos (\pi\alpha)} )^{\frac{1}{2}}\|u_{n}\|^{-1}\in(0,1)\), using (16) with \(R= (-\frac{4m}{\cos(\pi\alpha)} )^{\frac{1}{2}}\), we have
$$\begin{aligned} \varphi(t_{n}u_{n}) \geq& \varphi \biggl( \biggl(- \frac{4m}{\cos(\pi\alpha )} \biggr)^{\frac{1}{2}}\|u_{n}\|^{-1} u_{n} \biggr)=\varphi \biggl( \biggl(-\frac{4m}{\cos(\pi\alpha)} \biggr)^{\frac{1}{2}}w_{n} \biggr) \\ =&2m \int_{\vert \theta \vert =1}\bigl\vert D_{\theta}^{\alpha}w_{n}\bigr\vert _{2}^{2}M(d\theta)- \frac {2m}{\cos(\pi\alpha)} \int_{\mathbb{R}^{N}}V(x)w_{n}^{2}\,dx \\ &{}- \int _{\mathbb{R}^{N}}F \biggl(x, \biggl(-\frac{4m}{\cos(\pi\alpha)} \biggr)^{\frac{1}{2}}w_{n} \biggr)\,dx \\ \geq&2m \int_{\vert \theta \vert =1}\bigl\vert D_{\theta}^{\alpha}w_{n}\bigr\vert _{2}^{2}M(d\theta)+2m \int _{\mathbb{R}^{N}}V(x)w_{n}^{2}\,dx \\ &{}- \int_{\mathbb{R}^{N}}F \biggl(x, \biggl(-\frac{4m}{\cos(\pi\alpha)} \biggr)^{\frac{1}{2}}w_{n} \biggr)\,dx \\ \geq& m. \end{aligned}$$
That is, \(\varphi(t_{n}u_{n})\rightarrow+\infty\). But \(\varphi(0)=0\), \(\varphi(u_{n})\rightarrow c\), we see that \(t_{n}\in(0, 1)\), and
$$ \varphi'(t_{n}u_{n})t_{n}u_{n}=t_{n} \frac{d}{dt} \bigg|_{t=t_{n}}\varphi(tu_{n})=0. $$
Because of hypothesis (H
f
)(iii),
$$\begin{aligned} \int_{\mathbb{R}^{N}} \biggl(\frac{1}{2}f(x,u_{n})u_{n}-F(x,u_{n}) \biggr)\,dx \geq& \frac{1}{\vartheta} \int_{\mathbb{R}^{N}} \biggl(\frac {1}{2}f(x,t_{n}u_{n})t_{n}u_{n}-F(x,t_{n}u_{n}) \biggr)\,dx \\ =&\frac{1}{\vartheta} \biggl[\varphi(t_{n}u_{n})- \frac{1}{2}\varphi '(t_{n}u_{n})t_{n}u_{n} \biggr] \\ =&\frac{1}{\vartheta}\varphi(t_{n}u_{n})\rightarrow+\infty. \end{aligned}$$
This contradicts (9) and consequently we have proved that \(\{ u_{n}\}\) is bounded.
Let
$$\tau:=\lim_{n\rightarrow\infty}\sup_{y\in\mathbb {R}^{N}} \int_{B_{2}(y)}|u_{n}|^{2}\, dx. $$
If \(\tau= 0\), using Lemma 4, similarly to (16) we have
$$ \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}}F(x,u_{n})\, dx=0 \quad \mbox{and} \quad \lim _{n\rightarrow\infty} \int_{\mathbb {R}^{N}}f(x,u_{n})u_{n}\, dx=0. $$
(17)
Hence using (9) we have \(c= 0\), a contradiction. Therefore \(\tau>0\). Similarly to (12), we can choose a sequence \(\{z_{n}\} \subset Z^{N}\) such that setting \(\overline{u}_{n}(x)=u_{n}(x+z_{n})\), we have
$$ \int_{B_{2}(0)}|\overline{u}_{n}|^{2}\, dx= \int_{B_{2}(z_{n})}|u_{n}|^{2}\, dx\geq \frac {\tau}{2^{2N+1}}. $$
(18)
Note that \(\|\overline{u}_{n}\|=\|u_{n}\|\), we see that \(\{\overline {u}_{n}\}\) is bounded. Going if necessary to a subsequence, we obtain
$$\overline{u}_{n}\rightharpoonup\overline{u} \quad \mbox{in } E \quad \mbox{and} \quad \overline{u}_{n}\rightarrow\overline{u}\quad \mbox{in } L^{2}_{\mathrm{loc}}\bigl(\mathbb{R}^{N}\bigr). $$
Returning to (18), we see that \(\overline{u}\neq0\). Moreover, by the \(Z^{N}\) invariance of the problem, \(\{\overline{u}_{n}\}\) is also a \((C)_{c}\) sequence of φ. Thus for every \(v\in C_{0}^{\infty}(\mathbb{R}^{N})\), we have
$$\varphi'(\overline{u})v=\lim_{n\rightarrow\infty}\varphi '(\overline{u}_{n})v=0. $$
So \(\varphi'(\overline{u})= 0\) and u̅ is a nontrivial solution of (P).