In order to prove the main results, first we introduce the space
$$E:= \biggl\{ u\in H^{1}({\mathbb{R}}): \int_{{\mathbb{R}}} V(x)u^{2}\,dx< \infty \biggr\} , $$
which is a linear subspace of \(H^{1}({\mathbb{R}})\).
Let \(\widetilde{E}^{-}\) and \(\widetilde{E}^{+}\) be the negative space and positive space of the quadratic form
$$\int_{{\mathbb{R}}} \bigl(\bigl|u^{\prime}\bigr|^{2}+V(x)u^{2} \bigr)\,dx. $$
From (V), we deduce that \(E=\widetilde{E}^{-}\oplus\widetilde {E}^{+}\) and \(\widetilde{E}^{-}\) is a finite dimensional space which is spanned by the eigenfunctions with corresponding eigenvalues \(\lambda_{1}\leq\cdots\leq\lambda_{k}\). For any \(u, v\in E\), we define
$$(u, v)= \int_{{\mathbb{R}}} \bigl( \bigl({{\widetilde{u}}^{+}} \bigr)^{\prime} \bigl({{\widetilde {v}}^{+}} \bigr)^{\prime}+V(x) \widetilde{u}^{+}\widetilde{v}^{+} \bigr)\,dx - \int_{{\mathbb{R}}} \bigl( \bigl({{\widetilde{u}}^{-}} \bigr)^{\prime} \bigl({{\widetilde {v}}^{-}} \bigr)^{\prime}+V(x) \widetilde{u}^{-}\widetilde{v}^{-} \bigr)\,dx, $$
where \(u=\widetilde{u}^{-}+\widetilde{u}^{+}, v=\widetilde {v}^{-}+\widetilde{v}^{+}\in E=\widetilde{E}^{-}\oplus\widetilde {E}^{+}\). Then \((\cdot, \cdot)\) is an inner product on E. Therefore, E is a Hilbert space with the norm \(\|\cdot\|=(\cdot ,\cdot)^{1/2}\). It is easy to see that
$$\int_{{\mathbb{R}}} \bigl( \bigl|u^{\prime} \bigr|^{2}+V(x)u^{2} \bigr)\,dx= \bigl\| \widetilde {u}^{+} \bigr\| ^{2}- \bigl\| \widetilde{u}^{-} \bigr\| ^{2}. $$
For any \(r\in[2, \infty]\), the embedding \(E\hookrightarrow L^{r}({\mathbb{R}})\) is continuous.
Let \(I:E\rightarrow {\mathbb{R}}\) by
$$I(u)=\frac{1}{2} \int_{{\mathbb{R}}} \bigl( \bigl|u^{\prime} \bigr|^{2}+V(x)u^{2} \bigr)\,dx+ \int_{{\mathbb{R}}} \bigl|u^{\prime} \bigr|^{2}u^{2}- \int_{{\mathbb{R}}}F(x,u)\,dx. $$
Then under conditions (f1) and (f2), \(I\in C^{1}(E,{\mathbb{R}})\) and for all \(u, v\in E\)
$$\bigl\langle I^{\prime}(u),v \bigr\rangle = \int_{{\mathbb{R}}} \bigl(u^{\prime}v^{\prime }+V(x)uv \bigr) \,dx+2 \int_{{\mathbb{R}}} \bigl(\bigl|u^{\prime}\bigr|^{2}uv+u^{2}u^{\prime }v^{\prime} \bigr)\,dx- \int_{{\mathbb{R}}}f(x,u)v\,dx. $$
It is well known that the critical points of I are the solutions of problem (1.1).
Proof of Theorem 1.1
Step 1: To proof I is coercive. Suppose it is not true, then there exist \(M>0\) and \(\|u_{n}\| \rightarrow\infty\) such that \(I(u_{n})\leq M\).
Since \(h<\lambda_{\infty}\), we can choose \(h< h^{\star}<\lambda _{\infty}\) such that \(h^{\star}\neq{\lambda_{i}, 1\leq i<\infty}\). Let \(E^{-}\) be the space spanned by the eigenfunctions with corresponding eigenvalue less than \(h^{\star}\). By the choice of \(h^{\star}\), \(E^{-}\) is finite dimensional space. Let \(E^{+}={(E^{-})}^{\perp}\), then \(E=E^{-}\oplus E^{+}\). For \(u\in E\), we have a unique decomposition \(u=u^{+}+u^{-}\) with \(u^{+}\in E^{+}\) and \(u^{-}\in E^{-}\). By the choice of \(h^{\star}\), there exists an equivalent norm of E, still denoted by \(\|\cdot\|\), such that
$$\int_{{\mathbb{R}}} \bigl( \bigl|u_{n}^{\prime} \bigr|^{2}+V(x)u_{n}^{2}-h^{\star }u_{n}^{2} \bigr)\,dx= \bigl\| u_{n}^{+} \bigr\| ^{2}- \bigl\| u_{n}^{-} \bigr\| ^{2}. $$
So, from (f3), we have
$$\begin{aligned} I(u_{n})={}&\frac{1}{2} \int_{{\mathbb{R}}} \bigl( \bigl|u_{n}^{\prime } \bigr|^{2}+V(x)u_{n}^{2} \bigr)+ \int_{{\mathbb{R}}} \bigl|u_{n}^{\prime} \bigr|^{2}u_{n}^{2} \,dx- \int _{{\mathbb{R}}}F(x,u_{n})\,dx \\ ={}&\frac{1}{2} \int_{{\mathbb{R}}} \bigl( \bigl|u_{n}^{\prime } \bigr|^{2}+V(x)u_{n}^{2}-h^{\star}u^{2}_{n} \bigr)\,dx+ \int_{{\mathbb{R}}} \bigl|u_{n}^{\prime} \bigr|^{2}u_{n}^{2} \,dx \\ &{}+ \int_{{\mathbb{R}}} \biggl(\frac{1}{2}h^{\star }u^{2}_{n}-F(x,u_{n}) \biggr)\,dx \\ ={}&\frac{1}{2} \bigl\| u_{n}^{+} \bigr\| ^{2}- \frac{1}{2} \bigl\| u_{n}^{-} \bigr\| ^{2}+ \int_{{\mathbb{R}}} \bigl|u_{n}^{\prime} \bigr|^{2}u_{n}^{2} \,dx+ \int_{{\mathbb{R}}} \biggl(\frac{1}{2}h^{\star }u^{2}_{n}-F(x,u_{n}) \biggr)\,dx \\ \geq{}&\frac{1}{2} \bigl\| u_{n}^{+} \bigr\| ^{2}- \frac{1}{2} \bigl\| u_{n}^{-} \bigr\| ^{2}. \end{aligned}$$
(2.1)
Let \(v_{n}=u_{n}/\|u_{n}\|\), then \(\|v_{n}\|=1\). Multiplying both sides of (2.1) by \(\|u_{n}\|^{-2}\), since \(\|u_{n}\|\rightarrow\infty\) and \(I(u_{n})\leq M\), we get
$$ \bigl\| v^{+}_{n} \bigr\| ^{2}\leq \bigl\| v^{-}_{n} \bigr\| ^{2}+o_{n}(1). $$
(2.2)
Up to a subsequence, we assume that \(v_{n}\rightharpoonup v\) in E and \(v_{n}\rightarrow v\) a.e. in \({\mathbb{R}}\). If \(v=0\), for the finite dimension of \(E^{-}\), one has \(v_{n}^{-}\rightarrow0\) in E. By (2.2), \(v_{n}\rightarrow0\) in E. It is impossible, since \(\|v_{n}\|=1\). Hence, \(v\neq0\). Since \(\int_{{\mathbb{R}}}|u^{\prime}|^{2}u^{2}\,dx\) is weak sequential lower semi-continuous [3] (see also [7]), we have
$$ \liminf_{n\rightarrow\infty}\|u_{n}\|^{-4} \int_{{\mathbb{R}}} \bigl|u_{n}^{\prime } \bigr|^{2}u_{n}^{2} \,dx= \liminf_{n\rightarrow\infty} \int_{{\mathbb{R}}} \bigl|v_{n}^{\prime} \bigr|^{2}v_{n}^{2} \,dx \geq \int_{{\mathbb{R}}} \bigl|v^{\prime} \bigr|^{2}v^{2} \,dx>0. $$
(2.3)
By (f3), one gets
$$I(u_{n})\geq\frac{1}{2} \bigl\| u_{n}^{+} \bigr\| ^{2}-\frac{1}{2} \bigl\| u_{n}^{-} \bigr\| ^{2}+ \int_{{\mathbb{R}}} \bigl|u_{n}^{\prime} \bigr|^{2}u_{n}^{2} \,dx. $$
Multiplying both sides of the above inequality by \(\|u_{n}\|^{-4}\), by (2.3) and \(\|u_{n}\|^{-4}I(u_{n})\rightarrow0\) as \(n\rightarrow\infty \), we have
$$0\geq \int_{{\mathbb{R}}} \bigl|v^{\prime} \bigr|^{2}v^{2} \,dx>0. $$
This is a contradiction. So, I is coercive.
Step 2: We will show that I is weakly sequentially lower semi-continuous. Let \(u_{n}\rightharpoonup u\) in E. Then \(\liminf_{n\rightarrow \infty}\|u_{n}^{+}\|\geq\|u^{+}\|\) and \(u_{n}^{-}\rightarrow u^{-}\) in \(E^{-}\) since the dimension of \(E^{-}\) is finite. So, it follows from (f3) and \(\int_{{\mathbb{R}}}|u^{\prime}|^{2}u^{2}\,dx\) being weakly sequentially lower semi-continuous that
$$\begin{aligned} \liminf_{n\rightarrow\infty}I(u_{n})&= \liminf_{n\rightarrow\infty } \biggl(\frac{1}{2} \bigl\| u_{n}^{+} \bigr\| ^{2} -\frac{1}{2} \bigl\| u_{n}^{-} \bigr\| ^{2}+ \int_{{\mathbb{R}}}\bigl|u_{n}^{\prime }\bigr|^{2}u_{n}^{2} \,dx+ \int_{{\mathbb{R}}} \biggl(\frac{1}{2}h^{\star }u^{2}_{n}-F(x,u_{n}) \biggr)\,dx \biggr) \\ &\geq\frac{1}{2} \bigl\| u^{+} \bigr\| ^{2} -\frac{1}{2} \bigl\| u^{-} \bigr\| ^{2}+ \int_{{\mathbb{R}}} \bigl|u^{\prime} \bigr|^{2}u^{2}\,dx + \int_{{\mathbb{R}}} \biggl(\frac{1}{2}h^{\star}u^{2}-F(x,u) \biggr)\,dx \\ &=I(u). \end{aligned}$$
(2.4)
Hence, I is weakly sequentially lower semi-continuous.
Step 3: We will show that I is bounded from below and \(\inf_{E}I<0\). It is easy to see that I maps every bounded set in E into bounded set in \({\mathbb{R}}\). Hence by the Step 1, we see that I is bounded from below. Next, we will show that \(\inf_{E}I<0\). In fact, since (V), for any \(e\in E^{-}\setminus\{0\}\) we get
$$\int_{{\mathbb{R}}} \bigl( \bigl|e^{\prime} \bigr|^{2}+V(x)e^{2} \bigr)\,dx< 0. $$
From (f2), we have \(\int_{{\mathbb{R}}}F(x,te)\,dx=o(t^{2})\). Furthermore,
$$\int_{{\mathbb{R}}} \bigl|(te)^{\prime} \bigr|^{2}(te)^{2} \,dx=t^{4} \int_{{\mathbb{R}}} \bigl|e^{\prime } \bigr|^{2}e^{2} \,dx=O \bigl(t^{4} \bigr). $$
Therefore, we can choose some \(t>0\) small enough such that \(I(te)<0\). So, we get \(\inf_{E}I<0\).
By using direct variation method, Steps 1 and 2 show that I has a global minimizer. From Step 3, the global minimizer of I is not zero. Therefore, we get a nontrivial solution of problem (1.1). □