In order to prove the main results, first we introduce the space

$$E:= \biggl\{ u\in H^{1}({\mathbb{R}}): \int_{{\mathbb{R}}} V(x)u^{2}\,dx< \infty \biggr\} , $$

which is a linear subspace of \(H^{1}({\mathbb{R}})\).

Let \(\widetilde{E}^{-}\) and \(\widetilde{E}^{+}\) be the negative space and positive space of the quadratic form

$$\int_{{\mathbb{R}}} \bigl(\bigl|u^{\prime}\bigr|^{2}+V(x)u^{2} \bigr)\,dx. $$

From (V), we deduce that \(E=\widetilde{E}^{-}\oplus\widetilde {E}^{+}\) and \(\widetilde{E}^{-}\) is a finite dimensional space which is spanned by the eigenfunctions with corresponding eigenvalues \(\lambda_{1}\leq\cdots\leq\lambda_{k}\). For any \(u, v\in E\), we define

$$(u, v)= \int_{{\mathbb{R}}} \bigl( \bigl({{\widetilde{u}}^{+}} \bigr)^{\prime} \bigl({{\widetilde {v}}^{+}} \bigr)^{\prime}+V(x) \widetilde{u}^{+}\widetilde{v}^{+} \bigr)\,dx - \int_{{\mathbb{R}}} \bigl( \bigl({{\widetilde{u}}^{-}} \bigr)^{\prime} \bigl({{\widetilde {v}}^{-}} \bigr)^{\prime}+V(x) \widetilde{u}^{-}\widetilde{v}^{-} \bigr)\,dx, $$

where \(u=\widetilde{u}^{-}+\widetilde{u}^{+}, v=\widetilde {v}^{-}+\widetilde{v}^{+}\in E=\widetilde{E}^{-}\oplus\widetilde {E}^{+}\). Then \((\cdot, \cdot)\) is an inner product on *E*. Therefore, *E* is a Hilbert space with the norm \(\|\cdot\|=(\cdot ,\cdot)^{1/2}\). It is easy to see that

$$\int_{{\mathbb{R}}} \bigl( \bigl|u^{\prime} \bigr|^{2}+V(x)u^{2} \bigr)\,dx= \bigl\| \widetilde {u}^{+} \bigr\| ^{2}- \bigl\| \widetilde{u}^{-} \bigr\| ^{2}. $$

For any \(r\in[2, \infty]\), the embedding \(E\hookrightarrow L^{r}({\mathbb{R}})\) is continuous.

Let \(I:E\rightarrow {\mathbb{R}}\) by

$$I(u)=\frac{1}{2} \int_{{\mathbb{R}}} \bigl( \bigl|u^{\prime} \bigr|^{2}+V(x)u^{2} \bigr)\,dx+ \int_{{\mathbb{R}}} \bigl|u^{\prime} \bigr|^{2}u^{2}- \int_{{\mathbb{R}}}F(x,u)\,dx. $$

Then under conditions (f_{1}) and (f_{2}), \(I\in C^{1}(E,{\mathbb{R}})\) and for all \(u, v\in E\)

$$\bigl\langle I^{\prime}(u),v \bigr\rangle = \int_{{\mathbb{R}}} \bigl(u^{\prime}v^{\prime }+V(x)uv \bigr) \,dx+2 \int_{{\mathbb{R}}} \bigl(\bigl|u^{\prime}\bigr|^{2}uv+u^{2}u^{\prime }v^{\prime} \bigr)\,dx- \int_{{\mathbb{R}}}f(x,u)v\,dx. $$

It is well known that the critical points of *I* are the solutions of problem (1.1).

### Proof of Theorem 1.1

Step 1: To proof *I* is coercive. Suppose it is not true, then there exist \(M>0\) and \(\|u_{n}\| \rightarrow\infty\) such that \(I(u_{n})\leq M\).

Since \(h<\lambda_{\infty}\), we can choose \(h< h^{\star}<\lambda _{\infty}\) such that \(h^{\star}\neq{\lambda_{i}, 1\leq i<\infty}\). Let \(E^{-}\) be the space spanned by the eigenfunctions with corresponding eigenvalue less than \(h^{\star}\). By the choice of \(h^{\star}\), \(E^{-}\) is finite dimensional space. Let \(E^{+}={(E^{-})}^{\perp}\), then \(E=E^{-}\oplus E^{+}\). For \(u\in E\), we have a unique decomposition \(u=u^{+}+u^{-}\) with \(u^{+}\in E^{+}\) and \(u^{-}\in E^{-}\). By the choice of \(h^{\star}\), there exists an equivalent norm of *E*, still denoted by \(\|\cdot\|\), such that

$$\int_{{\mathbb{R}}} \bigl( \bigl|u_{n}^{\prime} \bigr|^{2}+V(x)u_{n}^{2}-h^{\star }u_{n}^{2} \bigr)\,dx= \bigl\| u_{n}^{+} \bigr\| ^{2}- \bigl\| u_{n}^{-} \bigr\| ^{2}. $$

So, from (f_{3}), we have

$$\begin{aligned} I(u_{n})={}&\frac{1}{2} \int_{{\mathbb{R}}} \bigl( \bigl|u_{n}^{\prime } \bigr|^{2}+V(x)u_{n}^{2} \bigr)+ \int_{{\mathbb{R}}} \bigl|u_{n}^{\prime} \bigr|^{2}u_{n}^{2} \,dx- \int _{{\mathbb{R}}}F(x,u_{n})\,dx \\ ={}&\frac{1}{2} \int_{{\mathbb{R}}} \bigl( \bigl|u_{n}^{\prime } \bigr|^{2}+V(x)u_{n}^{2}-h^{\star}u^{2}_{n} \bigr)\,dx+ \int_{{\mathbb{R}}} \bigl|u_{n}^{\prime} \bigr|^{2}u_{n}^{2} \,dx \\ &{}+ \int_{{\mathbb{R}}} \biggl(\frac{1}{2}h^{\star }u^{2}_{n}-F(x,u_{n}) \biggr)\,dx \\ ={}&\frac{1}{2} \bigl\| u_{n}^{+} \bigr\| ^{2}- \frac{1}{2} \bigl\| u_{n}^{-} \bigr\| ^{2}+ \int_{{\mathbb{R}}} \bigl|u_{n}^{\prime} \bigr|^{2}u_{n}^{2} \,dx+ \int_{{\mathbb{R}}} \biggl(\frac{1}{2}h^{\star }u^{2}_{n}-F(x,u_{n}) \biggr)\,dx \\ \geq{}&\frac{1}{2} \bigl\| u_{n}^{+} \bigr\| ^{2}- \frac{1}{2} \bigl\| u_{n}^{-} \bigr\| ^{2}. \end{aligned}$$

(2.1)

Let \(v_{n}=u_{n}/\|u_{n}\|\), then \(\|v_{n}\|=1\). Multiplying both sides of (2.1) by \(\|u_{n}\|^{-2}\), since \(\|u_{n}\|\rightarrow\infty\) and \(I(u_{n})\leq M\), we get

$$ \bigl\| v^{+}_{n} \bigr\| ^{2}\leq \bigl\| v^{-}_{n} \bigr\| ^{2}+o_{n}(1). $$

(2.2)

Up to a subsequence, we assume that \(v_{n}\rightharpoonup v\) in *E* and \(v_{n}\rightarrow v\) a.e. in \({\mathbb{R}}\). If \(v=0\), for the finite dimension of \(E^{-}\), one has \(v_{n}^{-}\rightarrow0\) in *E*. By (2.2), \(v_{n}\rightarrow0\) in *E*. It is impossible, since \(\|v_{n}\|=1\). Hence, \(v\neq0\). Since \(\int_{{\mathbb{R}}}|u^{\prime}|^{2}u^{2}\,dx\) is weak sequential lower semi-continuous [3] (see also [7]), we have

$$ \liminf_{n\rightarrow\infty}\|u_{n}\|^{-4} \int_{{\mathbb{R}}} \bigl|u_{n}^{\prime } \bigr|^{2}u_{n}^{2} \,dx= \liminf_{n\rightarrow\infty} \int_{{\mathbb{R}}} \bigl|v_{n}^{\prime} \bigr|^{2}v_{n}^{2} \,dx \geq \int_{{\mathbb{R}}} \bigl|v^{\prime} \bigr|^{2}v^{2} \,dx>0. $$

(2.3)

By (f_{3}), one gets

$$I(u_{n})\geq\frac{1}{2} \bigl\| u_{n}^{+} \bigr\| ^{2}-\frac{1}{2} \bigl\| u_{n}^{-} \bigr\| ^{2}+ \int_{{\mathbb{R}}} \bigl|u_{n}^{\prime} \bigr|^{2}u_{n}^{2} \,dx. $$

Multiplying both sides of the above inequality by \(\|u_{n}\|^{-4}\), by (2.3) and \(\|u_{n}\|^{-4}I(u_{n})\rightarrow0\) as \(n\rightarrow\infty \), we have

$$0\geq \int_{{\mathbb{R}}} \bigl|v^{\prime} \bigr|^{2}v^{2} \,dx>0. $$

This is a contradiction. So, *I* is coercive.

Step 2: We will show that *I* is weakly sequentially lower semi-continuous. Let \(u_{n}\rightharpoonup u\) in *E*. Then \(\liminf_{n\rightarrow \infty}\|u_{n}^{+}\|\geq\|u^{+}\|\) and \(u_{n}^{-}\rightarrow u^{-}\) in \(E^{-}\) since the dimension of \(E^{-}\) is finite. So, it follows from (f_{3}) and \(\int_{{\mathbb{R}}}|u^{\prime}|^{2}u^{2}\,dx\) being weakly sequentially lower semi-continuous that

$$\begin{aligned} \liminf_{n\rightarrow\infty}I(u_{n})&= \liminf_{n\rightarrow\infty } \biggl(\frac{1}{2} \bigl\| u_{n}^{+} \bigr\| ^{2} -\frac{1}{2} \bigl\| u_{n}^{-} \bigr\| ^{2}+ \int_{{\mathbb{R}}}\bigl|u_{n}^{\prime }\bigr|^{2}u_{n}^{2} \,dx+ \int_{{\mathbb{R}}} \biggl(\frac{1}{2}h^{\star }u^{2}_{n}-F(x,u_{n}) \biggr)\,dx \biggr) \\ &\geq\frac{1}{2} \bigl\| u^{+} \bigr\| ^{2} -\frac{1}{2} \bigl\| u^{-} \bigr\| ^{2}+ \int_{{\mathbb{R}}} \bigl|u^{\prime} \bigr|^{2}u^{2}\,dx + \int_{{\mathbb{R}}} \biggl(\frac{1}{2}h^{\star}u^{2}-F(x,u) \biggr)\,dx \\ &=I(u). \end{aligned}$$

(2.4)

Hence, *I* is weakly sequentially lower semi-continuous.

Step 3: We will show that *I* is bounded from below and \(\inf_{E}I<0\). It is easy to see that *I* maps every bounded set in *E* into bounded set in \({\mathbb{R}}\). Hence by the Step 1, we see that *I* is bounded from below. Next, we will show that \(\inf_{E}I<0\). In fact, since (V), for any \(e\in E^{-}\setminus\{0\}\) we get

$$\int_{{\mathbb{R}}} \bigl( \bigl|e^{\prime} \bigr|^{2}+V(x)e^{2} \bigr)\,dx< 0. $$

From (f_{2}), we have \(\int_{{\mathbb{R}}}F(x,te)\,dx=o(t^{2})\). Furthermore,

$$\int_{{\mathbb{R}}} \bigl|(te)^{\prime} \bigr|^{2}(te)^{2} \,dx=t^{4} \int_{{\mathbb{R}}} \bigl|e^{\prime } \bigr|^{2}e^{2} \,dx=O \bigl(t^{4} \bigr). $$

Therefore, we can choose some \(t>0\) small enough such that \(I(te)<0\). So, we get \(\inf_{E}I<0\).

By using direct variation method, Steps 1 and 2 show that *I* has a global minimizer. From Step 3, the global minimizer of *I* is not zero. Therefore, we get a nontrivial solution of problem (1.1). □