Existence and concentration of solutions for the nonlinear Kirchhoff type equations with steep well potential

Abstract

In this paper, we study the following nonlinear problem of Kirchhoff type:

$$ \textstyle\begin{cases} - ( a + b\int_{{\mathbb{R}^{3}}} \vert {\nabla u} \vert ^{2} ) \Delta u + \lambda V(x)u = {{ \vert u \vert } ^{p - 2}}u,\quad \text{in }{\mathbb{R}^{3}}, \\ {u \in{H^{1}}({\mathbb{R}^{3}}),} \end{cases} $$

where the parameter $\lambda > 0$ and $4 \leq p < 6$, constants $a, b>0$. By variational methods, the results of the existence of nontrivial solutions and the concentration phenomena of the solutions as $\lambda \to + \infty$ are obtained. It is worth pointing out that, for the case $p\in(4,6)$, the potential V is permitted to be sign-changing.

1 Introduction and main result

In this paper, we are concerned with the following Kirchhoff type problem:

$$ \textstyle\begin{cases} - ( {a + b{{\int_{{\mathbb{R}^{3}}} { \vert {\nabla u} \vert ^{2}} }}} ) \Delta u + \lambda V(x)u = {{ \vert u \vert }^{p - 2}}u,\quad\text{in }{\mathbb{R}^{3}},\\ {u \in{H^{1}}({\mathbb{R}^{3}}),} \end{cases} $$

(1.1)

where $a,b > 0$ are constants, $\lambda > 0$ is a parameter, $4 \leq p < 6$. We assume that $V(x)$ verifies the following hypotheses:

(V₁):

$V \in C({\mathbb{R}^{3}},{\mathbb{R}})$ and V is bounded from below.

(V₂):

There exists $b > 0$ such that $\operatorname{meas} \{ x \in{\mathbb{R}^{3}}:V(x) < b\}<\infty$.

(V₃):

The nonempty set $\Omega := \operatorname{int} {V^{ - 1}}(0)$ has a smooth boundary and $\bar{\Omega}= {V^{ - 1}}(0)$.

In recent years, more and more attention has been devoted to the study of the following Kirchhoff type problems:

$$ \textstyle\begin{cases} - ( {a + b{{\int_{{\mathbb{R}^{N}}} { \vert {\nabla u} \vert ^{2}} }}} )\Delta u + V(x)u = f(x,u),\quad\text{in }{\mathbb{R}^{N}},\\ {u \in{H^{1}}({\mathbb{R}^{N}}),} \end{cases} $$

(1.2)

where $V:{\mathbb{R}^{N}} \to\mathbb{R}$ and $a,b > 0$ are constants. (1.2) is a nonlocal problem as the appearance of the term ${{{\int _{{\mathbb{R}^{N}}} { \vert {\nabla u} \vert } }^{2}}}$, which implies that (1.2) is not a pointwise identity. This causes some mathematical difficulties which make the study of (1.2) particularly interesting. If we put $V(x) = 0$ and substitute ${\mathbb {R}^{N}}$ with a bounded domain $\Omega \subset{\mathbb{R}^{N}}$ in (1.2), then we obtain the following Kirchhoff-Dirichlet problem:

$$ \textstyle\begin{cases} - ( {a + b{{\int_{\Omega}{ \vert {\nabla u} \vert ^{2}} }}} )\Delta u = f(x,u),& x \in\Omega,\\ u = 0,& x \in\partial\Omega, \end{cases} $$

which is associated with the following stationary analogue of the equation:

$$ \rho\frac{{{\partial^{2}}u}}{{\partial{t^{2}}}} - \biggl( {\frac{{{P_{0}}}}{h} + \frac{E}{{2L}}{ \int_{0}^{L} { \biggl\vert {\frac{{\partial u}}{{\partial x}}} \biggr\vert } ^{2}}\,dx} \biggr) \frac{{{\partial^{2}}u}}{{\partial{x^{2}}}}=0, $$

(1.3)

presented by Kirchhoff in [16] as an extension of the classical D’Alembert wave equation for free vibrations of elastic strings. For more background, we refer to [4] and the references therein.

Equation (1.2) has been extensively studied in recent years under variant assumptions on V and f. In these works, various existence results of the nontrivial solutions to equation (1.2) were established by the variational method. About the existence of infinitely many radial solutions, Jin and Wu in [15] proved the result by applying a fountain theorem for $N=2, 3$, $V(x) \equiv1$ and $f(x,u)$ is subcritical, superlinear at the origin and 4-superlinear at infinity. When $f(x,u)$ is 4-superlinear at infinity and the potential $V(x)$ satisfies other conditions, Wu in [26] obtained the existence of nontrivial solutions to (1.2) by providing that $(PS)_{c}$ condition holds. In [14], He and Zou proved that (1.2) has a positive ground state solution by using the Nehari manifold. Wang et al. in [24] also proved the multiplicity of positive ground state solutions for (1.2) by the same methods in [14] when $N = 3$ and $f(x,u) = \lambda f(u) + { \vert u \vert ^{4}}u$. The existence of infinitely many solutions to (1.2) has been derived by a variant version of fountain theorem in [18]. In [19], by using a monotonicity trick and a global compactness lemma, Li and Ye obtained the positive ground state for problem (1.2) when $f(x,u)= \vert u \vert ^{p-2}u$ and $p\in(3,\frac{2N}{N-2})$. Recently, Liu and Guo in [17] extended the above result to $p\in (2,\frac{2N}{N-2})$. For more related results, we refer the readers to [1–3, 5–7, 9–13, 20, 21, 23, 28] and the references therein.

In some of the aforementioned references, the potential V is always assumed to be positive or vanishing at infinity. Here, we consider (1.1) with more general potential V, especially the potential V can be sign-changing. By a variational method like [27], the existence and concentration of nontrivial solutions of (1.1) are established. We need to overcome some new difficulties, which involves many technical estimates in our paper.

Our main result concerning problem (1.1) is the following.

Theorem 1.1

Suppose that conditions (V₁)-(V₃) hold and $4 < p < 6$. Then there exist positive constants $\Lambda > 0$ and $b_{\lambda}^{*} > 0$ such that problem (1.1) has at least one nontrivial solution $u_{\lambda}\in H^{1}(\mathbb{R}^{3})$ for $\lambda > \Lambda$ in the case of $b > b_{\lambda}^{*}$.

If $V \ge0$, the following result can be obtained.

Theorem 1.2

Suppose that conditions (V₁)-(V₃) hold. Moreover, $V(x) \ge0$ and $4 \le {p < 6}$. Then there exists a constant $\Lambda > 0$ such that problem (1.1) has at least one nontrivial solution $u_{\lambda}\in H^{1}(\mathbb{R}^{3})$ for $\lambda > \Lambda$.

For the concentration of the solutions of (1.1) as $\lambda \to + \infty$, we have the following.

Theorem 1.3

Let ${u_{\lambda}}$ be the solutions obtained in Theorem  1.2, then ${u_{\lambda}} \to\bar{u}$ in ${H^{1}}({\mathbb{R}^{3}})$ as $\lambda \to + \infty$, where $\bar{u} \in H_{0}^{1}(\Omega)$ is a nontrivial solution of

$$ \textstyle\begin{cases} - ( {a + b{{\int_{\Omega}{ \vert {\nabla u} \vert ^{2}} }}} )\Delta u = {{ \vert u \vert }^{p - 2}}u,&\textit{in }\Omega,\\ u = 0,&\textit{on }\partial\Omega. \end{cases} $$

(1.4)

The paper is organized as follows. In Section 2, we introduce some notations and the variational framework for (1.1), and then establish compactness conditions. In Section 3, we prove Theorem 1.1 and Theorem 1.2. In the last section, we study the concentration of solutions and prove Theorem 1.3.

2 Preliminary results

In this section, we introduce some notations and the variational framework for (1.1) and establish some decomposition of the space to apply the link theorem.

Let ${V^{\pm}}(x) = \max\{ \pm V(x),0\}$. Then $V(x) = {V^{+} }(x) - {V^{-} }(x)$. We consider the space

$$ E = \biggl\{ u \in{H^{1}}\bigl({{\mathbb{R}} ^{3}}\bigr): \int_{{{\mathbb{R}} ^{3}}} {{V^{+} } {u^{2}} < + \infty} \biggr\} $$

with respect to the inner product and norm defined through

$$ \langle{u,v} \rangle= \int_{{\mathbb{R}^{3}}} {\bigl(a\nabla u\nabla v + {V^{+} }uv} \bigr),\quad \Vert u \Vert = { \langle{u,u} \rangle^{\frac{1}{2}}}. $$

For $\lambda > 0$, we also consider the following inner product and norm:

$$ { \langle{u,v} \rangle_{\lambda}} = \int_{{\mathbb{R}^{3}}} {\bigl(a\nabla u\nabla v + \lambda{V^{+} }uv} \bigr),\quad { \Vert u \Vert _{\lambda}} = \langle{u,u} \rangle _{\lambda}^{\frac{1}{2}}. $$

We remark that $\Vert u \Vert \le{ \Vert u \Vert _{\lambda}}$ for $\lambda \ge1$. Set ${E_{\lambda}} = (E,{ \Vert \cdot \Vert _{\lambda}})$. By (V₁), (V₂) and the Poincaré inequality, we can claim that the Hilbert space E is embedded continuously into ${H^{1}}({\mathbb{R}^{3}})$. In fact, for any $u\in E$, letting $d=\min\{a,1\}$, $V_{b}=\{x\in\mathbb{R}^{3}: V(t)< b\}$, $V_{b}^{c}=\{x\in \mathbb{R}^{3}: V(x)\geq b\}$, we have $\int_{V_{b}}{u^{2}}\leq c\int_{V_{b}}{ \vert \nabla u \vert ^{2}}$ for some positive number c by the Poincaré inequality, and therefore

$$\begin{aligned} \frac{1}{d} \Vert u \Vert =&\frac{1}{d} \int_{\mathbb{R}^{3} } {\bigl(a \vert \nabla u \vert ^{2} + V^{+} u^{2} \bigr)} \\ \geq& \int_{\mathbb{R}^{3} } {\bigl( \vert \nabla u \vert ^{2} + V^{+} u^{2} \bigr)} \\ \geq& \int_{\mathbb{R}^{3} } { \vert \nabla u \vert ^{2} }+ \frac{1}{2} \int_{\mathbb{R}^{3} } {V^{+}u^{2} } \\ =& \int_{\mathbb{R}^{3} } { \vert \nabla u \vert ^{2} }+ \frac{1}{2} \int_{V_{b} } {V^{+}u^{2} }+\frac{1}{2} \int_{V_{b}^{c} } {V^{+}u^{2} } \\ \geq& \int_{\mathbb{R}^{3} } { \vert \nabla u \vert ^{2} }+ \frac{1}{2} \int_{V_{b} } {V^{+}u^{2} }+\frac{b}{2} \int_{V_{b}^{c} } {u^{2} } \\ =& \frac{1}{2} \int_{\mathbb{R}^{3} } { \vert \nabla u \vert ^{2} }+ \frac{1}{2} \int_{V_{b} } { \vert \nabla u \vert ^{2} }+ \frac{1}{2} \int_{V_{b}^{c} } { \vert \nabla u \vert ^{2} }+ \frac{1}{2} \int_{V_{b} } {V^{+}u^{2} }+\frac{b}{2} \int_{V_{b}^{c} } {u^{2} } \\ \geq& \frac{1}{2} \int_{\mathbb{R}^{3} } { \vert \nabla u \vert ^{2} }+ \frac{1}{2c} \int_{V_{b} } {u^{2} }+\frac{b}{2} \int_{V_{b}^{c} } {u^{2} } \\ \geq& \min\biggl\{ \frac{1}{2},\frac{1}{2c},\frac{b}{2} \biggr\} \int_{\mathbb{R}^{3} } {\bigl( \vert \nabla u \vert ^{2} +u^{2}\bigr)} \\ =&\min\biggl\{ \frac{1}{2},\frac{1}{2c},\frac{b}{2}\biggr\} \Vert u \Vert _{H^{1}}. \end{aligned}$$

Namely, there exists $\bar{c}>0$ such that $\Vert u \Vert _{H^{1}}\leq\bar{c} \Vert u \Vert$.

So, for every $t \in[2,6]$, there exists ${d_{t}} > 0$ (independent of λ for the case $\lambda\ge1$) such that

$$ { \vert u \vert _{t}} \le{d_{t}} \Vert u \Vert \le{d_{t}} { \Vert u \Vert _{\lambda}}\quad \text{for }u \in E_{\lambda}. $$

(2.1)

Set

$$ F_{\lambda}= \bigl\{ u \in E_{\lambda}: \operatorname{supp} u \subset {V^{ - 1}}\bigl([0,\infty)\bigr)\bigr\} $$

and $F_{\lambda}^{\bot}$ will be used to denote the orthogonal complement of $F_{\lambda}$ in ${E_{\lambda}}$. If $V \ge0$, then $E_{\lambda}= F_{\lambda}$, otherwise $F_{\lambda}^{\bot}\ne0$. Let ${A_{\lambda}} = - \Delta + \lambda V$, then ${A_{\lambda}}$ is formally self-adjoint in ${L^{2}}({\mathbb{R}^{3}})$, and the following associated bilinear form:

$$ {a_{\lambda}}(u,v) = \int_{{\mathbb{R}^{3}}} {\bigl(a\nabla u\nabla v + \lambda V(x)uv\bigr)} $$

is continuous in ${E_{\lambda}}$. For fixed $\lambda > 0$, study the eigenvalue problem in $F_{\lambda}^{\perp}$ as follows:

$$ - a\Delta u + \lambda{V^{+} }(x)u = \alpha\lambda{V^{-} }(x)u . $$

(2.2)

We can get that $u \mapsto\int_{{\mathbb{R}^{3}}} {\lambda{V^{-} }(x){u^{2}}\,dx}$ is weakly continuous since $\operatorname{supp} V^{ - }$ is of finite measure. According to the result in [25], we can obtain a sequence of positive eigenvalues ${\alpha_{k}}(\lambda)$, which is expressed by

$$ {\alpha_{k}}(\lambda) = \inf_{k\leq\dim G, G \subset F_{\lambda}^{\bot}} \sup\biggl\{ \Vert u \Vert _{\lambda}^{2}:u \in G,\lambda \int_{{\mathbb{R}^{3}}} {{V^{-} }(x){u^{2}} = 1} \biggr\} ,\quad k = 1,2,\ldots. $$

The eigenvalues admit the decompositions: $0<{\alpha_{1}}(\lambda) < {\alpha_{2}}(\lambda) \le \cdots \le{\alpha_{k}}(\lambda) \to + \infty$ as $k \to + \infty$, and the corresponding eigenfunctions ${e_{k}}$, which may be chosen so that $\langle{{e_{i}},{e_{j}}} \rangle = {\delta _{i,j}}$ are a basis for $F_{\lambda}^{\bot}$. Let

$$ {\widehat{E}_{\lambda}} = \operatorname{span}\bigl\{ {e_{k}}:{\alpha _{j}}(\lambda) \le1\bigr\} \quad\text{and}\quad E_{\lambda}^{+} = \operatorname{span} \bigl\{ {e_{k}}:{\alpha_{j}}(\lambda) > 1\bigr\} . $$

Then ${E_{\lambda}} = {\widehat{E}_{\lambda}} \oplus E_{\lambda}^{+} \oplus F_{\lambda}$ is an orthogonal decomposition with $\dim{\widehat{E}_{\lambda}} < + \infty$. The bilinear form ${a_{\lambda}}$ is negative semidefinite on ${\widehat{E}_{\lambda}}$ and positive definite on $E_{\lambda}^{+} \oplus F_{\lambda}$. If u, v are in different subspaces of the above decomposition of ${E_{\lambda}}$, then ${a_{\lambda}}(u,v) = 0$. These results will be used later.

The energy functional associated with (1.1) is

$$ {I_{\lambda}}(u) = \frac{1}{2} \int_{\mathbb{R}^{3}} {\bigl(a{{ \vert {\nabla u} \vert }^{2}} + \lambda V(x){u^{2}}\bigr) + \frac{b}{4}} { \biggl( { \int_{\mathbb{R}^{3}} {{{ \vert {\nabla u} \vert }^{2}}}} \biggr)^{2}} - \frac{1}{p}{ \int_{\mathbb{R}^{3}} { \vert u \vert } ^{p}}. $$

(2.3)

Let E be a real Banach space and $I:E\rightarrow\mathbb{R}$ be a function of class $C^{1}$. We say that $\{u_{n}\}\subset E$ is a $(C)_{c}$ sequence if $I(u_{n})\rightarrow c$ and $(1 + \Vert {{u_{n}}} \Vert )I'(u_{n})\rightarrow0$.

Lemma 2.1

Suppose that conditions (V₁)-(V₂) hold and $p \in(4, 6)$. Then any ${(C)_{c}}$ sequence of ${I_{\lambda}}$ is bounded in ${E_{\lambda}}$ for every $c \in\mathbb{R}$.

Proof

Assume that $\{ {u_{n}}\} \subset{E_{\lambda}}$ is a ${(C)_{c}}$ sequence of ${I_{\lambda}}$. Then

$$ {I_{\lambda}}({u_{n}}) \to c, \bigl(1+{ \Vert {{u_{n}}} \Vert _{\lambda}}\bigr)I'({u_{n}}) \to0 \quad \text{in }E_{\lambda}^{ - 1}. $$

(2.4)

Thus, for n large enough, we have

$$ \begin{aligned}[b] & {I_{\lambda}}({u_{n}}) - \frac{1}{p} \bigl\langle {{{I'}_{\lambda}}({u_{n}}),{u_{n}}} \bigr\rangle \\ &\quad= \biggl( {\frac{1}{2} - \frac{1}{p}} \biggr) \Vert {{u_{n}}} \Vert _{\lambda}^{2} - \biggl( { \frac{1}{2} - \frac{1}{p}} \biggr) \int_{\mathbb{R}^{3}} {\lambda{V^{-} }(x)u_{n}^{2} + \biggl( {\frac{1}{4} - \frac{1}{p}} \biggr)b{{ \biggl( { \int_{\mathbb{R}^{3}} {{{ \vert {\nabla{u_{n}}} \vert }^{2}}}} \biggr)}^{2}}} \\ &\quad\le c + 1. \end{aligned} $$

(2.5)

Combining (V₁) and (2.5), we deduce that

$$ \begin{aligned} \biggl( {\frac{1}{2} - \frac{1}{p}} \biggr) \Vert {{u_{n}}} \Vert _{\lambda}^{2} &\le c + 1 + \biggl( {\frac{1}{2} - \frac{1}{p}} \biggr) \int_{\mathbb{R}^{3}} {\lambda{V^{-} }(x)u_{n}^{2} - \biggl( {\frac{1}{4} - \frac{1}{p}} \biggr) b{{ \biggl( { \int_{\mathbb{R}^{3}} {{{ \vert {\nabla{u_{n}}} \vert }^{2}}} } \biggr)}^{2}}} \\ & \le c + 1 + \biggl( {\frac{1}{2} - \frac{1}{p}} \biggr) \int_{\mathbb{R}^{3}} {\lambda{V^{-} }(x)u_{n}^{2}} \\ & \le c + 1 + C \biggl( {\frac{1}{2} - \frac{1}{p}} \biggr) \int_{\mathbb{R}^{3}} {\lambda u_{n}^{2}}, \end{aligned} $$

where $C > 0$ is a constant.

Thus we get

$$ \Vert {{u_{n}}} \Vert _{\lambda}^{2} \le C \int_{\mathbb{R}^{3}} {\lambda u_{n}^{2} + } (c + 1) \frac{{2p}}{{p - 2}}. $$

Therefore, it is sufficient to show that $\{ {u_{n}}\}$ is bounded in ${L^{2}}({\mathbb{R}^{3}})$. Assume by contradiction that ${ \vert {{u_{n}}} \vert _{2}} \to+\infty$ as $n \to\infty$. Let ${v_{n}} = \frac{{{u_{n}}}}{{{{ \vert {{u_{n}}} \vert }_{2}}}}$, then ${ \vert {{v_{n}}} \vert _{2}} = 1$. By (2.5) we have

$$ \Vert {{v_{n}}} \Vert _{\lambda}^{2} - \lambda \int_{\mathbb{R}^{3}} {{V^{-} }(x)v_{n}^{2}} + \frac{{p - 4}}{{2(p - 2)}}b{ \biggl( {{ \int_{\mathbb{R}^{3}} { \vert {\nabla{v_{n}}} \vert } ^{2}}} \biggr)^{2}} { \int_{\mathbb{R}^{3}} {u_{n}^{2}}} \leq \frac{{c + 1}}{{(\frac{1}{2} - \frac{1}{p})\int_{\mathbb{R}^{3}} {u_{n}^{2}} }} $$

(2.6)

and therefore, the sequences ${ \Vert {{v_{n}}} \Vert _{\lambda}}$ and ${ (\int_{\mathbb{R}^{3}} {{{ \vert {\nabla{v_{n}}} \vert }^{2}}} )^{2}} \vert {{u_{n}}} \vert _{2}^{2}$ are both bounded. Up to a subsequence, we have

$$ {v_{n}} \rightharpoonup v\quad\text{in } {E_{\lambda}},\qquad {v_{n}} \to v\quad\text{in }L_{\mathrm{loc}}^{s}\bigl({ \mathbb{R}^{3}}\bigr),\qquad {v_{n}} \to v\quad \text{a.e. in } \mathbb{R}^{3} \text{ for }2 \le s < 6. $$

By (2.6) and noting that $\Vert {{v_{n}}} \Vert _{\lambda}^{2} - \lambda\int_{\mathbb{R}^{3}} {{V^{-} }(x)v_{n}^{2}}=\int_{\mathbb{R}^{3}} a \vert \nabla v_{n} \vert ^{2}+\lambda\int_{\mathbb{R}^{3}} V(x){v_{n}}^{2}$, we have

$$ \lambda \int_{\mathbb{R}^{3}} {V(x)v_{n}^{2} \le \frac{{c + 1}}{{(\frac{1}{2} - \frac{1}{p})\int_{\mathbb{R}^{3}} {u_{n}^{2}} }} \to0\quad\text{as }n \to\infty} . $$

(2.7)

By Fatou’s lemma together with (2.6), we see that

$$ {\biggl({ \int_{\mathbb{R}^{3}} { \vert {\nabla v} \vert } ^{2}} \biggr)^{2}} \le\lim_{n \to\infty} \inf{\biggl({ \int_{\mathbb{R}^{3}} { \vert {\nabla{v_{n}}} \vert } ^{2}}\biggr)^{2}} \le\lim_{n \to\infty} \inf \frac{C}{{ \vert {{u_{n}}} \vert _{2}^{4}}} = 0. $$

Hence $v \equiv \mathit{constant}$. Since $v \in{H^{1}}({\mathbb{R}^{3}})$, we infer that $v = 0$.

Let $V_{b}=\{x\in\mathbb{R}^{3}: V(x)< b\}$, $V_{b}^{c}=\{x\in\mathbb{R}^{3}: V(x)\geq b\}$. By (V₂), for any given $\varepsilon > 0$, there exists ${R_{\varepsilon}} > 0$ with $\operatorname{meas}(B_{{R_{\varepsilon}}}^{c}(0)\cap V_{b} ) < \varepsilon$, where ${B_{{R_{\varepsilon}}}}(0) = \{ x \in{\mathbb{R}^{3}}: \vert x \vert \le{R_{\varepsilon}}\}$, $B_{{{R_{\varepsilon}}}}^{c}(0) = {\mathbb{R}^{3}}\backslash {B_{{R_{\varepsilon}}}}(0)$. Therefore, for any fixed $t\in(1,3)$, as n is large enough, we have

$$ \begin{aligned}[b] \int_{V_{b} } V(x)v_{n}^{2}\,dx &\le \int_{{B_{{R_{\varepsilon}}}}(0) \cap V_{b} } {bv_{n}^{2}\,dx} + \int_{B_{{R_{\varepsilon}}(0)}^{c}(0) \cap V_{b} } {bv_{n}^{2}\,dx} \\ & \le\varepsilon+ b \vert {{v_{n}}} \vert _{2t}^{t} \operatorname{meas} {\bigl(B_{{R_{\varepsilon}}}^{c}(0) \cap V_{b} \bigr)^{\frac{{t - 1}}{t}}} \\ & \le c\varepsilon. \end{aligned} $$

(2.8)

Therefore, it follows from (2.8) and $\vert {{v_{n}}} \vert _{2}^{2} = 1$ that

$$ \begin{aligned} \int_{\mathbb{R}^{3}} V(x)v_{n}^{2}\,dx &= \int_{V_{b}^{c} } {V(x)v_{n}^{2}\,dx} + \int_{V_{b} } {V(x)v_{n}^{2}\,dx} \\ & \ge b \int_{V_{b}^{c} } {v_{n}^{2}\,dx} + o(1) \\ & \ge b \biggl( {1 - \int_{V_{b} } {v_{n}^{2}\,dx}} \biggr) + o(1) = b + o(1) > 0, \end{aligned} $$

which contradicts (2.7). This completes the proof. □

Now, we describe the following lemma for the case $p\in[4,6)$ and $V \geq0$.

Lemma 2.2

Assume that $p\in[4,6)$, $V \ge0$ and conditions (V₁)-(V₂) hold. Then there exists $\Lambda>0$ such that ${I_{\lambda}}$ satisfies ${(C)_{c}}$ condition for all $\lambda > \Lambda$ and $c=c_{\lambda}:= \inf_{\gamma \in\Gamma } \max_{t \in[0,1]} \mathrm{I}_{\lambda}(\gamma(t))$, which is showed in (3.7) later.

Proof

Let ${u_{n}}$ be a $(C)_{c}$ sequence. By Lemma 2.1, ${u_{n}}$ is bounded in ${E_{\lambda}}$ and there exists C such that ${ \Vert {{u_{n}}} \Vert _{\lambda}} \le{C}$ (for the case $p = 4$, that is also true by (2.5) and $V \ge0$).

Hence, without loss of generality, we can say that

$$ {u_{n}} \rightharpoonup u\quad\text{in } {E_{\lambda}},\qquad {u_{n}} \to u\quad \text{in }L_{\mathrm{loc}}^{s}\bigl({ \mathbb{R}^{3}}\bigr) \text{ for } 2 \le s < 6\quad \text{and}\quad { \int_{\mathbb{R}^{3}} { \vert {\nabla{u_{n}}} \vert } ^{2}} \to{A^{2}}. $$

Firstly, we can claim that ${I'_{\lambda}}(u) = 0$ for $4 \le p < 6$.

If $u \equiv0$, then the claim is finished.

If $u\not \equiv0$, then we see

$$ { \int_{\mathbb{R}^{3}} { \vert {\nabla u} \vert } ^{2}} \le \lim_{n \to\infty} \int_{\mathbb{R}^{3}} {{{ \vert {\nabla{u_{n}}} \vert }^{2}}} = {A^{2}}. $$

Suppose ${\int_{\mathbb{R}^{3}} { \vert {\nabla u} \vert } ^{2}} < {A^{2}}$, since ${I'_{\lambda}}({u_{n}}) \to0$ and ${\int_{\mathbb{R}^{3}} { \vert {\nabla{u_{n}}} \vert } ^{2}} \to{A^{2}}$, then

$$ \int_{\mathbb{R}^{3}} {\bigl(a\nabla u\nabla\varphi+ \lambda V(x)u\varphi \bigr)} + b{A^{2}} \int_{\mathbb{R}^{3}} {\nabla u\nabla\varphi- \int_{\mathbb{R}^{3}} {{{ \vert u \vert }^{p - 2}}u\varphi= 0,\quad \forall\varphi\in{E_{\lambda}}} }. $$

Then ${I'_{\lambda}}(u)u < 0$. Noting that ${I'_{\lambda}}(tu)(tu) > 0$ for small $t > 0$ and $\langle{{{I'}_{\lambda}}(tu),tu} \rangle$ is continuous on $t \in[0,1]$. Therefore, there exists ${t_{0}} \in(0,1)$ such that

$$ \bigl\langle {{I'_{\lambda}}({t_{0}}u),{t_{0}}u} \bigr\rangle = 0. $$

Observing the definition of $c_{\lambda}$ and ${I_{\lambda}}({t_{0}}u) = {\max_{t \in[0,1]}}I(tu)$, we have

$$ \begin{aligned} c_{\lambda}&\le{I_{\lambda}}({t_{0}}u) = {I_{\lambda}}({t_{0}}u) - \frac{1}{4} \bigl\langle {{I'_{\lambda}}({t_{0}}u),{t_{0}}u} \bigr\rangle \\ & = \frac{{t_{0}^{2}}}{4} \int_{\mathbb{R}^{3}} {\bigl(a{{ \vert {\nabla u} \vert }^{2}} + \lambda V(x){u^{2}}\bigr) + \biggl( { \frac{1}{4} - \frac{1}{p}} \biggr)t_{0}^{p} \int_{\mathbb{R}^{3}} {{{ \vert u \vert }^{p}}} } \\ & < \frac{1}{4} \int_{\mathbb{R}^{3}} {\bigl(a{{ \vert {\nabla u} \vert }^{2}} + \lambda V(x){u^{2}}\bigr) + \biggl( { \frac{1}{4} - \frac{1}{p}} \biggr) \int_{\mathbb{R}^{3}} {{{ \vert u \vert }^{p}}} } \\ & \le\lim_{n \to\infty} \inf\biggl[ {\frac{1}{4} \int_{\mathbb{R}^{3}} {\bigl(a{{ \vert {\nabla{u_{n}}} \vert }^{2}} + \lambda V(x)u_{n}^{2}\bigr)} + \biggl( {\frac{1}{4} - \frac{1}{p}} \biggr) \int_{\mathbb{R}^{3}} {{{ \vert {{u_{n}}} \vert }^{p}}} } \biggr] \\ & = \lim_{n \to\infty} \inf\biggl[{I_{\lambda}}({u_{n}}) - \frac{1}{4} \bigl\langle {{I'_{\lambda}}({u_{n}}),{u_{n}}} \bigr\rangle \biggr] = c_{\lambda}, \end{aligned} $$

which is impossible. Then $\int_{\mathbb{R}^{3}} {{{ \vert {\nabla u} \vert }^{2}}} = {A^{2}} = \lim_{n \to\infty} \int_{\mathbb{R}^{3}} {{{ \vert {\nabla u_{n}} \vert }^{2}}}$, and so ${I'_{\lambda}}(u) = 0$ for $4 \le {p < 6}$. Thus, the claim is got.

Furthermore, from $V \ge0$ and $p\in[4,6)$, it follows that ${a_{\lambda}}(u,u) = \Vert u \Vert _{\lambda}^{2}$ and

$$ {I_{\lambda}}(u) = {I_{\lambda}}(u) - \frac{1}{4} \bigl\langle {{I'_{\lambda}}(u),u} \bigr\rangle = \frac{1}{4} \Vert u \Vert _{\lambda}^{2} + \biggl( {\frac{1}{4} - \frac{1}{p}} \biggr) \vert u \vert _{p}^{p} \ge0. $$

(2.9)

Next, we show that ${u_{n}} \to u$ in ${E_{\lambda}}$. Let ${v_{n}}: = {u_{n}} - u$.

By (V₂) and a proof similar to (2.8), we have

$$ \vert {{v_{n}}} \vert _{2}^{2} = \int_{\{ V(x) \ge b\} } {v_{n}^{2}\,dx} + \int_{\{ V(x) < b\} } {v_{n}^{2}\,dx} \le \frac{1}{{\lambda b}} \Vert {{v_{n}}} \Vert _{\lambda}^{2} + o(1). $$

(2.10)

Then, by Hölder’s inequality and Sobolev’s embedding theorem, we have

$$ { \vert {{v_{n}}} \vert _{p}} = \vert {{v_{n}}} \vert _{2}^{\theta} \vert {{v_{n}}} \vert _{6}^{1 - \theta} \le d \vert {{v_{n}}} \vert _{2}^{\theta} \vert { \nabla{v_{n}}} \vert _{2}^{1 - \theta} \le d{(\lambda b)^{ - \frac{\theta}{2}}} { \Vert {{v_{n}}} \Vert _{\lambda}} + o(1), $$

(2.11)

as $n \to+\infty$, where $\theta = \frac{{6 - p}}{{2p}}$ and d is a constant independent of λ.

Applying the Brezis-Lieb lemma, we have

$$ \int_{\mathbb{R}^{3}} {{{ \vert {\nabla{v_{n}}} \vert }^{2}}} = \int_{\mathbb{R}^{3}} {{{ \vert {\nabla{u_{n}}} \vert }^{2}}} - \int_{\mathbb{R}^{3}} {{{ \vert {\nabla u} \vert }^{2}}} + o(1) $$

and

$$ \biggl( { \int_{\mathbb{R}^{3}} { \vert \nabla{v_{n}} \vert }^{2}} \biggr) ^{2} = \biggl( { \int_{\mathbb{R}^{3}}{{ \vert \nabla{u_{n}} \vert }^{2}}} \biggr)^{2} - \biggl( { \int_{\mathbb{R}^{3}}{{ \vert \nabla{u} \vert }^{2}}} \biggr)^{2} + o(1). $$

Moreover, we obtain

$$ {I_{\lambda}}({v_{n}}) = {I_{\lambda}}({u_{n}}) - {I_{\lambda}}(u) + o(1)\quad \text{and}\quad {I'_{\lambda}}({v_{n}}) \to0\quad \text{as } n \to\infty. $$

(2.12)

Therefore, by (2.9) we have

$$\begin{aligned} \frac{1}{4} \Vert {{v_{n}}} \Vert _{\lambda}^{2} + \biggl( {\frac{1}{4} - \frac{1}{p}} \biggr) \vert {{v_{n}}} \vert _{p}^{p} =& {I_{\lambda}}({v_{n}}) - \frac{1}{4} \bigl\langle {{{I'}_{\lambda}}({v_{n}}),{v_{n}}} \bigr\rangle \\ =& c_{\lambda}- {I_{\lambda}}(u) + o(1) \le c_{\lambda}+ o(1). \end{aligned}$$

(2.13)

Hence,

$$ \vert {{v_{n}}} \vert _{p}^{p-2} \le{ \biggl({\frac{{4p}}{{p - 4}}} \biggr)}^{\frac{p-2}{p}}c_{\lambda}^{\frac{p-2}{p}} + o(1)< { \biggl({\frac{{4p}}{{p - 3}}} \biggr)}^{\frac{p-2}{p}}c_{\lambda}^{\frac{p-2}{p}} + o(1). $$

(2.14)

If $p = 4$, it follows from (2.1) and (2.13) that

$$ \vert {{v_{n}}} \vert _{p}^{p-2} \le d_{p}^{p-2} \Vert {{v_{n}}} \Vert _{\lambda}^{p-2} \le{(2{d_{p}})^{p-2}} {{c_{\lambda}}^{\frac{p-2}{2}}} + o(1), $$

(2.15)

where a constant ${d_{p}} > 0$ is independent of $\lambda \ge1$. Hence, whenever $p>4$ or $p=4$, it follows from (2.14)-(2.15) that

$$ \vert v_{n} \vert _{p} ^{p - 2} \le\max\biggl\{ { \biggl( {\frac{{4p}}{{p - 3}}} \biggr)^{\frac{{p - 2}}{p}} c_{\lambda}^{\frac{{p - 2}}{p}} ,(2d_{p} )^{p - 2} c_{\lambda}^{\frac{{p - 2}}{p}} } \biggr\} +o(1). $$

(2.16)

Let $b_{\lambda}=\max \{ { ( {\frac{{4p}}{{p - 3}}} )^{\frac{{p - 2}}{p}} c_{\lambda}^{\frac{{p - 2}}{p}} ,(2d_{p} )^{p - 2} c_{\lambda}^{\frac{{p - 2}}{p}} } \}$. Then, in terms of (2.11), we have

$$ { \vert v_{n} \vert _{p}}^{p}={ \vert v_{n} \vert _{p}}^{p-2}{ \vert v_{n} \vert _{p}}^{2}\leq b_{\lambda}d^{2}( \lambda b)^{-\theta} \Vert v_{n} \Vert ^{2}+o(1). $$

(2.17)

Since $\langle{{I'_{\lambda}}({v_{n}}),{v_{n}}} \rangle = o(1)$, we have

$$\begin{aligned} o(1) =& \Vert {{v_{n}}} \Vert _{\lambda}^{2} + b{\biggl( \int_{\mathbb{R}^{3}} {{{ \vert {\nabla{v_{n}}} \vert }^{2}}} \biggr)^{2}} - \vert {{v_{n}}} \vert _{p}^{p} \\ \ge& \Vert {{v_{n}}} \Vert _{\lambda}^{2} - \vert {{v_{n}}} \vert _{p}^{p} \\ \ge& \bigl(1-b_{\lambda}d^{2}(\lambda b)^{-\theta} \bigr) \Vert {{v_{n}}} \Vert _{\lambda}^{2} + o(1). \end{aligned}$$

(2.18)

Hence, there exists a positive number Λ such that ${v_{n}} \to 0$ in ${E_{\lambda}}$ as $n \to\infty$ for $\lambda > \Lambda$. □

Remark 2.3

About the proof of Lemma 2.2, we can see that formula (2.9) is vital. Since V is sign-changing, for any critical point u of ${I_{\lambda}}$, it becomes more difficult to induce the result that ${I_{\lambda}}(u) \ge0$. Indeed, we have the following corollary.

Corollary 2.4

Suppose that conditions (V₁)-(V₂) hold and $p\in(4,6)$. Let $\{ {u_{n}}\}$ be a ${(C)_{c}}$ sequence of ${I_{\lambda}}$ with level $c=c_{\lambda}>0$, where $c_{\lambda}= \inf_{\gamma \in \Gamma} \max_{u \in Q} I_{\lambda}(\gamma(u))$, $\Gamma: = \{ C(Q,E_{\lambda}):\gamma\vert_{\partial Q} = I_{d} \}$, which is mentioned in Proposition  3.1. Then there exists $\Lambda >0$ such that, up to a subsequence, ${u_{n}} \to u$ in ${E_{\lambda}}$. Moreover, the nontrivial critical point of ${I_{\lambda}}$ satisfies ${I_{\lambda}}(u) \le c$ for all $\lambda > \Lambda$.

Proof

We adopt an approach similar to the proof of Lemma 2.2. In terms of Lemma 2.1, we know that $\{ {u_{n}}\}$ is bounded by ${c_{\lambda}}$ in ${E_{\lambda}}$. Then ${u_{n}} \rightharpoonup u$ in ${E_{\lambda}}$, and u is a critical point of ${I_{\lambda}}$. However, since V can be sign-changing and

$$ {I_{\lambda}}(u) = {I_{\lambda}}(u) - \frac{1}{4} \bigl\langle {{{I'}_{\lambda}}(u),u} \bigr\rangle = \frac{1}{4} \Vert u \Vert _{\lambda}^{2} - \frac{\lambda}{4} \int_{\mathbb{R}^{3}} {{V^{-} }(x){u^{2}}} + \biggl( { \frac{1}{4} - \frac{1}{p}} \biggr) \vert u \vert _{p}^{p}, $$

we cannot deduce that ${I_{\lambda}}(u) \ge0$. Next, we only need to consider the following two cases:

$$\mbox{(i)}\quad {I_{\lambda}}(u) < 0;\qquad \mbox{(ii)}\quad {I_{\lambda}}(u) \ge0. $$

In case (i), obviously, u is a nontrivial solution and the conclusion is obtained.

In case (ii), as in the proof of Lemma 2.2, we can see ${u_{n}} \to u$ in ${E_{\lambda}}$. Let ${v_{n}} = {u_{n}} - u$, indeed, by (V₂) and deduction similar to (2.8), we have

$$ \lambda \int_{\mathbb{R}^{3}} {{V^{-} }(x)v_{n}^{2}\,dx \to0}. $$

(2.19)

Therefore, similar to (2.13), we have

$$ \frac{1}{4} \Vert {{v_{n}}} \Vert _{\lambda}^{2} + \biggl(\frac{1}{4} - \frac{1}{p}\biggr) \vert {{v_{n}}} \vert _{p}^{p} + o(1) \le c_{\lambda}- {I_{\lambda}}(u) + o(1) \le c_{\lambda}+ o(1). $$

So we also have (2.18). Hence ${u_{n}} \to u$ in ${E_{\lambda}}$ and ${I_{\lambda}}(u) = c > 0$ and the proof is finished. □

3 Proof of Theorem 1.1 and Theorem 1.2

We first give the link theorem [22] under $(C)_{c}$ condition which is useful in the case of V is sign-changing. We will obtain the solutions of (1.1) and give the proofs of Theorem 1.1 and Theorem 1.2.

Proposition 3.1

Let $E = {E_{1}} \oplus{E_{2}}$ be a Banach space with $\operatorname{dim}{E_{2}} < \infty$, $\Phi \in{C^{1}}(E,{\mathbb{R}})$. If there exist $R > \rho > 0, \kappa > 0$ and ${e_{0}} \in{E_{1}}$ such that

$$ \kappa= \inf\Phi({E_{1}} \cap{S_{\rho}}) > \sup\Phi( \partial Q), $$

where ${S_{\rho}} = \{ u \in E: \Vert u \Vert = \rho\}$, $Q = \{ u = v + t{e_{0}}:v \in{E_{2}},t \ge0, \Vert u \Vert \le R\}$. Then $c\geq\kappa$ and Φ has a ${(C)_{c}}$ sequence, where $c=\inf_{\gamma \in\Gamma} \max_{u \in Q} I_{\lambda}(\gamma (u))$, $\Gamma: = \{ C(Q,E ):\gamma\vert_{\partial Q} = I_{d} \}$.

Here, we use Proposition 3.1 with ${E_{1}} = E_{\lambda}^{+} \oplus F_{\lambda}$ and ${E_{2}} = {\widehat{E}_{\lambda}}$. For every j fixed, by Lemma 2.1 in [8], we have ${\alpha _{j}}(\lambda) \to0$ as $\lambda \to\infty$. Hence, for $\lambda > {\Lambda_{0}}$, ${\widehat{E}_{\lambda}}$ is the finite dimensional space and there is ${\Lambda_{0}} > 0$ such that ${\widehat{E}_{\lambda}} \ne\emptyset$. All of this indicates that there exists ${\widehat{C}_{\lambda}} > 0$ with

$$ { \vert u \vert ^{p}} \ge{\widehat{C}_{\lambda}} \Vert u \Vert \quad \text{for } u \in{\widehat{E}_{\lambda}}, $$

(3.1)

where ${\widehat{C}_{\lambda}}$ is a constant dependent on λ. Now we will verify that the functional $I_{\lambda}$ satisfies the linking structure.

Lemma 3.2

For each $\lambda > {\Lambda_{0}}$, there exist ${\rho_{\lambda}} > 0$ and $\kappa_{\lambda}>0$ such that ${I_{\lambda}}(u) \ge{\kappa_{\lambda}}$ for all $u \in E_{\lambda}^{+} \oplus F_{\lambda}$ with ${ \Vert u \Vert _{\lambda}} = {\rho_{\lambda}}$. Furthermore, as $V \ge0$, we can choose the constants ρ and κ independent of λ for the case $\lambda\ge1$.

Proof

By the definition of $E_{\lambda}^{+}$, there exists ${\delta_{\lambda}} > 0$ such that

$$ {a_{\lambda}}(u,u) \ge{\delta_{\lambda}} \Vert u \Vert _{\lambda}^{2}\quad\text{for }u \in E_{\lambda}^{+} , $$

and

$$ {a_{\lambda}}(u,u) = \Vert u \Vert _{\lambda}^{2}\quad \text{for }u \in F_{\lambda}. $$

Therefore, for $u = v + w \in E_{\lambda}^{+} \oplus F_{\lambda}$, since ${ \langle{v,w} \rangle_{\lambda}} = 0$ and $a_{\lambda}(v, w)=0$ as mentioned before, we have

$$ \begin{aligned} {I_{\lambda}}(u) &= \frac{1}{2}{a_{\lambda}}(v,v) + \frac{1}{2}{a_{\lambda}}(w,w) + \frac{b}{4}{ \biggl( {{ \int_{\mathbb{R}^{3}} { \vert {\nabla u} \vert } ^{2}}} \biggr)^{2}} - \frac{1}{p} \vert u \vert _{p}^{p} \\ & \ge\frac{1}{2}\min\{ {\delta_{\lambda}},1\} \Vert u \Vert _{\lambda}^{2} - \bar{c} \Vert u \Vert _{\lambda}^{p}, \end{aligned} $$

where the constant c̄ is independent of $\lambda \ge1$.

By (2.1), we can choose ${\rho_{\lambda}} > 0$ and small $\kappa _{\lambda}$ such that the first half of the lemma holds. If $V \ge0$, note that ${a_{\lambda}}(u,u) = \Vert u \Vert _{\lambda}^{2}$, thus we finally have the conclusion. □

Now, we choose $e_{0}\in C_{0}^{\infty}(\Omega)$ which will be used in the following lemma, by (V₃), we have $e_{0}\in F_{\lambda}$.

Lemma 3.3

Suppose that assumptions in Theorem  1.1 hold. For each $\lambda > {\Lambda_{0}}$, there exist $b^{*} (\lambda) > 0$ and ${R_{\lambda}} > \rho_{\lambda}$ such that for $b < b^{*} (\lambda)$

$$ \sup_{u \in\partial Q} {I_{\lambda}}(u) < {\kappa_{\lambda}}, $$

where $Q = \{ u = v + t{e_{0}}:v \in{\widehat{E}_{\lambda}},t \ge0, \Vert u \Vert \le{R_{\lambda}}\}$, $\kappa_{\lambda}$ and $\rho_{\lambda}$ mentioned in Lemma  3.2.

Proof

(i) For $u = v + w \in{\widehat{E}_{\lambda}} \oplus\mathbb{R}{e_{0}}$, since $a_{\lambda}(v, w)=0$ as before, we have

$$ {a_{\lambda}}(u,u) = {a_{\lambda}}(v,v) + {a_{\lambda}}(w,w). $$

We show that $a_{\lambda}(v,v)\leq0$.

In fact, assume that $\hat{E}_{\lambda}= L(e_{1} ,e_{2} , \ldots,e_{m} )$, and $e_{j}$ is an eigenfunction corresponding to eigenvalue $\alpha _{j}(\lambda)$ with $0<\alpha_{j}(\lambda)\leq1$, $j=1,2,\ldots$ . It follows from (2.2) that

$$ \langle e_{j} ,\phi\rangle_{\lambda}= \alpha_{j} (\lambda)\lambda \int_{\mathbb{R}^{3}} V^{-} (x)e_{j} \phi,\quad \forall\phi\in E_{\lambda}. $$

(3.2)

Thus, noting that $0<\alpha_{j}(\lambda)\leq1$, we have

$$ \langle e_{j} , e_{j}\rangle_{\lambda}\leq\lambda \int_{\mathbb{R}^{3}} V^{-}(x)e_{j}^{2}, $$

(3.3)

and therefore $a_{\lambda}(e_{j}, e_{j})\leq0$. Similarly, by (3.2) we also have

$$ 0=\langle e_{j} , e_{i}\rangle_{\lambda}= \alpha_{j}(\alpha)\lambda \int_{\mathbb{R}^{3}} V^{-}(x)e_{j} e_{i},\quad i\neq j. $$

(3.4)

Now, noting that $\{e_{j}\}$ is a base of $\hat{E} _{\lambda}$, we can prove that $a_{\lambda}(v,v)\leq0$. Hence, we have

$$ {a_{\lambda}}(u,u) \leq{a_{\lambda}}(w,w) = a \vert {\nabla w} \vert _{2}^{2} \le{ \Vert u \Vert ^{2}}. $$

In view of the equivalence of all the norms on a finite dimensional space, we obtain

$$ {I_{\lambda}}(u) \le\frac{1}{2}{ \Vert u \Vert ^{2}} + \frac{b}{4} \vert {\nabla u} \vert _{2}^{4} - \frac{1}{p}{ \vert u \vert ^{p}} \to- \infty $$

for $u \in{\widehat{E}_{\lambda}} \oplus\mathbb{R}{e_{0} }$ with ${ \Vert u \Vert _{\lambda}} \to+\infty$. As a result, there exists ${R_{\lambda}} > 0$ such that ${I_{\lambda}}(u) \le\kappa_{\lambda}$ for $u \in{\widehat{E}_{\lambda}}\oplus\mathbb{R}{e_{0}}$ satisfying ${ \Vert u \Vert _{\lambda}} = {R_{\lambda}}$.

(ii) For $u \in{\widehat{E}_{\lambda}}$ with $\Vert u \Vert _{\lambda}\le{R_{\lambda}}$, we have

$$ {I_{\lambda}}(u) \le\frac{b}{4}{ \biggl( {{ \int_{\mathbb{R}^{3}} { \vert {\nabla u} \vert } ^{2}}} \biggr)^{2}} \le\frac{b}{4} \Vert u \Vert _{\lambda}^{4} \le\frac{b}{4}R_{\lambda}^{4}. $$

(3.5)

Therefore, taking $b^{*}(\lambda) = \frac{{4{\kappa_{\lambda}}}}{{R_{\lambda}^{4}}}$, we obtain the conclusion. □

Proof of Theorem 1.1

By Lemmas 3.2-3.3 and applying Proposition 3.1, it follows that for any $\lambda > {\Lambda_{0} }$ and $0 < b < b_{\lambda}^{*}$, ${I_{\lambda}}$ possesses a ${(C)_{c}}$ sequence $\{ u_{n}\}$ with $c =c_{\lambda}$. Now, by Lemma 2.1 and Corollary 2.4, we can obtain the conclusion of Theorem 1.1. □

Proof of Theorem 1.2

For the case $V \ge0$, we can easily prove that the functional I satisfies the conditions of mountain-pass theorem, and therefore, the existence of nontrivial solutions can be obtained.

Since $V(x) \ge0$, we have

$$ \begin{aligned} {I_{\lambda}}(u) &= \frac{1}{2} \Vert u \Vert _{\lambda}^{2} + \frac{b}{4}{ \biggl( {{ \int_{\mathbb{R}^{3}} { \vert {\nabla u} \vert } ^{2}}} \biggr)^{2}} - \frac{1}{p}{ \int_{\mathbb{R}^{3}} { \vert u \vert } ^{p}} \\ & \ge\frac{1}{2} \Vert u \Vert _{\lambda}^{2} - \frac{1}{p} \int_{\mathbb{R}^{3}} { \Vert u \Vert _{\lambda}^{p}}. \end{aligned} $$

Hence there exist two positive numbers α, ρ such that $I_{\lambda}(u) \ge\alpha$ for ${ \Vert u \Vert _{\lambda}} = \rho$ small enough.

Let $e_{0}\in C_{0}^{\infty}(\Omega)$, then

$$ \begin{aligned}[b] {I_{\lambda}}(te_{0}) &= \frac{{{t^{2}}}}{2} \int_{\mathbb{R}^{3}} {\bigl(a{ \vert {\nabla{e_{0}}} \vert ^{2}}+ \lambda V(x){e_{0}^{2}}\bigr)} + \frac{b}{4}{t^{4}} {\biggl({ \int_{\mathbb{R}^{3}} { \vert {\nabla e_{0}} \vert } ^{2}}\biggr)^{2}} - \frac{{{t^{p}}}}{4} \int_{\mathbb{R}^{3}} {{ \vert e_{0} \vert }^{p}}\,dx \\ &\to- \infty \end{aligned} $$

(3.6)

as $t \to\infty$. Then there exists ${t_{0}} > 0$ large such that

$$ {I_{\lambda}}({t_{0}}e_{0}) < 0\quad \text{and}\quad { \Vert {{t_{0}}e_{0}} \Vert _{\lambda}} > \rho. $$

By the mountain-pass theorem, there exists a ${(C)_{c}}$ sequence $\{ {u_{n}}\} \subset{E_{\lambda}}$ such that

$$ {I_{\lambda}}({u_{n}}) \to{c_{\lambda}}, {I'_{\lambda}}({u_{n}}) \to0\quad\text{in }E_{\lambda}^{ - 1}, $$

where

$$ 0< c_{\lambda} = : \inf_{\gamma \in\Gamma} \max _{t \in[0,1]} \mathrm{I}_{\lambda}\bigl(\gamma(t)\bigr) \le\sup _{t \ge0} {I_{\lambda}}(te_{0}) \le {C_{0}}, $$

(3.7)

$\Gamma = \{ \gamma \in C([0,1],{E_{\lambda}}):\gamma(0) = 0,{ \Vert {\gamma(1)} \Vert _{\lambda}} > \rho,{I_{\lambda}}(\gamma (1)) < 0\}$.

By Lemma 2.2, for λ large enough, we can get a nontrivial critical point u for $I_{\lambda}$ with $I_{\lambda }(u_{\lambda}) \in[c_{\lambda}, {C_{0}}]$. □

4 Concentration for solutions

Now, using the same notation as before, we are ready to investigate the concentration for solutions and give the proof of Theorem 1.3.

Proof

For any sequence ${\lambda_{n}} \to + \infty$, let ${u_{n}}: = {u_{{\lambda_{n}}}}$ be the critical points of ${I_{{\lambda_{n}}}}$ obtained in Theorem 1.2.

It follows from (3.7) and

$$ {I_{{\lambda_{n}}}}({u_{n}})= {I_{{\lambda_{n}}}}({u_{n}}) - \frac{1}{4} \bigl\langle {{I'_{{\lambda_{n}}}}({u_{n}}),{u_{n}}} \bigr\rangle = \frac{1}{4} \Vert {{u_{n}}} \Vert _{{\lambda_{n}}}^{2}+\biggl(\frac{1}{4} - \frac{1}{p} \biggr) \vert {{u_{n}}} \vert _{p}^{p} $$

that

$$ \sup_{n \ge1} \Vert {{u_{n}}} \Vert _{{\lambda_{n}}}^{2} \le4{C_{0}}, $$

(4.1)

where the constant ${C_{0}}$ is independent of ${\lambda_{n}}$.

Therefore, we may assume that ${u_{n}} \rightharpoonup\bar{u}$ in E and ${u_{n}} \to\bar{u}$ in $L_{\mathrm{loc}}^{s}({\mathbb{R}^{3}})$ for $2 \le s < 6$. By Fatou’s lemma, we deduce

$$ \int_{\mathbb{R}^{3}} {V(x){{\bar{u}}^{2}}} \le\lim_{n \to\infty}\inf \int_{\mathbb{R}^{3}} {V(x)u_{n}^{2} \le\lim _{n \to\infty} \inf\frac{{ \Vert {{u_{n}}} \Vert _{{\lambda _{n}}}^{2}}}{{{\lambda_{n}}}} = 0}. $$

Therefore $\bar{u} = 0$ a.e in ${\mathbb{R}^{3}}\backslash{V^{ - 1}}(0)$, and so $\bar{u} \in H_{0}^{1}(\Omega)$ by (V₃).

Now, for any $\varphi \in C_{0}^{\infty}(\Omega)$, since $\langle {{{I'}_{{\lambda_{n}}}}({u_{n}}),\varphi} \rangle = 0$, we obtain

$$ \int_{\Omega}a{\nabla\bar{u}} \nabla\varphi+ b \int_{\Omega}{{{ \vert {\nabla\bar{u}} \vert }^{2}}} \int_{\Omega}{\nabla\bar{u}} \nabla\varphi= { \int_{\Omega}{ \vert {\bar{u}} \vert } ^{p - 2}}\bar{u} \varphi. $$

By the density of $C_{0}^{\infty}(\Omega)$ in $H_{0}^{1}(\Omega)$, ū is a weak solution of (1.4).

Next, we need to prove that ${u_{n}} \to\bar{u}$ in ${L^{s}}({\mathbb {R}^{3}})$ for $s\in(2,6)$. If not, from the vanishing lemma, it follows that there exist two positive constants δ, ρ and ${x_{n}} \in{\mathbb{R}^{3}}$ such that

$$ \int_{{B_{\rho}}(x_{n})} {{{({u_{n}} - \bar{u})}^{2}}} \ge \delta. $$

Moreover, $\vert {x_{n}} \vert \to\infty$. Therefore $\operatorname{meas}({B_{\rho}}({x_{n}}) \cap\{ x \in{\mathbb{R}^{3}}:V(x) < b\} ) \to0$. By Hölder’s inequality and an argument similar to that used in the proof of (2.8), we have

$$ \int_{{B_{\rho}}({x_{n}}) \cap\{ V(x) < b\} } {{{({u_{n}} - \bar{u})}^{2}}\,dx} \to0. $$

Consequently,

$$ \begin{aligned} \Vert {{u_{n}}} \Vert _{{\lambda_{n}}}^{2} &\ge{\lambda_{n}}b \int_{{B_{\rho}}({x_{n}}) \cap\{ V(x) \ge b\} } {u_{n}^{2}\,dx} = {\lambda _{n}}b \int_{{B_{\rho}}({x_{n}}) \cap\{ V(x) \ge b\} } {{{({u_{n}} - \bar{u})}^{2}}\,dx} \\ & = {\lambda_{n}}b \biggl( {{ \int_{{B_{\rho}}({x_{n}})} {{{({u_{n}} - \bar{u})}^{2}}\,dx + o(1)} }} \biggr) \to+\infty, \end{aligned} $$

which contradicts (4.1).

Last, we only need to prove that ${u_{n}} \to\bar{u}$ in E. Since $\langle{{{I'}_{{\lambda_{n}}}}({u_{n}}),{u_{n}}} \rangle = \langle {{{I'}_{{\lambda_{n}}}}({u_{n}}),\bar{u}} \rangle = 0$, we have

$$ \Vert {{u_{n}}} \Vert _{{\lambda_{n}}}^{2} + b{ \biggl( { \int_{\mathbb{R}^{3}} {{{ \vert {\nabla{u_{n}}} \vert }^{2}}} } \biggr) ^{2}} = \int_{\mathbb{R}^{3}} {{{ \vert {{u_{n}}} \vert }^{p}}}, $$

(4.2)

and

$$ { \langle{{u_{n}},\bar{u}} \rangle_{{\lambda_{n}}}} + b \int_{\mathbb{R}^{3}} {{{ \vert {\nabla{u_{n}}} \vert }^{2}}} \int_{\mathbb{R}^{3}} {\nabla{u_{n}}\nabla\bar{u}} = \int_{\mathbb{R}^{3}} {{{ \vert {{u_{n}}} \vert }^{p - 2}} {u_{n}}\bar{u}} . $$

(4.3)

We can prove that

$$ { \biggl( { \int_{\mathbb{R}^{3}} {{{ \vert {\nabla{u_{n}}} \vert }^{2}}}} \biggr)^{2}} - \int_{\mathbb{R}^{3}} {{{ \vert {\nabla{u_{n}}} \vert }^{2}} \int_{\mathbb{R}^{3}} {\nabla{u_{n}}\nabla\bar{u}} } \to0. $$

(4.4)

Combining (4.2), (4.3) and (4.4), we obtain

$$ \lim_{n \to\infty} \Vert {{u_{n}}} \Vert _{{\lambda_{n}}}^{2} = \lim_{n \to\infty} { \langle {{u_{n}},\bar{u}} \rangle_{{\lambda_{n}}}} = \lim_{n \to\infty} \langle{{u_{n}},\bar{u}} \rangle= { \Vert \bar{u} \Vert ^{2}}. $$

Thanks to the weak lower semi-continuity, we have

$$ { \Vert \bar{u} \Vert ^{2}} \le\lim_{n \to\infty} \inf { \Vert {{u_{n}}} \Vert ^{2}} \le\lim _{n \to\infty} \inf \Vert {{u_{n}}} \Vert _{{\lambda_{n}}}^{2}, $$

so, up to a subsequence, $\Vert {u_{n}} \Vert \to \Vert \bar{u} \Vert$. Thus, it follows from $u_{n} \rightharpoonup\bar{u}$ in a Hilbert space E that $u_{n}\rightarrow\bar{u}$ in E.

Since ${u_{n}} \ne0$, by (4.2) we have

$$ { \Vert {{u_{n}}} \Vert ^{2}} \le \Vert {{u_{n}}} \Vert _{{\lambda_{n}}}^{2} \le \vert {{u_{n}}} \vert _{p}^{p} \le C{ \Vert {{u_{n}}} \Vert ^{p}}, $$

which implies that $\bar{u} \ne0$. Then we can obtain the conclusion. □

5 Conclusion

In this paper, by using the variational methods, the existence of nontrivial solutions and the concentration phenomena of the solutions to equation (1.1) were established. We consider (1.1) with more general potential V, especially the potential V can be sign-changing. (1.1) is a nonlocal problem as the appearance of the term ${{{\int _{{\mathbb{R}^{N}}} { \vert {\nabla u} \vert } }^{2}}}$, so we need to overcome some new difficulties, which involves many technical estimates in our paper.