Energy solutions and concentration problem of fractional Schrödinger equation

Abstract

In this paper, we consider a fractional Schrödinger equation with steep potential well and sublinear perturbation. By virtue of variational methods, the existence criteria of infinitely many nontrivial high or small energy solutions are established. In addition, the phenomenon of the concentration of solutions is also explored. We also give some examples to demonstrate the main results.

1 Introduction

In this paper, we are concerned with the following fractional Schrödinger equations:

$$ ( - \Delta)^{\alpha} u + \rho V(x)u = f(x,u) + h(x) \vert u \vert ^{p - 2}u,\quad x \in R^{N}, $$

(1.1)

where $0 < \alpha< 1$, $2\alpha< N$, $\rho> 0$, $1 \le p < 2$, $f \in C(R^{N} \times R,R)$, $h \in L^{\frac{2}{2 - p}}(R^{N})$, $( - \Delta)^{\alpha }$ is the so-called fractional Laplacian operator of order $\alpha\in(0,1)$ and it can be either defined point-wise for $x \in R^{N}$ by

$$( - \Delta)^{\alpha} u(x) = - \frac{1}{2} \int_{R^{N}} \frac{u(x + y) + u(x - y) - 2u(x)}{ \vert y \vert ^{N + 2\alpha}} \,dy $$

along any rapidly decaying function u of class $C^{\infty} (R^{N})$, or characterized by

$$( - \Delta)^{\alpha} u = \mathcal{F}^{ - 1}\bigl( \vert \xi \vert ^{2\alpha} \mathcal{F}u\bigr), $$

where $\mathcal{F}$ denotes the usual Fourier transform in $R^{N}$. The potential V satisfies the conditions:

($V_{1}$):

$V \in C(R^{N},R)$ and $V \ge0$ on $R^{N}$.

($V_{2}$):

There exists a constant $c > 0$ such that the set $\{ V < c\} = \{ x \in R^{N} \vert V(x) < c\}$ is nonempty and has finite measure.

($V_{3}$):

$\Omega = \operatorname{int} V^{ - 1}(0)$ is a nonempty open set and has smooth boundary with $\overline{\Omega} = V^{ - 1}(0)$.

From ($V_{1}$)–($V_{3}$), we can see ρV represents a steep potential well whose depth is controlled by ρ. Bartsch and Wang first introduced this problem for the case $\alpha = 1$ in [1], and since then it has attracted much attention. For more details, please see [2–5].

Over the past decades, the existence and multiplicity of nontrivial solutions for the integer order Schrödinger equation have been extensively investigated. In [6], the authors proved that the fractional Laplacian $( - \Delta)^{\alpha}$ reduces to the standard Laplacian −Δ, as $\alpha\to1$. When $\alpha = 1$, Eq. (1.1) becomes the integer order Schrödinger equation, which shows the results in the present paper are also valid for $\alpha= 1$.

Fractional calculus has played an important role in the description of hereditary properties of various materials and memory processes. Fractional differential equations provides a powerful tool for the research of many fields such as engineering, science, electrical circuits, diffusion, and applied mathematics; see [7–10] and so on. Recently, more and more attention has been focused on the study of problems involving the fractional Laplacian; see [11] and the references therein.

In the field of fractional quantum mechanics, the fractional Schrödinger equation is a fundamental equation, which was discovered by Laskin [12, 13] as a result of extending the Feynman path integral. Recently, for the different cases of the potential V and the nonlinearity f, some researchers have investigated the fractional Schrödinger equations under the appropriate assumptions:

$$ ( - \Delta)^{\alpha} u + V(x)u = f(x,u),\quad x \in R^{N}. $$

(1.2)

The variational method has been used in many works to study the fractional Schrödinger equations (1.2). For the basic theory of the variational method in a fractional setting, one can see [14]. Next, we list some work on (1.2) in the following.

In [15], Felmer et al. considered the regularity and existence of solutions under the famous Ambrosetti–Rabinowitz (A–R) condition, i.e., there exists $\theta> 2$ such that $0 < \theta F(x,t) < tf(x,t)$. In [16], the ground state solutions were obtained by Secchi under the A–R condition and

($V_{0}$):

$V \in C(R^{N})$, $\inf_{x \in R^{N}}V(x) = V_{0} > 0$ and $\lim_{ \vert x \vert \to\infty} V(x) = \infty$.

In [17], Chang obtained the existence and multiplicity of solutions when the nonlinear term f satisfies the asymptotically linear case and under the condition:

($V_{0}^{\prime}$):

There exists $r_{0} > 0$ such that, for all $M > 0$, $\vert \{ x \in B_{r_{0}}(y) \vert V \le M\} \vert \to0$ as $\vert y \vert \to\infty$.

In [18], by the variant fountain theorems, the author discussed the nontrivial high or small energy solutions; In [19], using nonlinear analysis techniques, the weak solutions are obtained. In [20], when $V = 1$ and $f(x,u) = f(u)$, the authors gave the existence of least two nontrivial radial solutions without the A–R condition by variational methods. In [21], the existence criteria of radial solutions are established under different conditions by variational methods. In [22], the authors also studied the existence of infinitely many nontrivial energy solutions by variational methods. In [23], in the asymptotically periodic case, a nontrivial solution is obtained by variational methods. For more related study, the interested reader may consult [24–34] and the references therein. It should also be noted that the concentration phenomena for the fractional Schrödinger equation have been investigated byDávila, del Pino et al. [35, 36], and Fall, Mahmoudi and Valdinoci [37].

But for the generalized fractional Schrödinger equations (1.1) with perturbation and steep potential well, there are very few results. Obviously, the form of (1.1), whose nonlinear term combines asymptotic linearity and superlinear term with sublinear term $h(x) \vert u \vert ^{p - 2}u$, is more general than that of (1.2). In [38], the existence of at least two nontrivial solutions of (1.1) is showed. In the present paper, by variational methods, we have established existence criteria of infinitely many nontrivial high or small energy solutions without A–R condition. In addition, we explore the concentration of solutions as well. Moreover, some examples are given to demonstrate our results. In fact, it is a challenging research to study the concentration problem of solutions for the fractional Schrödinger equations by variational methods and it is not trivial to construct the fractional Sobolev space, which is used to study the concentration phenomenon. As is well known, the compactness of the embedding fails when ρ is large enough. By using a new function space introduced in [38, 39] and constructing some inequalities, we can obtain the concentration of the solutions of (1.1) under different conditions. Hence our results can be viewed as an extension to the main results in [16–23, 38].

We list the following assumptions.

($f_{1}$):

$h \in L^{\frac{2}{2 - p}}(R^{N})$ and $h > 0$ on $R^{N}$.

($H_{1}$):

$f(t,s)$ is continuous on $R^{N} \times R$. There exist constants $a_{0} > 0$ and $v \in[1,2)$ such that

$$\bigl\vert f(x,u) \bigr\vert \le a_{0}\bigl(1 + \vert u \vert ^{v - 1}\bigr),\quad \forall(x,u) \in R^{N} \times R. $$

($H_{2}$):

$\lim_{ \vert u \vert \to0}\frac{f(x,u)}{ \vert u \vert } = 0$ uniformly in $x \in R^{N}$.

($H_{3}$):

$F(x,u) = \int_{0}^{u} f(x,\tau) \,d\tau\ge0$. There exist two constants $\sigma\in[1,v)$ and $b > 0$ such that $\lim_{ \vert u \vert \to\infty} \frac{F(x,u)}{ \vert u \vert ^{\sigma}} \ge b$ uniformly for $x \in R^{N}$.

($H_{4}$):

Let $f \in C(R^{N} \times R,R)$. There exist two constants $a_{1} > 0$, $q \in[2,2_{\alpha}^{*})$ such that

$$\bigl\vert f(x,u) \bigr\vert \le a_{1}\bigl(1 + \vert u \vert ^{q - 1}\bigr),\quad \forall(x,u) \in R^{N} \times R, $$

where $2_{\alpha}^{*} = \frac{2N}{N - 2\alpha}$ with $2\alpha< N$.

($H_{5}$):

$F(x,u) \ge0$ and $\lim_{ \vert u \vert \to\infty} \frac{F(x,u)}{ \vert u \vert ^{2}} = \infty$ uniformly in $x \in R^{N}$.

In the next of this paper, by variational methods, we will give the existence criteria of infinitely many nontrivial high or small energy solutions without A–R condition. Moreover, we will study the concentration phenomenon of (1.1). We will also give some examples to illustrate the main results.

2 Preliminaries

Now, we review some definitions and related lemmas. For $1 < q \le \infty$, we give the definition the following norms:

$$\Vert u \Vert _{\infty} = \max_{t \in R^{N}} \bigl\vert u(t) \bigr\vert ,\qquad \Vert u \Vert _{q} = \Vert u \Vert _{L^{q}( R^{N})} = \biggl( \int_{R^{N}} \bigl\vert u(s) \bigr\vert ^{q}\,ds \biggr)^{\frac{1}{q}}. $$

Let $D^{q}(R^{N})$ denote the completion of $C_{0}^{\infty} (R^{N})$ with respect to the Gagliardo (semi) norm

$$[ u ]_{\alpha} = \biggl( \int_{R^{N}} \int_{R^{N}} \frac{ \vert u(x) - u(z) \vert ^{2}}{ \vert x - z \vert ^{N + 2\alpha}} \,dx\,dz \biggr)^{1/2},$$

then the embedding $D^{q}(R^{N}) \mapsto L^{2_{\alpha}^{*}}(R^{N})$ is continuous with $\Vert u \Vert _{2_{\alpha}^{*}} \le\overline{S}^{ - 1} [ u ]_{\alpha}$, $\forall u \in D^{q}(R^{N})$, where $\overline{S}^{ - 1} = M(N,\alpha)\frac{\alpha(1 - \alpha)}{N - 2\alpha}$ and $\frac{1}{M(N,\alpha)} = \int_{R^{N}} \frac{1 - \cos \zeta}{ \vert \zeta \vert ^{2N + \alpha}} \,d\zeta$ depends only on N and α. For details, see [6].

Define the space

$$X = \biggl\{ u \in D^{q}\bigl(R^{N}\bigr): \int_{R^{N}} \int_{R^{N}} \frac{ \vert u(x) - u(z) \vert ^{2}}{ \vert x - z \vert ^{N + 2\alpha}} \,dx\,dz + \int_{R^{N}} V(x)u^{2}(x) \,dx < + \infty \biggr\} $$

with the inner product and the norm

$$\begin{gathered} \langle u,v \rangle= \int_{R^{N}} \int_{R^{N}} \frac{[u(x) - u(z)][v (x) - v (z)]}{ \vert x - z \vert ^{N + 2\alpha}} \,dx\,dz + \int_{R^{N}} V(x)u(x)v(x)\,dx, \\\Vert u \Vert ^{2} = \langle u,u \rangle\end{gathered} $$

(2.1)

for all $u,v \in X$. Then X is a Hilbert space with the inner product $\langle u,v \rangle$ and $X \subset L^{q}(R^{N})$ for all $q \in[2,2_{\alpha}^{*}]$ with the embedding being continuous. In order to consider (1.1), for $\rho> 0$, parallel to (2.1), we introduce another inner product and normal

$$\begin{gathered} \langle u,v \rangle_{\rho} = \int_{R^{N}} \int_{R^{N}} \frac{[u(x) - u(z)][v(x) - v(z)]}{ \vert x - z \vert ^{N + 2\alpha}} \, dx\,dz + \int_{R^{N}} \rho V(x)u(x)v(x)\,dx,\\ \Vert u \Vert _{\rho }^{2} = \langle u,u \rangle_{\rho}.\end{gathered} $$

Let $X_{\rho} = (X, \Vert u \Vert _{\rho} )$, obviously $X_{\rho}$ is still a Hilbert space and

$$ \Vert u \Vert \le \Vert u \Vert _{\rho}\quad\mbox{for } \rho \ge 1. $$

(2.2)

By a standard argument, we have the following.

Lemma 2.1

Assume that the condition ($V_{1}$)–($V_{2}$) hold. Then the embeddings $X_{\rho} \mapsto L_{\mathrm{loc}}^{q}(R^{N})$ are compact, for any $q \in[2,2_{\alpha}^{*})$.

Lemma 2.2

([38])

Suppose that ($V_{1}$)-($V_{2}$) hold. Then, for $r \in[2,2_{\alpha}^{*}]$,

$$ \int_{R^{N}} \vert u \vert ^{r}\,dx \le d_{r} \Vert u \Vert _{\rho}^{r}, \quad\forall\rho\ge \rho_{1}. $$

(2.3)

In particular,

$$ \int_{R^{N}} \vert u \vert ^{2}\,dx \le d_{2} \Vert u \Vert _{\rho}^{2}, \quad\forall\rho\ge \rho_{1}, $$

(2.4)

where $\rho_{1} = \frac{\overline{S}^{2}}{c} \vert \operatorname {meas}\{ V < c\} \vert ^{\frac{2 - 2_{\alpha}^{*}}{2_{\alpha}^{*}}}$, $d_{r} = \overline{S}^{ - r} \vert \operatorname{meas}\{ V < c\} \vert ^{\frac{2_{\alpha}^{*} - r}{2_{\alpha}^{*}}}$, c is given in ($V_{2}$).

Let $\{ e_{j}\}$ be a complete orthonormal basis of X. We define

$$X_{j}: = \operatorname{span}\{ e_{j}\},\qquad Y_{k}: = \bigoplus_{j = 1}^{k}X_{j}\quad \textit{and}\quad Z_{k}: = \overline{ \bigoplus_{j = k + 1}^{\infty} X_{j}},\quad k \in\mathbb{N}. $$

Also

$$B_{k} = \bigl\{ u \in Y_{k}: \Vert u \Vert \le \rho_{k}\bigr\} ,\qquad S_{k} = \bigl\{ u \in Z_{k}: \Vert u \Vert = r_{k}\bigr\} $$

for $\rho_{k} > r_{k} > 0$. Clearly, $X = Y_{k} \oplus Z_{k}$ with $\operatorname{dim}Y_{k} < \infty$.

The following variant fountain theorem will be used to prove Theorem 3.1.

Lemma 2.3

([40])

Let X be a Banach space, suppose that $\varphi_{\rho,\lambda} (u)$ satisfies:

($A_{1}$):

$\varphi_{\rho,\lambda} (u)$ maps bounded sets into bounded sets uniformly for $\lambda\in[1,2]$, and

$$\varphi_{\rho,\lambda} ( - u) = \varphi_{\rho,\lambda} (u)\quad\textit{for } ( \lambda,u) \in[1,2] \times X. $$

($A_{2}$):

$B(u) \ge0$ for all $u \in X$, and $B(u) \to\infty$ as $\Vert u \Vert \to\infty$ on any finite dimensional subspace of X.

($A_{3}$):

There exist $\rho_{k} > r_{k} > 0$ such that

$$a_{k}(\lambda): = \inf_{u \in Z_{k}, \Vert u \Vert = \rho_{k}}\varphi_{\rho,\lambda} (u) \ge0 > b_{k}(\lambda): = \max_{u \in Y_{k}, \Vert u \Vert = r_{k}} \varphi_{\rho,\lambda} (u), \quad\forall \lambda\in[1,2] $$

and

$$c_{k}(\lambda): = \inf_{u \in Z_{k}, \Vert u \Vert \le \rho_{k}}\varphi_{\rho,\lambda} (u) \to0,\quad\textit{as } k \to\infty \textit{ uniformly for } \lambda \in[1,2]. $$

Then there exist $\lambda_{n} \to1$, $u(\lambda_{n}) \in Y_{n}$ such that

$$\varphi'_{\rho,\lambda_{n}} \big\vert _{Y_{n}} \bigl(u( \lambda_{n})\bigr) = 0\quad\textit{and}\quad \varphi_{\rho,\lambda_{n}}\bigl(u( \lambda_{n})\bigr) \to \omega_{k} \in\bigl[c_{k}(2),b_{k}(1) \bigr],\quad\textit{as } n \to\infty. $$

In particular, if $\{ u(\lambda_{n})\}$ has a convergent subsequence for any $k \in N$, then $\varphi_{\rho, 1}$ has infinitely many nontrivial critical points $\{ u_{k}\} \in X\setminus\{ 0\}$ satisfying $\varphi_{\rho,1}(u_{k}) \to0^{ -}$ as $k \to\infty$.

Next, we give another variant fountain theorem which will be needed to prove Theorem 3.2.

Lemma 2.4

([40])

Let X be a Banach space, suppose that $\varphi_{\rho,\lambda} (u)$ defined above satisfies:

($B_{1}$):

$\varphi_{\rho,\lambda} (u)$ maps bounded sets into bounded sets uniformly for $\lambda\in[1,2]$, and

$$\varphi_{\rho,\lambda} ( - u) = \varphi_{\rho,\lambda} (u)\quad\textit{for every } (\lambda,u) \in[1,2] \times X. $$

($B_{2}$):

$B(u) \ge0$ for all $u \in X$, $A(u) \to \infty$ or $B(u) \to\infty$ as $\Vert u \Vert \to \infty$.

($B_{3}$):

There exist $\rho_{k} > r_{k} > 0$ such that

$$e_{k}(\lambda) = \inf_{u \in Z_{k}, \Vert u \Vert = r_{k}}\varphi_{\rho,\lambda} (u) > f_{k}(\lambda) = \max_{u \in Y_{k}, \Vert u \Vert = \rho_{k}} \varphi_{\rho,\lambda} (u), \quad\forall \lambda\in[1,2]. $$

Then

$$e_{k}(\lambda) \le g_{k}(\lambda) = \inf _{\gamma\in\Gamma_{k}}\max_{u \in B_{k}}\varphi_{\rho,\lambda} \bigl(\gamma(u)\bigr),\quad\forall\lambda\in [1,2], $$

where $\Gamma_{k} = \{ \gamma\in C(B_{k},X_{\lambda} ):\gamma\textit{ is odd}, \gamma \vert _{\partial B_{k}} = \mathit{id}\}$ ($k \ge2$). In addition, for almost every $\lambda\in[1,2]$, there exists a sequence $\{ u_{n}^{k}(\lambda)\}_{n = 1}^{\infty}$ such that

$$\sup_{n} \bigl\Vert u_{n}(\lambda) \bigr\Vert < \infty,\qquad\varphi '_{\rho,\lambda} \bigl(u_{n}^{k}( \lambda)\bigr) \to0\quad\textit{and}\quad \varphi_{\rho,\lambda} \bigl(u_{n}^{k}( \lambda)\bigr) \to g_{k}(\lambda),\quad \textit{as } n \to\infty. $$

The following famous Lions vanishing lemma is useful for the proofs of Theorems 3.3–3.4.

Lemma 2.5

([16])

Suppose that $\{ u_{n}\}$ is bounded in $D^{q}(R^{N})$ and satisfies

$$\lim_{n \to\infty}\sup_{ y \in R^{N}} \int_{B(y,R)} \bigl\vert u_{n}(x) \bigr\vert ^{2}\,dx = 0 $$

for some $R > 0$, where $B(y,R) = \{ x \in R^{n}: \vert x - y \vert \le R\}$. Then $u_{n} \to0$ strongly in $L^{q}(R^{N})$, for $q \in [2,2_{\alpha}^{*})$.

3 Main results

Without loss of generality, we use the same notation $\{ u_{n}\}$ for a sequence $\{ u_{n}\}$ and any of its subsequences. We denote

$$\begin{gathered} A(u) = \frac{1}{2} \Vert u \Vert _{\rho}^{2} = \frac{1}{2}\biggl([u]_{\alpha}^{2} + \int_{R^{N}} \rho Vu^{2}\,dx\biggr),\\ B(u) = \int_{R^{N}} F(x,u)\,dx + \frac{1}{p} \int_{R^{N}} h(x) \vert u \vert ^{p}\,dx\end{gathered} $$

and

$$ \varphi_{\rho,\lambda} (u) = A(u) - \lambda B(u) = \frac{1}{2} \Vert u \Vert _{\rho}^{2} - \lambda\biggl[ \int_{R^{N}} F(x,u)\,dx + \frac{1}{p} \int_{R^{N}} h(x) \vert u \vert ^{p}\,dx \biggr] $$

(3.1)

for all $u \in X_{\rho}$ and $\lambda\in[1,2]$.

It is easy to verify that $\varphi_{\lambda,\mu} (u):X_{\lambda} \to R$ is a $C^{1}$-functional for $\lambda\in[1,2]$ and

$$\begin{aligned}[b] \bigl\langle \varphi'_{\rho,\lambda} (u),v \bigr\rangle ={}& \int_{R^{N}} \int_{R^{N}} \frac{[u(x) - u(z)][v(x) - v(z)]}{ \vert x - z \vert ^{N + 2\alpha}} \,dx\,dz + \int_{R^{N}} \rho V(x)u(x)v(x)\,dx \\&- \lambda\biggl[ \int_{R^{N}} f(x,u)v\,dx + \int_{R^{N}} h(x) \vert u \vert ^{p - 2}uv\,dx \biggr]\end{aligned} $$

(3.2)

for all $u,v \in X$. Hence the critical points of $\varphi_{\rho,1}$ are solutions of (1.1). Next, we will discuss the existence of critical points of $\varphi_{\rho,1}$.

Lemma 3.1

Assume ($V_{1}$)–($V_{2}$), ($f_{1}$), ($H_{1}$)–($H_{3}$) hold. Then, for $\rho> \rho_{0}$, there exist $\rho_{k} > r_{k} > 0$ such that

$$a_{k}(\lambda): = \inf_{u \in Z_{k}, \Vert u \Vert = \rho_{k}}\varphi_{\rho,\lambda} (u) \ge0 > b_{k}(\lambda): = \max_{u \in Y_{k}, \Vert u \Vert = r_{k}} \varphi_{\rho,\lambda} (u),\quad \forall \lambda\in[1,2] $$

and

$$c_{k}(\lambda): = \inf_{u \in Z_{k}, \Vert u \Vert \le \rho_{k}}\varphi_{\rho,\lambda} (u) \to0,\quad\textit{as } k \to\infty \textit{ uniformly for } \lambda \in[1,2], $$

where $\rho_{0} = \max\{ 1,\rho_{1}\}$.

Proof

($H_{1}$)–($H_{3}$) imply that, for arbitrary $\delta> 0$ with $d_{2}\delta< \frac{1}{12}$, there exists a constant $c_{\delta}$ depending on δ such that

$$\begin{aligned}& F(x,u) \le\delta \vert u \vert ^{2} + c_{\delta} a_{0} \vert u \vert ^{v}\quad\mbox{for all } (x,u) \in R^{N} \times R, \end{aligned}$$

(3.3)

$$\begin{aligned}& F(x,u) \ge b \vert u \vert ^{\sigma} - \delta \vert u \vert ^{2} - c_{\delta} a_{0} \vert u \vert ^{v}\quad\mbox{for all } (x,u) \in R^{N} \times R. \end{aligned}$$

(3.4)

From (2.3), (3.3) and the Hölder inequality, for $u \in Z_{k}$, $v,p \in [1,2)$, $\rho> \rho_{0}$, we have

$$\begin{aligned}[b] \varphi_{\rho,\lambda} (u) &= \frac{1}{2} \Vert u \Vert _{\rho}^{2} - \lambda\biggl( \int_{R^{N}} F(x,u)\,dx + \frac{1}{p} \int_{R^{N}} h(x) \vert u \vert ^{p}\,dx \biggr) \\&\ge\frac{1}{2} \Vert u \Vert _{\rho}^{2} - \lambda \delta \int_{R^{N}} \vert u \vert ^{2}\,dx - \lambda c_{\delta} a_{0} \Vert u \Vert _{v}^{v} - \lambda \Vert h \Vert _{\frac{2}{2 - p}} \Vert u \Vert _{2}^{p} \\&\ge\frac{1}{2} \Vert u \Vert _{\rho}^{2} - \lambda d_{2}\delta \Vert u \Vert _{\rho}^{2} - \lambda c_{\delta} a_{0} \Vert u \Vert _{v}^{v} - \lambda \Vert h \Vert _{\frac{2}{2 - p}} \Vert u \Vert _{2}^{p}.\end{aligned} $$

(3.5)

Let

$$\alpha_{k} = \sup_{u \in Z_{k}, \Vert u \Vert _{\rho} = 1} \Vert u \Vert _{\sigma}\quad\mbox{for } \sigma\in[1,2_{\alpha}^{*}). $$

Since $X_{\rho} \mapsto L_{\mathrm{loc}}^{\sigma} (R^{N})$ is compact, by Lemma 3.8 of [41], we can deduce that $\alpha_{k} \to0$, as $k \to\infty$. Then, for $\lambda\in[1,2]$, it follows by (3.5) that

$$\varphi_{\rho,\lambda} (u) \ge\frac{1}{3} \Vert u \Vert _{\rho}^{2} - 2c_{\delta} a_{0} \alpha_{k}^{v} \Vert u \Vert _{\rho}^{v} - 2 \Vert h \Vert _{\frac{2}{2 - p}}\alpha_{k}^{p} \Vert u \Vert _{\rho}^{p}. $$

We denote

$$l_{k} = \max\bigl\{ 2c_{\delta} a_{0} \alpha_{k}^{v},2 \Vert h \Vert _{\frac{2}{2 - p}} \alpha_{k}^{p}\bigr\} ,\qquad \Vert u \Vert _{\rho}^{\eta} = \max\bigl\{ \Vert u \Vert _{\rho}^{v}, \Vert u \Vert _{\rho}^{p} \bigr\} . $$

Obviously, $l_{k} > 0$, $l_{k} \to0^{ +}$, as $k \to\infty$ and $\eta\in [1,2)$.

Then

$$ \varphi_{\rho,\lambda} (u) \ge\frac{1}{3} \Vert u \Vert _{\rho}^{2} - 2l_{0} \Vert u \Vert _{\rho}^{\eta}. $$

(3.6)

Choose $\rho_{k} = (12l_{k})^{\frac{1}{2 - \eta}}$, then $\rho_{k} \to0^{ +}$, as $k \to\infty$, for $\eta\in[1,2)$.

By a direct calculation, for any $\lambda\in[1,2]$, we have

$$a_{k}(\lambda): = \inf_{u \in Z_{k}, \Vert u \Vert = \rho_{k}}\varphi_{\rho,\lambda} (u) \ge\frac{1}{6}\rho_{k}^{2} > 0. $$

Moreover, from (3.6), for any $u \in Z_{k}$, $\Vert u \Vert \le \rho_{k}$, $\lambda\in[1,2]$, one has

$$\varphi_{\rho,\lambda} (u) \ge- 2l_{k} \Vert u \Vert _{\rho}^{\eta} \ge - 2l_{k}\rho_{k}^{\eta}, $$

which shows

$$\inf_{u \in Z_{k}, \Vert u \Vert \le\rho_{k}}\varphi_{\rho,\lambda} (u) \ge- 2l_{k} \rho_{k}^{\eta} \to0^{ +},\quad \mbox{as }k \to\infty. $$

Hence, for $l_{k} \to0^{ +}$, $\rho_{k} \to0^{ +}$, as $k \to\infty$, we have

$$c_{k}(\lambda): = \inf_{u \in Z_{k}, \Vert u \Vert \le \rho_{k}}\varphi_{\rho,\lambda} (u) \to0,\quad\mbox{as } k \to\infty \mbox{ uniformly for } \lambda \in[1,2]. $$

Now we are in the position to verify $b_{k}(\lambda): = \max_{u \in Y_{k}, \Vert u \Vert = r_{k}}\varphi_{\rho,\lambda} (u) < 0$, $\forall \lambda\in[1,2]$. From (2.3), (3.4), for any $u \in Y_{k}$ with $\operatorname{dim}Y_{k} < \infty$, $\lambda\in[1,2]$, we have

$$\varphi_{\rho,\lambda} (u) \le\frac{1}{2} \Vert u \Vert _{\rho}^{2} - \lambda \int_{R^{N}} F(x,u)\,dx - \frac{\lambda}{p} \int_{R^{N}} h(x) \vert u \vert ^{p}\,dx. $$

By the equivalence of any norm in finite dimensional space $Y_{k}$, one has

$$\begin{aligned} \varphi_{\rho,\lambda} (u) &\le\frac{1}{2} \Vert u \Vert _{\rho}^{2} - e_{1} \Vert u \Vert _{\rho}^{\sigma} + e_{2} \Vert u \Vert _{\rho}^{2} + e_{3} \Vert u \Vert _{\rho}^{v} \\&= \Vert u \Vert _{\rho}^{\sigma} \bigl[(1 - e_{2}) \Vert u \Vert _{\rho}^{2 - \sigma} + e_{3} \Vert u \Vert _{\rho}^{v - \sigma} - e_{1}d_{\sigma} \bigr],\end{aligned} $$

where $e_{1},e_{2},e_{3} > 0$ are positive constants.

Then, for $v \in[1,2)$, $\sigma\in[1,v)$, by choosing suitable $0 < \Vert u \Vert = r_{k} < \rho_{k}$, we can get

$$b_{k}(\lambda): = \max_{u \in Y_{k}, \Vert u \Vert = r_{k}}\varphi_{\rho,\lambda} (u) < 0,\quad \forall\lambda\in[1,2]. $$

We complete the proof of Lemma 3.1. □

Lemma 3.2

Suppose that ($V_{1}$)–($V_{2}$), ($f_{1}$), ($H_{1}$)–($H_{3}$) hold. Then, for $\rho\ge\rho_{0}$, $\lambda_{n} \in[1,2]$, $\lambda_{n} \to1$, $u(\lambda_{n}) \in Y_{n}$ with

$$\varphi'_{\rho,\lambda_{n}}\big\vert _{Y_{n}} \bigl(u( \lambda_{n})\bigr) = 0\quad\textit{and} \quad\varphi_{\rho,\lambda_{n}}\bigl(u( \lambda_{n})\bigr) \to \omega_{k} \in\bigl[c_{k}(2),b_{k}(1) \bigr],\quad\textit{as } n \to\infty, $$

$\{ u(\lambda_{n})\}$ has a convergent subsequence in $X_{\rho}$ for every $k \in N$.

Proof

Assume that, for each $k \in N$, $\lambda\in [1,2]$, there exist a subsequence $\lambda_{n} \to1$ and $u(\lambda_{n}) \in Y_{n}$ such that $\varphi'_{\rho,\lambda_{n}} \vert _{Y_{n}} (u(\lambda_{n})) = 0$, and $\varphi_{\rho,\lambda_{n}}(u(\lambda_{n})) \to\omega_{k} \in [c_{k}(2),b_{k}(1)]$, as $n \to\infty$.

From (2.2), (3.3), ($H_{3}$) and the Hölder inequality, for $\rho \ge \rho_{0}$, $\lambda_{n} \in[1,2]$, $\lambda_{n} \to1$ as $n \to \infty$ and $v,p \in[1,2)$, one has

$$\begin{aligned} \bigl\Vert u(\lambda_{n}) \bigr\Vert _{\rho}^{2} &= 2\varphi_{\rho,\lambda_{n}}\bigl(u(\lambda_{n})\bigr) + 2 \lambda_{n}\biggl[ \int_{R^{N}} F\bigl(x,u(\lambda_{n})\bigr)\,dx + \frac{1}{p} \int_{R^{N}} h(x) \bigl\vert u(\lambda_{n}) \bigr\vert ^{p}\,dx \biggr] \\&\le2(\omega_{k} + 1) + 2\lambda_{n}\biggl[ \int_{R^{N}} \bigl(\delta \bigl\vert u(\lambda_{n}) \bigr\vert ^{2} + c_{\delta} a_{0} \bigl\vert u( \lambda_{n}) \bigr\vert ^{v}\bigr)\,dx + \Vert h \Vert _{\frac{2}{2 - p}}d_{2}^{p/2} \bigl\Vert u( \lambda_{n}) \bigr\Vert _{\rho}^{p} \biggr] \\&\le2(\omega_{k} + 1) + \frac{1}{3} \bigl\Vert u( \lambda_{n}) \bigr\Vert _{\rho}^{2} + 4c_{\delta} a_{0}d_{v} \bigl\Vert u( \lambda_{n}) \bigr\Vert _{\rho}^{v} + 4 \Vert h \Vert _{\frac{2}{2 - p}}d_{2}^{p/2} \bigl\Vert u( \lambda_{n}) \bigr\Vert _{\rho}^{p},\end{aligned} $$

which shows that $\{ u(\lambda_{n})\}$ is bounded in $X_{\rho}$. Then we can obtain a weakly convergent subsequence of $\{ u(\lambda_{n})\}$. Assume $u(\lambda_{n}) \rightharpoonup u$ weakly in $X_{\rho}$. By Lemma 2.1, we know $u(\lambda_{n}) \to u$ strongly in $L^{q}(R^{N})$ for $q \in[2,2_{\alpha}^{*})$, which implies $\Vert u(\lambda_{n}) - u \Vert _{q} \to0$, as $n \to\infty$, for $q \in [2,2_{\alpha}^{*})$.

Next we prove that $u_{n} \to u$ in $X_{\rho}$. By (3.1)–(3.2), we have

$$\begin{aligned}[b] \bigl\Vert u(\lambda_{n}) - u \bigr\Vert _{\rho}^{2} \le{}& \bigl\langle \varphi '_{\rho,\lambda_{n}}\bigl(u( \lambda_{n})\bigr) - \varphi'_{\rho,1}(u),u( \lambda_{n}) - u \bigr\rangle \\&+ \int_{R^{N}} \bigl\vert \lambda_{n}f\bigl(x,u( \lambda_{n})\bigr) - f\bigl(x,u(x)\bigr) \bigr\vert \bigl\vert u( \lambda_{n}) - u \bigr\vert \,dx \\& + \int_{R^{N}} h(x) \bigl\vert \lambda_{n}u( \lambda_{n}) - u \bigr\vert ^{p - 1} \bigl\vert u( \lambda_{n}) - u \bigr\vert \,dx.\end{aligned} $$

(3.7)

Obviously,

$$ \bigl\langle \varphi'_{\rho,\lambda_{n}}\bigl(u(\lambda_{n}) \bigr) - \varphi '_{\rho,1}( u), u( \lambda_{n}) - u \bigr\rangle \to0,\quad n \to \infty. $$

(3.8)

From the condition $h \in L^{\frac{2}{2 - p}}(R^{N})$, $\lambda_{n} \to1$, $u(\lambda_{n}) \to u$ in $L^{q}(R^{N})$, as $n \to\infty$ and the Hölder inequality, we have

$$ \int_{R^{N}} h(x) \bigl\vert \lambda_{n}u ( \lambda_{n}) - u \bigr\vert ^{p - 1} \bigl\vert u ( \lambda_{n}) - u \bigr\vert \,dx \le \Vert h \Vert _{\frac {2}{2 - p}} \bigl\Vert u (\lambda_{n}) - u \bigr\Vert _{2}^{p} \to0,\quad n \to \infty. $$

(3.9)

From ($H_{1}$) and the Hölder inequality, we can get

$$\begin{aligned}[b] &\int_{R^{N}} \bigl[\lambda_{n}f\bigl(x,u( \lambda_{n})\bigr) - f(x,u)\bigr]\bigl(u(\lambda_{n}) - u \bigr)\,dx\\ &\quad \le \int_{R^{N}} \bigl( \bigl\vert 2f\bigl(x,u(\lambda_{n}) \bigr) \bigr\vert + \bigl\vert f(x,u) \bigr\vert \bigr) \bigl\vert u( \lambda_{n}) - u \bigr\vert \,dx \\&\quad\le \int_{R^{N}} a_{0}\bigl[\bigl( \vert u \vert + \bigl\vert 2u(\lambda_{n}) \bigr\vert \bigr) + \bigl( \vert u \vert ^{v - 1} + 2 \bigl\vert u(\lambda_{n}) \bigr\vert ^{v - 1}\bigr)\bigr] \bigl\vert u(\lambda_{n}) - u \bigr\vert \,dx \\&\quad\le3a_{0}\bigl( \Vert u \Vert _{2} \bigl\Vert u( \lambda_{n}) - u \bigr\Vert _{2} + \Vert u \Vert _{v}^{v - 1} \bigl\Vert u(\lambda_{n}) - u \bigr\Vert _{v}\bigr) \\ &\quad\to0,\quad n \to\infty.\end{aligned} $$

(3.10)

It follows from (3.7)–(3.10) that $\Vert u_{n} - u \Vert _{\rho}^{2} \to0$, which implies $u_{n} \to u$ in $X_{\rho}$. □

When the nonlinearity f is asymptotically linear, we have Theorem 3.1.

Theorem 3.1

Suppose that ($V_{1}$)–($V_{2}$), ($f_{1}$), ($H_{1}$)–($H_{3}$) hold, and $F(x, - u) = F(x,u)$ for all $(x,u) \in R^{N} \times R$, then, for $\rho> \rho_{0}$, (1.1) possesses infinitely many small energy solutions $u_{\rho}^{(k)} \in X_{\rho}$ for any $k \in N$, that is,

$$\frac{1}{2} \bigl\Vert u_{\rho}^{(k)} \bigr\Vert _{{\rho}}^{2} - \int_{R^{N}} F\bigl(x,u_{\rho}^{(k)}\bigr)\,dx - \frac{1}{p} \int_{R^{N}} h(x) \bigl\vert u_{\rho}^{(k)} \bigr\vert ^{p}\,dx \to0^{ -} ,\quad \textit{as }k \to\infty. $$

Proof

From ($f_{1}$) and ($H_{3}$), we have $B(u) \ge0$ and

$$B(u) = \int_{R^{N}} F(x,u)\,dx + \frac{1}{p} \int_{R^{N}} h(x) \vert u \vert ^{p}\,dx \ge \frac{1}{p} \int_{R^{N}} h(x) \vert u \vert ^{p}\, dx \to \infty,\quad\mbox{as } \Vert u \Vert \to\infty, $$

on any finite dimensional subspace of X, which shows that ($A_{2}$) of Lemma 2.3 holds.

It is easy to check that ($A_{1}$) of Lemma 2.3 holds. Lemma 3.1 implies that ($A_{3}$) is satisfied. Owing to Lemma 2.3, we know that, for each $k \in N$, there exist $\lambda_{n} \to1$, $u(\lambda_{n}) \in Y_{n}$ such that

$$\varphi'_{\rho,\lambda_{n}} \big\vert _{Y_{n}} \bigl(u( \lambda_{n})\bigr) = 0\quad\mbox{and}\quad \varphi_{\rho,\lambda_{n}}\bigl(u( \lambda_{n})\bigr) \to \omega_{k} \in\bigl[c_{k}(2),b_{k}(1) \bigr],\quad\mbox{as } n \to\infty. $$

By Lemma 3.2, we know $\{ u(\lambda_{n})\}$ has a convergent subsequence in $X_{\rho}$. In view of Lemma 2.3, $\varphi_{\rho, 1}$ has infinitely many nontrivial critical points $u_{\rho}^{(k)} \in X_{\rho} \setminus\{ 0\}$ satisfying

$$\frac{1}{2} \bigl\Vert u_{\rho}^{(k)} \bigr\Vert _{{\rho}}^{2} - \int_{R^{N}} F\bigl(x,u_{\rho}^{(k)}\bigr)\,dx - \frac{1}{p} \int_{R^{N}} h(x) \bigl\vert u_{\rho}^{(k)} \bigr\vert ^{p}\,dx \to0^{ -} , \quad\mbox{as }k \to\infty $$

for $\rho> \rho_{0}$, every $k \in N$, which implies (1.1) possesses infinitely many small energy solutions. □

Lemma 3.3

Assume ($V_{1}$)–($V_{2}$), ($f_{1}$), ($H_{2}$), ($H_{4}$)–($H_{5}$) hold. Then, for $\rho\ge\rho_{0}$, there exist $\rho_{k} > r_{k} > 0$ such that

$$e_{k}(\lambda) = \inf_{u \in Z_{k}, \Vert u \Vert = r_{k}}\varphi_{\rho,\lambda} (u) > f_{k}(\lambda) = \max_{u \in Y_{k}, \Vert u \Vert = \rho_{k}} \varphi_{\rho,\lambda} (u), \quad\forall \lambda\in[1,2]. $$

Proof

($H_{2}$) and ($H_{4}$) imply that, for arbitrary $\varepsilon> 0$ with $d_{2}\varepsilon< \frac{1}{12}$, there exists a positive constant $c_{\varepsilon}$ depending on ε such that

$$ F(x,u) \le\varepsilon \vert u \vert ^{2} + c_{\varepsilon} a_{1} \vert u \vert ^{q}\quad\mbox{for all } (x,u) \in R^{N} \times R. $$

(3.11)

By ($H_{5}$), we know that, for any $\theta> 0$ large enough, there exists a constant $\varsigma> 0$ such that, for $(x,u) \in R^{N} \times R$ and $\vert u \vert > \varsigma$,

$$ F(x,u) \ge\theta \vert u \vert ^{2}. $$

(3.12)

Combining (3.12) and ($H_{2}$), we see that there exists $S_{0} > 0$ such that

$$ F(x,u) \ge\theta' \vert u \vert ^{2}\quad\mbox{for all } (x,u) \in R^{N} \times R, $$

(3.13)

where $\theta' = \theta- S_{0} - \theta\varsigma$.

Choosing a suitable θ, we can get

$$ \theta'd_{2} > \frac{2}{3}. $$

(3.14)

From (2.3), (3.11) and the Hölder inequality, for any $\rho\ge \rho_{0}$, $u \in Z_{k}$, $q \in[2,2_{\alpha}^{*})$, we have

$$\begin{aligned} \varphi_{\rho,\lambda} (u) &= \frac{1}{2} \Vert u \Vert _{\rho}^{2} - \lambda \int_{R^{N}} F(x,u)\,dx - \frac{\lambda}{p} \int_{R^{N}} h(x) \vert u \vert ^{p}\,dx \\&\ge\frac{1}{2} \Vert u \Vert _{\rho}^{2} - \lambda d_{2}\varepsilon \Vert u \Vert _{\rho}^{2} - \lambda c_{\varepsilon} a_{1} \Vert u \Vert _{q}^{q} - \lambda \Vert h \Vert _{\frac{2}{2 - p}} \Vert u \Vert _{2}^{p} \\&\ge\biggl(\frac{1}{2} - \lambda d_{2}\varepsilon\biggr) \Vert u \Vert _{\rho}^{2} - \lambda c_{\varepsilon} a_{1}\alpha_{k}^{q} \Vert u \Vert _{\rho}^{q} - \lambda \Vert h \Vert _{\frac{2}{2 - p}} \alpha_{k}^{p} \Vert u \Vert _{\rho}^{p},\end{aligned} $$

where $\alpha_{k}$ is defined in the proof Lemma 3.1 and $\alpha_{k} \to 0$, as $n \to\infty$.

Let

$$M' = 2\alpha_{k}^{p}\bigl(c_{\varepsilon} a_{1}\alpha_{k}^{q - p} + \Vert h \Vert _{\frac{2}{2 - p}}\bigr). $$

We choose $r_{k} = (6M')^{\frac{1}{2 - q}}$, then $r_{k} \to\infty$, as $k \to\infty$, for $q \in[2,2_{\alpha}^{*})$, which implies there exists a positive constant $k_{0} \in N$ such that $r_{k} = (6M')^{\frac{1}{2 - q}} > 1$, for $k \ge k_{0}$, $k \in N$.

Then, for $\lambda\in[1,2]$, $k \ge k_{0}$, $k \in N$, $u \in Z_{k}$, $\Vert u \Vert _{\rho} = r_{k} = (6M')^{\frac{1}{2 - q}} > 1$, we have

$$\begin{aligned}[b] e_{k}(\lambda)& = \inf_{u \in Z_{k}, \Vert u \Vert = r_{k}}\varphi_{\rho,\lambda} (u) \ge\frac{1}{3} \Vert u \Vert _{\rho}^{2} - 2c_{\varepsilon} a_{1}\alpha_{k}^{q} \Vert u \Vert _{\rho}^{q} - 2 \Vert h \Vert _{\frac{2}{2 - p}} \alpha_{k}^{p} \Vert u \Vert _{\rho}^{p} \\&\ge\frac{1}{3} \Vert u \Vert _{\rho}^{2} - M' \Vert u \Vert _{\rho}^{q} = \frac{1}{6}r_{k}^{2} = \frac{1}{6} \bigl(6M'\bigr)^{\frac{2}{2 - q}} > 0. \end{aligned} $$

(3.15)

Next we prove $f_{k}(\lambda) = \max_{u \in Y_{k}, \Vert u \Vert = \rho_{k}}\varphi_{\rho,\lambda} (u) < 0$, $\forall\lambda\in[1,2]$.

From (2.3), (3.13)–(3.14) and the equivalence of the norms in the finite dimensional space $Y_{k}$, for $\lambda\in[1,2]$ and any $u \in Y_{k}$ with $\operatorname{dim}Y_{k} < \infty$, we have

$$\begin{aligned} \varphi_{\rho,\lambda} (u) &= \frac{1}{2} \Vert u \Vert _{\rho}^{2} - \lambda \int_{R^{N}} F(x,u)\,dx - \frac{\lambda}{p} \int_{R^{N}} h(x) \vert u \vert ^{p}\,dx \\&\le\frac{1}{2} \Vert u \Vert _{\rho}^{2} - \theta'd_{2} \Vert u \Vert _{\rho}^{2} \\&< - \frac{1}{6} \Vert u \Vert _{\rho}^{2}.\end{aligned} $$

Then, for any $\Vert u \Vert = \rho_{k} > r_{k} > 0$, one has

$$ f_{k}(\lambda) = \max_{u \in Y_{k}, \Vert u \Vert = \rho_{k}}\varphi_{\rho,\lambda} (u) < 0, \quad\forall\lambda\in[1,2]. $$

(3.16)

From (3.15)–(3.16), we obtain

$$e_{k}(\lambda) = \inf_{u \in Z_{k}, \Vert u \Vert = r_{k}}\varphi_{\rho,\lambda} (u) > 0 > f_{k}(\lambda) = \max_{u \in Y_{k}, \Vert u \Vert = \rho_{k}} \varphi_{\rho,\lambda} (u), \quad\forall \lambda\in[1,2]. $$

 □

Theorem 3.2 is about the existence of high energy solutions for (1.1) when the nonlinearity f is superlinear.

Theorem 3.2

Suppose ($V_{1}$)–($V_{2}$), ($f_{1}$), ($H_{2}$), ($H_{4}$)–($H_{5}$) hold, and $F(x, - u) = F(x,u)$ for all $(x,u) \in R^{N} \times R$, then, for $\rho> \rho_{0}$, (1.1) possesses infinitely many high energy solutions $u_{\rho}^{(k')} \in X_{\rho}$ for any $k' \ge k_{0}$, $k' \in N$, in the sense that

$$\frac{1}{2} \bigl\Vert u_{\rho}^{(k')} \bigr\Vert _{\rho}^{2} - \int_{R^{N}} F\bigl(x,u_{\rho}^{(k')}\bigr)\,dx - \frac{1}{p} \int_{R^{N}} h(x) \bigl\vert u_{\rho}^{(k')} \bigr\vert ^{p}\,dx \to+ \infty, \quad\textit{as }k' \to\infty. $$

Proof

From ($H_{5}$) and the definitions of $A(u)$ and $B(u)$, we know $B(u) \ge0$ for $u \in X$ and $A(u) \to\infty$ as $\Vert u \Vert \to\infty$, which implies ($A_{2}$) of Lemma 2.4 holds.

From ($f_{1}$), (3.1) and $F(x, - u) = F(x,u)$, it is easy to see that $\varphi_{\rho,\lambda} (u)$ maps a bounded set into bounded sets uniformly for $\lambda\in[1,2]$, and

$$\varphi_{\rho,\lambda} ( - u) = \varphi_{\rho,\lambda} (u)\quad\mbox{for all } (\lambda,u) \in[1,2] \times X, $$

which shows ($A_{1}$) of Lemma 2.4 is satisfied. Lemma 3.3 implies ($A_{3}$) holds. From Lemma 2.4, for almost any $\lambda\in[1,2]$, there exists a sequence $\{ u_{n}^{k'}(\lambda)\}_{n = 1}^{\infty}$ for $k' \ge k_{0}$, $k' \in N$, such that

$$\begin{gathered} \sup_{n} \bigl\Vert u_{n}^{k'}( \lambda) \bigr\Vert _{\rho} < \infty,\\ \varphi'_{\rho,\lambda} \bigl(u_{n}^{k'}(\lambda)\bigr) \to0\quad\mbox{and}\quad \varphi_{\rho,\lambda} \bigl(u_{n}^{k'}(\lambda)\bigr) \to g_{k}(\lambda),\quad \mbox{as } n \to\infty. \end{gathered} $$

(3.17)

From Lemma 2.4, we also have $g_{k'}(\lambda) = \inf_{\gamma\in \Gamma_{k'}}\max_{u \in B_{k'}}\varphi_{\rho,\lambda} (\gamma(u)) \ge e_{k'}(\lambda)$.

Let $\beta_{k'} = \frac{1}{6}(6M')^{\frac{2}{2 - q}} > 0$, then $\beta_{k'} \to\infty$, as $k' \to\infty$. For $k' \ge k_{0}$, $k' \in N$, it follows, see (3.15), that $g_{k'}(\lambda) \ge e_{k'}(\lambda) \ge\beta_{k'}$. Then

$$ g_{k'}(\lambda) \in\bigl[ \beta_{k'}, \beta'_{k'} \bigr], $$

(3.18)

where $\beta_{k}^{\prime} = \max_{u \in B_{k}}\varphi_{\rho,\lambda} (\gamma(u))$, $\Gamma_{k'} = \{ \gamma\in C(B_{k'},X_{\rho} ):\gamma \text{ is odd}, \gamma \vert _{\partial B_{k'}} = \mathit{id}\}$ ($k' \ge 2$) with

$$B_{k'} = \bigl\{ u \in Y_{k'}: \Vert u \Vert \le \rho_{k'}\bigr\} . $$

Choose $\lambda_{m} \to1$ as $m \to\infty$, for $\lambda_{m} \in[1,2]$. Owing to (3.17), we can get the boundedness of $\{ u_{n}^{k'}(\lambda_{m})\}$, which implies $\{ u_{n}^{k'}(\lambda_{m})\}$ has a weakly convergent subsequence. Similar to the proof of Lemma 3.2, we can prove that $\{ u_{n}^{k'}(\lambda_{m})\}$ has a strong convergent subsequence in $X_{\rho}$ as $n \to\infty$. Assume $\lim_{n \to\infty} u_{n}^{k'}(\lambda_{m}) = u^{k'}(\lambda_{m})$ for $m \in N$, then it follows, see (3.17)–(3.18), that, for $k' \ge k_{0}$, $k' \in N$,

$$ \varphi'_{\rho,\lambda_{m}}\bigl(u^{k'}( \lambda_{m})\bigr) = 0,\qquad \varphi_{\rho,\lambda_{m}} \bigl(u^{k'}(\lambda_{m})\bigr) \in\bigl[ \beta_{k'}, \beta '_{k'} \bigr]. $$

(3.19)

Next we show $\{ u^{k'}(\lambda_{m})\}$ is bounded in $X_{\rho}$. We argue by contradiction. In fact, if it is not the case, then

$$ \bigl\Vert u^{k'}(\lambda_{m}) \bigr\Vert _{\rho} \to\infty,\quad\mbox{as } m \to\infty. $$

(3.20)

Let $\varpi_{m} = \frac{u^{k'}(\lambda_{m})}{ \Vert u^{k'}(\lambda_{m}) \Vert _{\rho}}$, which shows $\Vert \varpi_{m} \Vert _{\rho} = 1$. Then $\varpi_{m}$ has a weakly convergent subsequence in $X_{\rho}$. By Lemma 2.1, we can see that $\varpi_{m}$ has a strong convergent subsequence in $L^{q}(R^{N})$, for $q \in[2,2_{\alpha}^{*}]$. Without loss of generality, we suppose $\varpi_{m} \to\varpi_{0}$ strongly a.e. in $L^{q}(R^{N})$, which also implies $\varpi_{m}(x) \to\varpi_{0}(x)$ for a.e. $x \in R^{N}$.

For $p \in[1,2)$, from ($f_{1}$), (3.20) and the Hölder inequality, we have

$$ 0 \le\lim_{m \to\infty} \frac{\int_{R^{N}} h(x) \vert u^{k'}(\lambda_{m}) \vert ^{p}\,dx}{ \Vert u^{k'}(\lambda_{m}) \Vert _{\rho}^{2}} \le \lim _{m \to\infty} \frac{ \Vert h \Vert _{\frac{2}{2 - p}}d_{2}^{p/2} \Vert u(\lambda_{m}) \Vert _{\rho}^{p}}{ \Vert u^{k'}(\lambda_{m}) \Vert _{\rho}^{2}} = 0. $$

(3.21)

Let $\mathrm{A} = \{ x \in R^{n}:\varpi_{0}(x) \ne0\}$. It’s easy to see that A is nonempty. For $x \in\mathrm{A}$, $\lambda_{m} \in [1,2]$, combining (3.1), ($H_{5}$), (3.19)–(3.21) and Fatou’s lemma, we can get

$$\begin{aligned} 0 &\leftarrow\frac{\varphi_{\rho,\lambda_{m}}(u^{k'}(\lambda _{m}))}{ \Vert u^{k'}(\lambda_{m}) \Vert _{\rho}^{2}} \\&= \frac{\frac{1}{2} \Vert u^{k'}(\lambda_{m}) \Vert _{\rho}^{2} - \lambda_{m}[\int_{R^{N}} F(x,u^{k'}(\lambda_{m}))\,dx + \frac{1}{p}\int_{R^{N}} h(x) \vert u^{k'}(\lambda_{m}) \vert ^{p}\,dx ]}{ \Vert u^{k'}(\lambda_{m}) \Vert _{\rho}^{2}} \\&\le\frac{1}{2} - \biggl[ \int_{\mathrm{A}} \frac{F(x,u^{k'}(\lambda_{m}))}{ \Vert u^{k'}(\lambda_{m}) \Vert _{\rho}^{2}}\,dx + \frac{1}{p} \int_{R^{N}} \frac{h(x) \vert u^{k'}(\lambda_{m}) \vert ^{p} }{ \Vert u^{k'}(\lambda_{m}) \Vert _{\rho}^{2}}\,dx \biggr] \\&= \frac{1}{2} - \int_{\mathrm{A}} \bigl\vert \varpi_{m}(x) \bigr\vert ^{2}\frac{F(x,u^{k'}(\lambda_{m}))}{ \vert u^{k'}(\lambda_{m}) \vert ^{2}}\,dx - \frac{1}{p} \int_{R^{N}} \frac{h(x) \vert u^{k'}(\lambda_{m}) \vert ^{p} }{ \Vert u^{k'}(\lambda_{m}) \Vert _{\rho}^{2}}\,dx \\&\to- \infty,\quad\mbox{as } m \to\infty.\end{aligned} $$

This is a contradiction. Hence $\{ u^{k'}(\lambda_{m})\}$ is bounded in $X_{\rho}$, which shows $\{ u^{k'}(\lambda_{m})\}$ has a weakly convergent subsequence. By a similar proof to that of Lemma 3.2, we know that $\{ u^{k'}(\lambda_{m})\}$ has a strongly convergent subsequence in $X_{\rho}$. Suppose

$$\lim_{m \to\infty} u^{k'}(\lambda_{m}) = u^{k'}(1) = u_{\rho}^{k'} \in X_{\rho}. $$

Then, for $k' \ge k_{0}$, $k' \in N$, from (3.19) and $\beta_{k'} \to \infty$, as $k' \to\infty$, we have

$$\varphi'_{\rho,1}\bigl(u_{\rho}^{k'}\bigr) = 0,\qquad \varphi_{\rho,1}\bigl(u_{\rho}^{k'}\bigr) \in \bigl[\beta_{k'},\beta_{k'}^{\prime} \bigr] \to \infty,\quad\mbox{as } k' \to\infty, $$

which shows $u_{\rho}^{k'}$ is a nontrivial critical point of $\varphi_{\rho,1}$. Consequently, for $k' \ge k_{0}$, $k' \in N$ is arbitrary, we obtain infinitely many nontrivial critical points $u_{\rho}^{(k')}$ of $\varphi_{\rho,1}$, which are also the nontrivial solutions of (1.1) with high energy, that is,

$$\frac{1}{2} \bigl\Vert u_{\rho}^{(k')} \bigr\Vert _{\rho}^{2} - \int_{R^{N}} F\bigl(x,u_{\rho}^{(k')}\bigr)\,dx - \frac{1}{p} \int_{R^{N}} h(x) \bigl\vert u_{\rho}^{(k')} \bigr\vert ^{p}\,dx \to+ \infty, \quad\mbox{as }k' \to\infty. $$

Next, we will explore the concentration phenomenon of (1.1).

We denote

$$ X_{0} = \biggl\{ u \in L^{2}(\Omega): \int \int_{Q} \frac{ \vert u(x) - u(y) \vert ^{2}}{ \vert x - y \vert ^{N + 2\alpha}} \,dx\,dy < \infty, u(x) = 0\mbox{ if }x \in\Omega^{c} \biggr\} , $$

(3.22)

where $Q = R^{2N}\setminus(\Omega^{c} \times\Omega^{c})$, $\Omega^{c} = R^{N}\setminus\Omega$, Ω is defined in ($V_{3}$). The function space $X_{0}$ was first defined by Servadei and Valdinoci in [39]. Obviously, $C_{0}^{\infty} (\Omega) \subset X_{0}$ and $X_{0}$ is a Hilbert space with the scalar product and normal

$$\langle u,v \rangle= \int \int_{Q} \frac{(u(x) - u(y))(v(x) - v(y))}{ \vert x - y \vert ^{N + 2\alpha}} \,dx\,dy,\qquad \Vert u \Vert ^{2} = \langle u,u \rangle. $$

 □

Theorem 3.3

Assume that ($V_{1}$)–($V_{3}$), ($f_{1}$), ($H_{1}$)–($H_{3}$) hold, and $F(x, - u) = F(x,u)$ for $(x,u) \in R^{N} \times R$. Let $u_{\rho}^{(k)}$ ($k \in N$) be solutions obtained by Theorem 3.1, then $u_{\rho}^{(k)} \to u_{0}^{(k)}$ strongly in $L^{q}(R^{N})$ as $\rho\to \infty$, for every $k \in N$, $q \in[2,2_{\alpha}^{*})$, where $u_{0}^{(k)} \in X_{0}$, for every $k \in N$, are the nontrivial solutions of the following fractional Schrödinger equation:

$$ \left \{ \textstyle\begin{array}{l} ( - \Delta)^{\alpha} u = f(x,u) + h(x) \vert u \vert ^{p - 2}u\quad \textit{in }\Omega, \\ u = 0 \quad \textit{in }R^{N}\setminus\Omega. \end{array}\displaystyle \right . $$

(3.23)

Proof

Let $u_{\rho_{n}}^{(k)}$ ($k \in N$) be the nontrivial solutions obtained by Theorem 3.1, then $u_{\rho_{n}}^{(k)}$ are the nontrivial critical points of the functional

$$\varphi_{\rho_{n},1}(u) = \frac{1}{2} \Vert u \Vert _{{{\rho_{n}}}}^{2} - \int_{R^{N}} F(x,u)\,dx - \frac{1}{p} \int_{R^{N}} h(x) \vert u \vert ^{p}\,dx $$

and $\{ u_{\rho_{n}}^{(k)}\}_{k = 1}^{\infty} \in X\setminus\{ 0\}$ satisfying $\varphi_{\rho,1}(u_{\rho_{n}}^{(k)}) \to0^{ -}$ as $k \to \infty$. From the proof of Lemma 3.2, we know the $u_{\rho_{n}}^{(k)}$ are bounded in $X_{\rho_{n}}$ for any $k \in N$. Then $\{ u_{\rho _{n}}^{(k)}\}$ has a weakly convergent subsequence in $X_{\rho}$ as $\rho_{n} \to \infty$. Assume $u_{\rho_{n}}^{(k)} \rightharpoonup u_{0}^{(k)}$, as $\rho_{n} \to\infty$. From Lemma 2.1, we know that $u_{\rho_{n}}^{(k)} \to u_{0}^{(k)}$ strongly in $L_{\mathrm{loc}}^{q}(R^{N})$ for $q \in [2,2_{\alpha}^{*}]$, as $\rho_{n} \to\infty$. By Lemma 2.5, we can verify that, as $\rho_{n} \to\infty$, $u_{\rho_{n}}^{(k)} \to u_{0}^{(k)}$ strongly in $L^{q}(R^{N})$ for $q \in[2,2_{\alpha}^{*}]$. And by a standard argument, we can prove $u_{\rho_{n}}^{(k)} \to u_{0}^{(k)}$ in $X_{\rho_{n}}$, as $\rho_{n} \to\infty$.

Combining the boundedness of $u_{\rho_{n}}^{(k)}$, the definition of $\Vert \cdot \Vert _{\rho_{n}}$ and Fatou’s lemma, for every $k \in N$, one can obtain

$$\int_{R^{N}} V(x) \bigl(u_{0}^{(k)} \bigr)^{2}\,dx \le\lim\inf_{\rho_{n} \to \infty} \int_{R^{N}} V(x) \bigl(u_{\rho_{n}}^{(k)} \bigr)^{2}\,dx \le\lim\inf_{\rho_{n} \to \infty} \frac{ \Vert u_{\rho_{n}}^{(k)} \Vert _{\rho_{n}}^{2}}{\rho_{n}} = 0. $$

Then we know that $u_{0}^{(k)} = 0$ a.e. in the set $D = \{ x \in R^{N}:V(x) \ne0\}$. It follows (3.22) that $u_{0}^{(k)} \in X_{0}$, for any $k \in N$. Hence, we have $\int_{R^{N}} V(x)u_{\rho _{n}}^{(k)}(x)v(x)\,dx = 0$, for any $v \in X_{0}$. Together with (3.2), $\langle\varphi '_{\rho_{n},1}(u_{\rho_{n}}^{(k)}),v \rangle= 0$, for any $k \in N$, $v \in X_{0}$, we have

$$\begin{gathered} \int_{R^{N}} \int_{R^{N}} \frac{[u_{\rho_{n}}^{(k)}(x) - u_{\rho_{n}}^{(k)}(z)][v(x) - v(z)]}{ \vert x - z \vert ^{N + 2\alpha}} \,dx\,dz\\\quad - \int_{R^{N}} f\bigl(x,u_{\rho_{n}}^{(k)}\bigr)v\,dx - \int_{R^{N}} h(x) \bigl\vert u_{\rho_{n}}^{(k)} \bigr\vert ^{p - 2}u_{\rho_{n}}^{(k)}v\,dx = 0.\end{gathered} $$

By taking the limit, we can get

$$\begin{gathered} \int_{R^{N}} \int_{R^{N}} \frac{[u_{0}^{(k)}(x) - u_{0}^{(k)}(z)][v(x) - v(z)]}{ \vert x - z \vert ^{N + 2\alpha}} \,dx\,dz\\\quad - \int_{R^{N}} f\bigl(x,u_{0}^{(k)}\bigr)v\,dx - \int_{R^{N}} h(x) \bigl\vert u_{0}^{(k)} \bigr\vert ^{p - 2}u_{0}^{(k)}v\,dx = 0,\end{gathered} $$

which implies the $u_{0}^{(k)} \in X_{0}$ are nontrivial weak solutions of (3.23), for every $k \in N$. □

Theorem 3.4

Assume that ($V_{1}$)–($V_{3}$), ($f_{1}$), ($H_{2}$), ($H_{4}$)–($H_{5}$) hold, and $F(x, - u) = F(x,u)$ for $(x,u) \in R^{N} \times R$. Let $u_{\rho}^{(k')}$ ($k' \ge k_{0}$, $k' \in N$) be solutions obtained by Theorem 3.2, then $u_{\rho}^{(k')} \to u_{0}^{(k')}$ strongly in $L^{q}(R^{N})$ as $\rho\to\infty$, for all $k' \ge k_{0}$, $k' \in N$, $q \in[2,2_{\alpha}^{*})$, where $u_{0}^{(k')} \in X_{0}$ are the nontrivial solutions of (3.23).

Proof

Similarly to the proof of Theorem 3.3, we can prove Theorem 3.4. In fact, we only should make some corresponding modifications for the high energy solutions $u_{\rho}^{(k')}$. □

4 Examples

Now, we give several examples as applications of our main results.

Example 4.1

$$\begin{aligned}[b] ( - \Delta)^{\frac{1}{2}}u + \rho V(x)u ={}& (2 + \sin x) \vert u \vert ^{\frac{5}{4}}\bigl(1 + \vert u \vert ^{\frac{1}{2}}\bigr)^{ - \frac{3}{2}} \operatorname{sgn} u\\& + \sqrt{\frac{\ln(1.1 + \cos^{2}x)}{e^{\sqrt{e^{1 - \sin x}}} (1 + x^{2})}} \vert u \vert ^{\frac{3}{2} - 2}u,\quad x \in R^{2}.\end{aligned} $$

(4.1)

From (4.1), we can see that

$$\begin{gathered} \alpha= 1/2,\qquad 2\alpha= 1 < N = 2,\qquad p = \frac{3}{2},\\ f(x,u) = (2 + \sin x) \vert u \vert ^{\frac{5}{4}}\bigl(1 + \vert u \vert ^{\frac{1}{2}} \bigr)^{ - \frac{3}{2}}\operatorname{sgn} u \in C\bigl(R^{2} \times R,R \bigr)\end{gathered} $$

and

$$h(x) = \sqrt{\frac{\ln(1.1 + \cos^{2}x)}{e^{\sqrt{e^{1 - \sin x}}} (1 + x^{2})}}.$$

Let

$$V(x) = \left \{ \textstyle\begin{array}{l@{\quad}l} 0,& \vert x \vert \le1; \\ 2( \vert x \vert - 1),& 1 < \vert x \vert \le2; \\ \vert x \vert , &\vert x \vert > 2. \end{array}\displaystyle \right . $$

Then we can easily show ($V_{1}$)–($V_{2}$) are satisfied.

Obviously, $h(x) > 0$ and $h^{2}(x) = \frac{\ln(1.1 + \cos^{2}x)}{e^{\sqrt{e^{1 - \sin x}}} (1 + x^{2})} \le\frac{1}{1 + x^{2}}$, which shows $h(x) \in L^{2}(R)$. Hence ($f_{1}$) holds.

From (4.1), we have

$$\bigl\vert f(x,u) \bigr\vert = \bigl\vert (2 + \sin x) \vert u \vert ^{\frac{5}{4}}\bigl(1 + \vert u \vert ^{\frac{1}{2}}\bigr)^{ - \frac{3}{2}} \bigr\vert \le3\bigl(1 + \vert u \vert ^{\frac{3}{2} - 1}\bigr), $$

with $a_{0} = 3$, $v = \frac{3}{2}$, which shows that ($H_{1}$) is satisfied.

Also we can get

$$\lim_{ \vert u \vert \to0}\frac{f(x,u)}{ \vert u \vert } = \lim_{ \vert u \vert \to0} \frac{(2 + \sin x) \vert u \vert ^{\frac{1}{4}}\operatorname{sgn} u}{(1 + \vert u \vert ^{\frac{1}{2}})^{\frac{3}{2}}} = 0, $$

which implies that ($H_{2}$) holds.

We also find $F(x,u) = \int_{0}^{u} f(x,\tau) \,d\tau\ge0$. For $\sigma \in[1,v = \frac{3}{2})$, it follows by the L’Hospital rule that

$$\lim_{ \vert u \vert \to\infty} \frac{F(x,u)}{ \vert u \vert ^{\sigma}} = \lim_{ \vert u \vert \to\infty} \frac{\int_{0}^{u} f(x,\tau) \, d\tau}{ \vert u \vert ^{\sigma}} = \lim_{ \vert u \vert \to\infty} \frac{ \vert f(x,u) \vert }{\sigma \vert u \vert ^{\sigma- 1}} > \lim _{ \vert u \vert \to\infty} \frac{ \vert u \vert ^{\frac{3}{4}}}{2(1 + \vert u \vert ^{\frac{1}{2}})^{\frac{3}{2}}} = \frac{1}{2}. $$

Then ($H_{3}$) also holds.

We can also easily verify that $F(x, - u) = F(x,u)$. Then all the conditions of Theorem 3.1 hold. Consequently, (4.1) has infinitely many small energy solutions $u_{\rho}^{(k)} \in X_{\rho}$, for $\rho> \rho_{0}$ and any $k \in N$.

We can also see that $\Omega = \operatorname{{{int}}} V^{ - 1}(0)$ is a nonempty open set and the boundary is smooth, which shows ($V_{3}$) holds. Hence, by Theorem 3.3, we know that we have the following equation:

$$\left \{ \textstyle\begin{array}{l} ( - \Delta)^{\frac{1}{2}}u = (2 + \sin x) \vert u \vert ^{\frac{5}{4}}(1 + \vert u \vert ^{\frac{1}{2}})^{ - \frac{3}{2}}\operatorname{sgn} u + \sqrt{\frac{\ln(1.1 + \cos^{2}x)}{e^{\sqrt{e^{1 - \sin x}}} (1 + x^{2})}} \vert u \vert ^{\frac{3}{2} - 2}u\quad \mbox{in }\Omega, \\ u = 0\quad \mbox{in }R^{N}\setminus\Omega. \end{array}\displaystyle \right . $$

It possesses infinitely many nontrivial solutions $u_{0}^{(k)} \in X_{0}$ satisfying $u_{\rho}^{(k)} \to u_{0}^{(k)}$ strongly in $L^{q}(R^{N})$ as $\rho\to\infty$, for $q \in[2,2_{\alpha}^{*}) = [2,4)$ and every $k \in N$.

Example 4.2

$$ ( - \Delta)^{\frac{1}{2}}u + \rho Vu = \frac{8(\sin^{2}x)u^{5/3}}{9} + \frac{\ln(1.3 + \vert \sin x^{2} \vert )}{e^{ \vert x \vert }\sqrt {(1 + x^{2})}} \vert u \vert ^{\frac{3}{2} - 2}u,\quad x \in R^{2}. $$

(4.2)

Obviously, from (4.2), we have

$$\begin{gathered} \alpha= \frac{1}{2},\qquad N = 2,\qquad p = \frac{3}{2},\\ f(x,u) = \frac{8(\sin^{2}x)u^{5/3}}{9} \in C\bigl(R^{2} \times R,R\bigr)\end{gathered}$$

and

$$h(x) = \frac{\ln(1.3 + \vert \sin x^{2} \vert )}{e^{ \vert x \vert }\sqrt{(1 + x^{2})}}. $$

Then $0 < h(x) = \frac{\ln(1.3 + \vert \sin x^{2} \vert )}{e^{ \vert x \vert }\sqrt{(1 + x^{2})}} \in L^{2}(R)$, which implies ($f_{1}$) is satisfied.

From (4.2), we know $2\alpha = 1 < N = 2$, $F(x,u) = \frac{(\sin^{2}x)u^{8/3}}{3} \ge0$, and

$$\begin{gathered} \bigl\vert f(x,u) \bigr\vert = \biggl\vert \frac{8(\sin^{2}x)u^{5/3}}{9} \biggr\vert < \bigl(1 + \vert u \vert ^{\frac{8}{3} - 1}\bigr),\qquad\lim _{ \vert u \vert \to 0}\frac{ \vert f(x,u) \vert }{ \vert u \vert } = \lim_{ \vert u \vert \to0} \frac{8(\sin^{2}x)u^{2/3}}{9} = 0, \\\lim_{ \vert u \vert \to\infty} \frac{F(x,u)}{ \vert u \vert ^{2}} = \lim_{ \vert u \vert \to\infty} \frac{(\sin^{2}x)u^{2/3}}{3} = \infty,\end{gathered} $$

with $q = \frac{8}{3} \in[2,2_{\alpha}^{*} = 4)$, $a_{1} = 1$, which shows that ($H_{4}$)–($H_{5}$) of Theorem 3.2 hold.

Let $V(x)$ be the same function as in (4.1), then the conditions ($V_{1}$)–($V_{3}$) are all satisfied.

It is also easy to check $F(x, - u) = F(x,u)$. In view of Theorem 3.2, for $\rho> \rho_{0}$, $k' \ge k_{0}$, $k' \in N$, (4.2) has infinitely many high energy solutions $u_{\rho}^{(k')} \in X_{\rho}$.

Owing to Theorem 3.4, we know that we have the following fractional Schrödinger equation:

$$\left \{ \textstyle\begin{array}{l} ( - \Delta)^{\frac{1}{2}}u = \frac{8(\sin^{2}x)u^{5/3}}{9} + \frac{\ln(1.3 + \vert \sin x^{2} \vert )}{e^{ \vert x \vert }\sqrt{(1 + x^{2})}} \vert u \vert ^{\frac{3}{2} - 2}u\quad \mbox{in }\Omega, \\ u = 0\quad \mbox{in }R^{N}\setminus\Omega. \end{array}\displaystyle \right . $$

It has infinitely many nontrivial solutions $u_{0}^{(k')} \in X_{0}$, satisfying $u_{\rho}^{(k')} \to u_{0}^{(k')}$, strongly in $L^{q}(R^{N})$ as $\rho\to\infty$, for $q \in[2,2_{\alpha}^{*}) = [2,4)$ and $k' \ge k_{0}$, $k' \in N$.