Concentrating standing waves for the fractional Schrödinger equation with critical nonlinearities

Abstract

We study the following nonlocal Schrödinger equations:

$$\begin{aligned}& \varepsilon^{2s}(-\Delta)^{s}u+V(x)u=W(x)f(u), \end{aligned}$$

(I)

$$\begin{aligned}& \varepsilon^{2s}(-\Delta)^{s}u+V(x)u=W(x) \bigl(f(u)+u^{2^{*}_{s}-1}\bigr), \end{aligned}$$

(II)

for $u\in H^{s}( \mathbb{R}^{N})$, where $f(u)$ is superlinear and subcritical, $2^{*}_{s}= \frac{2N}{N-2s}$ if $N>2s$. $V(x)$ and $W(x)$ are sufficiently smooth potential with $\inf V(x)>0$, $\inf W(x)>0$, and $\varepsilon>0$ is a small number. Under proper assumptions, we explore the existence, concentration phenomenon, convergence, and decay estimate of semiclassical solutions of (I) and (II), respectively. Compared with some existing issues, the most interesting results obtained here are therefore: the concentration phenomenon depends on competing potential functions; the nonlocal critical problem (II) is considered; unlike the classical case $s=1$, the decay estimate of solution to (I) or (II) is of polynomial instead of exponential form, due to the nonlocal effect.

1 Introduction and overview on main results

This paper is devoted to the study of the concentration phenomenon for the fractional Schrödinger equations with subcritical nonlinearity,

$$ \varepsilon^{2s}(-\Delta)^{s}u+V(x)u=W(x)f(u), \quad x\in \mathbb{R}^{N} , $$

(1.1)

or critical nonlinearity,

$$ \varepsilon^{2s}(-\Delta)^{s}u+V(x)u=W(x) \bigl(f(u)+u^{2^{*}_{s}-1}\bigr), \quad x\in \mathbb{R}^{N} , $$

(1.2)

where $V(x)$ and $W(x)$ are sufficiently smooth potentials with $\inf V(x)>0$, $\inf W(x)>0$, and $\varepsilon>0$ is a small parameter, $2^{*}_{s}=\frac{2N}{N-2s}$ ($N>2s$), f is superlinear and has subcritical growth at infinity.

A basic motivation for the study of (1.1) or (1.2) comes from looking for standing waves of the type

$$\psi(x,t)=e^{-\frac{iEt}{\varepsilon}}u(x), $$

for the following fractional nonlinear Schrödinger equation:

$$ i\varepsilon\psi_{t}=\varepsilon^{2s}(-\Delta)^{s} \psi+\bigl(V(x)+E\bigr)\psi -W(x)f(\psi), \quad (t,x)\in\mathbb{R}_{+} \times \mathbb{R}^{N}, $$

(1.3)

where $(-\Delta )^{s}$ ($0< s<1$) denotes the usual fractional Laplace operator, i is the imaginary unit, ε designates the usual Planck constant. Equation (1.3) was introduced by Laskin [1] as an extension of the classical nonlinear Schrödinger equation $s=1$ in which the Brownian motion of the quantum paths is replaced by a Levy flight. Here $\psi=\psi(x,t)$ represents the quantum mechanical probability amplitude for a given unit-mass particle to have position x at time t (the corresponding probability density is $|\psi|^{2}$), under a confinement due to the potential functions $V(x)$, $W(x)$. The nonlinear self-coupling $f(\psi)$, which describes a self-interaction in quantum electrodynamics, gives a closer description of many particles found in the real world. Typical examples can be found in the self-interacting theories, where the nonlinear function f can be both a polynomial and nonpolynomial (this includes the cases $|\psi|^{\lambda}$, $\sin|\psi|$, etc.). We assume throughout the paper that f satisfies $f(e^{i\theta}\psi)=f(\psi)$ for all $\theta\in[0,2\pi ]$. The function $V(x)$ represents the potential acting on the particle and $W(x)$ represents a particle-interaction term, which avoids spreading of the wave packets, in the time-dependent version of the above equation. We refer to [1–3] for detailed physical discussions and motivation.

A solution ψ is referred to as a bound state of (1.3) if $\psi \rightarrow0$ as $|x|\rightarrow\infty$. Bound states of (1.3) when $\varepsilon\ll1$ are called semiclassical states, which are relevant for the links between classical and quantum mechanics. An important feature of semiclassical states $u_{\varepsilon}$ is that they concentrate as $\varepsilon\rightarrow0$.

In the classical case $s=1$, there is a broad literature on the concentration phenomenon, for example, see [4–10] and the references therein. Investigations of the existence of solutions concentrating at certain points to nonlocal Schrödinger equations under different conditions have also appeared in [11–14]. In [11], Dávila et al. considered the following superlinear problem:

$$\varepsilon^{2s}(-\Delta)^{s}u+V(x)u=u^{p}, \quad \mbox{in } x\in \mathbb{R}^{N}, $$

and multi-peak solutions were obtained via a Lyapunov-Schmidt variational reduction.

In [12], for a smooth, bounded domain $\Omega\subset \mathbb{R}^{N}$, $p\in(1,\frac{N+2s}{N-2s})$, Dávila et al. constructed a family of solutions for the nonlocal equation

$$\varepsilon^{2s}(-\Delta)^{s}u+u=u^{p}, \quad \mbox{in } \Omega, $$

which shows concentration as $\varepsilon\rightarrow0$ at an interior point of the domain Ω in the form of a scaling of the ground state in the entire space.

Concentrating solutions for fractional problems involving critical or almost critical exponents were considered in [14]. See also [15] for some concentration phenomena in particular cases, and also [13] and the references therein for related problems about Schrödinger-type equations in a fractional setting.

The goal of this paper is to show, by variational techniques as developed by Rabinowitz [16], Wang [17], Ding and Liu [18] in the classical case, that semiclassical solutions concentrate around some certain points that depend on both linear and nonlinear potentials for the nonlocal superlinear problem (1.1) and the critical problem (1.2).

For any $p\in[1,+\infty)$, the fractional Sobolev space $W^{s,p}( \mathbb{R}^{N})$ is defined as follows:

$$W^{s,p}\bigl( \mathbb{R}^{N}\bigr):=\biggl\{ u\in L^{p}\bigl( \mathbb{R}^{N}\bigr):\frac {u(x)-u(y)}{|x-y|^{\frac{N}{p}+s}}\in L^{p}\bigl( \mathbb{R}^{N}\times \mathbb{R}^{N} \bigr)\biggr\} , $$

endowed with the natural norm

$$\|u\|_{W^{s,p}( \mathbb{R}^{N})}:=\biggl( \int_{ \mathbb{R}^{N}}|u|^{p}\,dx+ \int_{ \mathbb{R}^{N}} \int_{ \mathbb{R}^{N}}\frac {|u(x)-u(y)|^{p}}{|x-y|^{N+sp}}\,dx\,dy\biggr)^{\frac{1}{p}}, $$

where the term

$$[u]_{W^{s,p}( \mathbb{R}^{N})}:=\biggl( \int_{ \mathbb{R}^{N}} \int_{ \mathbb {R}^{N}}\frac{|u(x)-u(y)|^{p}}{|x-y|^{N+sp}}\,dx\,dy\biggr)^{\frac{1}{p}} $$

is the so-called Gagliardo (semi) norm of u.

For $p=2$, we take into account the definition of the space $H^{s}( \mathbb{R}^{N})=W^{s,2}( \mathbb{R}^{N})$ via the Fourier transform. Precisely, we may define

$$\widehat{H}^{s}\bigl( \mathbb{R}^{N}\bigr)=\biggl\{ u\in L^{2}\bigl( \mathbb{R}^{N}\bigr): \int_{ \mathbb {R}^{N}}\bigl(1+|\xi|^{2s}\bigr)\bigl\vert \hat{u}(\xi)\bigr\vert ^{2}\,d\xi< +\infty\biggr\} , $$

where $\widehat{\hphantom{\ \ \ }}$ denotes the Fourier transform. The dual space $H^{-s}( \mathbb{R}^{N})$ is defined in the standard way. The natural place to look for a bound state of (1.1) or (1.2) is the space $H^{s}( \mathbb{R}^{N})$. The fractional Laplace $(-\Delta )^{s}u$ of a function $u\in H^{s}( \mathbb{R}^{N})$ is defined in terms of its Fourier transform by the relation

$$\widehat{(-\Delta)^{s}u}=|\xi|^{2s}\hat{u}\in L^{2}\bigl( \mathbb{R}^{N}\bigr). $$

To describe our results, set

$$\begin{aligned}& \tau:=\min V, \qquad \mathscr{V}:=\bigl\{ x\in \mathbb{R}^{N}:V(x)=\tau \bigr\} , \qquad \tau_{\infty}:=\liminf_{|x|\rightarrow\infty}V(x); \\& \kappa:=\max W, \qquad \mathscr{W}:=\bigl\{ x\in \mathbb{R}^{N}:W(x)= \kappa\bigr\} , \qquad \kappa_{\infty}:=\limsup_{|x|\rightarrow\infty}W(x). \end{aligned}$$

Assume that the external linear and nonlinear potentials $V(x)$ and $W(x)$ satisfy:

(P₀):

$V,W\in L^{\infty}( \mathbb{R}^{N})$ are uniformly continuous and $\inf V>0$, $\inf W>0$.

(P₁):

Either (i) $\tau<\tau_{\infty}$, and there exist $R>0$, $x_{v}\in \mathscr{V}$ such that $W(x_{v})\geq W(x)$ for all $|x|\geq R$; or (ii) $\kappa>\kappa_{\infty}$, and there exist $R>0$, $x_{w}\in\mathscr{W}$ such that $V(x_{w})\leq V(x)$ for all $|x|\geq R$.

Observe that, in case (P₁)(i) we can assume $W(x_{v})=\max_{x\in \mathscr{V}}W(x)$ and set

$$\mathscr{A}_{v}:=\bigl\{ x\in\mathscr{V}:W(x)=W(x_{v})\bigr\} \cup\bigl\{ x\notin \mathscr{V}:W(x)>W(x_{v})\bigr\} ; $$

in case (P₁)(ii) we can assume $V(x_{w})=\min_{x\in\mathscr {W}}V(x)$ and set

$$\mathscr{A}_{w}:=\bigl\{ x\in\mathscr{W}:V(x)=V(x_{w})\bigr\} \cup\bigl\{ x\notin \mathscr{W}:V(x)< V(x_{w})\bigr\} . $$

Obviously, $\mathscr{A}_{v}$ and $\mathscr{A}_{w}$ are bounded. Moreover, $\mathscr{A}_{v}=\mathscr{A}_{w}= \mathscr{V}\cap\mathscr{W}$ if $\mathscr{V}\cap\mathscr{W}\neq\emptyset$. In particular, $\mathscr{A}_{v}=\mathscr{V}$ if W is constant, and $\mathscr{A}_{w}=\mathscr{W}$ if V is constant.

For the nonlinear fields, by writing $F(t):=\int_{0}^{t} f(\tau)\,d\tau$, we begin with the superlinear case:

(f₁):

$f\in C^{1}(\mathbb{R}^{+},\mathbb{R}^{+})$, $f(t)=0$ for $t\leq0$, $f(t)=o(t)$;

(f₂):

$\frac{f(t)}{t}$ is nondecreasing with respect to $t>0$;

(f₃):

there are $p\in(2,2^{*}_{s})$, $c_{1}>0$ such that $f(t)\leq c_{1}(1+t^{p-1})$ for $t\geq0$;

(f₄):

there exists $\mu>2$ such that $0<\mu F(t)\leq f(t)t$ for all $t>0$.

Our first result reads as follows.

Theorem 1.1

Let (P₀), (f₁)-(f₄) be satisfied.

• (A) Suppose (P₁)(i) holds, then for sufficiently small $\varepsilon>0$:

  • (i) (existence) a positive solution $w_{\varepsilon}\in\bigcap_{q\geq 2}W^{s,q}( \mathbb{R}^{N})$ to (1.1) exists;

  • (ii) (concentration) $w_{\varepsilon}$ possesses a (global) maximum point $x_{\varepsilon}$ such that

    $$\lim_{\varepsilon\rightarrow0} \operatorname{dist}( x_{\varepsilon}, \mathscr{A}_{v})=0; $$

  • (iii) (decay estimates) there exist constants $0< C_{1}< C_{2}$ such that

    $$\varepsilon^{(N+2s)} C_{1}|x-x_{\varepsilon}|^{-(N+2s)} \leq w_{\varepsilon}(x)\leq\varepsilon^{(N+2s)} C_{2}|x-x_{\varepsilon}|^{-(N+2s)}; $$

  • (iv) (convergence) setting $v_{\varepsilon}(x):=w_{\varepsilon}( \varepsilon x+x_{\varepsilon})$, for any sequence $x_{\varepsilon}\rightarrow x_{0} (\varepsilon\rightarrow0)$, $v_{\varepsilon}\rightarrow u$ in $H^{s}( \mathbb{R}^{N})$ with $u(x)$ being a least energy solution of

    $$(-\Delta)^{s}v+V(x_{0})v=W(x_{0})f(v), \quad v>0. $$

    In particular if $\mathscr{V}\cap\mathscr{W}\neq\emptyset$, then $\lim_{\varepsilon\rightarrow0} \operatorname{dist}( x_{\varepsilon},\mathscr {V}\cap\mathscr{W})=0$ and, up to a subsequence, $v_{\varepsilon}\rightarrow u$ in $H^{s}( \mathbb{R}^{N})$ with $u(x)$ being a least energy solution of

    $$(-\Delta)^{s}v+\tau v=\kappa f(v),\quad v>0. $$

• (B) Suppose (P₁)(ii) holds, then all the conclusions of (A) (with $\mathscr{A}_{v}$ replaced by $\mathscr{A}_{w}$) remain true.

For the critical problem (1.2), we strengthen (f₄) as follows:

($\mathrm{f}'_{4}$):

there exist $q>2$, $\mu>2$, and $c_{0}>0$ such that $c_{0} t^{q}\leq F(t)\leq\frac{1}{\mu}f(t)t$ for all $t>0$.

Our result concerned with the nonlocal critical problem (1.2) is as follows.

Theorem 1.2

Let (P₀), (f₁)-($\mathrm{f}'_{4}$) be satisfied.

• (A) Suppose (P₁)(i) with $\tau\in(0,\tau_{0})$ holds, then for sufficiently small $\varepsilon>0$, all the conclusions as in Theorem  1.1(A)(i)-(iv) are valid.

• (B) Suppose (P₁)(ii) with $\kappa>\kappa_{0}$ holds, then all the conclusions of (A) (with $\mathscr{A}_{v}$ replaced by $\mathscr{A}_{w}$) remain true. (See the definition of $\tau_{0}$, $\kappa_{0}$ in Section  4.)

In the case $s=1$, Theorems 1.1 and 1.2 were found by Ding and Liu [18]. The case of $W(x)=1$ was previously considered by Rabinowitz [16] and Wang [17].

The organization of this paper is as follows: In a preliminary section, Section 2, we describe the appropriate functional setting for the study of the problem (1.1) or (1.2), including the definition of an equivalent problem. In Section 3 and Section 4, we consider the superlinear problem (1.1) and the critical problem (1.2), respectively. The proof of the main results is variational and relies on an elementary idea entailing mountain-pass arguments.

2 Preliminaries and functional setting

Let $0< s<1$, as we have recalled in the introduction, for $\phi\in H^{s}( \mathbb{R}^{N})$ the standard definition of the fractional Laplacian $(-\Delta)^{s}\phi$ is given via the Fourier transform $\widehat{\hphantom{\ \ \ }}$. $(-\Delta)^{s}\phi\in L^{2}( \mathbb{R}^{N})$ is defined by the formula

$$ \widehat{(-\Delta)^{s}\phi}=|\xi|^{2s}\widehat{\phi(\xi)}. $$

(2.1)

When ϕ is assumed in addition sufficiently regular, we obtain the direct representation

$$ (-\Delta)^{s}\phi(x)=C_{N,s} \int_{{ \mathbb{R}^{N}}}\frac{\phi (x)-\phi(y)}{|x-y|^{N+2s}}\,dy, $$

(2.2)

for a suitable constant $C_{N,s}$ and the integral is understood in a principal value sense.

Another useful local representation, found by Caffarelli and Silverstre [19], is via the following boundary value problem in the half space $R_{+}^{N+1}=\{(x,y):x\in \mathbb{R}^{N},y>0\}$:

$$ \textstyle\begin{cases} \nabla\cdot(y^{1-2s}\nabla\tilde{\phi})=0, \quad \mbox{in } \mathbb {R}_{+}^{N+1}, \\ \tilde{\phi}(x,0)=\phi(x), \quad \mbox{on } \mathbb{R}^{N}. \end{cases} $$

(2.3)

Here ϕ̃ is called the s-harmonic extension of ϕ, the extension function belongs to the space $X_{0}^{s}(\mathbb{R}_{+}^{N+1})=\overline{C_{0}^{\infty}(R_{+}^{N+1})} ^{\| \cdot\|_{X_{0}^{s}(\mathbb{R}_{+}^{N+1})}}$, with

$$\|\tilde{\phi}\|_{X_{0}^{s}(\mathbb{R}_{+}^{N+1})}=\biggl(k_{s} \int_{\mathbb {R}_{+}^{N+1}} y^{1-2s}|\nabla\tilde{\phi}|^{2} \,dx\,dy\biggr)^{\frac{1}{2}}, $$

where $k_{s}$ is a normalization constant. With this constant, we have the extension operator to be an isometry between $H^{s}( \mathbb{R}^{N})$ and $X_{0}^{s}(\mathbb{R}_{+}^{N+1})$. That is,

$$\|\tilde{\phi}\|_{X_{0}^{s}(\mathbb{R}_{+}^{N+1})}=\|\phi\|_{H^{s}( \mathbb{R}^{N})}=\bigl\Vert (- \Delta)^{\frac{s}{2}}\phi\bigr\Vert _{2}^{2}. $$

Moreover, ϕ̃ can be explicitly given as a convolution integral with the s-Poisson kernel $P_{s}(x,y)$,

$$ \tilde{\phi}(x,y)= \int_{ \mathbb{R}^{N}}P_{s}(x-z,y)\phi(z)\, dz, $$

(2.4)

where $P_{s}(x,y)=d_{N,s}\frac{y^{2s}}{(|x|^{2}+|y|^{2})^{\frac {N+2s}{2}}}$, and $d_{N,s}$ achieves $\int_{ \mathbb{R}^{N}}P_{s}(x,y)\,dx=1$. Then under suitable regularity, $(-\Delta)^{s}\phi$ is the Dirichlet-to-Neumann map for this problem, namely

$$ \frac{1}{k_{s}}(-\Delta)^{s}\phi(x)=-\lim_{y\rightarrow 0^{+}}y^{1-2s} \partial_{y}\tilde{\phi}(x,y). $$

(2.5)

The characterizations (2.1), (2.2), and (2.4) are all equivalent, for instance, in Schwartz’s space of rapidly decreasing smooth functions. The constants in (2.2), (2.4), and (2.5) satisfy the identity $2s d_{N,s}k_{s}=C_{N,s}$. Their explicit values can be consulted for instance in [20].

It is standard that (1.1) or (1.2), by setting $u(x)=w(\varepsilon x)$, respectively, is equivalent to

$$ (-\Delta)^{s}u+V_{\varepsilon}(x)u=W_{\varepsilon}(x)f(u) $$

(2.6)

and

$$ (-\Delta)^{s}u+V_{\varepsilon}(x)u=W_{\varepsilon}(x)h(u), $$

(2.7)

where $h(t)=f(t)+t^{2^{*}_{s}-1}$, for $t\geq0$, $h(t)=0$ for $t<0$, and $H(u):=F(u)+\frac{1}{2^{*}_{s}}|u|^{2^{*}_{s}}$, $V_{\varepsilon}(x)=V(\varepsilon x)$, $W_{\varepsilon}(x)=W(\varepsilon x)$. We will in the sequel focus on these equivalent problems.

In the following we will denote

$$\|u\|_{\varepsilon}:=\biggl( \int_{ \mathbb{R}^{N}}\bigl\vert (-\Delta)^{\frac {s}{2}}u\bigr\vert ^{2}\,dx+ \int_{ \mathbb{R}^{N}}V(\varepsilon x)u^{2}\,dx \biggr)^{\frac{1}{2}} $$

as the norms in $H^{s}( \mathbb{R}^{N})$, which are all equivalent to the standard norm $\|\cdot\|_{H^{s}}$ of $H^{s}( \mathbb{R}^{N})$ because of the boundedness of $V(x)$ and $W(x)$. We will also denote by $|\cdot|_{p}$ the usual norm of $L^{p}( \mathbb {R}^{N})$. Associated to the problem (2.6) or (2.7) we consider the energy functional $I_{\varepsilon}$ or $I^{*}_{\varepsilon}$, respectively,

$$\begin{aligned}& I_{\varepsilon}(u)=\frac{1}{2} \int_{ \mathbb{R}^{N}}\bigl\vert (-\Delta)^{\frac {s}{2}}u\bigr\vert ^{2}\,dx+\frac{1}{2} \int_{ \mathbb{R}^{N}}V(\varepsilon x)u^{2}\,dx- \int_{ \mathbb{R}^{N}}W(\varepsilon x)F(u)\,dx, \\& I^{*}_{\varepsilon}(u)=\frac{1}{2} \int_{ \mathbb{R}^{N}}\bigl\vert (-\Delta )^{\frac{s}{2}}u\bigr\vert ^{2}\,dx+\frac{1}{2} \int_{ \mathbb{R}^{N}}V(\varepsilon x)u^{2}\,dx- \int_{ \mathbb{R}^{N}}W(\varepsilon x)H(u)\,dx. \end{aligned}$$

These functionals are well defined in $H^{s}( \mathbb{R}^{N})$, and, moreover, the critical points of $I_{\varepsilon}$ and $I^{*}_{\varepsilon}$ correspond to weak solutions to (2.6) and (2.7), respectively.

With the above extensions (2.3) and (2.5), we can reformulate our problem (2.6) or (2.7), respectively, as

$$\begin{aligned}& \textstyle\begin{cases} \nabla\cdot(y^{1-2s}\nabla\tilde{\phi})=0, \quad \mbox{in } \mathbb {R}_{+}^{N+1}, \\ \tilde{\phi}(x,0)=\phi(x), \quad \mbox{on } \mathbb{R}^{N}, \\ -k_{s}\lim_{y\rightarrow0^{+}}y^{1-2s}\partial_{y}\tilde{\phi }(x,y)=W(\varepsilon x)f(\tilde{\phi})-V(\varepsilon x)\tilde {\phi}, \quad \mbox{in } \mathbb{R}^{N+1}_{+}. \end{cases}\displaystyle \end{aligned}$$

(2.8)

$$\begin{aligned}& \textstyle\begin{cases} \nabla\cdot(y^{1-2s}\nabla\tilde{\phi}^{*})=0, \quad \mbox{in } \mathbb{R}_{+}^{N+1}, \\ \tilde{\phi}^{*}(x,0)=\phi^{*}(x),\quad \mbox{on } \mathbb{R}^{N}, \\ -k_{s}\lim_{y\rightarrow0^{+}}y^{1-2s}\partial_{y}\tilde{\phi }^{*}(x,y)=W(\varepsilon x)h(\tilde{\phi}^{*})-V(\varepsilon x)\tilde{\phi}^{*}, \quad \mbox{in } \mathbb{R}^{N+1}_{+}. \end{cases}\displaystyle \end{aligned}$$

(2.9)

An energy solution to problem (2.8) is a function $\tilde{\phi }\in X_{0}^{s}(\mathbb{R}_{+}^{N+1})$ such that

$$k_{s} \int_{\mathbb{R}_{+}^{N+1}}y^{1-2s}\langle\nabla\tilde{\phi },\nabla \varphi\rangle \,dx\,dy+ \int_{ \mathbb{R}^{N}}V(\varepsilon x)\tilde{\phi}\varphi \,dx = \int_{ \mathbb{R}^{N}}W(\varepsilon x)f(\tilde{\phi})\varphi \,dx, $$

for all $\varphi\in X_{0}^{s}(\mathbb{R}_{+}^{N+1})$. For any energy solution $\tilde{\phi}\in X_{0}^{s}(\mathbb{R}_{+}^{N+1})$ of problem (2.8), the function $\phi=\tilde{\phi}(\cdot,0)$ defined in the sense of traces, belongs to the space $H^{s}( \mathbb{R}^{N})$ and is an energy solution to the problem (2.6). The converse is also true. Therefore, the formulations of (2.6) and (2.8) are equivalent. This is the same as for (2.7) and (2.9).

The associated energy functional to the problem (2.8) or (2.9) is, respectively,

$$J_{\varepsilon}(\tilde{\phi})=\frac{k_{s}}{2} \int_{\mathbb {R}_{+}^{N+1}}y^{1-2s}|\nabla\tilde{\phi}|^{2} \,dx\,dy+\frac{1}{2} \int _{ \mathbb{R}^{N}}V(\varepsilon x)|\tilde{\phi}|^{2}\,dx - \int_{ \mathbb{R}^{N}}W(\varepsilon x)F(\tilde{\phi})\,dx $$

or

$$\begin{aligned} J_{\varepsilon}^{*}\bigl(\tilde{\phi}^{*}\bigr) =&\frac{k_{s}}{2} \int_{\mathbb {R}_{+}^{N+1}}y^{1-2s}\bigl|\nabla\tilde{\phi}^{*}\bigr|^{2} \,dx\,dy+\frac {1}{2} \int_{ \mathbb{R}^{N}}V(\varepsilon x)\bigl|\tilde{\phi}^{*}\bigr|^{2}\,dx \\ &{}- \int_{ \mathbb{R}^{N}}W(\varepsilon x)H\bigl(\tilde{\phi}^{*}\bigr)\,dx. \end{aligned}$$

Clearly, critical points of $J_{\varepsilon}$ and $J_{\varepsilon}^{*}$ in $X_{0}^{s}(\mathbb{R}_{+}^{N+1})$ correspond to critical points of $I_{\varepsilon}$ and $I_{\varepsilon}^{*}$ in $H^{s}( \mathbb{R}^{N})$, respectively.

Remark 2.1

In the sequel, and in view of the above equivalence, we will find both formulations of the problem, in $\mathbb{R}^{N}$ or in $\mathbb{R}_{+}^{N+1}$, whenever we may take some advantage. In particular, we will use the extension version (2.8) or (2.9) respectively, when dealing with the fractional operator acting on products of functions, since it is not clear how to calculate this action.

Another tool which is very useful in the following is the trace inequality,

$$ \int_{\mathbb{R}_{+}^{N+1}}y^{1-2s}\bigl\vert \nabla z(x,y)\bigr\vert ^{2}\,dx\,dy\geq C_{1}\biggl( \int _{ \mathbb{R}^{N}}\bigl\vert z(x,0)\bigr\vert ^{r}\,dx \biggr)^{\frac{2}{r}}, $$

(2.10)

for any $1\leq r\leq\frac{2N}{N-2s}$, $N>2s$, and any $z\in X_{0}^{s}(\mathbb{R}_{+}^{N+1})$, where $C_{1}=C_{1}(s,r,N)>0$. It is equivalent to the fractional Sobolev inequality

$$ \int_{ \mathbb{R}^{N}}\bigl\vert (-\Delta)^{\frac{s}{2}}u\bigr\vert ^{2}\,dx\geq C_{2}\biggl( \int_{ \mathbb{R}^{N}}\vert u\vert ^{r}\,dx \biggr)^{\frac{2}{r}}, $$

(2.11)

for any $1\leq r\leq\frac{2N}{N-2s}$, $N>2s$, and any $u\in H^{s}( \mathbb{R}^{N})$. In the following we will denote the critical fractional Sobolev exponent $2^{*}_{s}=\frac{2N}{N-2s}$.

Remark 2.2

When $r=2^{*}_{s}$, the best constant in (2.10) will be denoted by $S(s,N)$. This constant is explicit and independent of the domain; its exact value is

$$S(s, N)=\frac{2\pi^{s}\Gamma(\frac{N+2s}{2})\Gamma(1-s)(\Gamma (\frac{N}{2}))^{\frac{2s}{N}}}{\Gamma(s)\Gamma(\frac{N-2s}{2}) (\Gamma(N))^{s}}. $$

So we have

$$ \int_{R_{+}^{N+1}}y^{1-2s}\bigl\vert \nabla z(x,y)\bigr\vert ^{2}\,dx\,dy\geq S(s,N) \biggl( \int_{ \mathbb{R}^{N}}\bigl\vert z(x,0)\bigr\vert ^{\frac{2N}{N-2s}}\,dx \biggr)^{\frac{N-2s}{N}}, $$

(2.12)

for any $z\in X_{0}^{s}(\mathbb{R}_{+}^{N+1})$. This will be used in Section 4, the best constant in (2.11) is then $k_{s} S(s,N)$.

Recall that we say that $u\in H^{s}( \mathbb{R}^{N})$ is a weak solution of (2.6) if

$$ \int_{ \mathbb{R}^{N}}(-\Delta)^{\frac{s}{2}}u(-\Delta)^{\frac {s}{2}} \varphi \,dx+ \int_{ \mathbb{R}^{N}}V(\varepsilon x)u\varphi \,dx= \int_{ \mathbb{R}^{N}}W(\varepsilon x)f(u)\varphi \,dx, $$

(2.13)

for all $\varphi\in H^{s}( \mathbb{R}^{N})$.

First of all, under our assumptions, we have the following lemma.

Lemma 2.1

Any weak solution $u\in H^{s}( \mathbb {R}^{N})$ of (2.6) or (2.7) is positive.

Proof

Since $f(t)=0$ on $\mathbb{R}^{-}$, choosing $\varphi=u^{-}\in H^{s}( \mathbb{R}^{N})$ in the variational formulation (2.13) yields

$$\begin{aligned} \int_{ \mathbb{R}^{N}}(-\Delta)^{\frac{s}{2}}u(-\Delta)^{\frac {s}{2}}u^{-} \,dx =&- \int_{ \mathbb{R}^{N}}V(\varepsilon x)uu^{-} \,dx+ \int_{ \mathbb{R}^{N}}W(\varepsilon x)f(u)u^{-} \,dx \\ =& \int_{ \mathbb{R}^{N}}V(\varepsilon x) \bigl(u^{-}\bigr)^{2} \,dx. \end{aligned}$$

Hence, it follows from the definition (2.2) of $(-\Delta)^{s}u$,

$$\begin{aligned}& \int_{ \mathbb{R}^{N}}(-\Delta)^{\frac{s}{2}}u(-\Delta)^{\frac {s}{2}}u^{-} \,dx \\& \quad = \int_{ \mathbb{R}^{N}}u^{-}(-\Delta)^{s}u^{+} \,dx-\bigl\vert (-\Delta)^{\frac {s}{2}}u^{-}\bigr\vert _{2}^{2} \\& \quad = \frac{C_{N,s}}{2} \int_{ \mathbb{R}^{N}} \int_{ \mathbb{R}^{N}}\frac {(u^{+}(x)-u^{+}(y))(u^{-}(x)-u^{-}(y))}{\vert x-y\vert ^{N+2s}}\,dx\,dy-\bigl\vert (-\Delta )^{\frac{s}{2}}u^{-}\bigr\vert _{2}^{2} \\& \quad = -C_{N,s} \int_{ \mathbb{R}^{N}} \int_{ \mathbb{R}^{N}}\frac {u^{+}(x)u^{-}(y)}{\vert x-y\vert ^{N+2s}}\,dx\,dy-\bigl\vert (- \Delta)^{\frac {s}{2}}u^{-}\bigr\vert _{2}^{2} \leq-\bigl\| (-\Delta)^{\frac{s}{2}}u^{-}\bigr\| _{2}^{2}. \end{aligned}$$

In turn, we get $\|u^{-}\|_{\varepsilon}^{2}=|(-\Delta)^{\frac {s}{2}}u^{-}|_{2}^{2}+\int_{ \mathbb{R}^{N}}V(\varepsilon x)(u^{-})^{2}\,dx\leq0$, namely $u^{-}=0$, hence the weak solution $u\in H^{s}( \mathbb{R}^{N})$ for (2.6) is nonnegative. Moreover, u cannot vanish at an interior point as follows from the maximum principle [21]. This completes the proof. □

3 Study of the nonlocal superlinear problem (1.1)

In this section, we will prove Theorem 1.1. We only need to give the details for (B) because the argument for (A) is similar to that for (B).

Suppose that (f₁)-(f₄) hold and let (P₀) and (P₁)(ii) be satisfied; without loss of generality, we assume that $x_{w}=0\in \mathscr{W}$ ($x_{w}=0\in\mathscr{V}\cap\mathscr{W}$ if $\mathscr{V}\cap\mathscr{W}\neq\emptyset$) and $a:=V(0)=\min_{y\in\mathscr{W}}V(y)\leq V(x)$ for all $|x|\geq R$, then $V(0)=a$, $W(0)=\kappa$. We remark by (P₀) that $V_{\varepsilon}(x)\rightarrow V(0) =a$, $W_{\varepsilon}(x)\rightarrow W(0)=\kappa$ uniformly on bounded sets of $\mathbb{R}^{N}$ as $\varepsilon\rightarrow0$.

Before proving the main results, we denote the Nehari manifold, the critical set, the least energy, and the set of least energy solutions of $I_{\varepsilon}$ as follows:

$$\begin{aligned}& \mathscr{N}_{\varepsilon}:=\bigl\{ u\in H^{s}\setminus\{0 \}:I_{\varepsilon}'(u)u=0\bigr\} , \\& \mathscr{H}_{\varepsilon}:=\bigl\{ u\in H^{s}: I_{\varepsilon}'(u)=0 \bigr\} , \\& \gamma_{\varepsilon}:=\inf_{u\in\mathscr{N}_{\varepsilon}}I_{\varepsilon}(u), \\& \mathscr{R}_{\varepsilon}:=\bigl\{ u\in\mathscr{H}_{\varepsilon}:I_{\varepsilon}(u)=\gamma_{\varepsilon}\bigr\} . \end{aligned}$$

Observe that, in virtue of (f₄), we have

$$ F(t)\geq a_{1}|t|^{\mu}-a_{2}|t|^{2}, \quad \mbox{for all } t\geq0. $$

(3.1)

By (f₁) and (f₃), for any $\delta>0$, there is $C_{\delta}>0$ such that

$$ F(t)\leq\delta|t|^{2}+C_{\delta}|t|^{p},\quad \mbox{for all } t\in R. $$

(3.2)

These inequalities imply $\mu\leq p$. Setting $\widehat{F(t)}:=\frac {1}{2}f(t)t-F(t)$, we have

$$ \widehat{F(t)}\geq\frac{\mu-2}{2\mu}f(t)t\geq\frac{\mu-2}{2}F(t). $$

(3.3)

3.1 The function $I_{\varepsilon}$

In this subsection, we are going to establish some results for the function $I_{\varepsilon}$.

It is easy to check by (3.1) and (3.2) that functional $I_{\varepsilon}$ possesses the mountain-pass structure.

Lemma 3.1

There exist $\alpha>0$ and an open set $B\subset H^{s}( \mathbb{R}^{N})$ (both independent of ε), such that:

• (i) $I_{\varepsilon}(u)\geq\alpha$ for $u\in\partial B$;

• (ii) $\lim_{t\rightarrow+\infty}I_{\varepsilon}(tu)=-\infty$ if $u(x)\geq0$, $u\neq0$.

Consequently, let us consider the family

$$\Gamma_{\varepsilon}:=\bigl\{ \gamma\in C\bigl([0,1],H^{s}\bigl( \mathbb{R}^{N}\bigr)\bigr):\gamma (0)=0,I_{\varepsilon}\bigl(\gamma(1) \bigr)< 0\bigr\} , $$

and the minimax schemes $c_{\varepsilon}:=\inf_{\gamma\in\Gamma _{\varepsilon}}\max_{t\in[0,1]}I_{\varepsilon}(\gamma(t))$. Moreover, there is $\overline{c}>0$ independent of ε such that $\alpha\leq c_{\varepsilon}<\overline{c}$.

Using a standard argument as in the classical case in [18, 22], we have the following.

Lemma 3.2

$c_{\varepsilon}=\inf_{u\in H^{s}( \mathbb {R}^{N})\setminus\{0\}}\max_{t\geq0}I_{\varepsilon}(tu)=\inf_{u\in \mathscr{N}_{\varepsilon}}I_{\varepsilon}(u)$.

The following lemma is clear by the assumptions.

Lemma 3.3

• (i) For each $u\in H^{s}( \mathbb {R}^{N})\setminus\{0\}$, there is a unique $t_{\varepsilon}=t_{\varepsilon}(u)>0$ such that $t_{\varepsilon}u\in\mathscr{N}_{\varepsilon}$.

• (ii) Moreover, there is $T>0$ independent of $\varepsilon>0$ such that $t_{\varepsilon}< T$.

Proof

Since the proof of (i) is standard, we only need to prove (ii).

Indeed, by (i), for any fixed $u\in H^{s}(\mathbb{R}^{N})\setminus\{0\}$, there exists unique $t_{\varepsilon}u\in\mathscr{N}_{\varepsilon}$ so that

$$\begin{aligned} C_{1}t_{\varepsilon}^{2}\|u\|^{2} \geq& \|t_{\varepsilon}u\|_{\varepsilon}^{2}= \int_{ \mathbb{R}^{N}}W(\varepsilon x)f(t_{\varepsilon}u)t_{\varepsilon}u\, dx \\ \geq& C_{2} \inf W\cdot t_{\varepsilon}^{\mu}|u|_{\mu}^{\mu}-C_{3} \inf W\cdot t_{\varepsilon}^{2}|u|_{2}^{2} \quad (\mu>2). \end{aligned}$$

This proves that there is $T>0$ only dependent of u such that $t_{\varepsilon}\leq T$. This completes the proof. □

Lemma 3.4

There is $\theta>0$ independent of $\varepsilon\in(0,1)$ such that $\|u\|_{H^{s}}\geq\theta$ for all $u\in\mathscr{N}_{\varepsilon}$.

Proof

Since $V(x)\geq\tau$, there is $\gamma_{1}>0$ independent of ε such that

$$\gamma_{1} \Vert u \Vert _{H^{s}}^{2}\leq \int_{ \mathbb{R}^{N}}\bigl(\bigl\vert (-\Delta)^{\frac {s}{2}}u\bigr\vert ^{2}+V_{\varepsilon}(x)\vert u\vert ^{2}\bigr)\, dx, \quad \mbox{for all } u\in H^{s}\bigl( \mathbb{R}^{N}\bigr). $$

Since $W(x)\leq\kappa$, it follows from (3.2) that, for any $\delta >0$, there is $C_{\delta}$ independent of ε such that, for all $u\in H^{s}( \mathbb{R}^{N})$,

$$\int_{ \mathbb{R}^{N}}W_{\varepsilon}(x)f(u)\, dx\leq C_{1} \delta\|u \| _{H^{s}}^{2}+C_{2}C_{\delta}\|u \|_{H^{s}}^{p}. $$

Now, for $u\in\mathscr{N}_{\varepsilon}$,

$$\gamma_{1}\|u\|_{H^{s}}^{2}\leq\|u \|_{\varepsilon}^{2}= \int_{ \mathbb {R}^{N}}W_{\varepsilon}(x)f(u)\,dx\leq C_{1} \delta\|u\|_{H^{s}}^{2}+C_{2}C_{\delta}\|u \|_{H^{s}}^{p}, $$

taking $\delta=\frac{\gamma_{1}}{2C_{1}}$, there is $\theta>0$ independent of ε such that $\|u\|_{H^{s}}\geq\theta$. Thus we complete the proof. □

For any $a>0$, $b>0$, consider the constant coefficient equation

$$ (-\Delta)^{s}u+au=bf(u),\quad u\in H^{s}\bigl( \mathbb{R}^{N}\bigr). $$

(3.4)

The solutions of (3.4) are critical points of the functional

$$I_{ab}(u)=\frac{1}{2}\bigl\vert (-\Delta)^{\frac{s}{2}}u \bigr\vert _{2}^{2}+\frac {a}{2}\vert u\vert _{2}^{2}-b \int_{ \mathbb{R}^{N}}F(u)\,dx, $$

defined for $u\in H^{s}( \mathbb{R}^{N})$. Let $\gamma_{ab}$ be the mountain-pass level and $\mathscr{N}_{ab}$ the Nehari manifold of $I_{ab}$.

The following lemma is similar to the one of [18].

Lemma 3.5

For equation (3.4) we have:

• (i) $\mathscr{H}_{ab}:=\{u\in H^{s}( \mathbb{R}^{N})\setminus\{0\} :I'_{ab}(u)=0\}\neq\emptyset$.

• (ii) $\gamma_{ab}=\inf\{I_{ab}(u):u\in\mathscr{N}_{ab}\}=\inf\{ I_{ab}(u):u\in\mathscr{H}_{ab}\setminus\{0\}\}$.

• (iii) $\gamma_{ab}$ is attained.

• (iv) Let $a_{j}>0$ and $b_{j}>0$ ($j=1,2$) with $\min\{a_{2}-a_{1},b_{1}-b_{2}\}\geq 0$, then $\gamma_{a_{1}b_{1}}\leq\gamma_{a_{2}b_{2}}$. If additionally, $\min\{a_{2}-a_{1},b_{1}-b_{2}\}> 0$, then $\gamma_{a_{1}b_{1}}< \gamma_{a_{2}b_{2}}$.

Next, we state the regularity results whose proofs are the same as the ones in [23].

Lemma 3.6

Suppose that $u\in H^{s}( \mathbb{R}^{N})$ is a weak solution to (3.4) and f satisfies conditions (f₁)-(f₄). Then $u\in L^{q}( \mathbb{R}^{N})$ for all $q\in[2,+\infty)$ and $u\in C^{0,\mu}( \mathbb{R}^{N})$ for some $\mu\in(0,1)$. Moreover, $|u(x)|\rightarrow0$ as $|x|\rightarrow\infty$.

Using the same iterative argument as for Lemma 3.6, we obtain $u_{\varepsilon}\in\bigcap_{q\geq2} W^{s,q}( \mathbb{R}^{N})$.

Using Lemma 3.5, we have the following energy comparison between $c_{\varepsilon}$ and $\gamma_{a\kappa}$, which will be useful for the existence and concentration results.

Lemma 3.7

$\limsup_{\varepsilon\rightarrow 0}c_{\varepsilon}\leq\gamma_{a\kappa}$.

Proof

Denote $V^{c}(x)=\max\{c,V(x)\}$, $W^{d}(x)=\min\{ d,W(x)\}$, $V^{c}_{\varepsilon}(x)=V^{c}(\varepsilon x)$, and $W^{d}_{\varepsilon}(x) =W^{d}(\varepsilon x)$, where c, d are positive constants.

Define the auxiliary functional as follows:

$$I_{\varepsilon}^{cd}(u):=\frac{1}{2}\bigl\vert (- \Delta)^{\frac {s}{2}}u\bigr\vert _{2}^{2}+ \frac{1}{2} \int_{ \mathbb{R}^{N}}V_{\varepsilon}^{c}(x)\vert u\vert ^{2}\,dx- \int_{ \mathbb{R}^{N}}W_{\varepsilon}^{d}(x)F(u)\,dx, $$

for any $u\in H^{s}( \mathbb{R}^{N})$, which implies that $I_{\varepsilon}(u)\leq I_{\varepsilon}^{cd}(u)$, and thus $\gamma_{cd}\leq c_{\varepsilon}^{cd}$, where $c_{\varepsilon}^{cd}$ is the least energy of $I_{\varepsilon}^{cd}$. By the definition of τ and κ, we get $V_{\varepsilon}^{\tau}(x)=V_{\varepsilon}( x)$, $W_{\varepsilon}^{\kappa}(x)=W_{\varepsilon}( x)$. Therefore,

$$ I_{\varepsilon}^{\tau\kappa}(u)=I_{\varepsilon}(u), $$

(3.5)

and $V_{\varepsilon}^{\tau}(x)\rightarrow V(0)=a$, $W_{\varepsilon}^{\kappa}(x)\rightarrow W(0)=\kappa$ uniformly on bounded sets of x as $\varepsilon\rightarrow0$.

Now, we claim $\limsup_{\varepsilon\rightarrow0}I_{\varepsilon}^{\tau \kappa}(u)\leq\gamma_{a\kappa}$.

Indeed, let e is a ground state of $I_{a\kappa}$, that is, $I_{a\kappa}(e)=\gamma_{a\kappa}$, there exists $t_{\varepsilon}>0$ such that $t_{\varepsilon}e\in\mathscr{N}_{\varepsilon}^{\tau\kappa}$ for sufficiently small ε, where $\mathscr{N}_{\varepsilon}^{\tau\kappa}$ is Nehari manifold for function $I_{\varepsilon}^{\tau\kappa}$. Thus

$$c_{\varepsilon}^{\tau\kappa}\leq I_{\varepsilon}^{\tau\kappa }(t_{\varepsilon}e)=\max_{t\geq0}I_{\varepsilon}^{\tau\kappa}(te). $$

One has

$$ I_{\varepsilon}^{\tau\kappa}(t_{\varepsilon}e)=I_{a\kappa }(t_{\varepsilon}e)+\frac{1}{2} \int_{ \mathbb{R}^{N}}\bigl(V_{\varepsilon}^{\tau}(x)-a \bigr)|t_{\varepsilon}e|^{2}\,dx+ \int_{ \mathbb{R}^{N}}\bigl(\kappa-W_{\varepsilon}^{\kappa}(x) \bigr)F(t_{\varepsilon}e)\,dx. $$

(3.6)

We can assume that $t_{\varepsilon}\rightarrow t_{0}$ (as $\varepsilon \rightarrow0$) by Lemma 3.3. This, together with the decay of $t_{0}e$, implies

$$\int_{ \mathbb{R}^{N}}\bigl(V_{\varepsilon}^{\tau}(x)-a \bigr)|t_{\varepsilon}e|^{2}\,dx=o(1) $$

and

$$\int_{ \mathbb{R}^{N}}\bigl(\kappa-W_{\varepsilon}^{\kappa}(x) \bigr)F(t_{\varepsilon}e)\,dx=o(1). $$

Notice from (3.6) that

$$I_{\varepsilon}^{\tau\kappa}(t_{\varepsilon}e)=I_{a\kappa }(t_{\varepsilon}e)+o(1)\rightarrow I_{a\kappa}(t_{0} e) \quad \mbox{as } \varepsilon\rightarrow0. $$

Consequently

$$c_{\varepsilon}^{\tau\kappa}\leq I_{\varepsilon}^{\tau\kappa }(t_{\varepsilon}e)\rightarrow I_{a\kappa}(t_{0}e)\leq\max_{t\geq 0}I_{a\kappa}(te) =I_{a\kappa}(e)=\gamma_{a\kappa}. $$

From (3.5), we obtain $c_{\varepsilon}^{\tau\kappa}=c_{\varepsilon}$. Thus, we complete the proof. □

3.2 Existence results

Lemma 3.8

$c_{\varepsilon}$ is attained at some $u_{\varepsilon}\in\mathscr{R}_{\varepsilon}$ for all small $\varepsilon>0$.

Proof

Given $\varepsilon>0$, let $u_{k}\in\mathscr {N}_{\varepsilon}$ be a minimizing sequence of $I_{\varepsilon}$, which is clearly a $(\mathit{PS})_{c_{\varepsilon}}$ sequence for $I_{\varepsilon}$: $I_{\varepsilon}(u_{k})\rightarrow c_{\varepsilon}$ and $I_{\varepsilon}'(u_{k})\rightarrow0$ as $k\rightarrow\infty$. It is easy to see that $\{u_{k}\}$ is bounded in $H^{s}( \mathbb{R}^{N})$. Assume that $u_{k}\rightharpoonup u_{\varepsilon}\in\mathscr {H}_{\varepsilon}$ in $H^{s}( \mathbb{R}^{N})$. If $u_{\varepsilon}\neq0$, then clearly $I_{\varepsilon}(u_{\varepsilon})=c_{\varepsilon}$.

Next we check that $u_{\varepsilon}\neq0$ for all $\varepsilon>0$ small.

Assume that there exists a sequence $\varepsilon_{j}\rightarrow0$ with $u_{\varepsilon_{j}}=0$, then $u_{k}\rightharpoonup0$ in $H^{s}( \mathbb {R}^{N})$, and thus $u_{k}\rightarrow0$ in $L^{p}_{\mathrm{loc}}$ for $q\in(1,2^{*}_{s})$ and $u_{k}(x)\rightarrow0$ a.e. in $x\in \mathbb{R}^{N}$.

Choose by (P₁)(ii) $b\in(\kappa_{\infty},\kappa)$ and consider the functional $I_{\varepsilon}^{ab}$, let $t_{k}>0$ be such that $t_{k}u_{k}\in \mathscr{N}_{\varepsilon}^{ab}$, this implies that $t_{k}\leq C$ for some constant $C>0$. Assume $t_{k}\rightarrow t_{0}$ as $k\rightarrow\infty$. By (P₁)(ii), the set $O_{\varepsilon}:=\{x\in \mathbb{R}^{N}:V_{\varepsilon}(x)< a \mbox{ or } W_{\varepsilon}(x)\geq b\}$ is bounded. Remark that $I_{\varepsilon _{j}}(t_{k}u_{k})\leq I_{\varepsilon_{j}}(u_{k})$. We obtain

$$\begin{aligned} c_{\varepsilon_{j}}^{ab} \leq& I_{\varepsilon _{j}}^{ab}(t_{k}u_{k}) \\ =&I_{\varepsilon_{j}}(t_{k}u_{k})+\frac{1}{2} \int_{ \mathbb {R}^{N}}\bigl(V_{\varepsilon_{j}}^{a}(x)-V_{\varepsilon_{j}}(x) \bigr) |t_{k}u_{k}|^{2}\,dx \\ &{} + \int_{ \mathbb{R}^{N}}\bigl(W_{\varepsilon_{j}}(x)-W_{\varepsilon _{j}}^{b}(x) \bigr)F(t_{k}u_{k})\,dx \\ =&I_{\varepsilon_{j}}(t_{k}u_{k})+\frac{1}{2} \int_{O_{\varepsilon _{j}}}\bigl(a-V_{\varepsilon_{j}}(x)\bigr)|t_{k}u_{k}|^{2} \,dx \\ &{} + \int_{O_{\varepsilon_{j}}} \bigl(W_{\varepsilon_{j}}(x)-b\bigr)F(t_{k}u_{k}) \,dx \\ \leq&I_{\varepsilon_{j}}(t_{k}u_{k})+o(1)\leq I_{\varepsilon _{j}}(u_{k})+o(1)=c_{\varepsilon_{j}}. \end{aligned}$$

Notice that $\gamma_{ab}\leq c_{\varepsilon_{j}}^{ab}$, hence $\gamma _{ab}\leq c_{\varepsilon_{j}}$. In virtue of Lemma 3.7, letting $\varepsilon_{j}\rightarrow0$ yields

$$\gamma_{ab}\leq\gamma_{a\kappa}, $$

which contradicts $\gamma_{a\kappa}<\gamma_{ab}$ (see Lemma 3.5(iv)). Therefore, $c_{\varepsilon}$ is attained at $0\neq u_{\varepsilon}\in\mathscr{R}_{\varepsilon}$, which ends the proof. □

3.3 Concentration and convergence of ground state

Lemma 3.9

Assume that (f₁)-(f₄), (P₀), (P₁)(ii) and for all ε sufficiently small, let $u_{\varepsilon}\in\mathscr{R}_{\varepsilon}$, then $u_{\varepsilon}$ possesses a (global) maximum $x_{\varepsilon}$ such that $\lim_{\varepsilon\rightarrow0}\operatorname{dist}(\varepsilon x_{\varepsilon}, \mathscr{A}_{w})=0$, and for any sequence $\varepsilon x_{\varepsilon}\rightarrow x_{0}$, $v_{\varepsilon}(x):=u_{\varepsilon}(x+x_{\varepsilon})$ converges in $H^{s}( \mathbb{R}^{N})$ to $u(x)$, which is a least energy solution of

$$(-\Delta)^{s}u+V(x_{0})u=W(x_{0})f(u). $$

In particular, $\mathscr{V}\cap\mathscr{W}\neq\emptyset$, then $\lim_{\varepsilon\rightarrow0}\operatorname{dist}(\varepsilon x_{\varepsilon}, \mathscr{V}\cap\mathscr{W})=0$, and up to subsequences, $v_{\varepsilon}$ converges in $H^{s}( \mathbb{R}^{N})$ to u being a least energy solution of

$$(-\Delta)^{s}u+\tau u=\kappa f(u). $$

Remark

The proof of this lemma will be lengthy but will be along the main lines of the proof of the corresponding results in the classical case in [18, 22]. We shall first show that there exists a sequence of points $\{x_{\varepsilon}\}$ in $\mathbb{R}^{N}$ such that (i) most of the ‘mass’ of $u_{\varepsilon}$ is contained in a ball (of fixed size) centered at $x_{\varepsilon}$ and (ii) $\varepsilon x_{\varepsilon}$ is bounded. This will be done in Step 1 and Step 2. Then in Step 3, we show that any limit point of $\varepsilon x_{\varepsilon}$ belongs to $\mathscr {A}_{w}$, and Step 4 together with Step 1 shows that $u_{\varepsilon}(x+x_{\varepsilon})$ converges to the least energy solution of corresponding limit equation. Furthermore, Step 5 tells us such solution $u_{\varepsilon}$ is at least a singular peak bound state.

Proof

Step 1. Let $u_{\varepsilon}\in H^{s}( \mathbb{R}^{N})$ be the critical point of $I_{\varepsilon}$ so that $I_{\varepsilon}(u_{\varepsilon})=c_{\varepsilon}$, we see that $\{u_{\varepsilon}\}$ is a bounded set in $H^{s}( \mathbb{R}^{N})$. A concentration argument and Lemma 3.4 show that there exist a sequence $\{x_{\varepsilon}\}\subset \mathbb{R}^{N}$ and constants $R>0$, $\sigma>0$ such that $\lim_{\varepsilon\rightarrow 0}\int_{B_{R}(x_{\varepsilon})}u_{\varepsilon}^{2}\geq\sigma$.

Set $v_{\varepsilon}(x):=u_{\varepsilon}(x+x_{\varepsilon})$, then $v_{\varepsilon}$ satisfies

$$ (-\Delta)^{s}v_{\varepsilon}+\widehat{V}_{\varepsilon}(x)v_{\varepsilon}=\widehat{W}_{\varepsilon}(x)f(v_{\varepsilon}), $$

(3.7)

where $\widehat{V}_{\varepsilon}(x)=V(\varepsilon(x+x_{\varepsilon}))$, $\widehat{W}_{\varepsilon}(x)=W(\varepsilon(x+x_{\varepsilon}))$, with energy

$$\begin{aligned} \hat{I}_{\varepsilon}(v_{\varepsilon}) =&\frac{1}{2}\bigl\vert (- \Delta )^{\frac{s}{2}}v_{\varepsilon}\bigr\vert ^{2}_{2} +\frac{1}{2} \int_{ \mathbb{R}^{N}}\widehat{V}_{\varepsilon}(x)v_{\varepsilon}^{2}- \int_{ \mathbb{R}^{N}}\widehat{W}_{\varepsilon}(x)F(v_{\varepsilon}) \\ =&\hat{I}_{\varepsilon}(v_{\varepsilon})-\frac{1}{2}\hat {I}_{\varepsilon}'(v_{\varepsilon})v_{\varepsilon}\\ =& \int_{ \mathbb{R}^{N}}\widehat{W}_{\varepsilon}(x) \biggl( \frac {1}{2}f(v_{\varepsilon})v_{\varepsilon}-F(v_{\varepsilon}) \biggr) \\ =&I_{\varepsilon}(u_{\varepsilon})-\frac{1}{2}I_{\varepsilon}'(u_{\varepsilon})u_{\varepsilon}=I_{\varepsilon}(u_{\varepsilon})=c_{\varepsilon}. \end{aligned}$$

We may assume $v_{\varepsilon}\rightharpoonup u$ in $H^{s}( \mathbb {R}^{N})$, and $v_{\varepsilon}\rightarrow u$ in $L_{\mathrm{loc}}^{q}$ for $q\in [1,2^{*}_{s})$ with $u\neq0$.

By $V,W\in L^{\infty}$, without loss of generality, we may assume that $V(\varepsilon x_{\varepsilon})\rightarrow V_{0}$ and $W(\varepsilon x_{\varepsilon})\rightarrow W_{0}$ as $\varepsilon \rightarrow0$. Furthermore, since V, W are uniformly continuous, for any $x\in B_{r}(0)$, one has

$$\bigl\vert V\bigl(\varepsilon(x+x_{\varepsilon})\bigr)-V(\varepsilon x_{\varepsilon})\bigr\vert \rightarrow0\quad \mbox{and} \quad \bigl\vert W\bigl( \varepsilon(x+x_{\varepsilon})\bigr)-W(\varepsilon x_{\varepsilon})\bigr\vert \rightarrow0. $$

Therefore $\widehat{V}_{\varepsilon}(x)\rightarrow V_{0}$ and $\widehat {W}_{\varepsilon}(x)\rightarrow W_{0}$ as $\varepsilon\rightarrow0$ uniformly on bounded sets of $x\in \mathbb{R}^{N}$.

Consequently, by (3.7), for any $\varphi\in C_{0}^{\infty}( \mathbb{R}^{N})$,

$$\begin{aligned} 0 =&\lim_{\varepsilon\rightarrow0} \int_{ \mathbb{R}^{N}}\bigl((-\Delta )^{s}v_{\varepsilon}+ \widehat{V}_{\varepsilon}(x)v_{\varepsilon}-\widehat {W}_{\varepsilon}(x)f(v_{\varepsilon}) \bigr)\varphi \,dx \\ =& \int_{ \mathbb{R}^{N}}\bigl((-\Delta)^{s}u+V_{0}u-W_{0}f(u) \bigr)\varphi \,dx, \end{aligned}$$

which implies that u solves

$$ (-\Delta)^{s}u+V_{0}u=W_{0}f(u), $$

(3.8)

with the energy

$$\begin{aligned} I_{V_{0}W_{0}}(u) :=&\frac{1}{2}\bigl|(-\Delta)^{\frac{s}{2}}u\bigr|_{2}^{2}+ \frac {1}{2}V_{0} \int_{ \mathbb{R}^{N}}u^{2}-W_{0} \int_{ \mathbb{R}^{N}}F(u) \\ =& \int_{ \mathbb{R}^{N}}W_{0}\biggl(\frac{1}{2}f(u)u-F(u) \biggr)\geq\gamma_{V_{0}W_{0}}. \end{aligned}$$

By Fatou’s lemma and Lemma 3.7,

$$\begin{aligned} \gamma_{V_{0}W_{0}} \leq& \int_{ \mathbb{R}^{N}}W_{0}\biggl(\frac {1}{2}f(u)u-F(u) \biggr) \\ \leq&\liminf_{\varepsilon\rightarrow0} \int_{ \mathbb {R}^{N}}\widehat{W}_{\varepsilon}(x) \biggl( \frac{1}{2}f(v_{\varepsilon}) v_{\varepsilon}-F(v_{\varepsilon}) \biggr) \\ =&\liminf_{\varepsilon\rightarrow0}\hat{I}_{\varepsilon}(v_{\varepsilon}) \leq\limsup_{\varepsilon\rightarrow0}I_{\varepsilon}(u_{\varepsilon}) \leq \gamma_{V_{0}W_{0}}. \end{aligned}$$

Therefore,

$$ \lim_{\varepsilon\rightarrow0}\hat{I}_{\varepsilon}(v_{\varepsilon})=\lim _{\varepsilon\rightarrow0}c_{\varepsilon}=I_{V_{0}W_{0}} \quad \mbox{and} \quad \Gamma_{V_{0}W_{0}}(u)=\gamma_{V_{0}W_{0}}. $$

(3.9)

As a consequence, u is the least energy solution of the limit equation (3.8).

Step 2. $\{\varepsilon x_{\varepsilon}\}$ is bounded.

Assume that $\varepsilon|x_{\varepsilon}|\rightarrow+\infty$, by $V(\varepsilon x_{\varepsilon})\rightarrow V_{0}$, $a=V(0)\leq V(x)$, $|x|\geq R$, and $W(\varepsilon x_{\varepsilon})\rightarrow W_{0}$, $\kappa=\max W$, we deduce that $V_{0}\geq a$ and $W_{0}\leq\kappa$. So it follows from Lemma 3.5 that $\gamma_{V_{0}W_{0}}>\gamma_{a\kappa}$.

However, by Step 1 and Lemma 3.7, $c_{\varepsilon}\rightarrow\gamma _{V_{0}W_{0}}\leq\gamma_{a\kappa}$, a contradiction. Therefore, we can assume $\varepsilon x_{\varepsilon}\rightarrow x_{0}$ (as $\varepsilon \rightarrow0$), then $V_{0}=V(x_{0})$, $W_{0}=W(x_{0})$, and we read (3.8) as

$$(-\Delta)^{s}u+V(x_{0})u=W(x_{0})f(u), $$

where u is the least energy solution.

Step 3. $\varepsilon x_{\varepsilon}\rightarrow\mathscr{A}_{w}$ as $\varepsilon\rightarrow0$, that is, $x_{0}\in\mathscr{A}_{w}$.

Assume that $x_{0}\notin\mathscr{A}_{w}$, by the definition of $\mathscr {A}_{w}$, we have $V(x_{0})>V(0)=a$, which, combined with $W(x_{0})<\kappa$, leads to $\gamma_{V(x_{0})W(x_{0})}>\gamma_{a\kappa}$. However, by Lemma 3.7,

$$\lim_{\varepsilon\rightarrow0}c_{\varepsilon}=\gamma _{V(x_{0})W(x_{0})}> \gamma_{a\kappa}\geq\lim_{\varepsilon\rightarrow 0}c_{\varepsilon}, $$

a contradiction.

Step 4. Let $v_{\varepsilon}$, u be defined in Step 1, then $v_{\varepsilon}\rightarrow u$ in $H^{s}( \mathbb{R}^{N})$.

It suffices to prove that there is a subsequence $\{v_{\varepsilon_{j}}\}$ so that $v_{\varepsilon_{j}}\rightarrow u$ in $H^{s}( \mathbb{R}^{N})$.

Recall that, as the argument shows, u is a least energy solution to

$$(-\Delta)^{s}u+V(x_{0})u=W(x_{0})f(u). $$

Let $\eta\in C_{0}^{\infty}(\mathbb{R}_{+}^{N+1})$ be a nonincreasing cut-off function verifying $\eta=1$ in $B_{1}^{+}(0)$, $\eta=0$ in $B_{2}^{+}(0)^{c}$. Let now $w_{j}(x,y)=\eta(\frac{2|x|}{j},y)\tilde{u}(x,y)$, where $\operatorname{Tr}(\tilde{u})=u$. One has $w_{j}(x,0)\rightarrow u(x)$ in $H^{s}( \mathbb{R}^{N})$ and $w_{j}(x,0)\rightarrow u(x)$ in $L^{q}$, $q\in[2,2^{*}_{s})$. Denote $\tilde{z}_{j}(x,y)=\tilde{v}_{\varepsilon _{j}}(x,y)-w_{j}(x,y)$, where $\operatorname{Tr}(\tilde{z}_{j})=z_{j}$, $\operatorname{Tr}(\tilde{v}_{\varepsilon_{j}})=v_{\varepsilon_{j}}$, $\operatorname{Tr}(w_{j})=w_{j}(x,0)$.

Next we prove $z_{j}\rightarrow0$ in $H^{s}( \mathbb{R}^{N})$.

Firstly, we remark that $\{z_{j}\}$ is bounded in $H^{s}( \mathbb{R}^{N})$ and using similar argument to [22], one has

$$ \lim_{j\rightarrow\infty}\biggl\vert \int_{ \mathbb{R}^{N}}\widehat {W}_{\varepsilon_{j}}(x) \bigl(F( \tilde{v}_{\varepsilon_{j}})-F(\tilde {z}_{j})-F(w_{j})\bigr) \,dx\biggr\vert =0 $$

(3.10)

and

$$ \lim_{j\rightarrow\infty}\biggl\vert \int_{ \mathbb{R}^{N}}\widehat {W}_{\varepsilon_{j}}(x) \bigl(f( \tilde{v}_{\varepsilon_{j}})-f(\tilde{z}_{j}) -f(w_{j})\bigr) \phi \,dx \biggr\vert =0, $$

(3.11)

uniformly in $\phi\in X_{0}^{s}(R_{+}^{N+1})$ with $\|\phi\|_{X_{0}^{s}(\mathbb {R}R_{+}^{N+1})}\leq1$. By the decay of u, (3.10), (3.11), and the facts that $\widehat{V}_{\varepsilon_{j}}(x)\rightarrow V(x_{0})$, $\widehat {W}_{\varepsilon_{j}}(x)\rightarrow W(x_{0})$ as $j\rightarrow\infty$ uniformly on bounded sets of x, one checks directly the following:

$$\begin{aligned} \hat{J}_{\varepsilon_{j}}(\tilde{z}_{j}) =&\frac{k_{s}}{2} \int _{\mathbb{R}_{+}^{N+1}} y^{1-2s}\langle\nabla\tilde{z}_{j}, \nabla \tilde{z}_{j}\rangle \,dx\,dy+\frac{1}{2} \int_{ \mathbb{R}^{N}} \widehat{V}_{\varepsilon_{j}}(x) \langle \tilde{z}_{j},\tilde{z}_{j}\rangle \,dx \\ &{}- \int_{ \mathbb{R}^{N}} \widehat{W}_{\varepsilon_{j}}(x)F(\tilde {z}_{j})\,dx \\ =&\frac{k_{s}}{2} \int_{R_{+}^{N+1}} y^{1-2s}\bigl(|\nabla\tilde {v}_{\varepsilon_{j}}|^{2}-2\langle\nabla \tilde{v}_{\varepsilon _{j}}, \nabla w_{j}\rangle+|\nabla w_{j}|^{2}\bigr)\,dx \,dy \\ &{}+\frac{1}{2} \int_{ \mathbb{R}^{N}}\widehat{V}_{\varepsilon _{j}}(x) \bigl(| \tilde{v}_{\varepsilon_{j}}|^{2}-2\tilde {v}_{\varepsilon_{j}}w_{j}+w_{j}^{2} \bigr)\,dx - \int_{ \mathbb{R}^{N}}\widehat{W}_{\varepsilon_{j}}(x)F(\tilde {z}_{j})\,dx \\ =&\hat{J}_{\varepsilon_{j}}( \tilde{v}_{\varepsilon _{j}})-\Gamma_{V_{0}W_{0}}( \tilde{u})+ \int_{ \mathbb{R}^{N}} \widehat {W}_{\varepsilon_{j}}(x) \bigl(F( \tilde{v}_{\varepsilon _{j}})-F(\tilde{z}_{j})-F(w_{j})\bigr) \,dx \\ &{}+o(1)=o(1) \end{aligned}$$

as $j\rightarrow\infty$, which implies that $\hat {J}_{\varepsilon_{j}}(\tilde{z}_{j})\rightarrow0$, and thus $\hat{I}_{\varepsilon_{j}}(z_{j})\rightarrow0$, where $\hat{J}_{\varepsilon_{j}}$ is the extension function of the problem as in (2.7) corresponding to $\hat{I}_{\varepsilon_{j}}$.

Similarly,

$$\begin{aligned} \hat{J}_{\varepsilon_{j}}'(\tilde{z}_{j}) \phi =&k_{s} \int _{\mathbb{R}_{+}^{N+1}} y^{1-2s}\langle\nabla\tilde{z}_{j}, \nabla \phi\rangle \,dx\,dy+ \int_{ \mathbb{R}^{N}} \widehat{V}_{\varepsilon_{j}}(x) \langle \tilde{z}_{j},\phi\rangle \,dx \\ &{}- \int\widehat{W}_{\varepsilon_{j}}(x)f(\tilde{z}_{j})\phi \,dx \\ =&\hat{J}_{\varepsilon_{j}}'( \tilde{v}_{\varepsilon _{j}}) \phi-k_{s} \int_{\mathbb{R}_{+}^{N+1}} y^{1-2s}\langle\nabla w_{j}, \nabla\phi\rangle \,dx\,dy- \int_{ \mathbb{R}^{N}} \widehat {V}_{\varepsilon_{j}}(x) \langle w_{j},\phi\rangle \,dx \\ &{}+ \int_{ \mathbb{R}^{N}} \widehat{W}_{\varepsilon _{j}}(x) \bigl(f( \tilde{v}_{\varepsilon_{j}})-f(\tilde{z}_{j})\bigr)\phi \,dx \\ =&o(1)+ \int_{ \mathbb{R}^{N}} \widehat{W}_{\varepsilon _{j}}(x) \bigl(f( \tilde{v}_{\varepsilon_{j}})-f(\tilde {z}_{j})-f(w_{j})\bigr) \phi \,dx=o(1) \end{aligned}$$

as $j\rightarrow\infty$ uniformly in $\|\phi\|_{X_{0}^{s}(\mathbb {R}_{+}^{N+1})}\leq1$, which implies $\hat{J}'_{\varepsilon _{j}}(\tilde{z}_{j})\rightarrow0$, and thus $\hat{I}'_{\varepsilon_{j}}(z_{j})\rightarrow0$. Therefore,

$$o(1)=\hat{I}_{\varepsilon_{j}}(z_{j})-\frac{1}{\mu}\hat {I}'_{\varepsilon_{j}}(z_{j})z_{j}\geq\biggl( \frac{1}{2}-\frac{1}{\mu}\biggr)\| z_{j}\|_{\varepsilon}. $$

Consequently, $z_{j}\rightarrow0$ in $H^{s}( \mathbb{R}^{N})$.

Step 5. $v_{\varepsilon}(x)\rightarrow0$ as $|x|\rightarrow \infty$ uniformly for all small ε.

Since $u,v_{\varepsilon}\in\bigcap_{q\geq2}W^{s,q}( \mathbb{R}^{N})$ and $v_{\varepsilon}\rightarrow u$ in $H^{s}( \mathbb{R}^{N})=W^{s,2}( \mathbb{R}^{N})$, for any $r\in(2,q)$, we infer that

$$\|v_{\varepsilon}-u\|_{L^{r}}\leq\|v_{\varepsilon}-u \|_{L^{2}}^{1-\lambda }\cdot\|v_{\varepsilon}-u\|_{L^{q}}^{\lambda}, $$

where $\frac{1-\lambda}{2}+\frac{\lambda}{q}=\frac{1}{r}$.

Therefore,

$$\|v_{\varepsilon}-u\|_{W^{s,r}}\leq C\|v_{\varepsilon}-u\| _{W^{s,2}}^{\theta}\cdot\|v_{\varepsilon}-u\|_{W^{s,q}}^{1-\theta}, $$

for some constant $C>0$ and $\theta>0$.

Consequently, $v_{\varepsilon}-u\rightarrow0$ in $\bigcap_{q\geq 2}W^{s,q}( \mathbb{R}^{N})$. Moreover, by a Sobolev embedding, $W^{s,q}( \mathbb {R}^{N})\hookrightarrow C^{0,\alpha}( \mathbb{R}^{N})$ (for q large enough), we deduce that $v_{\varepsilon}-u\rightarrow0$ in $C^{0,\alpha}( \mathbb{R}^{N})$, it follows from the decay of u that $|v_{\varepsilon}(x)|\rightarrow0$ as $|x|\rightarrow\infty$ uniformly in $\varepsilon>0$. Thus, we complete the proof. □

By virtue of Step 5, it is clear that one may assume the sequence $\{ x_{\varepsilon}\}$ in Step 1 to be the maximum points of $u_{\varepsilon}$. Moreover, from the above argument, we readily see that any sequence of such points satisfies $\varepsilon x_{\varepsilon}$ converging to some point in $\mathscr{A}_{w}$ as $\varepsilon\rightarrow0$.

3.4 Decay estimates

Step 5 in the previous lemma shows a uniform decay estimate; unlike the classical case $s=1$, we find suitable comparison functions as in [23] based on the Bessel kernel $\mathcal{K}$ to see that the solution $v_{\varepsilon}$ has a power-type decay at infinity instead of exponential.

Lemma 3.10

There exist $0< C_{1}\leq C_{2}$ and $R>1$ such that, for all small $\varepsilon>0$,

$$\frac{C_{1}}{|x-x_{\varepsilon}|^{N+2s}}\leq u_{\varepsilon}(x)\leq\frac {C_{2}}{|x-x_{\varepsilon}|^{N+2s}}, $$

for all $|x|\geq R$.

Before starting to give proof, let us consider for $m>0$ and $g\in L^{2}( \mathbb{R}^{N})$ the equation

$$(-\Delta)^{s}\phi+m\phi=g,\quad \mbox{in } \mathbb{R}^{N}. $$

Then in terms of the Fourier transform, this problem, for $\phi\in L^{2}$, reads

$$\bigl(|\xi|^{2s}+m\bigr)\hat{\phi}=\hat{g} $$

and has a unique solution $\phi\in H^{s}( \mathbb{R}^{N})$ given by the convolution

$$\phi(x)=\mathcal{K}*g= \int_{ \mathbb{R}^{N}}\mathcal{K}(x-z)g(z)\, dz, $$

where $\mathcal{K}$ is the fundamental solution of $(-\Delta)^{s}+m$, called the Bessel kernel,

$$\widehat{\mathcal{K}(\xi)}=\frac{1}{|\xi|^{2s}+m}. $$

Moreover, the decay properties of the kernel are obtained in [23] using the basic idea of [24, 25], that is,

$$ \frac{C_{1}}{|x|^{N+2s}}\leq\mathcal{K}(x)\leq\frac{C_{2}}{|x|^{N+2s}}, $$

(3.12)

for $|x|\geq1$ and $C_{2}>C_{1}>0$.

Proof of Lemma 3.10

First of all, we have the following claim.

• (i) There is a continuous function $v_{1}$ in $\mathbb{R}^{N}$ satisfying

  $$ (-\Delta)^{s}v_{1}+a v_{1}=0, \quad \mbox{if } |x|>1 $$

  (3.13)

  and

  $$ v_{1}(x)\geq\frac{C_{1}}{|x|^{N+2s}}, $$

  (3.14)

  for an appropriate $C_{1}>0$, where $a=\sup\widehat{V}_{\varepsilon}(x)$.

• (ii) There is a continuous function $v_{2}$ in $\mathbb{R}^{N}$ satisfying

  $$ (-\Delta)^{s}v_{2}+\tau v_{2}=0, \quad \mbox{if } |x|>1 $$

  (3.15)

  and

  $$ v_{2}(x)\leq\frac{C_{2}}{|x|^{N+2s}}, $$

  (3.16)

  for an appropriate $C_{2}>0$, where $\tau<\inf V(x)$.

Indeed, consider the function $v_{1}=\mathcal{K}_{1}\ast\mathcal {X}_{B_{1}}$, where $\mathcal{X}_{B_{1}}$ is the characteristic function of the unit ball $B_{1}$, and $\mathcal{K}_{1}=\mathscr{F}^{-1}(\frac{1}{a+|\xi |^{2s}})$ is a fundamental solution of $(-\Delta)^{s}+a$. Clearly $v_{1}$ satisfies equation (3.13) outside $B_{1}$ and the decaying estimate (3.14) thanks to (3.12).

Similarly, we consider the function $v=\mathcal{K}_{2}\ast\mathcal {X}_{B_{r}}$, where $B_{r}$ is the ball of radius $r=\tau^{\frac{1}{2s}}$ and $\mathcal{K}_{2}=\mathscr{F}^{-1}(\frac{1}{1+|\xi|^{2s}})$ is a fundamental solution of $(-\Delta)^{s}+1$. Then, by scaling, $v_{2}(x)=v(rx)$ satisfies equation (3.15) and using (3.12), we obtain (3.16).

By the continuity of $v_{\varepsilon}$ and $v_{1}$, there exists a constant $C_{1}>0$ so that $w_{\varepsilon}(y)=v_{\varepsilon}(y)-C_{1}v_{1}(y)\geq0$ in $\partial B_{1}$. Moreover, $((-\Delta)^{s}+a)w_{\varepsilon}(y)\geq0$ in $B_{1}^{c}$. By the maximum principle [20] we can conclude that $w_{\varepsilon}(y)\geq0$ in $B_{1}^{c}$. As a consequence, $v_{\varepsilon}(y)\geq\frac{C_{1}}{|y|^{N+2s}}$ for $|y|\geq1$, that is,

$$u_{\varepsilon}(x)\geq\frac{C_{1}}{|x-x_{\varepsilon}|^{N+2s}}. $$

On the other hand, the uniform decay estimate of $v_{\varepsilon}$ in Lemma 3.9, Step 5, and (f₁) allows us to take $R_{1}>0$ sufficiently large such that

$$(-\Delta)^{s}v_{\varepsilon}+\tau v_{\varepsilon}= \widehat{W}_{\varepsilon}(x)f(v_{\varepsilon})+\bigl(\tau- \widehat{V}_{\varepsilon}(x)\bigr)v_{\varepsilon}\leq0,\quad \mbox{in } B_{R_{1}}^{c}, $$

now we consider the function $v_{2}$ and the claim we found, which satisfies (3.15) in $B_{1}^{c}$ and then in $B_{R_{1}}^{c}$.

In view of the continuity of $v_{\varepsilon}$ and $v_{2}$, there exist constants $C_{2}>C_{1}>0$ such that

$$w_{\varepsilon}(y):=v_{\varepsilon}(y)-C_{2}v_{2}(y) \leq0,\quad \mbox{in } \partial B_{R_{1}}. $$

Moreover,

$$\bigl((-\Delta)^{s}+\tau\bigr)w_{\varepsilon}(y)\leq0, \quad \mbox{in } B_{R_{1}}^{c}. $$

Using a similar comparison argument, we conclude that $u_{\varepsilon}(x)\leq\frac{C_{2}}{|x-x_{\varepsilon}|^{N+2s}}$ for $|x|\geq R_{1}$ and all $\varepsilon>0$ small. The proof is completed. □

Proof of Theorem 1.1

(B) Define $\omega _{\varepsilon}(x)=u_{\varepsilon}(\frac{x}{\varepsilon})$. Then $\omega_{\varepsilon}$ is a solution of (1.1) for all $\varepsilon >0$. Since $z_{\varepsilon}$ is a maximum point of $|\omega_{\varepsilon}|$, we have

$$\frac{C_{1}\varepsilon^{N+2s}}{|x-z_{\varepsilon}|^{N+2s}}\leq\omega _{\varepsilon}(x)\leq\frac{C_{2}\varepsilon^{N+2s}}{|x-z_{\varepsilon}|^{N+2s}}, $$

for some constants $0< C_{1}< C_{2}$, and

$$\lim_{\varepsilon\rightarrow0} \operatorname{dist}( z_{\varepsilon}, \mathscr{A}_{w})=0. $$

Then we proceed similarly to (A). □

4 Study of the nonlocal critical problem (1.2)

In this section, we will prove Theorem 1.2. We only need to give the details for (A) because the argument for (B) is similar to that for (A).

Suppose that (f₁)-($\mathrm{f}'_{4}$) hold and let (P₀) and (P₁)(i) be satisfied; without loss of generality, we assume that $x_{v}=0\in \mathscr{V}$ ($x_{v}=0\in\mathscr{W}\cap\mathscr{V}$ if $\mathscr{V}\cap\mathscr{W}\neq\emptyset$) and $b:=W(0)=\min_{y\in\mathscr{V}}W(y)\leq W(x)$ for all $|x|\geq R$, then $V(0)=\tau$, $W(0)=b$. We remark by (P₀) that $V_{\varepsilon}(x)\rightarrow V(0) =\tau$, $W_{\varepsilon}(x)\rightarrow W(0)=b$ uniformly on bounded sets of $\mathbb{R}^{N}$ as $\varepsilon\rightarrow0$.

Plainly one only verifies that $I^{*}_{\varepsilon}$ possesses the mountain-pass structure as Lemma 3.1. As in Section 3 we define replacing $I_{\varepsilon}$ by $I^{*}_{\varepsilon}$ the notations: mountain-pass level $c^{*}_{\varepsilon}$; the Nehari manifold $\mathscr {N}^{*}_{\varepsilon}$; the critical set $\mathscr{H}^{*}_{\varepsilon}$, and the least energy solution set $\mathscr{R}^{*}_{\varepsilon}$. Observe that

$$c^{*}_{\varepsilon}=\inf\bigl\{ I^{*}_{\varepsilon}(u):u\in\mathscr {N}^{*}_{\varepsilon}\bigr\} $$

and $0< c^{*}_{\varepsilon}\leq c_{\varepsilon}$.

4.1 Autonomous equation

In this subsection, we give some results for the autonomous problem

$$ \varepsilon^{2s}(-\Delta)^{s}u+au=bh(u),\quad u\in H^{s}\bigl( \mathbb{R}^{N}\bigr), $$

(4.1)

which are useful for the study of the nonlocal critical problem (1.2). Here $h(u)=f(u)+u^{2^{*}_{s}-1}$, f as before satisfies the assumptions (f₁)-(f₄), $a,b>0$ are constants.

Consider the functional $I^{*}_{ab}:H^{s}( \mathbb{R}^{N})\rightarrow R$ associated to this equation,

$$I^{*}_{ab}(u):=\frac{1}{2}\bigl\vert (-\Delta)^{\frac{s}{2}}u \bigr\vert _{2}^{2}+\frac {a}{2}|u|_{2}^{2}-b \int_{ \mathbb{R}^{N}}F(u)\,dx-\frac{b}{2^{*}_{s}}|u|^{2^{*}_{s}}, $$

then $I^{*}_{ab}$ is of class $C^{1}$ on $H^{s}( \mathbb{R}^{N})$.

Let $c^{*}_{ab}$ be the mountain-pass level; $\mathscr{N}^{*}_{ab}$ the Nehari manifold; $\mathscr{H}^{*}_{ab}$ the critical set, $\gamma^{*}_{ab}$ the least energy, and $\mathscr{R}^{*}_{ab}$ the least energy solution set.

Let us notice that inequalities (3.1) and (3.2) imply that $I^{*}_{ab}$ satisfies the mountain-pass conditions, we can define the mountain-pass level

$$c^{*}_{ab}:=\inf_{\gamma\in\Gamma^{*}_{ab}}\max_{t\in [0,1]}I^{*}_{ab} \bigl(\gamma(t)\bigr), $$

where $\Gamma^{*}_{ab}:=\{\gamma\in C([a,b],H^{s}),\gamma (0)=0,I^{*}_{ab}(\gamma(1))<0\}$. It is easy to verify

$$c^{*}_{ab}=\inf_{u\in H^{s}\setminus\{0\}}\max_{t\geq 0}I^{*}_{ab}(tu)= \gamma^{*}_{ab}. $$

Next we show $\gamma^{*}_{ab}$ is attained, under proper assumptions for $a,b>0$.

Proposition 4.1

$\gamma^{*}_{ab}$ is attained if $\gamma^{*}_{ab}< l_{b}:= \frac{(S_{0})^{\frac{N}{2s}}\cdot s}{N\cdot b^{\frac{N-2s}{2s}}}$, where $S_{0}=k_{s} S(s,N)$ is the Sobolev constant as in (2.11).

Proof

Since $\gamma^{*}_{ab}=\inf_{u\in\mathscr {N}^{*}_{ab}}I^{*}_{ab}(u)$, let $\{u_{n}\}\in\mathscr{N}^{*}_{ab}$ be a minimizing sequence of $I^{*}_{ab}$, which is clearly a $(\mathit{PS})_{\gamma^{*}_{ab}}$ sequence: $I^{*}_{ab}(u_{n})\rightarrow\gamma^{*}_{ab}$ and $I^{*\prime}_{ab}(u_{n})\rightarrow0$ as $n\rightarrow\infty$.

Remark that $2<\mu<p<2^{*}_{s}$ and

$$\gamma^{*}_{ab}+o(1)=I^{*}_{ab}(u_{n})- \frac{1}{\mu }I^{*\prime}_{ab}(u_{n})u_{n} \geq\biggl(\frac{1}{2}-\frac{1}{\mu}\biggr)\|u_{n} \|_{H^{s}}^{2}. $$

Thus, $\{u_{n}\}$ is bounded in $H^{s}( \mathbb{R}^{N})$. By Lion’s concentration principle, $\{u_{n}\}$ is either vanishing or non-vanishing. Assume that $\{u_{n}\}$ is vanishing, then $|u_{n}|_{p}\rightarrow0$, $p\in(2,2^{*}_{s})$. Since $a>0$, by (f₁)-($\mathrm{f}'_{4}$), one gets

$$ o(1)=I^{*\prime}_{ab}(u_{n})u_{n}\geq\bigl\vert (-\Delta)^{\frac {s}{2}}u_{n}\bigr\vert _{2}^{2}-b|u_{n}|^{2^{*}_{s}}_{2^{*}_{s}}+o(1). $$

(4.2)

By the Sobolev embedding inequality (2.11),

$$ S_{0}|u_{n}|_{2^{*}_{s}}^{2}\leq\bigl\vert (-\Delta)^{\frac{s}{2}}u_{n}\bigr\vert _{2}^{2}. $$

(4.3)

Note that

$$ \gamma^{*}_{ab}+o(1)=I^{*}_{ab}(u_{n})- \frac{1}{2}I^{*\prime}_{ab}(u_{n})u_{n} \geq \frac{sb}{N}|u_{n}|^{2^{*}_{s}}_{2^{*}_{s}}. $$

(4.4)

It follows from (4.2)-(4.4) that

$$\gamma^{*}_{ab}\geq\frac{(S_{0})^{\frac{N}{2s}}\cdot s}{N\cdot b^{\frac {N-2s}{2s}}}, $$

a contradiction. Therefore, $\{u_{n}\}$ is non-vanishing, that is, there exist $\gamma,\delta>0$, and $x_{n}\in \mathbb{R}^{N}$ such that, setting $v_{n}(x)=u_{n}(x+x_{n})$, along a subsequence $\int_{B_{r}(0)}|v_{n}|^{2}\geq \delta$. Without loss of generality we assume $v_{n}\rightharpoonup v$. Then $v\neq0$ is a solution of (4.1), and so $\gamma^{*}_{ab}$ is attained. Thus, we complete the proof. □

Next we claim $\gamma^{*}_{ab}< l_{b}$ if $a< b\mathcal{R}_{q}$ where

$$\mathcal{R}_{q}:=\biggl(\frac{2qc_{0}^{\frac{2}{q-2}}s(S_{0})^{\frac {N}{2s}}}{S_{q}^{\frac{q}{q-2}}(q-2)N}\biggr)^{\frac{2s(q-2)}{2(sq+N)-Nq}}\quad \bigl(N>2s,2< q< 2^{*}_{s}\bigr). $$

Indeed, let u be a ground state of

$$ (-\Delta)^{s}u+au=bg(u) $$

(4.5)

with g as in (f₁), which is equivalent to

$$ (-\Delta)^{s}z+z=\frac{b}{a}g(z), $$

(4.6)

after a change of variable, $z(x)=u(\frac{x}{a^{\frac{1}{2s}}})$, with the least energy

$$\begin{aligned} \gamma_{\frac{b}{a}} =&\frac{1}{2}\bigl\vert (-\Delta)^{\frac {s}{2}}z \bigr\vert _{2}^{2}+\frac{1}{2}\vert z\vert _{2}^{2}-\frac{b}{a} \int_{ \mathbb {R}^{N}}G(z)\,dx \\ =&\frac{1}{2}\biggl\vert (-\Delta)^{\frac{s}{2}}u\biggl( \frac{x}{a^{\frac {1}{2s}}}\biggr)\biggr\vert _{2}^{2}+ \frac{1}{2}\biggl\vert u\biggl(\frac{x}{a^{\frac{1}{2s}}}\biggr)\biggr\vert _{2}^{2} -\frac{b}{a} \int_{ \mathbb{R}^{N}}G\biggl(u\biggl(\frac{x}{a^{\frac {1}{2s}}}\biggr)\biggr)\,dx \\ =&a^{\frac{N}{2s}-1}\gamma_{ab}. \end{aligned}$$

Denote

$$S_{q}:=\inf_{u\in H^{s}\setminus\{0\}}\frac{|(-\Delta)^{\frac {s}{2}}u|_{2}^{2}+|u|_{2}^{2}}{|u|_{q}^{2}}. $$

If $g(t)=c_{0}t^{q-2}$, then by the mountain-pass theorem, the least energy corresponding to (4.6) denoted by $\gamma_{\frac{b}{a}}(q)$ satisfies

$$\gamma_{\frac{b}{a}}(q)\leq\frac{q-2}{2q}\biggl(\frac{a}{bc_{0}} \biggr)^{\frac {2}{q-2}}S_{q}^{\frac{q}{q-2}}. $$

This implies that the least energy corresponding to (4.5) denoted by $\gamma_{ab}(q)$ satisfies

$$ \gamma_{ab}(q)=a^{\frac{2s-N}{2s}}\gamma_{\frac{b}{a}}(q)\leq \lambda(q,a,b):= a^{\frac{2s-N}{2s}}\cdot \frac{q-2}{2q}\biggl(\frac{a}{bc_{0}} \biggr)^{\frac{2}{q-2}}\cdot S_{q}^{\frac{q}{q-2}}. $$

(4.7)

Note that $l_{b}$ is strictly decreasing with respect to $b>0$, in virtue of (4.7), setting

$$\mathcal{R}_{q}:=\biggl(\frac{2qc_{0}^{\frac{2}{q-2}}s\cdot(S_{0})^{\frac {N}{2s}}}{S_{q}^{\frac{q}{q-2}}\cdot(q-2)N}\biggr) ^{\frac{2s(q-2)}{2(sq+N)-Nq}}, $$

if $a< b\mathcal{R}_{q}$, then $\gamma_{ab}(q)< l_{b}$. By ($\mathrm{f}'_{4}$), we have

$$\begin{aligned} I^{*}_{ab}(u) \leq&\frac{1}{2}\bigl|(-\Delta)^{\frac{s}{2}}u\bigr|_{2}^{2}+ \frac {a}{2}|u|_{2}^{2}-bc_{0}|u|_{q}^{q}- \frac{b}{2^{*}_{s}}|u|_{2^{*}_{s}}^{2^{*}_{s}} \\ \leq&\frac{1}{2}\bigl|(-\Delta)^{\frac{s}{2}}u\bigr|_{2}^{2}+ \frac {a}{2}|u|_{2}^{2}-bc_{0}|u|_{q}^{q}. \end{aligned}$$

This implies that $\gamma^{*}_{ab}\leq\gamma_{ab}(q)$ and thus $\gamma ^{*}_{ab}< l_{b}$ if $a< b\mathcal{R}_{q}$.

In order to prove Theorem 1.2, next we will be concerned with $a\in [\tau,\tau_{\infty}]$ and $b\in[\kappa_{\infty},\kappa]$. In particular we are looking for $\tau_{0}>\tau$ and $\kappa_{0}<\kappa$ such that

$$ \gamma_{ab}(q)< l_{k}\leq l_{b}, $$

(4.8)

for either $a\in[\tau,\tau_{0})$ and $b\in[\kappa_{\infty},\kappa]$ or $a\in[\tau,\tau_{\infty}]$ and $b\in(\kappa_{0},\kappa]$.

Observe that $\lambda(q,a,b)\leq\lambda(q,a,\kappa_{\infty})$ and $l_{b}\geq l_{\kappa}$ if $b\in[\kappa_{\infty},\kappa]$. Furthermore, $\lambda(q,a,b)< l_{\kappa}$ if

$$a< \tau_{0}:=\biggl(\frac{\kappa_{\infty}^{\frac{2}{q-2}}}{\kappa^{\frac {N-2s}{2s}}}\biggr)^{\frac{2s(q-2)}{2(sq+N)-Nq}} \mathcal{R}_{q}. $$

Also $\lambda(q,a,b)\leq\lambda(q,\tau_{\infty},b)$ and $l_{b}\geq l_{\kappa}$ if $b\in[\kappa_{\infty},\kappa]$. Also, $\lambda(q,\tau _{\infty},b)< l_{\kappa}$ if

$$b>\kappa_{0}:=\biggl(\frac{q-2}{2q}(\tau_{\infty})^{\frac {2(sq+N)-Nq}{2s(q-2)}}S_{q}^{\frac{q}{q-2}} \cdot\frac{N\kappa^{\frac {N-2s}{2s}}}{(S_{0})^{\frac{N}{2s}} \cdot s}c_{0}^{\frac{2}{2-q}}\biggr)^{\frac{q-2}{2}}. $$

In conclusion, (4.8) holds if either $\tau<\tau_{0}$, $a\in[\tau,\tau _{0})$, and $b\in[\kappa_{\infty},\kappa]$, or $\kappa>\kappa_{0}$, $b\in (\kappa_{0},\kappa]$ and $a\in[\tau,\tau_{\infty}]$.

From the above argument and Proposition 4.1, we have the following results.

Lemma 4.1

If $\tau<\tau_{0}$, $a\in[\tau, \tau_{0})$, and $b\in[\kappa_{\infty},\kappa]$, or $\kappa>\kappa_{0}$, $b\in (\kappa_{0},\kappa]$, and $a\in[\tau,\tau_{\infty}]$, then $\gamma^{*}_{ab}< l_{b}$ and, consequently, $\mathscr{R}^{*}_{ab}\neq\emptyset$, $\gamma^{*}_{ab}$ is attained.

The following comparison result is similar to the one in Lemma 3.5(iv).

Lemma 4.2

Let for $j=1,2$, either $\tau<\tau_{0}$, $a_{j}\in[\tau, \tau_{0})$, and $b_{j}\in[\kappa_{\infty},\kappa]$ or $\kappa>\kappa_{0}$, $b_{j}\in(\kappa_{0},\kappa]$, $a_{j}\in[\tau,\tau _{\infty}]$. Assume $\min\{a_{2}-a_{1},b_{1}-b_{2}\}\geq0$, then $\gamma^{*}_{a_{1}b_{1}}\leq\gamma^{*}_{a_{2}b_{2}}$.

4.2 The function $I_{\varepsilon}^{*}$

In this subsection, we will discuss the properties of the functional $I^{*}_{\varepsilon}$.

Plainly one easily verifies that $I^{*}_{\varepsilon}$ possesses the mountain-pass structure and

$$c^{*}_{\varepsilon}:=\inf_{\gamma\in\Gamma^{*}_{\varepsilon}}\max_{t\in [0,1]}I^{*}_{\varepsilon}\bigl(\gamma(t)\bigr) =\inf_{u\in H^{s}\setminus\{0\}}\max_{t\geq0}I^{*}_{\varepsilon}(tu)=\inf_{u\in\mathscr{N}^{*}_{\varepsilon}}I^{*}_{\varepsilon}(u), $$

where $c^{*}_{\varepsilon}$ is the mountain-pass minimax value associated with $I^{*}_{\varepsilon}$, $\Gamma^{*}_{\varepsilon}=\{\gamma\in C([0,1], H^{s}( \mathbb{R}^{N})):\gamma(0)=0,I^{*}_{\varepsilon}(\gamma(1))<0\}$.

Similar to the proof of Lemma 3.6, we also have the following energy comparison between $c^{*}_{\varepsilon}$ and $\gamma_{\tau b}$.

Lemma 4.3

If $\tau\in(0,\tau_{0})$, $b\in[\kappa _{\infty},\kappa]$, then $\limsup_{\varepsilon\rightarrow 0}c^{*}_{\varepsilon}\leq\gamma^{*}_{\tau b}< l_{\kappa}\leq l_{b}$.

4.3 Existence results

Lemma 4.4

If $\tau\in(0,\tau_{0})$, $c^{*}_{\varepsilon}$ is attained at some $u^{*}_{\varepsilon}\in\mathscr{R}_{\varepsilon}$ for all small $\varepsilon>0$.

Proof

Given $\varepsilon>0$, let $u^{*}_{k}\in \mathscr{N}^{*}_{\varepsilon}$ be a minimizing sequence of $I^{*}_{\varepsilon}$, which is clearly a $(\mathit{PS})_{c^{*}_{\varepsilon}}$ sequence for $I^{*}_{\varepsilon}$. It is easy to see that $\{u^{*}_{k}\}$ is bounded in $H^{s}( \mathbb{R}^{N})$. Assume that $u^{*}_{k}\rightharpoonup u^{*}_{\varepsilon}\in\mathscr {H}^{*}_{\varepsilon}$ in $H^{s}( \mathbb{R}^{N})$. If $u^{*}_{\varepsilon}\neq 0$, then clearly $I^{*}_{\varepsilon}(u^{*}_{\varepsilon})=c^{*}_{\varepsilon}$.

Next we check that $u^{*}_{\varepsilon}\neq0$ for all $\varepsilon>0$ small.

Assume that there exists a sequence $\varepsilon_{j}\rightarrow0$ with $u^{*}_{\varepsilon_{j}}=0$, then $u^{*}_{k}\rightharpoonup0$ in $H^{s}( \mathbb{R}^{N})$ and thus $u^{*}_{k}\rightarrow0$ in $L^{p}_{\mathrm{loc}}$ for $q\in(1,2^{*}_{s})$ and $u^{*}_{k}(x)\rightarrow0$ a.e. in $x\in \mathbb{R}^{N}$.

Since $\tau<\tau_{\infty}$ and $\tau<\tau_{0}$ by the assumptions, choose $\min\{\tau_{\infty},\tau_{0}\}>a>\tau$, and consider the functional $I_{\varepsilon}^{*ab}$ as in Lemma 3.7, let $t_{k}>0$ be such that $t_{k}u^{*}_{k}\in \mathscr{N}^{*ab}_{\varepsilon}$, then $\{t_{k}\}$ is bounded and we may assume $t_{k}\rightarrow t_{0}$ as $k\rightarrow\infty$. By (P₀), (P₁)(i), the set $O_{\varepsilon}:=\{x\in \mathbb{R}^{N}:V_{\varepsilon}(x)\leq a \mbox{ or } W_{\varepsilon}(x)> b\}$ is bounded. Remark that $I^{*}_{\varepsilon _{j}}(t_{k}u^{*}_{k})\leq I^{*}_{\varepsilon_{j}}(u^{*}_{k})$. We obtain

$$\begin{aligned} c^{*ab}_{\varepsilon_{j}} \leq& I^{*ab}_{\varepsilon_{j}} \bigl(t_{k}u^{*}_{k}\bigr) \\ \leq& I^{*}_{\varepsilon_{j}}\bigl(t_{k}u^{*}_{k}\bigr)+ \frac{1}{2} \int _{O_{\varepsilon_{j}}}\bigl(a-V_{\varepsilon_{j}}(x)\bigr)\bigl\vert t_{k}u^{*}_{k}\bigr\vert ^{2}\,dx \\ &{}+ \int_{O_{\varepsilon_{j}}} \bigl(W_{\varepsilon_{j}}(x)-b\bigr)H \bigl(t_{k}u^{*}_{k}\bigr)\,dx \\ \leq&I^{*}_{\varepsilon_{j}}\bigl(t_{k}u^{*}_{k} \bigr)+o(1)=c^{*}_{\varepsilon_{j}} \end{aligned}$$

as $k\rightarrow\infty$, hence $c^{*ab}_{\varepsilon_{j}}\leq c^{*}_{\varepsilon_{j}}$. Notice that $\gamma^{*}_{ab}\leq c^{*ab}_{\varepsilon_{j}}$, hence $\gamma^{*}_{ab}\leq c^{*}_{\varepsilon_{j}}$. In virtue of Lemma 4.3, letting $\varepsilon_{j}\rightarrow0$ yields

$$\gamma^{*}_{ab}\leq\gamma^{*}_{\tau b}, $$

which contradicts with $\gamma^{*}_{\tau b}<\gamma^{*}_{ab}$ (see Lemma 4.2). Therefore, $c^{*}_{\varepsilon}$ is attained at $0\neq u^{*}_{\varepsilon}\in\mathscr{R}^{*}_{\varepsilon}$, which ends the proof. □

4.4 Concentration and convergence of ground state

Lemma 4.5

Assuming (f₁)-($\mathrm{f}'_{4}$), (P₀), (P₁)(i) with $\tau<\tau_{0}$, and, for all ε sufficiently small, let $u^{*}_{\varepsilon}\in\mathscr{R}^{*}_{\varepsilon}$, then $u^{*}_{\varepsilon}$ possesses a (global) maximum $x_{\varepsilon}$ such that $\lim_{\varepsilon\rightarrow0}\operatorname{dist}(\varepsilon x_{\varepsilon}, \mathscr{A}_{w})=0$, and for any sequence $\varepsilon x_{\varepsilon}\rightarrow x_{0}$, $v^{*}_{\varepsilon}(x):=u^{*}_{\varepsilon}(x+x_{\varepsilon})$ converges in $H^{s}( \mathbb{R}^{N})$ to $u^{*}(x)$, which is a least energy solution of

$$(-\Delta)^{s}u+V(x_{0})u=W(x_{0})h(u), $$

note that $h(u)=f(u)+u^{2^{*}_{s}-1}$. In particular, $\mathscr{V}\cap\mathscr{W}\neq\emptyset$, then $\lim_{\varepsilon\rightarrow0}\operatorname{dist}(\varepsilon x_{\varepsilon}, \mathscr{V}\cap\mathscr{W})=0$, and up to subsequences, $v^{*}_{\varepsilon}$ converges in $H^{s}( \mathbb{R}^{N})$ to $u^{*}$ being a least energy solution of

$$(-\Delta)^{s}u+\tau u=\kappa h(u). $$

The proof of this lemma will be along the main lines of the proof of Theorem 1.1. We argue step by step.

Proof

Step 1. Let $u^{*}_{\varepsilon}\in H^{s}( \mathbb{R}^{N})$ be the critical point of $I^{*}_{\varepsilon}$ so that $I^{*}_{\varepsilon}(u^{*}_{\varepsilon})=c^{*}_{\varepsilon}$, we see that $\{u^{*}_{\varepsilon}\}$ is a bounded set in $H^{s}( \mathbb{R}^{N})$. Then $\{u^{*}_{\varepsilon}\}$ is non-vanishing.

Indeed, assume $\{u^{*}_{\varepsilon}\}$ is vanishing, then $|u^{*}_{\varepsilon}|_{p}\rightarrow0$ for $p\in(2,2^{*}_{s})$. By (f₁)-($\mathrm{f}'_{4}$), one gets

$$ o(1)=I^{*\prime}_{\varepsilon}\bigl(u^{*}_{\varepsilon}\bigr)u^{*}_{\varepsilon}\geq \bigl\vert (-\Delta)^{\frac{s}{2}}u_{\varepsilon}^{*} \bigr\vert _{2}^{2}- \int_{ \mathbb {R}^{N}}W_{\varepsilon}(x)\bigl\vert u^{*}_{\varepsilon}\bigr\vert ^{2^{*}_{s}}\,dx. $$

(4.9)

On the other hand

$$ c^{*}_{\varepsilon}+o(1)=I^{*}_{\varepsilon}\bigl(u^{*}_{\varepsilon}\bigr)- \frac {1}{2}I^{*\prime}_{\varepsilon}\bigl(u^{*}_{\varepsilon}\bigr)u^{*}_{\varepsilon}\geq\frac{s}{N} \int_{ \mathbb{R}^{N}}W_{\varepsilon}(x)\bigl\vert u^{*}_{\varepsilon}\bigr\vert ^{2^{*}_{s}}\,dx. $$

(4.10)

Recall that

$$ \int_{ \mathbb{R}^{N}}\bigl\vert (-\Delta)^{\frac{s}{2}}u^{*}_{\varepsilon}\bigr\vert ^{2}\,dx\geq S_{0}\biggl( \int_{ \mathbb{R}^{N}}\bigl\vert u^{*}_{\varepsilon}\bigr\vert ^{2^{*}_{s}}\,dx\biggr)^{\frac{2}{2^{*}_{s}}}. $$

(4.11)

It follows from (4.9)-(4.11) that

$$\liminf_{\varepsilon\rightarrow0}c^{*}_{\varepsilon}\geq l_{k}= \frac {N}{s}S_{0}^{\frac{N}{2s}}\kappa^{\frac{2s-N}{2s}}, $$

contradicting Lemma 4.3. Therefore $\{u^{*}_{\varepsilon}\}$ is non-vanishing, that is, there exist a sequence $\{x_{\varepsilon}\}\subset \mathbb{R}^{N}$ and constant $R>0$, $\sigma>0$ such that $\lim_{\varepsilon\rightarrow0}\int_{B_{R}(x_{\varepsilon})}u^{*2}_{\varepsilon}\geq\sigma$.

Set $v^{*}_{\varepsilon}(x):=u^{*}_{\varepsilon}(x+x_{\varepsilon})$, then $v^{*}_{\varepsilon}$ satisfies

$$ (-\Delta)^{s}v^{*}_{\varepsilon}+\widehat{V}_{\varepsilon}(x)v^{*}_{\varepsilon}=\widehat{W}_{\varepsilon}(x)h\bigl(v^{*}_{\varepsilon}\bigr), $$

(4.12)

where $\widehat{V}_{\varepsilon}(x)=V(\varepsilon(x+x_{\varepsilon}))$, $\widehat{W}_{\varepsilon}(x)=W(\varepsilon(x+x_{\varepsilon}))$, with energy

$$\begin{aligned} \hat{I}^{*}_{\varepsilon}\bigl(v^{*}_{\varepsilon}\bigr) =&\frac{1}{2} \bigl\vert (-\Delta )^{\frac{s}{2}}v^{*}_{\varepsilon}\bigr\vert ^{2}_{2} +\frac{1}{2} \int_{ \mathbb{R}^{N}}\widehat{V}_{\varepsilon}(x)v^{*2}_{\varepsilon}- \int_{ \mathbb{R}^{N}}\widehat{W}_{\varepsilon}(x)H\bigl(v^{*}_{\varepsilon}\bigr) \\ =&\hat{I}^{*}_{\varepsilon}\bigl(v^{*}_{\varepsilon}\bigr)-\frac{1}{2} \hat {I}^{*\prime}_{\varepsilon}\bigl(v^{*}_{\varepsilon}\bigr)v^{*}_{\varepsilon}\\ =& \int_{ \mathbb{R}^{N}}\widehat{W}_{\varepsilon}(x)\biggl[\biggl( \frac {1}{2}f\bigl(v^{*}_{\varepsilon}\bigr)v^{*}_{\varepsilon}-F \bigl(v^{*}_{\varepsilon}\bigr)\biggr)+ \biggl(\frac{1}{2}- \frac{1}{2^{*}_{s}}\biggr)\bigl\vert v^{*}_{\varepsilon}\bigr\vert ^{2^{*}_{s}}\biggr] \\ =&I^{*}_{\varepsilon}\bigl(u^{*}_{\varepsilon}\bigr)-\frac{1}{2}I^{*\prime}_{\varepsilon}\bigl(u^{*}_{\varepsilon}\bigr)u^{*}_{\varepsilon}=I^{*}_{\varepsilon}\bigl(u^{*}_{\varepsilon}\bigr)=c^{*}_{\varepsilon}. \end{aligned}$$

We may assume $v^{*}_{\varepsilon}\rightharpoonup u^{*}$ in $H^{s}( \mathbb {R}^{N})$, and $v^{*}_{\varepsilon}\rightarrow u^{*}$ in $L_{\mathrm{loc}}^{q}$ for $q\in [1,2^{*}_{s})$ with $u^{*}\neq0$.

Since $V,W\in L^{\infty}$, without loss of generality, we assume that $V(\varepsilon x_{\varepsilon})\rightarrow V_{0}$ and $W(\varepsilon x_{\varepsilon})\rightarrow W_{0}$ as $\varepsilon \rightarrow0$. It is easy to check that $\widehat{V}_{\varepsilon}(x)\rightarrow V_{0}$ and $\widehat{W}_{\varepsilon}(x)\rightarrow W_{0}$ as $\varepsilon \rightarrow0$ uniformly on bounded sets of $x\in \mathbb{R}^{N}$.

Consequently, by (4.12), for any $\varphi\in C_{0}^{\infty}( \mathbb{R}^{N})$,

$$\begin{aligned} 0 =&\lim_{\varepsilon\rightarrow0} \int_{ \mathbb{R}^{N}}\bigl((-\Delta )^{s}v^{*}_{\varepsilon}+ \widehat{V}_{\varepsilon}(x)v^{*}_{\varepsilon}-\widehat{W}_{\varepsilon}(x)h \bigl(v^{*}_{\varepsilon}\bigr)\bigr)\varphi \,dx \\ =& \int_{ \mathbb{R}^{N}}\bigl((-\Delta)^{s}u^{*}+V_{0}u^{*}-W_{0}h \bigl(u^{*}\bigr)\bigr)\varphi \,dx, \end{aligned}$$

which implies that $u^{*}$ solves

$$ (-\Delta)^{s}u^{*}+V_{0}u^{*}=W_{0}h\bigl(u^{*}\bigr), $$

(4.13)

with the energy

$$\begin{aligned} \Gamma^{*}_{V_{0}W_{0}}\bigl(u^{*}\bigr) :=&\frac{1}{2}\bigl|(- \Delta)^{\frac {s}{2}}u^{*}\bigr|_{2}^{2}+\frac{1}{2}V_{0} \int_{ \mathbb{R}^{N}}u^{*2}-W_{0} \int_{ \mathbb{R}^{N}}H\bigl(u^{*}\bigr) \\ =& \int_{ \mathbb{R}^{N}}W_{0}\biggl[\biggl(\frac{1}{2}f \bigl(u^{*}\bigr)u^{*}-F\bigl(u^{*}\bigr)\biggr)+ \biggl(\frac{1}{2}- \frac{1}{2^{*}_{s}}\biggr)\bigl|u^{*}\bigr|^{2^{*}_{s}}\biggr]\geq\gamma^{*}_{V_{0}W_{0}} \end{aligned}$$

by Fatou’s lemma and Lemma 4.3,

$$\begin{aligned} \gamma^{*}_{V_{0}W_{0}} \leq& \int_{ \mathbb{R}^{N}}W_{0}\biggl[\biggl(\frac {1}{2}f \bigl(u^{*}\bigr)u^{*}-F\bigl(u^{*}\bigr)\biggr)+ \biggl(\frac{1}{2}- \frac{1}{2^{*}_{s}}\biggr)\bigl\vert u^{*}\bigr\vert ^{2^{*}_{s}}\biggr] \\ \leq&\liminf_{\varepsilon\rightarrow0} \int_{ \mathbb {R}^{N}}\widehat{W}_{\varepsilon}(x)\biggl[\biggl( \frac{1}{2}f\bigl(v^{*}_{\varepsilon}\bigr)v^{*}_{\varepsilon}-F \bigl(v^{*}_{\varepsilon}\bigr)\biggr)+ \biggl(\frac{1}{2}- \frac{1}{2^{*}_{s}}\biggr)\bigl\vert v^{*}_{\varepsilon}\bigr\vert ^{2^{*}_{s}}\biggr] \\ =&\liminf_{\varepsilon\rightarrow0}\hat{I}^{*}_{\varepsilon}\bigl(v^{*}_{\varepsilon}\bigr)\leq\limsup_{\varepsilon\rightarrow 0}I^{*}_{\varepsilon}\bigl(u^{*}_{\varepsilon}\bigr) \leq\gamma^{*}_{V_{0}W_{0}}. \end{aligned}$$

Therefore,

$$\lim_{\varepsilon\rightarrow0}\hat{I}^{*}_{\varepsilon}\bigl(v^{*}_{\varepsilon}\bigr)=\lim_{\varepsilon\rightarrow0}c^{*}_{\varepsilon}=\gamma^{*}_{V_{0}W_{0}} \quad \mbox{and} \quad \Gamma^{*}_{V_{0}W_{0}}\bigl(u^{*}\bigr)= \gamma^{*}_{V_{0}W_{0}}. $$

As a consequence, $u^{*}$ is the least energy solution of the limit equation (4.4).

Step 2. $\{\varepsilon x_{\varepsilon}\}$ is bounded.

Assume that $\varepsilon|x_{\varepsilon}|\rightarrow+\infty$, by $W(\varepsilon x_{\varepsilon})\rightarrow W_{0}$, $b=W(0)\geq W(x)$, $|x|\geq R$, and $V(\varepsilon x_{\varepsilon})\rightarrow V_{0}$, we deduce that $V_{0}\geq\tau$ and $W_{0}< b$. So it follows from Lemma 4.2 that $\gamma^{*}_{V_{0}W_{0}}>\gamma^{*}_{\tau b}$.

However, by Step 1 and Lemma 4.3, $c^{*}_{\varepsilon}\rightarrow\gamma ^{*}_{V_{0}W_{0}}\leq\gamma^{*}_{\tau b}$, a contradiction. Therefore, we can assume $\varepsilon x_{\varepsilon}\rightarrow x_{0}$ (as $\varepsilon \rightarrow0$), then $V_{0}=V(x_{0})$, $W_{0}=W(x_{0})$, and we read (4.13) as

$$(-\Delta)^{s}u^{*}+V(x_{0})u^{*}=W(x_{0})h\bigl(u^{*} \bigr), $$

where $u^{*}$ is the least energy solution.

Step 3. $\varepsilon x_{\varepsilon}\rightarrow\mathscr{A}_{v}$ as $\varepsilon\rightarrow0$, that is, $x_{0}\in\mathscr{A}_{v}$.

Assume that $x_{0}\notin\mathscr{A}_{v}$, by the definition of $\mathscr {A}_{v}$, we have $W(x_{0})< W(0)=b$, which combined with $V(x_{0})>\tau$, leads to $\gamma^{*}_{V(x_{0})W(x_{0})}>\gamma^{*}_{a\kappa}$. However, by Lemma 4.3,

$$\lim_{\varepsilon\rightarrow0}c^{*}_{\varepsilon}=\gamma ^{*}_{V(x_{0})W(x_{0})}> \gamma^{*}_{\tau b}\geq\lim_{\varepsilon\rightarrow 0}c^{*}_{\varepsilon}, $$

a contradiction.

Step 4. Let $v^{*}_{\varepsilon}$, $u^{*}$ be defined in Step 1, then $v^{*}_{\varepsilon}\rightarrow u^{*}$ in $H^{s}( \mathbb{R}^{N})$. See Step 4 of the proof of Lemma 3.9.

Step 5. $v_{\varepsilon}(x)\rightarrow0$ as $|x|\rightarrow \infty$ uniformly for all small ε. See Step 5 of the proof of Lemma 3.9. □

4.5 Decay estimates

Now repeating the arguments of Lemma 3.10 we obtain the following lemma.

Lemma 4.6

There exist $0< C_{1}\leq C_{2}$ and $R>1$ such that, for all small $\varepsilon>0$,

$$\frac{C_{1}}{|x-x_{\varepsilon}|^{N+2s}}\leq u_{\varepsilon}(x)\leq\frac {C_{2}}{|x-x_{\varepsilon}|^{N+2s}}, $$

for all $|x|\geq R$.

Proof of Theorem 1.2

(A) Define $\omega_{\varepsilon}(x)=u_{\varepsilon}(\frac{x}{\varepsilon})$, then $\omega_{\varepsilon}$ is a solution of (1.2) for all $\varepsilon >0$. Since $z_{\varepsilon}$ is a maximum point of $|\omega_{\varepsilon}|$, we have

$$\frac{C_{1}\varepsilon^{N+2s}}{|x-z_{\varepsilon}|^{N+2s}}\leq\omega _{\varepsilon}(x)\leq\frac{C_{2}\varepsilon^{N+2s}}{|x-z_{\varepsilon}|^{N+2s}}, $$

for some constants $0< C_{1}< C_{2}$, and

$$\lim_{\varepsilon\rightarrow0} \operatorname{dist}( z_{\varepsilon}, \mathscr{A}_{v})=0. $$

Proceed similar to (B). □