Existence of positive solutions for a Schrödinger-Poisson system with bounded potential and weighted functions in $\mathbb{R}^{3}$

Abstract

In this paper we study the Schrödinger-Poisson system

$$ \textstyle\begin{cases} -\Delta u+V(x)u+K(x)\phi u=a(x)|u|^{m-2}u+\lambda b(x)|u|^{q-2}u, &\mbox{in } \mathbb {R}^{3} \\ -\Delta\phi=K(x)u^{2},\qquad \lim_{|x|\to\infty}\phi(x)=0, & \mbox{in } \mathbb {R}^{3}, \end{cases} $$

(0.1)

where the potential $V(x)$ and the weighted functions $a(x), b(x)$ are positive and bounded in $\mathbb {R}^{3}$, $K(x)\in L^{2}(\mathbb {R}^{3})\cup L^{\infty}(\mathbb {R}^{3})$ and $K(x)\ge0$ in $\mathbb {R}^{3}$. We prove the existence of a positive solution $(u,\phi)\in W^{1,2}(\mathbb {R}^{3})\times\mathcal{D}^{1,2}(\mathbb {R}^{3})$ for $4< q< m<2^{*}=6$ and $\lambda\in \mathbb {R}$.

Introduction and main results

In this paper, we study the existence of positive solutions for the Schrödinger-Poisson system

$$ \textstyle\begin{cases} -\Delta u+V(x)u+K(x)\phi u=a(x)|u|^{m-2}u+\lambda b(x)|u|^{q-2}u, & \mbox{in } \mathbb {R}^{3} \\ -\Delta\phi=K(x)u^{2}, \qquad \lim_{|x|\to\infty}\phi(x)=0, & \mbox{in } \mathbb {R}^{3} \end{cases} $$

(1.1)

where $V(x)$, $a(x)$ and $b(x)$ are positive and bounded in $\mathbb {R}^{3}$, $K(x)\in L^{2}(\mathbb {R}^{3})\cup L^{\infty}(\mathbb {R}^{3})$ and $K(x)\ge0$ in $\mathbb {R}^{3}$. We will prove the existence of a positive solution $(u,\phi)\in W^{1,2}(\mathbb {R}^{3})\times\mathcal{D}^{1,2}(\mathbb {R}^{3})$ for $\lambda\in \mathbb {R}$ and $4< q< m<2^{*}$, where $2^{*}=6$ is the critical exponent for the Sobolev embedding in dimension 3. The assumption ‘$4< q< m<6$’ implies that the nonlinear term $f(x,u)=a(x)|u|^{m-2}u+\lambda b(x)|u|^{q-2}u$ in (1.1) is superlinear, which is similar to those in [1].

Such a system, also known as the Schrödinger-Maxwell system, arises in many fields of physics. For example, the Schrödinger-Poisson system can describe the interaction of a charged particle with its own electrostatic field in quantum mechanics. The unknowns u and ϕ represent the wave functions associated with the particle and electric potential, and the functions V and K are, respectively, an external potential and nonnegative density charge. We refer to Benci and Fortunato [2] for more details on the physical aspects. This model can also appear in semiconductor theory to describe solitary waves [3].

In recent years, the Schrödinger-Poisson system

$$ \textstyle\begin{cases} -\Delta u+V(x)u+K(x)\phi u=f(x,u), & \mbox{in } \mathbb {R}^{3} \\ -\Delta\phi=K(x)u^{2},\qquad \lim_{|x|\to\infty}\phi(x)=0,& \mbox{in } \mathbb {R}^{3} \end{cases} $$

(1.2)

has been widely studied under various assumptions on V, K, and f via variational methods, and existence, nonexistence and multiplicity results have been obtained in many papers, see [4–10].

Very recently, Cerami and Vaira [11] considered problem (1.2) with $K(x)\in L^{2}(\mathbb {R}^{3})$. They proved that (1.2) with $V(x)=1$ and $f(x,u)=a(x)|u|^{p-1}u$ ($3< p<5$) possesses a positive ground state solution by minimization on the Nehari manifold when $a(x), K(x):\mathbb {R}^{3}\to \mathbb {R}$ are nonnegative functions such that

$$ \lim_{|x|\to\infty}a(x)=a_{\infty}>0,\qquad \lim _{|x|\to\infty}K(x)=0. $$

(1.3)

Similar results for $V(x)=\lambda>0$ can be found in [4, 12, 13].

Liu et al. [14] also considered the existence of a solution for problem (1.2) with the potential $V(x)\in C(\mathbb{R}^{3})$ satisfying $\inf_{x\in\mathbb{R}^{3}}V(x)>-\infty$ and for every $M>0$, $\operatorname{meas}(\{x\in\mathbb{R}^{3} |V(x)\le M\})<\infty$. It is well known that this assumption guarantees that the embedding $W^{1,2}(\mathbb{R}^{3})\hookrightarrow L^{p}(\mathbb{R}^{3})$ is compact for each $2\le p<6$. For problem (1.2), the function $f(x,u)$ verifies $uf(x,u)\ge4F(x,u)$ with $F(x,u)=\int_{0}^{u}f(x,t)dt$. Similar assumptions can be found in [15–19].

However, to the best of our knowledge, there are few results on problem (1.1) when the potential $V(x)$ and the weighted functions $a(x)$, $b(x)$ are bounded in $\mathbb {R}^{N}$. In this paper, we are interested in the existence of a solution to problem (1.1) with $V(x)$ satisfying

(H₁):

The function $V(x)\in C(\mathbb{R}^{3})$ and $0<\alpha _{0}:=\inf_{x\in\mathbb{R}^{3}}V(x)<\sup_{x\in\mathbb {R}^{3}}V(x)=:\alpha<\infty$;

(H₂):

$\lim_{|x|\to\infty}V(x)=\alpha$.

Clearly, $V(x)$ is not necessarily radial and coercive. For these assumptions, the embedding $W^{1,2}(\mathbb{R}^{3})\hookrightarrow L^{p}(\mathbb{R}^{3})$ is not compact. Furthermore, for problem (1.1), the function $f(x,u)$ fails to satisfy the assumption $uf(x,u)\ge4F(x,u)$. So the variational technique for problem (1.1) becomes more delicate. Arguing as in [20, 21], to preserve this compactness in some extent for our problem, we split a minimizing sequence $\{u_{n}\}$ into two parts: $u_{n}=u'_{n}+u''_{n}$ ($n\in \mathbb{N}$) such that $u'_{n} \to u, u''_{n}\to0$ in $L^{m}(\mathbb{R}^{3},a)\cap L^{q}(\mathbb{R}^{3},b)$. We will obtain a positive solution by using the Nehari manifold method.

In order to state our main results, we introduce some Sobolev spaces and norms. For $p\ge1$, let $L^{p}(\mathbb{R}^{3})$ be a usual Lebesgue space with the norm $\|\cdot\|_{p}$. Denote

$$ E= \biggl\{ u\in W^{1,2}\bigl(\mathbb{R}^{3}\bigr) : \int_{\mathbb{R}^{3}}\bigl(|\nabla u|^{2}+V(x)|u|^{2} \bigr)\,dx< +\infty \biggr\} $$

endowed with the norm

$$ \|u\|_{E}= \biggl( \int_{\mathbb{R}^{3}}\bigl(|\nabla u|^{2}+V(x)|u|^{2} \bigr)\,dx \biggr)^{1/2}. $$

This norm is equivalent to the standard norm on $W^{1,2}(\mathbb{R}^{3})$ under assumption (H₁).

In general, let $\|u\|_{p,\rho}=(\int_{\mathbb {R}^{3}}\rho|u|^{p}\,dx)^{1/p}$ with $p\ge1$ and $\rho=\rho(x)\ge0, \neq0$ a.e. in $\mathbb {R}^{3}$. In particular, denote $\|u\|_{p}=(\int_{\mathbb {R}^{3}}|u|^{p}\,dx)^{1/p}$ or $\|u\|_{L^{p}(\Omega)}=(\int_{\Omega}|u|^{p}\,dx)^{1/p}$ with the domain $\Omega\subset \mathbb {R}^{3}$.

Let $\mathcal{D}^{1,2}(\mathbb {R}^{3})$ be the completion of $C_{0}^{\infty}(\mathbb {R}^{3})$ with respect to the norm

$$ \|u\|_{\mathcal{D}}^{2}=\|u\|_{\mathcal{D}^{1,2}}^{2}= \int_{\mathbb {R}^{3}}|\nabla u|^{2}\,dx. $$

The following Sobolev inequality [22] is well known. There is a constant $S>0$ such that for every $u\in \mathcal{D}^{1,2}(\mathbb{R}^{3})$,

$$ \|u\|_{6}\le S\|\nabla u\|^{2}_{2}=S \|u\|_{\mathcal{D}}^{2}. $$

(1.4)

Hence, inequality (1.4) holds in $W^{1,2}(\mathbb{R}^{N})$ and E. Furthermore, there exists $S_{p}>0$ such that for $2\le p\le 6$,

$$ \|u\|_{p}\le S_{p}\|u\|_{E}, \quad \forall u\in E. $$

(1.5)

It is well known that problem (1.1) can be reduced to a single equation with a nonlocal term, see [11, 23]. In fact, for every $u\in E$, we define the linear functional $\mathcal{L}_{u}$ by

$$ \mathcal{L}_{u}(v)= \int_{\mathbb {R}^{3}}K(x)u^{2}v\,dx,\quad \forall v\in \mathcal{D}^{1,2}\bigl(\mathbb {R}^{3}\bigr). $$

(1.6)

If $K(x)\in L^{\infty}(\mathbb {R}^{3})$, by the Hölder inequality and the Sobolev inequalities (1.4) and (1.5), we get

$$ \bigl\vert \mathcal{L}_{u}(v) \bigr\vert \le\|K \|_{\infty}\|u\|_{12/5}^{2}\|v\|_{6}\le S\|K \|_{\infty}\|u\|_{12/5}^{2}\|v\|_{\mathcal{D}},\quad \forall v\in \mathcal{D}^{1,2}(\mathbb {R}). $$

(1.7)

Similarly, if $K\in L^{2}(\mathbb {R}^{3})$, we have

$$ \bigl\vert \mathcal{L}_{u}(v) \bigr\vert \le\|K \|_{2}\|u\|_{6}^{2}\|v\|_{6}\le S\|K \|_{2}\|u\|_{6}^{2}\|v\|_{\mathcal{D}}. $$

(1.8)

Hence, by the Lax-Milgram theorem, there exists a unique $\phi_{u}\in \mathcal{D}^{1,2}(\mathbb {R}^{3})$ (see [11, 22]) such that

$$ -\Delta\phi_{u}=K(x)u^{2},\quad \mbox{in } \mathbb{R}^{3}. $$

(1.9)

Moreover, $\phi_{u}$ has the following integral expression:

$$ \phi_{u}(x)=\frac{1}{4\pi} \int_{\mathbb {R}^{3}}\frac{K(y)u^{2}(y)}{|x-y|}\,dy,\quad \mbox{in } \mathbb {R}^{3} $$

(1.10)

and $\phi_{u}(x)>0$ in $\mathbb {R}^{3}$ if $u\neq0$ and $K(x)\ge0$. Therefore, we have from (1.10) that

$$ \frac{1}{4\pi} \int_{\mathbb {R}^{3}} \int_{\mathbb {R}^{3}}\frac {K(x)K(y)}{|x-y|}u^{2}(x)u^{2}(y) \,dx\,dy= \int_{\mathbb {R}^{3}}K(x)\phi_{u}(x)u^{2}(x)\,dx. $$

(1.11)

If $K\in L^{\infty}(\mathbb {R}^{3})$, it follows (1.6)-(1.9) that

$$ \begin{aligned}[b] \|\nabla\phi_{u}\|_{2}^{2}&= \int_{\mathbb {R}^{3}}K(x)\phi_{u}u^{2}\,dx\le d_{0} \int_{\mathbb {R}^{3}}\phi_{u}u^{2}\,dx \\ &\le d_{0}\|\phi_{u}\|_{6}\|u\|_{\frac{12}{5}}^{2} \le d_{0}S\|\nabla\phi_{u}\|_{2}\|u \|_{E}^{2}, \end{aligned} $$

(1.12)

and if $K\in L^{2}(\mathbb {R}^{3})$, we have

$$ \|\nabla\phi_{u}\|_{2}^{2}= \int_{\mathbb {R}^{3}}K(x)\phi_{u}u^{2}\,dx\le \|K \|_{2}\|\phi_{u}\|_{6}\|u\|_{6}^{2} \le S \|K\|_{2}\|\nabla\phi_{u}\|_{2}\|u \|_{E}^{2}, $$

(1.13)

where and in the sequel, $d_{0}=\|K\|_{\infty}$. So, it follows from (1.12) and (1.13) that there exists a constant $c_{1}>0$ such that

$$ \|\phi_{u}\|_{\mathcal{D}}^{2}\le c_{1}\|u\|_{E}^{4},\quad \forall u\in E, \mbox{if } K\in L^{\infty}\bigl(\mathbb {R}^{3}\bigr)\cup L^{2}\bigl( \mathbb {R}^{3}\bigr). $$

(1.14)

Furthermore, inserting $\phi_{u}$ into the first equation in (1.1), we obtain

$$ -\Delta u+V(x)u+K(x)\phi_{u}u=a(x)|u|^{m-2}u+ \lambda b(x)|u|^{q-2}u,\quad \mbox{in } \mathbb {R}^{3}. $$

(1.15)

Let $J(u):E\to \mathbb {R}$ be the energy functional associated to (1.15) defined by

$$ J(u)=\frac{1}{2}\|u\|_{E}^{2}+ \frac{1}{4}F(u)-\frac{1}{m} \int_{\mathbb {R}^{3}}a(x)|u|^{m}\,dx-\frac{\lambda}{q} \int_{\mathbb {R}^{3}}b(x)|u|^{q}\,dx,\quad u\in E, $$

(1.16)

where

$$ F(u)= \int_{\mathbb {R}^{3}}K(x)\phi_{u}(x)u^{2}(x)\,dx. $$

(1.17)

The following assumptions will be used in this paper.

(H₃):

The parameters q, m and λ satisfy $4< q< m<6$ and $\lambda\in \mathbb {R}$.

(H₄):

The function $K(x)\in L^{\infty}(\mathbb {R}^{3})\cup L^{2}(\mathbb {R}^{3})$ and $K(x)\ge0$ in $\mathbb {R}^{3}$.

(H₅):

The functions $a(x),b(x)\in C(\mathbb {R}^{3})$ satisfy $a_{0}\le a(x),b(x)\le a_{1}$ in $\mathbb {R}^{3}$ for some constants $a_{0},a_{1}>0$.

Under assumptions (H₁)-(H₅), it is easy to verify $J\in C^{1}(E,\mathbb {R})$, and for any $v \in E$, there holds

$$\begin{aligned} J'(u)v =& \int_{\mathbb{R}^{3}}\bigl(\nabla u\nabla v+V(x)uv\bigr)\,dx+ \int_{\mathbb{R}^{3}}K(x)u\phi_{u}v\,dx \\ &{}- \int_{\mathbb{R}^{3}}\bigl(a(x)|u|^{m-2}+\lambda b(x)|u|^{q-2}\bigr)uv\,dx. \end{aligned}$$

(1.18)

Hence, if $u\in E$ is a critical point of J, that is, $J'(u)v=0$ for $\forall v\in E$, then the pair $(u,\phi_{u})$ is a solution of problem (1.1)(see [4, 23]). For the sake of simplicity in many cases, we just say that $u\in E$, instead of $(u,\phi_{u})\in E\times\mathcal{D}^{1,2}(\mathbb {R}^{3})$, is a weak solution of problem (1.1).

Our main result in this paper is as follows.

Theorem 1.1

Assume that (H₁)-(H₅) hold. Then problem (1.1) admits at least a positive solution $u\in E$.

Open problem

For $K(x)\in L_{\mathrm{loc}}^{\infty}(\mathbb{R}^{3})$, does equation (1.1) admit a positive solution $u\in E$? Denis and Carlo in [1] considered this problem with unbounded and vanishing potentials $V(x)$. As far as we know, there is no result on the existence of positive solutions for (1.1) in the case $K(x)\in L_{\mathrm{loc}}^{\infty}(\mathbb{R}^{3})$ and $V(x)$, $a(x)$, $b(x)$ satisfy (H₁), (H₂), (H₃), (H₅). Hence, this case should be also an interesting topic for future research.

This paper is organized as follows. In Section 2, we set up the variational framework and establish some lemmas, which will be used in the proof of Theorem 1.1. In Section 3, we prove Theorem 1.1.

Preliminaries

In this section, we are going to establish a series of lemmas to prove Theorem 1.1, in which we tacitly assume that the conditions in Theorem 1.1 are satisfied. We first set up the variational framework for problem (1.1).

Let $J(u):E\rightarrow\mathbb{R}$ be the energy functional associated with problem (1.1) defined by (1.16), and its Gateaux derivative is given by (1.18).

Since the functional J is not bounded from below on E, a good candidate of an appropriate subset to study J is the so-called Nehari manifold for problem (1.1):

$$ \mathcal{N}= \bigl\{ u\in E\setminus\{0\} : J'(u)u=0 \bigr\} = \bigl\{ u\in E\setminus\{0\} : \|u\|_{E}^{2}+F(u)= \|u\|_{m,a}^{m}+\lambda\|u\|_{q,b}^{q} \bigr\} . $$

(2.1)

Notice that, if $u\in\mathcal{N}$, then

$$\begin{aligned} J(u) =&\biggl(\frac{1}{2}-\frac{1}{m}\biggr)\|u \|_{E}^{2}+\biggl(\frac{1}{4}-\frac {1}{m} \biggr)F(u)+\biggl(\frac{1}{m}-\frac{1}{q}\biggr)\lambda\|u \|_{q,b}^{q} \\ =&\biggl(\frac{1}{2}-\frac{1}{q}\biggr)\|u\|_{E}^{2}+ \biggl(\frac{1}{4}-\frac {1}{q}\biggr)F(u)+\biggl(\frac{1}{q}- \frac{1}{m}\biggr)\|u\|_{m,a}^{m} \\ =&\frac{1}{2}\|u\|_{E}^{2}+\biggl( \frac{1}{4}-\frac{1}{m}\biggr)\|u\|_{m,a}^{m}+ \biggl(\frac {1}{4}-\frac{1}{q}\biggr)\lambda\|u\|_{q,b}^{q}. \end{aligned}$$

(2.2)

Lemma 2.1

The Nehari manifold $\mathcal{N}\neq\emptyset$.

Proof

Let $u\in E$, $u\not\equiv0$ in $\mathbb{R}^{3}$. We consider the function

$$ h(t)=J'(tu)tu=t^{2}\|u \|_{E}^{2}+t^{4}F(u)-t^{m}\|u \|_{m,a}^{m}-\lambda t^{q} \|u\|_{q,b}^{q}, \quad t>0. $$

(2.3)

Since $4< q< m$, it follows that $h(t)>0$ for small $t>0$ and $h(t)\to-\infty$ as $t\to\infty$. Then there exists $t_{1}>0$ such that $h(t_{1})=0$. Obviously, $t_{1}u\not\equiv0$. Thus, we conclude that $t_{1}u\in\mathcal{N}$ and $\mathcal{N}\neq\emptyset$. □

Lemma 2.2

The functional J is coercive and bounded from below on $\mathcal{N}$. Moreover,

$$ d=\inf_{u\in\mathcal{N}}J(u)>0. $$

(2.4)

Proof

Let $u\in\mathcal{N}$. Then, from (1.4) and (2.1), it follows that

$$ \|u\|_{E}^{2}\le\|u\|_{E}^{2}+F(u)= \|u\|_{m,a}^{m}+\lambda\|u\|_{q,b}^{q}\le c_{1}\bigl(\|u\|_{E}^{m}+\|u\|_{E}^{q} \bigr) $$

(2.5)

and

$$ 1\le c_{1}\bigl(\|u\|_{E}^{m-2}+\|u \|_{E}^{q-2}\bigr) $$

(2.6)

with some $c_{1}>1$. If $\|u\|_{E}\le1$, inequality (2.6) implies $1\le2c_{1}\|u\|_{E}^{q-2}$. Hence,

$$ \|u\|_{E}\ge (2c_{1})^{\frac{1}{2-q}}. $$

(2.7)

If $\|u\|_{E}\ge1$, inequality (2.6) gives $1\le 2c_{1}\|u\|_{E}^{m-2}$ and

$$ \|u\|_{E}\ge\max\bigl\{ 1, (2c_{1})^{\frac{1}{2-m}} \bigr\} . $$

(2.8)

Hence, if $u\in\mathcal{N}$, it follows from (2.7) and (2.8) that there exists $c_{2}>0$ such that

$$ \|u\|_{E}\ge c_{2},\quad \forall u\in \mathcal{N}. $$

(2.9)

Moreover, it follows from (2.2) that

$$\begin{aligned} J(u)&=\biggl(\frac{1}{2}-\frac{1}{q}\biggr)\|u \|_{E}^{2}+\biggl(\frac{1}{4}-\frac {1}{q} \biggr)F(u)+\biggl(\frac{1}{q}-\frac{1}{m}\biggr)\|u \|_{m,a}^{m} \\ &\ge\biggl(\frac{1}{2}-\frac{1}{q} \biggr)\|u\|_{E}^{2}\ge \biggl(\frac{1}{2}- \frac{1}{q}\biggr)c_{2}^{2}\equiv c_{3}. \end{aligned}$$

This shows that the functional J is coercive and bounded from below on $\mathcal{N}$ and $d\ge c_{3}>0$. Then the proof of Lemma 2.2 is completed. □

Let $\{u_{n}\}$ be a minimizing sequence for d in $\mathcal{N}$, that is, $J(u_{n})\to d$ as $n\to\infty$ and

$$ \|u_{n}\|_{E}^{2}+F(u_{n})= \|u_{n}\|_{m,a}^{m}+\lambda\|u_{n} \|_{q,b}^{q}, \quad \forall n\in\mathbb{N}. $$

(2.10)

Furthermore, it follows from (2.2) that

$$ J(u_{n})\ge \biggl(\frac{1}{2}-\frac{1}{q} \biggr)\|u_{n}\|_{E}^{2},\quad \forall n\in \mathbb{N}. $$

(2.11)

This shows that $\{u_{n}\}$ is bounded in E and, from (2.10), $\{u_{n}\}$ is bounded in $L^{m}(\mathbb{R}^{3},a)$ and $L^{q}(\mathbb{R}^{3},b)$. Therefore, up to a subsequence, there exists $u\in E$ such that as $n\to\infty$,

$$\begin{aligned} \textstyle\begin{cases} u_{n}\rightharpoonup u \quad \mbox{weakly in } E, L^{m}(\mathbb{R}^{3},a), L^{q}(\mathbb{R}^{3},b); \\ u_{n}\to u\quad \mbox{strongly in } L^{p}_{\mathrm{loc}}(\mathbb{R}^{3}), 1\le p< 6, \\ u_{n}(x)\to u(x)\quad \mbox{a.e. in } \mathbb{R}^{3};\qquad \|u_{n}\|_{E}, \|u_{n}\|_{p} \le M_{0},\quad \forall n\ge1, 2\le p\le6, \end{cases}\displaystyle \end{aligned}$$

(2.12)

with some $M_{0}>0$. Since $J(u_{n})=J(|u_{n}|)$, we assume $u_{n}(x)\ge0$ a.e. in $\mathbb{R}^{3}$ for every $n\ge1$ and thus $u(x)\ge0$ a.e. in $\mathbb{R}^{3}$. By the weak lower semi-continuity of the norm, we get

$$ \|u\|_{m,a}^{m}\le\beta:=\liminf _{n\to\infty}\|u_{n}\|_{m,a}^{m}. $$

(2.13)

By extracting a further subsequence, if necessary, we assume

$$ \beta=\lim_{n\to\infty}\|u_{n} \|_{m,a}^{m}. $$

(2.14)

Denote

$$ \theta=\|u\|_{m,a}^{m}= \int_{\mathbb{R}^{3}}a(x)|u|^{m}\,dx,\qquad \sigma=\|u \|_{q,b}^{q}= \int_{\mathbb{R}^{3}}b(x)|u|^{q}\,dx. $$

(2.15)

By weak convergence, it is obvious that $\theta\in[0, \beta]$. First, we have the following.

Lemma 2.3

There results $\beta>0$.

Proof

Obviously, $\beta\ge0$. If $\beta=0$, we have $\lim_{n\to\infty}\|u_{n}\|_{m,a}^{m}=0$. Let $t\in(0,1)$ be such that $q=2t+(1-t)m$. Then, by the Hölder inequality and (H₅), we derive

$$ \|u_{n}\|_{q,b}^{q}\le b_{1}\|u_{n}\|_{2}^{2t} \|u_{n}\|_{m,a}^{m(1-t)},\quad \forall n\in\mathbb{N} $$

(2.16)

with the constant $b_{1}=a_{0}^{t-1}a_{1}$. Since $\{u_{n}\}$ is bounded in E, so it is bounded in $L^{p}(\mathbb{R}^{3})$ ($2\le p\le6$), and it follows from (2.16) that $\lim_{n\to\infty}\|u_{n}\|_{q,b}^{q}=0$. Then (2.10) implies that $u_{n}\to0$ in E and $F(u_{n})\to0$ as $n\to\infty$ and so $d=0$. By Lemma 2.2, it is impossible. The proof of Lemma 2.3 is completed. □

Lemma 2.4

If $\beta=\theta$, then $u\in\mathcal{N}$ and $J(u)=d$.

Proof

If $\beta=\theta$, then $\lim_{n\to\infty}\|u_{n}\|_{m,a}^{m}=\|u\|_{m,a}^{m}$. Since $u_{n}\rightharpoonup u$ in $L^{m}(\mathbb{R}^{3})$, it follows from the Brezis-Lieb lemma that $u_{n}\to u$ in $L^{m}(\mathbb{R}^{3},a)$. Similar to (2.16), we have

$$ \|u_{n}-u\|_{q,b}^{q}\le b_{1}\|u_{n}-u\|_{2}^{2t} \|u_{n}-u\|_{m,a}^{m(1-t)},\quad \forall n\ge1, $$

(2.17)

and $u_{n}\to u$ in $L^{q}(\mathbb{R}^{3},b)$. Hence, by the weak lower semi-continuity of the norm, we obtain

$$\begin{aligned} J(u)&=\frac{1}{2}\|u\|_{E}^{2}+\frac{1}{4}F(u)- \frac{1}{m}\|u\| _{m,a}^{m}-\frac{\lambda}{q}\|u \|_{q,b}^{q} \\ &\le \frac{1}{2}\liminf_{n\to\infty}\|u_{n} \|_{E}^{2}+\frac{1}{4}\liminf_{n\to\infty}F(u_{n}) -\frac{1}{m}\lim_{n\to\infty}\|u_{n} \|_{m,a}^{m}-\frac{\lambda }{q}\lim_{n\to\infty} \|u_{n}\|_{q,b}^{q} \\ &\le\liminf_{n\to\infty} \biggl(\frac{1}{2} \|u_{n}|_{E}^{2}+\frac{1}{4}F(u_{n}) -\frac{1}{m}\|u_{n}\|_{m,a}^{m}- \frac{\lambda}{q}\|u_{n}\|_{q,b}^{q} \biggr) \\ &= \liminf_{n\to\infty}J(u_{n})=d. \end{aligned}$$

(2.18)

Furthermore, we have from (2.10) that

$$ \|u\|_{E}^{2}+F(u)\le\|u\|_{m,a}^{m}+ \lambda\|u\|_{q,b}^{q}. $$

(2.19)

If the equality in (2.19) holds, then $u\in\mathcal{N}$ and the lemma is proved.

We now assume that the equality in (2.19) fails to hold. Let

$$ h(t)=J'(tu) (tu)=t^{2}\|u \|_{E}^{2}+t^{4}F(u)- t^{m}\|u \|_{m,a}^{m}-\lambda t^{q}\|u\|_{q,b}^{q}, \quad t\ge0. $$

(2.20)

Clearly, $h(t)>0$ for small $t>0$ and $h(1)<0$. Then there exists $t\in(0,1)$ such that $h(t)=0$, and then $tu\in\mathcal{N}$ and

$$\begin{aligned} d&\le J(tu)= p_{1}t^{2}\|u\|_{E}^{2}+p_{2}t^{4}F(u)+p_{3}t^{m} \|u\|_{m,a}^{m} \\ &< p_{1}\|u\|_{E}^{2}+p_{2}F(u)+p_{3} \|u\|_{m,a}^{m}\le\liminf_{n\to \infty}J(u_{n})=d. \end{aligned}$$

(2.21)

Here and in the sequel,

$$ p_{1}=2^{-1}-q^{-1}>0,\qquad p_{2}=4^{-1}-q^{-1}>0,\qquad p_{3}=q^{-1}-m^{-1}>0. $$

(2.22)

Relation (2.21) is a contradiction, and thus the proof of Lemma 2.4 is completed. □

We now turn to study the value of θ. First we introduce the Sobolev space

$$ Y= \bigl\{ u\in W^{1,2}\bigl(\mathbb{R}^{3} \bigr) : \|\nabla u\|_{2}^{2}+\alpha\|u\|_{2}^{2}< \infty \bigr\} $$

(2.23)

endowed with the norm

$$ \|u\|_{Y}:=\bigl(\|\nabla u\|_{2}^{2}+ \alpha\|u\|_{2}^{2}\bigr)^{1/2}= \biggl( \int_{\mathbb{R}^{3}}\bigl(|\nabla u|^{2}+\alpha|u|^{2} \bigr)\,dx \biggr)^{1/2}, $$

(2.24)

where α is the positive number defined in (H₂). Consider the Schrödinger-Poisson system

$$ \textstyle\begin{cases} -\Delta u+\alpha u+K(x)\phi u=a(x)|u|^{m-2}u+\lambda b(x)|u|^{q-2}u, & \mbox{in } \mathbb {R}^{3}, \\ -\Delta\phi=K(x)u^{2},\qquad \lim_{|x|\to\infty}\phi(x)=0, &\mbox{in } \mathbb {R}^{3}. \end{cases} $$

(2.25)

The functional associated with problem (2.25) is

$$ J_{\alpha}(u)=\frac{1}{2}\|u\|_{Y}^{2}+ \frac{1}{4}F(u)-\frac{1}{m}\|u\| _{m,a}^{m}- \frac{\lambda}{q}\|u\|_{q,b}^{q}, $$

(2.26)

and the associated Nehari manifold is

$$ \mathcal{N}_{\alpha}= \bigl\{ u\in Y\setminus\{0\} : J'_{\alpha}(u)u=0 \bigr\} = \bigl\{ u\in Y\setminus\{0\} : \|u \|_{Y}^{p}+F(u)=\|u\|_{m,a}^{m}+\lambda\|u \|_{q,b}^{q} \bigr\} . $$

(2.27)

Notice that $u\in\mathcal{N}_{\alpha}$,

$$\begin{aligned} J_{\alpha}(u)&=\biggl(\frac{1}{2}-\frac{1}{m}\biggr)\|u \|_{Y}^{2}+\biggl(\frac {1}{4}-\frac{1}{m} \biggr)F(u)+\biggl(\frac{1}{m}-\frac{1}{q}\biggr)\lambda\|u \|_{q,b}^{q} \\ &=\biggl(\frac{1}{2}-\frac{1}{q}\biggr)\|u\|_{Y}^{2}+ \biggl(\frac{1}{4}-\frac {1}{q}\biggr)F(u)+\biggl(\frac{1}{q}- \frac{1}{m}\biggr)\|u\|_{m,a}^{m} \\ &=\frac{1}{2}\|u\|_{Y}^{2}+\biggl(\frac{1}{4}- \frac{1}{m}\biggr)\|u\|_{m,a}^{m}+\biggl( \frac {1}{4}-\frac{1}{q}\biggr)\lambda\|u\|_{q,b}^{q}. \end{aligned}$$

(2.28)

Finally, we define

$$ d_{\alpha}=\inf_{u\in\mathcal{N}_{\alpha}}J_{\alpha}(u). $$

(2.29)

Lemma 2.5

There results $d< d_{\alpha}$, that is, $\inf_{u\in \mathcal{N}}J(u)<\inf_{u\in \mathcal{N}_{\alpha}}J_{\alpha}(u)$.

Proof

Similar to the proofs of Lemmas 2.1 and 2.2, we can obtain that $\mathcal{N}_{\alpha}\neq\emptyset$ and $d_{\alpha}>0$. Arguing in Lemma 3.1.9 in [20], we see that there exists the nonnegative function $u_{0}\in\mathcal{N}_{\alpha}$ such that $J_{\alpha}(u_{0})=d_{\alpha}$. On the other hand, from (H₁)-(H₂), we infer

$$ \int_{\mathbb{R}^{3}}V(x)|u_{0}|^{2}\,dx< \alpha \int_{\mathbb{R}^{3}}|u_{0}|^{2}\,dx, $$

(2.30)

which implies that

$$ \int_{\mathbb{R}^{3}}\bigl(|\nabla u_{0}|^{2}+V(x)|u_{0}|^{2} \bigr)\,dx< \int_{\mathbb{R}^{3}}\bigl(|\nabla u_{0}|^{2}+ \alpha|u_{0}|^{2}\bigr)\,dx, \quad \mbox{or}\quad \|u_{0}\|_{E}^{2}< \|u_{0} \|_{Y}^{2}. $$

(2.31)

Then we have

$$\begin{aligned} J'(u_{0})u_{0}&=\|u_{0} \|_{E}^{2}+F(u_{0})-\|u_{0} \|_{m,a}^{m}-\lambda\|u_{0}\|_{q,b}^{q} \\ &< \|u_{0}\|_{Y}^{2}+F(u_{0})- \|u_{0}\|_{m,a}^{m}-\lambda\|u_{0} \|_{q,b}^{q}=J'_{\alpha }(u_{0})u_{0}=0. \end{aligned}$$

(2.32)

As the argument of Lemma 2.4, there exists $t\in(0, 1)$ such that $tu_{0}\in\mathcal{N}$, so that

$$\begin{aligned} d&\le J(tu_{0})=p_{1}t^{2}\|u_{0} \|_{E}^{2}+ p_{2}t^{4}F(u_{0})+p_{3}t^{m} \|u_{0}\|_{m,a}^{m} < p_{1} \|u_{0}\|_{E}^{2}+ p_{2}F(u_{0})+p_{3} \|u_{0}\|_{m,a}^{m} \\ &< p_{1}\|u_{0}\|_{Y}^{2}+ p_{2}F(u_{0})+p_{3}\|u_{0} \|_{m,a}^{m} =J_{\alpha}(u_{0})=d_{\alpha}, \end{aligned}$$

(2.33)

where $p_{i}$ is given by (2.22). Then (2.33) finishes the proof of Lemma 2.5. □

Lemma 2.6

There results $\theta>0$.

Proof

If $\theta=0$, then $u=0$, which implies in particular that $u_{n}\to 0$ in $L^{p}_{\mathrm{loc}}(\mathbb{R}^{3})$ ($1\le p<6$). We first prove the following claim:

$$ \lim_{n\to\infty} \int_{\mathbb{R}^{3}}\bigl(\alpha-V(x)\bigr)|u_{n}|^{2} \,dx=0. $$

(2.34)

By (H₂), for any $\varepsilon>0$, there exists $R_{\varepsilon}>0$ such that $0<\alpha-V(x)\le\varepsilon, \forall|x|\ge R_{\varepsilon}$. Then

$$\begin{aligned} \int_{\mathbb{R}^{3}}\bigl(\alpha-V(x)\bigr)|u_{n}|^{2} \,dx&= \int_{|x|\le R_{\varepsilon}}\bigl(\alpha-V(x)\bigr)|u_{n}|^{2} \,dx+ \int_{|x|\ge R_{\varepsilon}}\bigl(\alpha-V(x)\bigr)|u_{n}|^{2} \,dx \\ &\le\varepsilon \int_{|x|\ge R_{\varepsilon}}|u_{n}|^{2}\,dx+C_{1} \int_{|x|\le R_{\varepsilon}}|u_{n}|^{2}\,dx \\ &\le M_{1}\varepsilon+C_{1} \int_{|x|\le R_{\varepsilon}}|u_{n}|^{2}\,dx, \end{aligned}$$

(2.35)

where

$$ C_{1}=\max_{x\in\mathbb{R}^{3}} \bigl\vert \alpha-V(x) \bigr\vert ,\qquad M_{1}=\sup_{n\ge 1} \|u_{n}\|_{2}^{2}< \infty. $$

(2.36)

Noticing that

$$ \lim_{n\to\infty} \int_{|x|\le R_{\varepsilon}}|u_{n}|^{2}\,dx=0, $$

we have from (2.35) that

$$ \limsup_{n\to\infty} \int_{\mathbb{R}^{3}}\bigl(\alpha-V(x)\bigr)|u_{n}|^{2} \,dx\le M_{1}\varepsilon. $$

(2.37)

Since ε is arbitrarily small, (2.37) implies that (2.34), and then

$$ \|u_{n}\|_{Y}^{2}= \|u_{n}\|_{E}^{2}+\epsilon_{n}, $$

(2.38)

where $\epsilon_{n}\to0$ as $n\to\infty$.

On the other hand, the fact $u_{n}\in \mathcal{N}$ shows that

$$ \|u_{n}\|_{m,a}^{m}+\lambda \|u_{n}\|_{q,b}^{q}=\|u_{n} \|_{E}^{2}+F(u_{n})\le \|u_{n} \|_{Y}^{2}+F(u_{n}). $$

(2.39)

Denote

$$ \gamma_{n}(t)=J'_{\alpha}(tu_{n})tu_{n}=t^{2} \|u_{n}\|_{Y}^{2}+t^{4}F(u_{n})- t^{m}\|u_{n}\|_{m,a}^{m}-\lambda t^{q}\|u_{n}\|_{q,b}^{q},\quad t>0. $$

(2.40)

Since $4< q< m$ and (2.39), it follows that $\gamma_{n}(1)\ge0$ and $\gamma_{n}(t)\to-\infty$ as $t\to\infty$. Then there exists $t_{n}\ge1$ such that $\gamma_{n}(t_{n})=0$, and then $t_{n}u_{n}\in \mathcal{N}_{\alpha}$, that is,

$$ t_{n}^{2}\|u_{n} \|_{Y}^{2}+t_{n}^{4}F(u_{n})=t_{n}^{m} \|u_{n}\|_{m,a}^{m}+\lambda t_{n}^{q} \|u_{n}\|_{q,b}^{q}. $$

(2.41)

Using the facts that $\|u_{n}\|_{Y}$ and $\|u_{n}\|_{q,b}$ are bounded and $\lim_{n\to\infty}\|u_{n}\|_{m,a}^{m}=\beta$, we deduce from (2.41) that the sequence $\{t_{n}\}$ is bounded. If necessary, up to a subsequence, we can assume $t_{n}\to t_{0}\ge1$. Then it follows from (2.38), (2.39) and (2.41) that

$$ \bigl(t_{n}^{2}-t_{n}^{q} \bigr)\|u_{n}\|_{E}^{2}+\epsilon_{n}t_{n}^{2}+ \bigl(t_{n}^{4}-t_{n}^{q} \bigr)F(u_{n})= \bigl(t_{n}^{m}-t_{n}^{q} \bigr)\|u_{n}\|_{m,a}^{m}. $$

(2.42)

Since $t_{n}^{4}\le t_{n}^{q}$ and $F(u_{n})\ge0$, we have

$$ \bigl(t_{n}^{m}-t_{n}^{q} \bigr)\|u_{n}\|_{m,a}^{m}-\epsilon_{n}t_{n}^{2} \le \bigl(t_{n}^{2}-t_{n}^{q}\bigr) \|u_{n}\|_{E}^{2},\quad \forall n\ge1. $$

(2.43)

Letting $n\to\infty$ in (2.43) yields

$$ \bigl(t_{0}^{m}-t_{0}^{q} \bigr)\beta\le\bigl(t_{0}^{2}-t_{0}^{q} \bigr)\nu, $$

(2.44)

where $\nu=\limsup_{n\to\infty}\|u_{n}\|_{E}^{2}>0$ and $\beta=\lim_{n\to\infty}\|u_{n}\|_{m,a}^{m}$. Since $t_{0}\ge1$ and $4< q< m$, we have $t_{0}=1$. That is, $t_{n}\to1$ as $n\to\infty$. Therefore, it follows from (2.28) and (2.38) that

$$\begin{aligned} d_{\alpha}&\le J_{\alpha}(t_{n}u_{n}) =p_{1}t_{n}^{2}\|u\|_{Y}^{2}+p_{2}t_{n}^{4}F(u_{n}) +p_{3}t_{n}^{m}\|u_{n} \|_{m,a}^{m} \\ &=t_{n}^{2} \bigl[p_{1}\|u_{n} \|_{E}^{2}+p_{2}F(u_{n})+p_{3} \|u_{n}\|_{m,a}^{m} \bigr]+\gamma _{n}=t_{n}^{2}J(u_{n})+ \gamma_{n}, \end{aligned}$$

(2.45)

where $\gamma_{n}=p_{1}\varepsilon_{n}t_{n}^{2}+p_{2}(t_{n}^{4}-t_{n}^{2})F(u_{n}) +p_{3}(t_{n}^{m}-t_{n}^{2})\|u_{n}\|_{m,a}^{m}$. Since $t_{n}\to1$, we get $\gamma_{n}\to0$. Moreover, the facts $J(u_{n})\to d$ and $t_{n}\to1$ in (2.45) imply that $d_{\alpha}\le d$. This contradicts the result in Lemma 2.5. Therefore, we have $\theta>0$ and complete the proof of Lemma 2.6. □

In the following, we consider the case $\theta\in(0, \beta)$. As in [20, 21], we let $\theta\in(0, \beta)$ and take $\{u_{n}\}$ as a minimizing sequence for d on $\mathcal{N}$, which satisfies (2.12) and

$$ \begin{aligned} & \int_{B_{r_{n}}}a(x)|u_{n}|^{m}\,dx= \theta+o_{n}(1),\qquad \int_{\Omega_{n}}a(x)|u_{n}|^{m} \,dx=o_{n}(1),\\ &\int_{B_{r_{n}}}b(x)|u|^{q}\,dx=\sigma+o_{n}(1),\qquad \int_{\Omega_{n}}b(x)|u|^{q}\,dx=o_{n}(1), \\ & \int_{B^{c}_{r_{n}}}|u_{n}|^{2}\,dx=o_{n}(1), \qquad \int_{\Omega_{n}}|\nabla u_{n}|^{2} \,dx=o_{n}(1), \\ & \int_{B^{c}_{r_{n}}} \bigl\vert u_{n}(x) \bigr\vert ^{6}\,dx=o_{n}(1),\qquad \|u_{n} \|_{E}, \|u_{n}\|_{p}\le M_{0},\quad \forall n\ge1, 2\le p\le6, \end{aligned} $$

(2.46)

where and in the sequel, $B_{r}=\{x\in\mathbb{R}^{3}:|x|< r\}$, $B_{r}^{c}=\{x\in\mathbb{R}^{3}:|x|\ge r\}$, $\Omega_{n}=\{x\in \mathbb{R}^{3}: r_{n}\le|x|< r_{n+1}\}$ with $r_{n}\uparrow\infty$ and $o_{n}(1)$ is a quantity which goes to zero as $n\to\infty$.

Since $J(u_{n})=J(|u_{n}|)$ in E, we assume $u_{n}\ge0$ in $\mathbb{R}^{3}$. Furthermore, we also consider, for every $n\in\mathbb{N}$, a function $\varphi_{n}\in C_{0}^{\infty}(\mathbb{R}^{3})$ such that

• (1) $0\le\varphi_{n}(x)\le1$, $\forall x\in \mathbb{R}^{3}$,

• (2) $\varphi_{n}(x)=1$ if $|x|\le r_{n}$, $\varphi_{n}(x)=0$ if $|x|\ge r_{n+1}$,

• (3) $|\nabla\varphi_{n}(x)|\le C_{0}$, $\forall x\in\mathbb {R}^{3}$ and $\forall n\ge1$,

where $C_{0}$ is some positive number independent of n. Furthermore, we set

$$ u'_{n}(x)=\varphi_{n}(x)u_{n}(x), \qquad u''_{n}(x)=\bigl(1- \varphi_{n}(x)\bigr)u_{n}(x),\quad x\in\mathbb{R}^{3}. $$

(2.47)

Then $u'_{n}, u''_{n}\ge0$ and $u_{n}=u'_{n}+u''_{n}$ in $\mathbb{R}^{3}$ for every $n\ge1$.

Lemma 2.7

The following properties for $u_{n}$, $u'_{n}$, $u''_{n}$ hold:

(P₁):

$u'_{n}\rightharpoonup u$ weakly in $E=W^{1,2}(\mathbb{R}^{3})$, $u'_{n}\to u$ strongly in $L^{m}(\mathbb{R}^{3},a)\cap L^{q}(\mathbb{R}^{3},b)$.

(P₂):

$\int_{\mathbb{R}^{3}}a(x)|u_{n}|^{m}\,dx=\int_{\mathbb {R}^{3}}a(x)|u'_{n}|^{m}\,dx+\int_{\mathbb{R}^{3}}a(x)|u''_{n}|^{m}\,dx+o_{n}(1)$.

(P₃):

$\int_{\mathbb{R}^{3}}b(x)|u_{n}|^{q}\,dx=\int_{\mathbb {R}^{3}}b(x)|u'_{n}|^{q}\,dx+\int_{\mathbb{R}^{3}}b(x)|u''_{n}|^{q}\,dx+o_{n}(1)$.

(P₄):

$\|u_{n}\|_{E}^{2}\ge\|u'_{n}\|_{E}^{2}+\|u''_{n}\|_{E}^{2}+o_{n}(1)$.

(P₅):

$F(u_{n})\ge F(u'_{n})+F(u''_{n})$, $\forall n\ge1$.

(P₆):

$F(u_{n})\le F(u'_{n})+F(u''_{n})+\sum_{j=1}^{5}A_{n}^{j}$, $\forall n\ge 1$, in which

$$\begin{aligned}& A_{n}^{1}\equiv \int_{\Omega_{n}}K(x)\phi_{u_{n}}(x)u_{n}^{2}(x) \,dx=o_{n}(1), \\& A_{n}^{2}\equiv \int_{\Omega_{n}}K(x)\phi_{u'_{n}}(x)u_{n}^{2}(x) \,dx=o_{n}(1), \\& A_{n}^{3}\equiv \int_{\Omega_{n}}K(x)\phi_{u''_{n}}(x)u_{n}^{2}(x) \,dx=o_{n}(1), \\ & A_{n}^{4}\equiv \int_{B^{c}_{r_{n+1}}}K(x)\phi_{u'_{n}}(x)u_{n}^{2}(x) \,dx=o_{n}(1), \\ & A_{n}^{5}\equiv \int_{B_{r_{n}}}K(x)\phi_{u''_{n}}(x)u_{n}^{2}(x) \,dx=o_{n}(1),\quad n=1,2,\ldots, \end{aligned}$$

(2.48)

where $\phi_{u}$ is defined by (1.10).

Proof

The proof of properties (P₁)-(P₄) is similar to that in [20, 21] and is omitted. Here, we prove (P₅) and (P₆). Note that $u'_{n}(x)=u_{n}(x)$, $u''_{n}(x)=0$ if $x\in B_{r_{n}}$ and $u''_{n}(x)=u_{n}(x)$, $u'_{n}(x)=0$ if $x\in B_{r_{n+1}}^{c}$. So,

$$ \bigl\vert u'_{n}(x) \bigr\vert ^{2}+ \bigl\vert u''_{n}(x) \bigr\vert ^{2}= \bigl\vert u_{n}(x) \bigr\vert ^{2},\quad x\in B_{r_{n}}\cup B_{r_{n+1}}^{c}. $$

(2.49)

Moreover, if $x\in\Omega_{n}$, we have $0\le\varphi_{n}(x)\le1$ and

$$ \bigl\vert u'_{n}(x) \bigr\vert ^{2}+ \bigl\vert u''_{n}(x) \bigr\vert ^{2}= \bigl\vert u_{n}(x) \bigr\vert ^{2}\bigl( \bigl\vert \varphi_{n}(x) \bigr\vert ^{2}+ \bigl\vert 1-\varphi _{n}(x) \bigr\vert ^{2}\bigr)\le \bigl\vert u_{n}(x) \bigr\vert ^{2}. $$

(2.50)

Thus, one sees that

$$ \bigl\vert u_{n}(x) \bigr\vert ^{2}\ge \bigl\vert u'_{n}(x) \bigr\vert ^{2}+ \bigl\vert u''_{n}(x) \bigr\vert ^{2},\quad \forall x\in \mathbb{R}^{3}. $$

(2.51)

Clearly,

$$\begin{aligned} \phi_{u_{n}}(x) =&\frac{1}{4\pi} \int_{\mathbb {R}^{3}}\frac{K(y)u_{n}^{2}(y)}{|x-y|}\,dy \\ =& \frac{1}{4\pi} \int_{B_{r_{n}}}\frac{K(y)(u'_{n}(y))^{2}}{|x-y|}\,dy+\frac {1}{4\pi} \int_{B^{c}_{r_{n+1}}}\frac{K(y)(u''_{n}(y))^{2}}{|x-y|}\,dy \\ &{}+\frac{1}{4\pi} \int_{\Omega_{n}}\frac{K(y)(u_{n}(y))^{2}}{|x-y|}\,dy \\ =&\frac{1}{4\pi} \int_{\Omega_{n}}\frac{K(y)}{|x-y|} \bigl[\bigl(u_{n}(y) \bigr)^{2}-\bigl(u'_{n}(y)\bigr)^{2}- \bigl(u''_{n}(y)\bigr)^{2} \bigr] \,dy +\phi_{u'_{n}}(x)+\phi_{u''_{n}}(x) \\ \ge&\phi_{u'_{n}}(x)+\phi_{u''_{n}}(x),\quad \forall x\in \mathbb {R}^{3}. \end{aligned}$$

(2.52)

So, we have

$$\begin{aligned} F(u_{n}) =& \int_{\mathbb {R}^{3}}K(x)\phi_{u_{n}}(x)u_{n}^{2}(x) \,dx \\ \ge& \int_{\mathbb {R}^{3}}K(x)\phi_{u'_{n}}(x)u_{n}^{2}(x) \,dx+ \int_{\mathbb {R}^{3}}K(x)\phi _{u''_{n}}(x)u_{n}^{2}(x) \,dx \\ =&F\bigl(u'_{n}\bigr)+F\bigl(u''_{n} \bigr)+ \int_{\Omega_{n}}K(x)\phi_{u'_{n}} \bigl[u_{n}^{2}- \bigl(u'_{n}\bigr)^{2} \bigr]\,dx+ \int_{\Omega_{n}}K(x)\phi_{u''_{n}} \bigl[u_{n}^{2}- \bigl(u''_{n}\bigr)^{2} \bigr] \,dx \\ &{} + \int_{B_{r_{n}}}K(x)\phi_{u''_{n}}\bigl(u'_{n} \bigr)^{2}\,dx + \int_{B^{c}_{r_{n+1}}}K(x)\phi_{u'_{n}}\bigl(u''_{n} \bigr)^{2}\,dx \\ \ge& F\bigl(u'_{n}\bigr)+F\bigl(u''_{n} \bigr),\quad \forall n\ge1, \end{aligned}$$

(2.53)

which proves (P₅).

On the other hand, it follows from (2.52) that

$$ \phi_{u_{n}}(x)\le\phi_{u'_{n}}(x)+ \phi_{u''_{n}}(x)+\frac{1}{4\pi} \int_{\Omega_{n}}\frac{K(y)u^{2}_{n}(y)}{|x-y|}\,dy. $$

(2.54)

Noticing that

$$\begin{aligned}& \frac{1}{4\pi} \int_{\mathbb {R}^{3}} \biggl( \int_{\Omega_{n}}\frac{K(y)u_{n}^{2}(y)}{|x-y|} \,dy \biggr)K(x)u_{n}^{2}(x) \,dx \\& \quad =\frac{1}{4\pi} \int_{\Omega_{n}} \biggl( \int _{R^{3}}\frac{K(x)u_{n}^{2}(x)}{|x-y|}\,dx \biggr)K(y)u_{n}^{2}(y) \,dy \\& \quad = \int_{\Omega_{n}}K(x)\phi_{u_{n}}(x)u_{n}^{2}(x) \,dx, \end{aligned}$$

(2.55)

we have from (2.54) and (2.55) that

$$\begin{aligned} \int_{\mathbb {R}^{3}}K(x)\phi_{u_{n}}(x)u_{n}^{2}(x) \,dx \le& \int_{\mathbb {R}^{3}}K(x)u_{n}^{2}(x) \bigl( \phi_{u'_{n}}(x)+\phi_{u''_{n}}(x)\bigr)\,dx+ \int_{\Omega_{n}}K(x)\phi_{u_{n}}(x)u_{n}^{2}(x) \,dx \\ \le& F\bigl(u'_{n}\bigr)+F\bigl(u''_{n} \bigr)+\sum_{j=1}^{5}A_{n}^{j}. \end{aligned}$$

(2.56)

If $K\in L^{\infty}(\mathbb {R}^{3})$ and $K\ge0$, we have from (2.12) that

$$\begin{aligned} A_{n}^{1}&= \int_{\Omega_{n}}K\phi_{u_{n}}u_{n}^{2}\,dx \le d_{0}\|\phi_{u_{n}}\|_{L^{6}(\Omega_{n})}\|u_{n} \|^{2}_{L^{\frac{12}{5}}(\Omega _{n})} \\ &\le c_{2}\|u_{n} \|_{E}^{2}\|u_{n}\|^{2}_{L^{\frac{12}{5}}(\Omega_{n})} \le c_{2}M_{0}^{2}\|u_{n} \|^{2}_{L^{\frac{12}{5}}(\Omega_{n})}. \end{aligned}$$

(2.57)

From (2.46), we have $\|u_{n}\|^{2}_{L^{\frac{12}{5}}(\Omega_{n})}=o_{n}(1)$ and so $A_{n}^{1}=o_{n}(1)$.

If $K\in L^{2}(\mathbb {R}^{3})$ and $K\ge0$, we derive from the Hölder inequality and the Sobolev inequality that

$$\begin{aligned} A_{n}^{1}&\le\|\phi_{u_{n}} \|_{L^{6}(\Omega_{n})}\|K\|_{L^{2}(\Omega_{n})}\|u_{n}\| _{L^{6}(\Omega_{n})}^{2} \\ &\le c_{1}\|u_{n}\|_{E}^{2} \|u_{n}\|_{L^{6}(\Omega_{n})}^{2}\|K\|_{L^{2}(\Omega_{n})}\le c_{1}M_{0}^{4}\|K\|_{L^{2}(\Omega_{n})}. \end{aligned}$$

(2.58)

Similarly, the assumption $K\in L^{2}(\mathbb {R}^{3})$ implies that $\|K\|_{L^{2}(\Omega_{n})}=o_{n}(1)$ and $A_{n}^{1}=o_{n}(1)$.

Furthermore, since $u_{n}(x)\ge u'_{n}(x), u''_{n}(x)$, we have $\phi_{u_{n}}(x)\ge\phi_{u'_{n}}(x), \phi_{u''_{n}}(x)$, and then $0\le A_{n}^{2}, A_{n}^{3}\le A_{n}^{1}=o_{n}(1)$. Noticing that

$$ A_{n}^{4}= \int_{B^{c}_{r_{n+1}}}K(x)\phi_{u'_{n}}(x)u_{n}^{2}(x) \,dx\le \int_{B^{c}_{r_{n}}}K(x)\phi_{u_{n}}(x)u_{n}^{2}(x) \,dx, $$

(2.59)

we argue as in the proof of (2.57) and (2.58) and obtain $A_{n}^{4}=o_{n}(1)$. Similarly,

$$\begin{aligned} \begin{aligned}[b] A_{n}^{5}&= \int_{B_{r_{n}}}K(x)\phi_{u''_{n}}(x)u_{n}^{2}(x) \,dx=\frac{1}{4\pi} \int_{B_{r_{n}}} \int_{\mathbb {R}^{3}}\frac{K(y)(u''_{n}(y))^{2}}{|x-y|}K(x)u_{n}^{2}(x) \,dy\,dx \\ &\le \frac{1}{4\pi} \int_{B_{r_{n}}} \int_{B_{r_{n}}^{c}}\frac {K(y)u^{2}_{n}(y)}{|x-y|}K(x)u_{n}^{2}(x) \,dy\,dx= \int_{B^{c}_{r_{n}}}K(y)\phi _{u_{n}}(y)u_{n}^{2}(y) \,dy, \end{aligned} \end{aligned}$$

(2.60)

we get from (2.59) and $A_{n}^{4}=o_{n}(1)$ that $A_{n}^{5}=o_{n}(1)$ if $K\in L^{2}(\mathbb {R}^{3})\cup L^{\infty}(\mathbb {R}^{3})$ and finish the proof of (P₆). Then the proof of Lemma 2.7 is completed. □

Proof of Theorem 1.1

In order to prove Theorem 1.1, we first show that the weak limit u of the minimizing sequence $\{u_{n}\}$ in Section 2 verifies

$$ p_{1}\|u\|_{E}^{p}+p_{2}F(u)+p_{3} \|u\|_{m,a}^{m}\le d $$

(3.1)

and

$$ \|u\|_{E}^{2}+F(u)=\|u\|_{m,a}^{m}+ \lambda\|u\|_{q,b}^{q}, $$

(3.2)

where $p_{i}$ is given by (2.22).

Let $\{u_{n}\}$ be a minimizing sequence for d on $\mathcal{N}$ in Section 2. Then, if necessary, up to a subsequence, we have $u_{n} \rightharpoonup u$ in E and $L^{m}(\mathbb{R}^{3},a)$. By the weak lower semi-continuity of norms, we obtain

$$\begin{aligned} p_{1}\|u\|_{E}^{2}+p_{2}F(u)+p_{3} \|u\|_{m,a}^{m} \le& p_{1}\liminf _{n\to\infty}\|u_{n}\|_{E}^{2}+p_{2} \liminf_{n\to\infty }F(u_{n})+p_{3}\liminf _{n\to\infty}\|u_{n}\|_{m,a}^{m} \\ \le& \liminf_{n\to\infty}\bigl(p_{1}\|u_{n} \|_{E}^{2}+p_{2}F(u_{n})+p_{3} \|u_{n}\| _{m,a}^{m}\bigr) \\ =&\liminf _{n\to\infty}J(u_{n})=d. \end{aligned}$$

(3.3)

This implies relation (3.1). By Lemma 2.6, we get $u\ge 0, u\not\equiv0$ in $\mathbb{R}^{3}$. We now prove (3.2). If

$$ \|u\|_{E}^{2}+F(u)< \|u\|_{m,a}^{m}+ \lambda\|u\|_{q,b}^{q}, $$

(3.4)

we set

$$ h(t)=J'(tu)tu=t^{2}\|u \|_{E}^{2}+t^{4}F(u)-t^{m}\|u \|_{m,a}^{m}-\lambda t^{q}\|u\|_{q,b}^{q} \quad \mbox{for } t\ge0. $$

(3.5)

Then one sees that $h(t)>0$ for small $t>0$ and $h(1)<0$, so that there exists $t\in(0,1)$ such that $h(t)=0$ and $tu\in\mathcal{N}$ and

$$\begin{aligned} d&\le J(tu)= p_{1}t^{2}\|u\|_{E}^{2}+p_{2} t^{4}F(u)+p_{3}t^{m}\|u\|_{m,a}^{m} \\ &< p_{1} \|u\|_{E}^{2}+p_{2}F(u)+p_{3}\|u \|_{m,a}^{m}\le d. \end{aligned}$$

(3.6)

This contradiction shows that (3.4) cannot occur. Now we assume

$$ \|u\|_{E}^{2}+F(u)>\|u\|_{m,a}^{m}+ \lambda\|u\|_{q,b}^{q} $$

(3.7)

and choose small $\delta_{1}>0$ such that

$$ \|u\|_{E}^{2}+F(u)>\|u\|_{m,a}^{m}+ \lambda\|u\|_{q,b}^{q}+\delta_{1}. $$

(3.8)

Since $u'_{n}\rightharpoonup u$ weakly in E, one has

$$ \|u\|_{E}^{2}\le\liminf_{n\to\infty} \bigl\Vert u'_{n} \bigr\Vert _{E}^{2}. $$

(3.9)

Moreover, it follows from (P₆) that

$$\begin{aligned} F(u)&\le\liminf_{n\to\infty}F(u_{n})\le \liminf_{n\to\infty}F\bigl(u'_{n}\bigr)+ \limsup_{n\to\infty}F\bigl(u''_{n} \bigr)+\sum_{j=1}^{5}\limsup _{n\to\infty }A_{n}^{j} \\ &=\liminf _{n\to\infty}F\bigl(u'_{n}\bigr). \end{aligned}$$

(3.10)

Then, from (3.8)-(3.10), there exist $\delta_{2}\in (0,\delta_{1})$ and $n_{0}\ge1$ such that

$$ \bigl\Vert u'_{n} \bigr\Vert _{E}^{2}+F\bigl(u'_{n}\bigr)> \bigl\Vert u'_{n} \bigr\Vert _{m,a}^{m}+ \lambda \bigl\Vert u'_{n} \bigr\Vert _{q,b}^{q}+\delta_{2}\quad \mbox{for } n\ge n_{0}. $$

(3.11)

Now, by Lemma 2.7, we derive from (3.10), (3.11) and the fact $u_{n}\in\mathcal{N}$ that

$$\begin{aligned} \bigl\Vert u'_{n} \bigr\Vert _{E}^{2}+ \bigl\Vert u''_{n} \bigr\Vert _{E}^{2}&\le \Vert u_{n} \Vert _{E}^{2}+o_{n}(1)= \Vert u_{n} \Vert _{m,a}^{m}+\lambda \Vert u_{n} \Vert _{q,b}^{q}-F(u_{n})+o_{n}(1) \\ &\le \bigl\Vert u'_{n} \bigr\Vert _{m,a}^{m}+\lambda \bigl\Vert u'_{n} \bigr\Vert _{q,b}^{q}+ \bigl\Vert u''_{n} \bigr\Vert _{m,a}^{m}+\lambda \bigl\Vert u''_{n} \bigr\Vert _{q,b}^{q}-F \bigl(u'_{n}\bigr)-F\bigl(u''_{n} \bigr)+o_{n}(1) \\ &< \bigl\Vert u'_{n} \bigr\Vert _{E}^{2}+ \bigl\Vert u''_{n} \bigr\Vert _{m,a}^{m}+\lambda \bigl\Vert u''_{n} \bigr\Vert _{q,b}^{q}-F \bigl(u''_{n}\bigr)-\delta_{2}+o_{n}(1). \end{aligned}$$

(3.12)

This implies that

$$ \bigl\Vert u''_{n} \bigr\Vert _{E}^{2}+F\bigl(u''_{n} \bigr)\le \bigl\Vert u''_{n} \bigr\Vert _{m,a}^{m}+\lambda \bigl\Vert u''_{n} \bigr\Vert _{q,b}^{q}-\delta_{2}+o_{n}(1). $$

(3.13)

The application of (2.38) and (3.13) shows that there exist $\delta_{3}\in(0,\delta_{2})$ and $n_{1}\ge n_{0}$ such that

$$ \bigl\Vert u''_{n} \bigr\Vert _{Y}^{2}+F\bigl(u''_{n} \bigr)\le \bigl\Vert u''_{n} \bigr\Vert _{m,a}^{m}+\lambda \bigl\Vert u''_{n} \bigr\Vert _{q,b}^{q}-\delta_{3}+o_{n}(1) \quad \mbox{for } n\ge n_{1}. $$

Again, we set an auxiliary function

$$ \gamma_{n}(t)=J'_{\alpha} \bigl(tu''_{n}\bigr) \bigl(tu''_{n} \bigr)=t^{2} \bigl\Vert u''_{n} \bigr\Vert _{Y}^{2}+t^{4}F\bigl(u''_{n} \bigr)- t^{m} \bigl\Vert u''_{n} \bigr\Vert _{m,a}^{m}-\lambda t^{q} \bigl\Vert u''_{n} \bigr\Vert _{q,b}^{q} \quad \mbox{for } t>0. $$

Obviously, $\gamma_{n}(t)>0$ for small $t>0$ and $\gamma_{n}(1)<0$ for $n\ge n_{1}$. Then there exists $t_{n}\in(0,1)$ such that $\gamma_{n}(t_{n})=0$ and $t_{n}u''_{n}\in\mathcal{N}_{\alpha}$. Then it follows from (2.28) and (2.38) that

$$\begin{aligned} d_{\alpha}&\le J_{\alpha}\bigl(t_{n}u''_{n} \bigr)=p_{1}t_{n}^{2} \bigl\Vert u''_{n} \bigr\Vert _{Y}^{2}+p_{2}t_{n}^{4}F \bigl(u''_{n}\bigr)+ p_{3}t_{n}^{m} \bigl\Vert u''_{n} \bigr\Vert _{m,a}^{m} \\ &< p_{1} \bigl\Vert u''_{n} \bigr\Vert _{Y}^{2}+p_{2}F\bigl(u''_{n} \bigr)+p_{3} \bigl\Vert u''_{n} \bigr\Vert _{m,a}^{m} \\ &\le p_{1} \bigl\Vert u''_{n} \bigr\Vert _{E}^{2}+p_{2}F\bigl(u''_{n} \bigr)+p_{3} \bigl\Vert u''_{n} \bigr\Vert _{m,a}^{m}+o_{n}(1) \\ &\le p_{1} \Vert u_{n} \Vert _{E}^{2}+p_{2}F(u_{n})+p_{3} \Vert u_{n} \Vert _{m,a}^{m}+o_{n}(1)=J(u_{n})+o_{n}(1) \to d \quad \mbox{as } n\to\infty. \end{aligned}$$

This shows that $d_{\alpha}\le d$ and contradicts the result in Lemma 2.5. Therefore, (3.7) fails to be true and (3.2) is satisfied.

Since u satisfies (3.1) and (3.2), it follows that $u\in\mathcal{N}$ and so that $J(u)=d$ and u is a minimum point for J on $\mathcal{N}$.

To finish the proof of Theorem 1.1, it is sufficient to prove that u is a critical point for the functional $J(u)$ in E, that is, $J'(u)v=0$ for all $v\in E$, and thus $J'(u)=0$ in $E^{*}$.

For every fixed $v\in E$, there exists $\varepsilon>0$ such that $u+sv \not\equiv0$ for all $s\in(-\varepsilon,\varepsilon)$. From Lemma 2.1, there is $t(s)\in\mathbb{R}$ such that $t(s)(u+sv )\in \mathcal{N}$. We will show that $t=t(s)$ is a $C^{1}$ function. So, we consider the function $\varphi: (-\varepsilon, \varepsilon)\times \mathbb{R}\rightarrow\mathbb{R}$ as

$$\begin{aligned} \varphi(s,t) =&J'\bigl(t(s) (u+\varepsilon v)\bigr)t(s) (u+sv )=t^{2}\|u+sv\|_{E}^{2}+t^{4}F(u+sv) \\ &{}-t^{m}\|u+sv\|_{m,a}^{m}-\lambda t^{q}\|u+sv\|_{q,b}^{q}. \end{aligned}$$

Since $u\in\mathcal{N}$, we have

$$ \varphi(0,1)=\|u\|_{E}^{2}+F(u)-\|u \|_{m,a}^{m}-\lambda\|u\|_{q,b}^{q}=0. $$

Moreover, we derive

$$\begin{aligned} \frac{\partial\varphi}{\partial t}(0,1)&=2\|u\|_{E}^{2}+4F(u)-m \|u\|_{m,a}^{m}-\lambda q\|u\|_{q,b}^{q} \\ &=(2-q)\|u\|_{E}^{2}+(4-q)F(u)+(q-m)\|u \|_{m,b}^{m}< 0. \end{aligned}$$

Then, by the implicit function theorem, there exists a $C^{1}$ function $t(s)$ such that $\varphi(s,t(s))=0$ and $t(0)=1$ for every $s\in(-\varepsilon_{0},\varepsilon_{0})\subset (-\varepsilon,\varepsilon)$. This also shows that $t(s)\neq0$ and $t(s)(u+sv)\in\mathcal{N}$. Then the function $h(s)=J(t(s)(u+sv))$, $s\in(-\varepsilon_{0}, \varepsilon_{0})$ is in $C^{1}$ and has a minimum at $s=0$. Therefore,

$$ 0=h'(0)=t'(0) \bigl(\|u \|_{E}^{2}+F(u)-\|u\|_{m,a}^{m}-\lambda \|u\|_{q,b}^{q} \bigr)+J'(u)v. $$

(3.14)

Then, from (3.14), it follows for every $v\in E$ that $J'(u)v=0$. Thus $J'(u)=0$ in $E^{*}$. So, u is a critical point for J, and then u is a weak solution of problem (1.1) in E, that is, u is a solution of (1.1). Since $J(u)=J(|u|)=d>0$, we can assume $u\ge0$ a.e. in $\mathbb{R}^{3}$. Furthermore, the application of maximum principle in [22] yields $u(x)>0$ in $\mathbb{R}^{3}$. Then the proof of Theorem 1.1 is finished.