Ground state solutions for a class of generalized quasilinear Schrödinger–Poisson systems

Abstract

This paper is concerned with the existence of ground state solutions for a class of generalized quasilinear Schrödinger–Poisson systems in $\mathbb {R}^{3}$ which have appeared in plasma physics, as well as in the description of high-power ultrashort lasers in matter. By employing a change of variables, the generalized quasilinear systems are reduced to a semilinear one, whose associated functionals are well defined in the usual Sobolev space and satisfy the mountain-pass geometric. Finally, we use Ekeland’s variational principle and the mountain-pass theorem to obtain the ground state solutions for the given problem.

1 Introduction and main results

The aim of this paper is to establish the existence of ground state solutions to the following generalized quasilinear Schrödinger–Poisson system:

$$ \textstyle\begin{cases} -\operatorname{div}(g^{2}(u)\nabla u)+g(u)g^{\prime}(u)|\nabla u|^{2}+a(x)u+\phi G(u)g(u)=k(x,u), & x\in \mathbb {R}^{3}, \\ -\Delta\phi=G^{2}(u), & x\in \mathbb {R}^{3}, \end{cases} $$

(1.1)

where $g:\mathbb {R}\to \mathbb {R}^{+}=[0,\infty)$ is an even differential function, $g^{\prime}(s)\geq0$ for all $s\geq0$, and $G(t)=\int_{0}^{t}g(s)\,ds$, $k:\mathbb {R}^{3}\times \mathbb {R}\to \mathbb {R}$ is continuous, $a(x):\mathbb {R}^{3}\to \mathbb {R}^{+}$ is continuous.

If $\phi=0$ in (1.1), solutions of this type are related to the existence of solitary wave solutions for quasilinear Schrödinger equations of the form

$$ i\partial_{t}z=-\Delta z+W(x)z-k(x,z)-\omega\Delta l \bigl(|z|^{2}\bigr)l^{\prime }\bigl(|z|^{2}\bigr)z,\quad x\in \mathbb {R}^{N}, $$

(1.2)

where ω is a real constant, $N\geq3$, $z:\mathbb {R}\times \mathbb {R}^{N}\to \mathbb{C}$, $W:\mathbb {R}^{N}\to \mathbb {R}$ is a given potential, $l:\mathbb {R}\to \mathbb {R}$ and $k:\mathbb {R}^{N}\times \mathbb {R}\to \mathbb {R}$ are suitable functions.

The semilinear case corresponding to $\omega=0$ has been studied extensively by many scholars in recent years (see [1–5]). Quasilinear equations of the form (1.2) have been derived as models of several physical phenomena corresponding to various types of $l(s)$. For instance, the case $l(s)=s$ models the time evolution of the condensate wave function in a superfluid film [6, 7], and it is called the superfluid film equation in fluid mechanics by Kurihara [6]. In the case $l(s)=(1+s)^{1/2}$, problem (1.2) models the self-channeling of a high-power ultra short laser in matter, the propagation of a high-irradiance laser in a plasma creates an optical index depending nonlinearly on the light intensity and this leads to an interesting new nonlinear wave equation (see [8–11]). Equation (1.2) also appears in plasma physics and fluid mechanics [12–15], in dissipative quantum mechanics [16] and in condensed matter theory [17].

Recently, Deng–Peng–Yan [18] introduced a class of generalized quasilinear critical Schrödinger equations,

$$ -\operatorname{div}\bigl(g^{2}(u)\nabla u \bigr)+g(u)g^{\prime}(u)|\nabla u|^{2}+a(x)u =k(x,u),\quad x\in \mathbb {R}^{N} , $$

(1.3)

to study the existence of positive soliton solutions. The reason we call Eq. (1.3) a generalized quasilinear Schrödinger equation is that if we take

$$g^{2}(u)=1+\frac{ [l^{\prime}(u^{2}) ]^{2}}{2}, $$

then the following quasilinear equation:

$$-\Delta u+a(x)u-\Delta l\bigl(u^{2}\bigr)l^{\prime} \bigl(u^{2}\bigr)u=k(x,u),\quad x\in \mathbb {R}^{N}, $$

turns into it (see [18, 19]). Equation (1.3) also arises in biological models and propagation of laser beams when $g(u)$ is a positive constant. If we set $g^{2}(u) =1 +2u^{2}$, i.e. $l(s) =s$, we get the superfluid film equation in plasma physics:

$$-\Delta u+V(x)u-\Delta\bigl(u^{2}\bigr)u=k(x,u),\quad x\in \mathbb {R}^{N}. $$

If we set $g^{2}(u) =1 + \frac{u^{2}}{2(1+u^{2})}$, i.e. $l(s)=(1+s)^{1/2}$, we get the equation

$$-\Delta u+V(x)u- \bigl[\Delta\bigl(1+u^{2}\bigr)^{\frac{1}{2}} \bigr] \frac {1}{2(1+u^{2})^{\frac{1}{2}}}u=k(x,u), \quad x\in \mathbb {R}^{N}, $$

which models the self-channeling of a high-power ultrashort laser in matter. For the related and important results on quasilinear Schrödinger equations, we refer the reader to [19–26] and the references therein.

We call problem (1.1) the generalized quasilinear Schrödinger–Poisson system because of the coupling of the Poisson equation with (1.3). Indeed, if we choose $g(t)=1$ for all $t\in \mathbb {R}$, then (1.1) transforms to the following classical Schrödinger–Poisson system:

$$\textstyle\begin{cases} -\Delta u+a(x)u+\phi u=k(x,u), & x\in \mathbb {R}^{3}, \\ -\Delta\phi=u^{2}, & x\in \mathbb {R}^{3}, \end{cases} $$

proposed by Benci–Fortunato [27, 28] to represent solitary waves for nonlinear Schrödinger type equations and look for the existence of standing waves interacting with an unknown electrostatic field. We refer the reader to [29–34] for some related and important results. In view of this, it is also reasonable to consider the generalized quasilinear Schrödinger–Poisson system.

According to Ruiz [35], for any $u\in H^{1}(\mathbb {R}^{3})$ we can define

$$\phi_{u}(x)=\frac{1}{4\pi} \int_{\mathbb {R}^{3}}\frac{u^{2}(y)}{|x-y|}\, dy, $$

which is a weak solution to $-\Delta\phi=u^{2}$ in $\mathbb {R}^{3}$. Therefore the weak solution of $-\Delta\phi=G^{2}(u)$ can be represented as

$$\phi_{G(u)}(x)=\frac{1}{4\pi} \int_{\mathbb {R}^{3}}\frac{G^{2}(u(y))}{|x-y|}\,dy $$

and then (1.1) can be reduced to a single equation:

$$ -\operatorname{div}\bigl(g^{2}(u)\nabla u \bigr)+g(u)g^{\prime}(u)|\nabla u|^{2}+a(x)u+\phi _{G(u)} G(u)g(u)=k(x,u),\quad x\in \mathbb {R}^{3}. $$

(1.4)

In this paper, we establish the existence of ground state solutions for problem (1.1). To this end, we assume $k(x,t)=b(x)|G(u)|^{p-2}G(u)g(u)-c(x)|G(u)|^{q-2}G(u)g(u)$. Hence the problem (1.4) can be rewritten in the following form:

$$\begin{aligned}& -\operatorname{div}\bigl(g^{2}(u)\nabla u\bigr)+g(u)g^{\prime}(u) \vert \nabla u \vert ^{2}+a(x)u+\phi _{G(u)} G(u)g(u) \\& \quad =b(x) \bigl\vert G(u) \bigr\vert ^{p-2}G(u)g(u)-c(x) \bigl\vert G(u) \bigr\vert ^{q-2}G(u)g(u), \end{aligned}$$

(1.5)

whose corresponding variational functional is given by

$$\begin{aligned} I(u) =&\frac{1}{2} \int_{\mathbb {R}^{3}}g^{2}(u) \vert \nabla u \vert ^{2}\,dx+\frac{1}{2} \int _{\mathbb {R}^{3}}a(x)u^{2}\,dx +\frac{1}{4} \int_{\mathbb {R}^{3}}\phi_{G(u)}G^{2}(u)\,dx \\ &{}-\frac{1}{p} \int_{\mathbb {R}^{3}}b(x) \bigl\vert G(u) \bigr\vert ^{p} \,dx+\frac{1}{q} \int_{\mathbb {R}^{3}}c(x) \bigl\vert G(u) \bigr\vert ^{q} \,dx. \end{aligned}$$

Unfortunately, the above functional I may be not well defined in $H^{1}(\mathbb {R}^{3})$. To overcome this difficulty, we make a change of variable constructed by Shen–Wang [19],

$$v=G(u)= \int^{u}_{0}g(\tau)\,d\tau. $$

Then we get

$$\begin{aligned} J(v) =&\frac{1}{2} \int_{\mathbb {R}^{3}} \vert \nabla v \vert ^{2}\,dx+ \frac {1}{2} \int_{\mathbb {R}^{3}}a(x) \bigl\vert G^{-1}(v) \bigr\vert ^{2}\,dx + \frac{1}{4} \int_{\mathbb {R}^{3}}\phi_{v}v^{2}\,dx \\ &{}-\frac{1}{p} \int_{\mathbb {R}^{3}}b(x) \vert v \vert ^{p}\,dx+ \frac{1}{q} \int_{\mathbb {R}^{3}}c(x) \vert v \vert ^{q} \,dx. \end{aligned}$$

(1.6)

Since g is a nondecreasing positive function, we get $|G^{-1}(v)|\leq |v|/g(0)$. It is clear that J is well defined in $H^{1}(\mathbb {R}^{3})$ and $J\in C^{1}$ if assumption (H₁) holds.

If u is a nontrivial solution of (1.5), then it should satisfy

$$\begin{aligned}& \int_{\mathbb {R}^{3}} \bigl[g^{2}(u)\nabla u\nabla \varphi+g(u)g^{\prime }(u) \vert \nabla u \vert ^{2} \varphi+a(x)u\varphi +\phi_{G(u)}G(u)g(u)\varphi \\& \quad {}-b(x) \bigl\vert G(u) \bigr\vert ^{p-2}G(u)g(u)\varphi+c(x) \bigl\vert G(u) \bigr\vert ^{q-2}G(u)g(u)\varphi \bigr]\,dx=0, \end{aligned}$$

for any $\varphi\in C^{\infty}_{0}(\mathbb {R}^{3})$. Let $\varphi=\psi/g(u)$, we know that the above formula is equivalent to

$$\begin{aligned} \bigl\langle J^{\prime}(v),\psi\bigr\rangle =& \int_{\mathbb {R}^{3}} \biggl[ \nabla v \nabla\psi+a(x)\frac{G^{-1}(v)}{g(G^{-1}(v))} \psi+\phi_{v} v\psi -b(x)|v|^{p-2}v\varphi+c(x)|v|^{q-2}v \varphi \,dx \biggr] \\ =&0,\quad \forall\psi\in C^{\infty}_{0}\bigl(\mathbb {R}^{3} \bigr). \end{aligned}$$

Therefore, in order to find the nontrivial solutions of (1.5), it suffices to study the existence of the nontrivial solutions of the following equations:

$$ -\Delta v+a(x)\frac{G^{-1}(v)}{g(G^{-1}(v))}+\phi_{v} v=b(x)|v|^{p-2}v-c(x)|v|^{q-2}v. $$

(1.7)

It is easy to verify that the problem (1.5) is equivalent to problem (1.7) and the nontrivial critical points of $J(v)$ are the nontrivial solutions of problem (1.7). Inspired by all the work described above, particularly, by the results in [18, 25], we intend to show the existence of ground state solutions of problem (1.7). To this end, we first give some assumptions on g, a, b and c.

(g):

$g\in C^{1}(\mathbb {R})$ is an even positive function and $g^{\prime}(t)\geq0$ for all $t\geq0$ and $g(0) =1$;

(H₁):

$a(x)$, $b(x)$ and $c(x)$ are continuous and nonnegative and bounded;

(H₂):

$a(x)\leq\lim_{|x|\to\infty}a(x)\triangleq a_{\infty}$, $b(x)\geq\lim_{|x|\to\infty}b(x)\triangleq b_{\infty}$ and $c(x)\leq\lim_{|x|\to\infty}c(x)\triangleq c_{\infty}$ and one of these inequalities is strict on a set of positive measure.

Our main result is as follows.

Theorem 1.1

Suppose (g) and (H₁)–(H₂) hold. Problem (1.1) admits at least a ground state solution if $2< q<4<p<6$.

To prove our main theorem, we need to introduce the limiting equation at infinity related to problem (1.7)

$$ -\Delta v+a_{\infty}\frac{G^{-1}(v)}{g(G^{-1}(v))}+\phi_{v} v=b_{\infty}|v|^{p-2}v-c_{\infty}|v|^{q-2}v, $$

(1.8)

which plays a vital role.

The outline of this paper is as follows. In Sect. 2, we introduce and provide several lemmas. In Sect. 3, we prove the limiting equation (1.8) has a ground state solution. The proof of Theorem 1.1 is completed in Sect. 4.

Notations

Throughout this paper we shall denote by C and $C_{i}$ ($i=1, 2,\ldots$) various positive constants whose exact value may change from line to line but are not essential to the analysis of the problem. $L^{p}(\mathbb {R}^{3})$ ($1\leq p\leq+\infty$) is the usual Lebesgue space with the standard norm $|u|_{p}$. We use “→” and “⇀” to denote the strong and weak convergence in the related function space, respectively. For any $\rho>0$ and any $x\in \mathbb {R}^{3}$, $B_{\rho}(x)$ denotes the ball of radius ρ centered at x, that is, $B_{\rho}(x):=\{y\in \mathbb {R}^{3}:|y-x|<\rho \}$.

2 Variational settings and preliminaries

In this section, we will give some lemmas which are useful for the main results. To solve problem (1.1), we firstly introduce some function spaces. Throughout the paper, we consider the Hilbert space $H^{1}(\mathbb {R}^{3})$ with the inner product and the norm as follows:

$$\begin{aligned}& (u,v)= \int_{\mathbb {R}^{3}}\nabla u\nabla v+a(x)uv\,dx\quad \text{and} \\& \|u\|= \biggl( \int_{\mathbb {R}^{3}}|\nabla u|^{2}+ a(x)u^{2}\,dx \biggr)^{\frac{1}{2}}, \quad \forall u,v\in H^{1}\bigl(\mathbb {R}^{3} \bigr) \end{aligned}$$

which are equivalent to the usual inner product and the norm in $H^{1}(\mathbb {R}^{3})$ because of the assumption (H₁).

Lemma 2.1

Assume (g), the functions $g(s)$ and $G(s)$ have the following properties:

• (1) both G and $G^{-1}$ are odd and for all $s\geq0$, $t\geq0$, we have

  $$G(t)\leq g(t)t,\qquad s/g\bigl(G^{-1}(s)\bigr)\leq G^{-1}(s) \leq s; $$

• (2) for all $s\geq0$, $G^{-1}(s)/s$ is non-increasing and

  $$\lim_{s\to0}\frac{G^{-1}(s)}{s}=\frac{1}{g(0)}=1\quad \textit{and}\quad \lim_{s\to\infty}\frac{G^{-1}(s)}{s}= \textstyle\begin{cases} \frac{1}{g(\infty)}, & \textit{if } g \textit{ is bounded}, \\ 0, & \textit{if } g \textit{ is unbounded}. \end{cases} $$

Proof

The proof is standard, see [18, 25] for example. □

Denote

$$ f(x,s)\triangleq b(x)|s|^{p-2}s-c(x)|s|^{q-2}s+a(x)s-a(x) \frac {G^{-1}(s)}{g(G^{-1}(s))} $$

(2.1)

and

$$ F(x,s)\triangleq\frac{1}{p}b(x)|s|^{p}- \frac{1}{q}c(x)|s|^{q}+\frac {1}{2}a(x)s^{2}- \frac{1}{2}a(x) \bigl\vert G^{-1}(s) \bigr\vert ^{2}. $$

(2.2)

Lemma 2.2

The functions $f(x,s)$ and $F(x,s)$ satisfy the following properties under the assumptions (g) and (H₁)–(H₂).

• (1) $f(x,s)=o(s)$ and $F(x,s)=o(s^{2})$ as $s\to0^{+}$ uniformly in $x\in \mathbb {R}^{3}$;

• (2) $f(x,s)=o(s^{5})$ and $F(x,s)=o(s^{6})$ as $s\to+\infty$ uniformly in $x\in \mathbb {R}^{3}$;

• (3) $\frac{1}{4}f(x,s)s-F(x,s)+\frac{1}{4}a(x)s^{2}\geq\frac {1}{4}a(x)|G^{-1}(s)|^{2}$ uniformly in $x\in \mathbb {R}^{3}$;

• (4) $\lim_{|x|\to\infty}f(x,s)=f_{\infty}(s)$ exists and

  $$f_{\infty}(s)\triangleq b_{\infty}|s|^{p-2}s-c_{\infty}|s|^{q-2}s+a_{\infty}s-a_{\infty}\frac{G^{-1}(s)}{g(G^{-1}(s))}. $$

  Furthermore, we have

  $$ 3f_{\infty}(s)s-f^{\prime}_{\infty}(s)s^{2}-2a_{\infty}s^{2}\leq0\quad \textit {for any } s\in \mathbb {R}. $$

  (2.3)

Proof

Points (1)–(2) are obvious, see [18] for example. Recalling that $2< q<4<p<6$ and using Lemma 2.1(1), we have

$$ \begin{aligned}[b] \frac{1}{4}f(x,s)s-F(x,s)+ \frac{1}{4}a(x)s^{2}&=\frac{p-4}{4p}b(x) \vert s \vert ^{p} +\frac{4-q}{4q}c(x) \vert s \vert ^{q} + \frac{1}{4}a(x) \bigl\vert G^{-1}(s) \bigr\vert ^{2} \\ &\quad {}+\frac{1}{4}a(x) \biggl[ \bigl\vert G^{-1}(s) \bigr\vert ^{2}-\frac {G^{-1}(s)s}{g(G^{-1}(s))} \biggr], \end{aligned} $$

(2.4)

which yields the point (3). The first part of point (4) follows from our assumption (H₂). Using $2< q<4<p<6$ and Lemma 2.1(1) again, we obtain

$$\begin{aligned}& 3f_{\infty}(s)s-f^{\prime}_{\infty}(s)s^{2}-2a_{\infty}s^{2} \\& \quad = (4-p)b_{\infty}|s|^{p}+(q-4)c_{\infty}|s|^{q} \\& \qquad {}+a_{\infty}\frac{g(G^{-1}(s))s^{2}-G^{-1}(s)g^{\prime }(G^{-1}(s))s^{2}-3G^{-1}(s)g^{2}(G^{-1}(s))s}{g^{3}(G^{-1}(s))} \\& \quad \leq a_{\infty}\frac{g(G^{-1}(s))s^{2}-G^{-1}(s)g^{\prime }(G^{-1}(s))s^{2}-3G^{-1}(s)g^{2}(G^{-1}(s))s}{g^{3}(G^{-1}(s))} \\& \quad \leq a_{\infty}\frac{-2g(G^{-1}(s))s^{2}-G^{-1}(s)g^{\prime }(G^{-1}(s))s^{2}}{g^{3}(G^{-1}(s))}\leq0, \end{aligned}$$

which gives the second part of point (4). The proof is complete. □

We collect some properties of the functions $\phi_{v}$.

Lemma 2.3

(see [29, Lemmas 2.1–2.2])

For any $v\in H^{1}(\mathbb {R}^{3})$, we have:

• (1) There exists a constant $C>0$ such that $\int_{\mathbb {R}^{3}}\phi _{v}v^{2}\,dx\leq C|v|_{12/5}^{4}$;

• (2) $\phi_{v}(x)\geq0$, $\phi_{tv}(x)=t^{2}\phi_{v}(x)$ and $\phi _{v(\cdot+y)}=\phi_{v}(\cdot+y)$;

• (3) If $v_{n}\rightharpoonup v$ in $H^{1}(\mathbb {R}^{3})$ and $v_{n}\to v$ a.e. in $\mathbb {R}^{3}$, we have

  $$ \lim_{n\to\infty} \biggl[ \int_{\mathbb {R}^{3}}\phi_{v_{n}}v_{n}^{2}\,dx- \int_{\mathbb {R}^{3}}\phi_{v_{n}-v}(v_{n}-v)^{2} \,dx- \int_{\mathbb {R}^{3}}\phi_{v}v^{2}\,dx \biggr]=0 $$

  (2.5)

  and

  $$ \lim_{n\to\infty} \biggl[ \int_{\mathbb {R}^{3}}\phi_{v_{n}}v_{n}\varphi \,dx- \int _{\mathbb {R}^{3}}\phi_{v}v\varphi \,dx \biggr]=0 \quad \textit{for any } \varphi\in C^{\infty}_{0}\bigl(\mathbb {R}^{3} \bigr). $$

  (2.6)

We now introduce some definitions. Let $(X,\|\cdot\|)$ be a Banach space with its dual space $(X^{-1},\|\cdot\|_{*})$, and Φ be its functional on X. The $(\mathit{Ce})$ sequence (or $(\mathit{PS})$ sequence) at level $c\in \mathbb {R}$ ($(\mathit{Ce})_{c}$ sequence ($(\mathit{PS})_{c}$ sequence) in short) corresponding to Φ assumes that $\Phi(x_{n})\to c$ and $(1+\|x_{n}\| )\|\Phi^{\prime}(x_{n})\|_{*}\to0$ ($\Phi^{\prime}(x_{n})\to0$) as $n\to\infty$, where $\{x_{n}\}\subset X$. If for any $(\mathit{Ce})_{c}$ sequence $\{x_{n}\}$ in X, there exists a subsequence $\{x_{n_{k}}\}$ such that $x_{n_{k}}\to x_{0}$ in X for some $x_{0}\in X$, then we say that the functional Φ satisfies the so called $(\mathit{Ce})_{c}$ condition.

Lemma 2.4

The functional $J(v)$ satisfies the mountain-pass geometry, that is,

• (i) there exist $\eta,\rho>0$ such that $J(v)\geq\eta>0$ when $\| v\|=\rho$;

• (ii) there exists $e\in H^{1}(\mathbb {R}^{3})$ with $\|e\|>\rho$ such that $J(e)<0$.

Proof

(i) From Lemma 2.2(1)–(2), for any $\varepsilon >0$, there exists $C_{\varepsilon }>0$ such that

$$\begin{aligned} J(v) =& \frac{1}{2}\|v\|^{2}+\frac{1}{4} \int_{\mathbb {R}^{3}}\phi_{v}v^{2}\,dx- \int _{\mathbb {R}^{3}}F(x,v)\,dx \\ \geq& \frac{1}{2}\|v\|^{2}-\varepsilon \|v\|^{2}-C_{\varepsilon }\|v\|^{6}. \end{aligned}$$

It follows that

$$J(v) \geq C\|v\|^{2}-C\|v\|^{6} $$

if we choose sufficiently small $\varepsilon >0$ and $\rho>0$, which implies the result (i).

(ii) Choosing $v_{0}\in H^{1}(\mathbb {R}^{3})\setminus \{0\}$ and using Lemma 2.1(1), one has

$$\begin{aligned} J(tv_{0}) \leq&\frac{t^{2}}{2}\|v_{0}\|^{2}+ \frac{t^{4}}{4} \int_{\mathbb {R}^{3}}\phi _{v}v^{2}\,dx- \frac{t^{p}}{p} \int_{\mathbb {R}^{3}}b(x)|v_{0}|^{p}\,dx \\ &{}+ \frac{t^{q}}{q} \int _{\mathbb {R}^{3}}c(x)|v_{0}|^{q}\,dx \to-\infty \end{aligned}$$

as $t\to+\infty$. Hence letting $e=t_{0}v_{0}\in H^{1}(\mathbb {R}^{3})\setminus \{ 0\}$ with $t_{0}$ sufficiently large, we have $\|e\|>\rho$ and $J(e)<0$. □

By Lemma 2.4 and the variant mountain-pass theorem [36, Theorem 1], a $(\mathit{Ce})_{c}$ sequence of the functional $J(v)$ at the level

$$ c:=\inf_{\gamma\in\Gamma}\max_{t\in[0,1]}J\bigl( \gamma(t)\bigr)>0 $$

(2.7)

can be constructed, where the set of paths is defined as

$$\Gamma:= \bigl\{ \gamma\in C \bigl([0,1],H^{1}\bigl(\mathbb {R}^{3} \bigr) \bigr):\gamma (0)=0, J\bigl(\gamma(1)\bigr)< 0 \bigr\} . $$

In other words, there exists a sequence $\{v_{n}\}\subset H^{1}(\mathbb {R}^{3})$ such that

$$ J(v_{n})\to c, \bigl(1+ \Vert v_{n} \Vert \bigr) \bigl\Vert J^{\prime}(v_{n}) \bigr\Vert _{*}\to0 \quad \text{as } n\to \infty. $$

(2.8)

Lemma 2.5

Any sequence $\{v_{n}\}\subset H^{1}(\mathbb {R}^{3})$ verifying (2.8) is bounded.

Proof

Since $\{v_{n}\}\subset H^{1}(\mathbb {R}^{3})$ is a $(\mathit{Ce})_{c}$ sequence, we have

$$\begin{aligned} c+1 \geq& J(v_{n})-\frac{1}{4}\bigl\langle J^{\prime }(v_{n}),v_{n}\bigr\rangle \\ =&\frac{1}{4} \int_{\mathbb {R}^{3}}|\nabla v_{n}|^{2}\,dx+ \int_{\mathbb {R}^{3}} \biggl[\frac{1}{4}f(x,v_{n})v_{n}-F(x,v_{n})+ \frac{1}{4}a( x)v_{n}^{2} \biggr]\,dx \\ \stackrel{\text{(2.4)}}{\geq}&\frac{1}{4} \int_{\mathbb {R}^{3}}|\nabla v_{n}|^{2}\,dx+ \frac{1}{4} \int_{\mathbb {R}^{3}}a( x) \bigl\vert G^{-1}(v_{n}) \bigr\vert ^{2} \end{aligned}$$

(2.9)

and by Lemma 2.1(1),

$$\begin{aligned} \begin{aligned}[b] \int_{\mathbb {R}^{3}}a(x)v_{n}^{2}\,dx &= \int _{|G^{-1}(v_{n})|>1}a(x)v_{n}^{2}\,dx+ \int_{|G^{-1}(v_{n})|\leq1}a(x)v_{n}^{2}\,dx \\ & \leq C \int_{\mathbb {R}^{3}}v_{n}^{6}\,dx+g^{2}(1) \int_{|G^{-1}(v_{n})|\leq 1}a(x)\frac{v_{n}^{2}}{g^{2}(G^{-1}(v_{n}))}\,dx \\ &\leq C \int_{\mathbb {R}^{3}}v_{n}^{6}\,dx+g^{2}(1) \int_{\mathbb {R}^{3}}a(x) \bigl\vert G^{-1}(v_{n}) \bigr\vert ^{2}\,dx. \end{aligned} \end{aligned}$$

(2.10)

Combining (2.9)–(2.10) and the Sobolev inequality, the sequence $\{v_{n}\}$ is bounded. □

Let $\{v_{n}\}$ be a $(\mathit{Ce})$ sequence of J. Going to a subsequence if necessary, we may assume that $v_{n}\rightharpoonup v$ in $H^{1}(\mathbb {R}^{3})$, $v_{n}\to v$ in $L^{r}_{\mathrm{loc}}(\mathbb {R}^{3})$ with $r\in[1,6)$ and $v_{n}\to v$ a.e. in $\mathbb {R}^{3}$. Set $w_{n}=v_{n}-v$, by Lemma 2.2 we have the following lemma.

Lemma 2.6

(see [37, Lemma 1.3])

If $v_{n}\rightharpoonup v$ in $H^{1}(\mathbb {R}^{3})$ and $v_{n}\to v$ a.e. in $\mathbb {R}^{3}$, then

$$\lim_{n\to\infty} \biggl[ \int_{\mathbb {R}^{3}}F(x,v_{n})\,dx- \int_{\mathbb {R}^{3}}F(x,v)\,dx- \int_{\mathbb {R}^{3}}F(x,w_{n})\,dx \biggr]=0. $$

As a consequence of Lemma 2.6, we have the following lemma.

Lemma 2.7

Let $\{v_{n}\}$ be a $(\mathit{Ce})$ sequence of J at the level c, and set $w_{n}=v_{n}-v$, then $\{w_{n}\}$ is a $(\mathit{PS})$ sequence of J at the level $c-J(v)$.

Proof

We claim first that $J^{\prime}(v)=0$. In fact, it is enough to show that $\langle J^{\prime}(v),\varphi\rangle=0$ for any $\varphi\in C^{\infty}_{0}(\mathbb {R}^{3})$. By Lemma 2.2(1)–(2), it is easy to verify

$$\lim_{n\to\infty} \int_{\mathbb {R}^{3}}f(x,w_{n})\varphi \,dx=\lim _{n\to\infty } \biggl[ \int_{\mathbb {R}^{3}}f(x,v_{n})\varphi \,dx- \int_{\mathbb {R}^{3}}f(x,v)\varphi \,dx \biggr]=0. $$

Using the above formula and (2.6), we have

$$ 0=\lim_{n\to\infty}\bigl\langle J^{\prime}(v_{n}), \varphi\bigr\rangle =\bigl\langle J^{\prime}(v),\varphi\bigr\rangle $$

(2.11)

which yields the claim. By (2.11), we derive

$$\begin{aligned}& \lim_{n\to\infty}J(w_{n})=\lim_{n\to\infty} \bigl[J(v_{n})-J(v) \bigr]=c-J(v), \\& \lim_{n\to\infty}\bigl\langle J^{\prime}(w_{n}), \varphi\bigr\rangle =\lim_{n\to\infty} \bigl[\bigl\langle J^{\prime}(v_{n}),\varphi\bigr\rangle -\bigl\langle J^{\prime}(v),\varphi\bigr\rangle \bigr]=0, \end{aligned}$$

which show that $\{w_{n}\}$ is a $(\mathit{PS})$ sequence of J at the level $c-J(v)$. □

The proofs of the following lemmas can be found in the corresponding references.

Lemma 2.8

(see [38, 39])

Let $\{\rho_{n}\}$ be a sequence of nonnegative functions satisfying $|\rho_{n}|_{1}=\lambda$ and $\lambda >0$ is fixed, then there exists a subsequence, still denoted by $\{\rho_{n}\}$, satisfying one of the following two possibilities:

• (i) (Vanishing) for any fixed $R > 0$, we have

  $$\lim_{n\to\infty}\sup_{y\in \mathbb {R}^{N}} \int_{B_{R}(y)}\rho_{n}(x)\,dx=0. $$

• (ii) (Nonvanishing) there exist $\beta>0$, $\overline{R}\in (0,+\infty)$ and $\{y_{n}\}\subset \mathbb {R}^{N}$ such that

  $$\lim_{n\to\infty} \int_{B_{\overline{R}}(y_{n})}\rho_{n}(x)\,dx\geq \beta>0. $$

Lemma 2.9

(see [38, 39])

Assume that $\{u_{n}\}$ is bounded in $H^{1}(\mathbb {R}^{3})$ and satisfies

$$\lim_{n\to\infty}\sup_{y\in \mathbb {R}^{3}} \int_{B_{R}(y)}|u_{n}|^{2}\,dx=0, $$

for some $R>0$. Then $u_{n}\to0$ in $L^{r}(\mathbb {R}^{3})$ for every $2< r<6$.

3 The existence of ground state solution for limit equation at infinity

In this section, by employing Ekeland’s variational principle [40], we prove the existence of ground state solution for problem (1.8) which is the limit equation of problem (1.7) at infinity. To establish the ground sate solution of problem (1.8), we set

$$m_{\infty}=\inf_{u\in\mathcal{N}_{\infty}}J_{\infty}(u) \quad \text{and}\quad \mathcal{N}_{\infty}= \bigl\{ u\in H^{1}\bigl( \mathbb {R}^{3}\bigr)\setminus \{0\}:\bigl\langle J_{\infty}^{\prime}(u),u \bigr\rangle =0 \bigr\} . $$

Since $a_{\infty}$ is positive, the norm and inner product in this section are not distinguished by the norm and inner product used in the previous section. To show that the Nehair manifold $\mathcal{N}_{\infty}$ is nonempty and $m_{\infty}$ is well defined, we prove the following lemma.

Lemma 3.1

Assume (g) and (H₁)–(H₂), then we have the following properties:

• (a) For any $v\in H^{1}(\mathbb {R}^{3})\setminus \{0\}$, there exists a unique $t_{v}>0$ such that $t_{v}v\in\mathcal{N}_{\infty}$ and $J_{\infty}(t_{v}v)=\max_{t\geq0}J_{\infty}(tv)$. In particular, if $v\in\mathcal {N}_{\infty}$ we have $J_{\infty}(v)=\max_{t\geq0}J_{\infty}(tv)$;

• (b) There exists $\alpha> 0$ such that $\|u\|\geq\alpha$ for all $u\in\mathcal{N}_{\infty}$;

• (c) $J_{\infty}$ is bounded from below on $\mathcal{N}_{\infty}$ by a positive constant;

• (d) $J_{\infty}$ is coercive on $\mathcal{N}_{\infty}$, i.e. $J_{\infty}(v)\to\infty$ as $\|v\|\to\infty$ when $v\in\mathcal {N}_{\infty}$.

Proof

(a) Given $v\in H^{1}(\mathbb {R}^{3})\setminus \{0\}$, by [41, Lemma 2.4] and Lemma 2.2(1)–(2) we derive that

$$\xi(t)=c_{1}t^{2}+c_{2}t^{4}- \int_{\mathbb {R}^{3}}F_{\infty}(tv)\,dx $$

has a unique positive critical point which corresponds to its maximum, where the fact $F_{\infty}(tv)/t^{2}\to\infty$ as $t\to\infty$ is used. Let $\xi(t)=J_{\infty}(tv)$, then we can conclude the result (a).

(b) If $v\in\mathcal{N}_{\infty}$, using Lemma 2.2(1)–(2) one has

$$\begin{aligned} \|v\|^{2} &\leq\|v\|^{2}+ \int_{\mathbb {R}^{3}}\phi_{v}v^{2}\,dx= \int_{\mathbb {R}^{3}}f_{\infty}(v)v\,dx\leq \varepsilon \int_{\mathbb {R}^{3}}v^{2}\,dx+C_{\varepsilon }\int_{\mathbb {R}^{3}}v^{6}\,dx \\ & \leq C\varepsilon \|v\|^{2}+CC_{\varepsilon }\|v\|^{6}= \frac{1}{2}\|v\|^{2}+C\|v\|^{6} \end{aligned}$$

if we choose $C\varepsilon =1/2$ and the result (b) follows.

(c) If $v\in\mathcal{N}_{\infty}$, then

$$J_{\infty}(v)=J_{\infty}(v)-\frac{1}{4}\bigl\langle J_{\infty}^{\prime }(v),v\bigr\rangle \geq \frac{1}{4} \int_{\mathbb {R}^{3}}|\nabla v|^{2}\,dx+\frac{1}{4} \int_{\mathbb {R}^{3}}a_{\infty}\bigl\vert G^{-1}(v) \bigr\vert ^{2}\,dx, $$

which together with the result (b) and (2.10) gives the result (c).

(d) Combining the above formula and (2.10), the result (d) is obvious. □

Lemma 3.2

Assume (g), then $m_{\infty}=\inf_{u\in\mathcal{N}_{\infty}}J_{\infty}(u)$ can be attained.

Proof

If it is possible to verify that a minimizing sequence of $m_{\infty}$ is radially symmetric, the minimizer may be easily obtained. In particular, the minimizer is a critical point of $J_{\infty}$ restricted on $\mathcal{N}_{\infty}$. Finally, we proceed as follows.

Step 1: Any minimizing sequence of $m_{\infty}$ can be radially symmetric.

Let $\{v_{n}\}\subset H^{1}(\mathbb {R}^{3})$ be a minimizing sequence of $m_{\infty}$, that is, $\langle J^{\prime}_{\infty}(v_{n}),v_{n}\rangle=0$ and $J_{\infty}(v_{n})\to m_{\infty}$ as $n\to\infty$. According to the Schwarz symmetrization $v_{n}^{*}$ of $v_{n}$, we know that $v_{n}^{*}$ is continuous and nonnegative and satisfies

$$\begin{aligned}& \int_{\mathbb {R}^{3}} \bigl\vert \nabla v_{n}^{*} \bigr\vert ^{2}\,dx\leq \int_{\mathbb {R}^{3}} \vert \nabla v_{n} \vert ^{2}\,dx \quad \text{and} \\& \int_{\mathbb {R}^{3}}h\bigl(v_{n}^{*}\bigr)\,dx= \int_{\mathbb {R}^{3}}h(v_{n})\,dx\quad \text{for any } h(v_{n})\in L^{1}\bigl(\mathbb {R}^{3}\bigr), \end{aligned}$$

which give $\langle J^{\prime}_{\infty}(v_{n}^{*}),v_{n}^{*}\rangle\leq \langle J^{\prime}_{\infty}(v_{n}),v_{n}\rangle=0$. It is obvious that $\langle J^{\prime}_{\infty}(tv_{n}^{*}),tv_{n}^{*}\rangle>0$ for sufficiently small $t>0$. Hence there is $t_{0}\in(0,1]$ satisfying $\langle J^{\prime}_{\infty}(t_{0}v_{n}^{*}),t_{0}v_{n}^{*}\rangle=0$ and then

$$\begin{aligned} m_{\infty} \leq&J_{\infty}\bigl(t_{0}v_{n}^{*} \bigr)=J_{\infty}\bigl(t_{0}v_{n}^{*}\bigr)- \frac {1}{4}\bigl\langle J^{\prime}_{\infty}\bigl(t_{0}v_{n}^{*}\bigr),t_{0}v_{n}^{*} \bigr\rangle \\ =&\frac{t_{0}^{2}}{4} \int_{\mathbb {R}^{3}} \bigl\vert \nabla v_{n}^{*} \bigr\vert ^{2}\,dx+\frac{1}{4} \int _{\mathbb {R}^{3}}a_{\infty}\bigl\vert G^{-1} \bigl(t_{0}v_{n}^{*}\bigr) \bigr\vert ^{2}\,dx+ \frac{p-4}{4p}t_{0}^{p} \int_{\mathbb {R}^{3}}b_{\infty}\bigl\vert v_{n}^{*} \bigr\vert ^{p}\,dx \\ &{}+\frac{4-q}{4q}t_{0}^{q} \int_{\mathbb {R}^{3}}c_{\infty}\bigl\vert v_{n}^{*} \bigr\vert ^{q}\,dx \\ \leq&\frac{1}{4} \int_{\mathbb {R}^{3}} \bigl\vert \nabla v_{n}^{*} \bigr\vert ^{2}\,dx+\frac{1}{4} \int _{\mathbb {R}^{3}}a_{\infty}\bigl\vert G^{-1} \bigl(v_{n}^{*}\bigr) \bigr\vert ^{2}\,dx+\frac{p-4}{4p} \int_{\mathbb {R}^{3}}b_{\infty}\bigl\vert v_{n}^{*} \bigr\vert ^{p}\,dx \\ &{}+\frac{4-q}{4q} \int_{\mathbb {R}^{3}}c_{\infty}\bigl\vert v_{n}^{*} \bigr\vert ^{q}\,dx \\ \leq&\frac{1}{4} \int_{\mathbb {R}^{3}} \vert \nabla v_{n} \vert ^{2}\,dx+\frac{1}{4} \int_{\mathbb {R}^{3}}a_{\infty}\bigl\vert G^{-1}(v_{n}) \bigr\vert ^{2}\,dx+\frac{p-4}{4p} \int_{\mathbb {R}^{3}}b_{\infty} \vert v_{n} \vert ^{p}\,dx \\ &{}+\frac{4-q}{4q} \int_{\mathbb {R}^{3}}c_{\infty} \vert v_{n} \vert ^{q}\,dx \\ =&J_{\infty}(v_{n})-\frac{1}{4}\bigl\langle J^{\prime}_{\infty}(v_{n}),v_{n}\bigr\rangle =J_{\infty}(v_{n})\to m_{\infty} \end{aligned}$$

which yields $t_{0}=1$. Therefore we conclude that $\langle J^{\prime }_{\infty}(v_{n}^{*}),v_{n}^{*}\rangle=0$ and $J_{\infty}(v_{n}^{*})\to m_{\infty}$ as $n\to\infty$. So the proof of Step 1 is complete.

Step 2: Any minimizing sequence of $m_{\infty}$ contains a strongly convergent subsequence.

Let $\{v_{n}\}\subset H^{1}(\mathbb {R}^{3})$ be a minimizing sequence of $m_{\infty}$, then similar to Lemma 2.5 we know that $\{v_{n}\}$ is bounded in $H^{1}(\mathbb {R}^{3})$. From the Step 1, we know that $\{v_{n}\}$ is radially symmetric. Up to a subsequence, there exists $v\in H^{1}(\mathbb {R}^{3})$ such that $v_{n}\rightharpoonup v$ in $H^{1}(\mathbb {R}^{3})$, $v_{n}\to v$ in $L^{r}(\mathbb {R}^{3})$ with $r\in(2,6)$ and $v_{n}\to v$ a.e. in $\mathbb {R}^{3}$. By using Fatou’s lemma, one has

$$\begin{aligned}& \int_{\mathbb {R}^{3}}|\nabla v|^{2}\,dx+ \int_{\mathbb {R}^{3}}a_{\infty}\bigl|G^{-1}(v)\bigr|^{2} \,dx+ \int _{\mathbb {R}^{3}}\phi_{v}v^{2}\,dx \\& \quad \leq\liminf_{n\to\infty} \biggl( \int_{\mathbb {R}^{3}}|\nabla v_{n}|^{2}\,dx+ \int_{\mathbb {R}^{3}}a_{\infty}\bigl\vert G^{-1}(v_{n}) \bigr\vert ^{2}\,dx+ \int_{\mathbb {R}^{3}}\phi _{v_{n}}{v_{n}}^{2} \,dx \biggr) \\& \quad = \liminf_{n\to\infty} \biggl( \int_{\mathbb {R}^{3}}b_{\infty}| v_{n}|^{p}\,dx- \int_{\mathbb {R}^{3}}c_{\infty}|v_{n}|^{q}\,dx \biggr) \\& \quad = \int_{\mathbb {R}^{3}}b_{\infty}|v|^{p}\,dx- \int_{\mathbb {R}^{3}}c_{\infty}|v|^{q}\,dx, \end{aligned}$$

which gives $\langle J^{\prime}_{\infty}(v),v\rangle\leq0$. It is easy to check that $\langle J^{\prime}_{\infty}(tv),tv\rangle>0$ for sufficiently small $t>0$. Hence there exists $\overline{t}\in(0,1]$ such that $\langle J^{\prime}_{\infty}(\overline{t}v),\overline {t}v\rangle=0$ and then

$$\begin{aligned} m_{\infty} \leq&J_{\infty}(\overline{t}v)=J_{\infty}(\overline {t}v)-\frac{1}{4}\bigl\langle J^{\prime}_{\infty}( \overline{t}v),\overline {t}v\bigr\rangle \leq J_{\infty}(v)- \frac{1}{4}\bigl\langle J^{\prime}_{\infty}(v),v\bigr\rangle \\ \leq& \liminf_{n\to\infty} \biggl[J_{\infty}(v_{n})- \frac {1}{4}\bigl\langle J^{\prime}_{\infty}(v_{n}),v_{n} \bigr\rangle \biggr]=m_{\infty}, \end{aligned}$$

which indicates that $\overline{t}=1$. Thus we have $\langle J^{\prime }_{\infty}(v),v\rangle=0$ and $J_{\infty}(v)=m_{\infty}$. □

Proposition 3.3

Assume (g), any minimizer of $m_{\infty}$ is a critical point of $J_{\infty}$ in $H^{1}(\mathbb {R}^{3})$.

Proof

If v is a minimizer of $m_{\infty}$, according to Lemma 3.2 we know that v is a critical point of ${J_{\infty }}|_{\mathcal{N}_{\infty}}$, that is, $v\in\mathcal{N}_{\infty}$ and ${J_{\infty}^{\prime}}|_{\mathcal{N}_{\infty}}(v)=0$. Hence there is a Lagrange multiplier $\lambda\in \mathbb {R}$ such that $J_{\infty}^{\prime }(v)=\lambda\Psi_{\infty}^{\prime}(v)$, where $\Psi_{\infty}(v)=\langle J_{\infty}^{\prime}(v),v\rangle$. To end the proof, it is enough to show that $\lambda=0$.

In fact, using $0=\langle J_{\infty}^{\prime}(v),v\rangle=\lambda \langle\Psi_{\infty}^{\prime}(v),v\rangle$ and $v\in H^{1}(\mathbb {R}^{3})\setminus \{0\}$ we have

$$\begin{aligned} \bigl\langle \Psi_{\infty}^{\prime}(v),v\bigr\rangle =&2 \int_{\mathbb {R}^{3}}|\nabla v|^{2}+a_{\infty}v^{2}\,dx+4 \int_{\mathbb {R}^{3}}\phi_{v}v^{2}\,dx - \int_{\mathbb {R}^{3}} \bigl[f_{\infty}^{\prime}(v)v^{2}+f_{\infty}(v)v \bigr]\,dx \\ =&-2 \int_{\mathbb {R}^{3}}|\nabla v|^{2}\,dx+ \int_{\mathbb {R}^{3}} \bigl[3f_{\infty}(v)v-f_{\infty}^{\prime}(v)v^{2}-2a_{\infty}v^{2} \bigr]\,dx \\ \stackrel{\text{(2.3)}}{\leq}&-2 \int_{\mathbb {R}^{3}}|\nabla v|^{2}\,dx< 0, \end{aligned}$$

which implies that $\lambda=0$. Hence the proof is complete. □

Theorem 3.4

Assume (g), the system (1.8) has a ground state solution.

Proof

In view of Lemma 3.1, we know that the Nehair manifold $\mathcal{N}_{\infty}$ is nonempty and $m_{\infty}$ is well defined. It follows from Lemma 3.2 that $m_{\infty}$ is attained by some $v\in\mathcal{N}_{\infty}$ and v is a critical point of ${J_{\infty}}|{\mathcal{N}_{\infty}}$. By Proposition 3.4, we have $J_{\infty}^{\prime}(v)=0$ in the whole space $H^{1}(\mathbb {R}^{3})$. Consequently, we have shown that $J(v)=m_{\infty}>0$ and $J_{\infty}^{\prime}(v)=0$, which show that v is a ground state solution of problem (1.8). The proof is complete. □

4 Proof of Theorem 1.1

In this section, we prove Theorem 1.1 by applying the variant mountain-pass theorem [36]. In the following, we will verify that the mountain-pass value c given by (2.7) is satisfied.

Lemma 4.1

Suppose that (g) and (H₁)–(H₂) hold, then $0< c< m_{\infty}$.

Proof

By Theorem 3.4, we know that there exists $v\in H^{1}(\mathbb {R}^{3})$ such that $J_{\infty}^{\prime}(v)=0$ and $J_{\infty}(v)=m_{\infty}>0$. Since $J(0)=0$ and $\lim_{t\to\infty}J(tv)=-\infty$, there exists $\tilde{t}>0$ such that

$$J(\tilde{t}v)=\max_{t\geq0}J(tv). $$

Choosing a sufficient large $t_{0}>0$ to satisfy $J(t_{0}v)<0$, then $\gamma_{0}(t)=tt_{0}v\in\Gamma$ and hence

$$c\leq\max_{t\in[0,1]}J\bigl(\gamma_{0}(t)\bigr)\leq\max _{t\geq 0}J(tv)=J(\tilde{t}v)< J_{\infty}(\tilde{t}v) \leq J_{\infty}(v)=m_{\infty}, $$

where (H₂) yields the strict inequality. The proof is complete. □

Proposition 4.2

Assume that (g) and (H₁)–(H₂) hold, then $J(v)$ satisfies the $(\mathit{Ce})_{c}$ condition if $c\in(0,m_{\infty})$.

Proof

Let $\{v_{n}\}$ be a $(\mathit{Ce})_{c}$ sequence of $J(v)$. Similar to Lemma 2.5, we find that $\{v_{n}\}$ is bounded in $H^{1}(\mathbb {R}^{3})$ and there exists a subsequence, still denoted by $\{v_{n}\}$, such that

$$\textstyle\begin{cases} v_{n}\rightharpoonup v, &\text{in } H^{1}(\mathbb {R}^{3}), \\ v_{n}\to v, &\text{in } L^{r}_{\mathrm{loc}}(\mathbb {R}^{3}) \text{ for } 1\leq r< 6, \\ v_{n}\to v, &\text{a.e. in } \mathbb {R}^{3}, \end{cases} $$

and $J^{\prime}(v)=0$ with $J(v)\geq0$ by (2.4). Denote $w_{n}=v_{n}-v$, then it follows from Lemma 2.7 that $\{w_{n}\}$ is a $(\mathit{PS})$ sequence of $J(v)$ at the level $c-J(v)$. To prove that $\|w_{n}\|\to0$ as $n\to\infty$, we divide our proof into the following two steps.

Step 1: The nonvanishing case for $\rho_{n}=|w_{n}|_{2}^{2}$ in Lemma 2.8 can never occur.

Arguing indirectly, by Lemma 2.8 we know that there exist $\beta>0$, $\overline{R}\in(0,+\infty)$ and $\{y_{n}\}\subset \mathbb {R}^{N}$ such that

$$ \liminf_{n\to\infty} \int_{B_{\overline{R}}(y_{n})}|w_{n}|^{2}\,dx\geq \beta>0. $$

(4.1)

Without loss of generality, we choose $|y_{n}|\to\infty$ as $n\to \infty$. Otherwise, $\{w_{n}\}$ is tight, and thus $|w_{n}|_{2}\to0$ as $n\to\infty$, which yields a contradiction to (4.1). Denote $\overline{w}_{n}(x)=w_{n}(x+y_{n})$. Since $\limsup_{n\to\infty}\| \overline{w}_{n}\|=\limsup_{n\to\infty}\|w_{n}\|\leq C<+\infty$, we may assume that there exists $w_{0}\in H^{1}(\mathbb {R}^{3})$ such that $\overline{w}_{n}\rightharpoonup w_{0}$ in $H^{1}(\mathbb {R}^{3})$, $\overline{w}_{n}\to w_{0}$ in $L^{r}_{\mathrm{loc}}(\mathbb {R}^{3})$ with $r\in[1,6)$ and $\overline{w}_{n}\to w_{0}$ a.e. in $\mathbb {R}^{3}$. We then claim that $\{ \overline{w}_{n}\}$ is a $(\mathit{PS})$ sequence of $J_{\infty}(v)$ at the level $c-J(v)$. In fact, in view of $\lim_{n\to\infty}a(x+y_{n})=a_{\infty}$, we have

$$\int_{\mathbb {R}^{3}}a(x) \bigl\vert G^{-1}(w_{n}) \bigr\vert ^{2}\,dx= \int_{\mathbb {R}^{3}}a(x+y_{n}) \bigl\vert G^{-1}( \overline{w}_{n}) \bigr\vert ^{2}\,dx = \int_{\mathbb {R}^{3}}a_{\infty}\bigl\vert G^{-1}( \overline{w}_{n}) \bigr\vert ^{2}\,dx+o(1) $$

and by $\lim_{n\to\infty}b(x+y_{n})=b_{\infty}$ and $\lim_{n\to\infty }c(x+y_{n})=c_{\infty}$,

$$\begin{aligned}& \int_{\mathbb {R}^{3}}b(x)|w_{n}|^{p}\,dx= \int_{\mathbb {R}^{3}}b_{\infty}|\overline{w}_{n}|^{p} \,dx+o(1), \\& \int_{\mathbb {R}^{3}}c(x)|w_{n}|^{q}\,dx= \int_{\mathbb {R}^{3}}c_{\infty}|\overline{w}_{n}|^{q} \,dx+o(1), \end{aligned}$$

which give

$$\begin{aligned} J(w_{n}) &=\frac{1}{2} \int_{\mathbb {R}^{3}}|\nabla w_{n}|^{2}+a(x)w_{n}^{2} \,dx+\frac {1}{4} \int_{\mathbb {R}^{3}}\phi_{w_{n}}w_{n}^{2}\,dx- \int_{\mathbb {R}^{3}}F(x,w_{n})\,dx \\ & =\frac{1}{2} \int_{\mathbb {R}^{3}}|\nabla\overline{w}_{n}|^{2}+a_{\infty}\overline{w}_{n}^{2}\,dx+\frac{1}{4} \int_{\mathbb {R}^{3}}\phi_{\overline{w}_{n}}\overline{w}_{n}^{2} \,dx- \int_{\mathbb {R}^{3}}F_{\infty}(\overline{w}_{n}) \,dx+o(1) \\ &=J_{\infty}(\overline{w}_{n})+o(1). \end{aligned}$$

On the other hand, since $\lim_{n\to\infty}a(x+y_{n})=a_{\infty}$, for any $\phi\in C^{\infty}_{0}(R^{N})$ we have

$$\begin{aligned} &\biggl\vert \int_{\mathbb {R}^{3}} \bigl[a(x+y_{n})-a_{\infty}\bigr] \frac {G^{-1}(\overline{w}_{n})}{g(G^{-1}(\overline{w}_{n}))}\phi \,dx \biggr\vert \\ &\quad \leq \int_{\mathbb {R}^{3}} \bigl\vert \bigl[a(x+y_{n})-a_{\infty}\bigr]\overline{w}_{n}\phi \bigr\vert \,dx \\ &\quad \leq C \biggl( \int_{\mathbb {R}^{3}} \bigl\vert a(x+y_{n})-a_{\infty}\bigr\vert ^{2}|\phi|^{2} \biggr)^{\frac{1}{2}}\to0. \end{aligned}$$

Denote $\phi_{n}(x)=\phi(x-y_{n})$, we can deduce that

$$\begin{aligned} &\int_{\mathbb {R}^{3}}a(x)\frac{G^{-1}(w_{n})}{g(G^{-1}(w_{n}))}\phi _{n}\,dx \\ &\quad = \int_{\mathbb {R}^{3}}a(x+y_{n})\frac{G^{-1}(\overline{w}_{n})}{g(G^{-1}(\overline{w}_{n}))}\phi \,dx \\ &\quad = \int_{\mathbb {R}^{3}}a_{\infty}\frac{G^{-1}(\overline{w}_{n})}{g(G^{-1}(\overline{w}_{n}))}\phi \,dx+ \int_{\mathbb {R}^{3}} \bigl[a(x+y_{n})-a_{\infty}\bigr] \frac{G^{-1}(\overline{w}_{n})}{g(G^{-1}(\overline{w}_{n}))}\phi \,dx \\ &\quad = \int_{\mathbb {R}^{3}}a_{\infty}\frac{G^{-1}(\overline{w}_{n})}{g(G^{-1}(\overline{w}_{n}))}\phi \,dx+o(1). \end{aligned}$$

(4.2)

Similarly, we derive

$$ \begin{aligned} & \int_{\mathbb {R}^{3}}b(x)|w_{n}|^{p-2}w_{n} \phi_{n}\,dx= \int_{\mathbb {R}^{3}}b_{\infty}|\overline{w}_{n}|^{p-2} \overline{w}_{n}\phi \,dx +o(1), \\ & \int_{\mathbb {R}^{3}}c(x)|w_{n}|^{q-2}v_{n} \phi_{n}\,dx= \int_{\mathbb {R}^{3}}c_{\infty}|\overline{w}_{n}|^{q-2} \overline{w}_{n}\phi \,dx +o(1). \end{aligned} $$

(4.3)

Combining (4.2) and (4.3), we have

$$\bigl\langle J_{\infty}^{\prime}(\overline{w}_{n}),\phi \bigr\rangle = \bigl\langle J^{\prime}(w_{n}),\phi_{n} \bigr\rangle +o(1)=o(1). $$

Hence the claim is true. Furthermore, we can conclude that $J_{\infty}^{\prime}(w_{0})=0$. We now use (4.1) to show that $w_{0}\not \equiv0$. In the contrary case $\overline{w}_{n}\to0$ in $L^{2}_{\mathrm{loc}}(\mathbb {R}^{3})$, we have

$$\begin{aligned} 0 =&\lim_{n\to\infty} \int_{B_{\overline{R}}(0)}|\overline{w}_{n}|^{2}\,dx= \lim _{n\to\infty} \int_{B_{\overline{R}}(y_{n})}|w_{n}|^{2}\,dx \\ \geq&\liminf _{n\to\infty} \int_{B_{\overline {R}}(y_{n})}|w_{n}|^{2}\,dx\geq\beta>0, \end{aligned}$$

which yields a contradiction. Thus $w_{0}\not\equiv0$.

Denote $z_{n}=\overline{w}_{n}-w_{0}$, by the Brézis–Lieb lemma [42] we easily get

$$J_{\infty}(\overline{w}_{n})=J_{\infty}(z_{n})+J_{\infty}(w_{0})+o(1) $$

and

$$J_{\infty}^{\prime}(\overline{w}_{n})=J_{\infty}^{\prime}(z_{n})+J_{\infty}^{\prime}(w_{0})+o(1)=J_{\infty}^{\prime}(z_{n})+o(1). $$

Hence we have

$$\liminf_{n\to\infty}J_{\infty}(z_{n}) = \liminf _{n\to\infty} \biggl[J_{\infty}(z_{n})- \frac{1}{4}\bigl\langle J_{\infty}^{\prime}(z_{n}),z_{n} \bigr\rangle \biggr] \stackrel{\text{(2.4)}}{\geq} 0 $$

and then

$$\begin{aligned} c & =J(v_{n})+o(1)=J(v)+J(w_{n})+o(1)=J(v)+J_{\infty}( \overline{w}_{n})+o(1) \\ &=J(v)+J_{\infty}(z_{n})+J_{\infty}(w_{0})+o(1), \end{aligned}$$

which implies that

$$c\geq J_{\infty}(w_{0})\geq m_{\infty}, $$

a contradiction. So the proof of Step 1 is complete.

Step 2: $\|w_{n}\|\to0$ as $n\to\infty$.

In fact, as a consequence of Step 1 and Lemma 2.8, for any fixed $R>0$, we have

$$\lim_{n\to\infty}\sup_{y\in \mathbb {R}^{3}} \int_{B_{R}(y)}w_{n}^{2}\,dx=0. $$

By Lemma 2.9, we have

$$\lim_{n\to\infty} \int_{\mathbb {R}^{3}}|w_{n}|^{r}\,dx=0 \quad \text{for all } 2< r< 6. $$

Since $2< q< p<6$, by (H₁) we have

$$\lim_{n\to\infty} \int_{\mathbb {R}^{3}}b(x)|w_{n}|^{p}\,dx=\lim _{n\to\infty} \int _{\mathbb {R}^{3}}c(x)|w_{n}|^{q}\,dx=0. $$

Therefore

$$\lim_{n\to\infty} \biggl( \int_{\mathbb {R}^{3}}|\nabla w_{n}|^{2}\,dx + \int_{\mathbb {R}^{3}}a(x) \bigl\vert G^{-1}(w_{n}) \bigr\vert ^{2}\,dx \biggr)=\lim_{n\to\infty}\bigl\langle J^{\prime}(w_{n}),w_{n}\bigr\rangle =0, $$

which together with (2.10) yields $\|w_{n}\|\to0$ as $n\to \infty$.

Summing the above two steps, we obtain $v_{n}\to v$ in $H^{1}(\mathbb {R}^{3})$ as $n\to\infty$. □

Complement of proof of Theorem 1.1

By Lemma 2.4 and the variant mountain-pass theorem [36], a sequence verifying (2.8) can be obtained. Using Lemma 4.1 and Proposition 4.2, there exists $v\in H^{1}(\mathbb {R}^{3})$ such that $J^{\prime}(v)=0$ and $J(v)=c>0$, which imply that v is a nontrivial solution to (1.1). To show the existence of ground state solution, we set

$$m=\inf_{u\in\mathcal{N}}J(u) \quad \text{and}\quad \mathcal{N}= \bigl\{ u \in H^{1}\bigl(\mathbb {R}^{3}\bigr)\setminus \{0\}:\bigl\langle J^{\prime}(u),u\bigr\rangle =0 \bigr\} . $$

Obviously, we have $c\geq m$. On the other hand, for any $v\in\mathcal {N}$, choosing a sufficient large $t_{0}>0$ to satisfy $J(t_{0}v)<0$, then $\gamma_{0}(t)=tt_{0}v\in\Gamma$ and similar to the proof of Lemma 3.1 $(a)$ we obtain

$$c\leq\max_{t\in[0,1]}J\bigl(\gamma_{0}(t)\bigr)\leq\max _{t\geq0}J(tv)=J(v), $$

which indicates that $m\geq c$. Hence $J(v)=m>0$. The proof is complete. □

5 Conclusion

In this paper, we consider the existence of ground state solutions for a class of generalized quasilinear Schrödinger–Poisson systems in $\mathbb {R}^{3}$. By employing a change of variables constructed by Shen–Wang [19] and used in [18, 25] and the references therein, the generalized quasilinear systems are reduced to a semilinear one, whose associated functionals are well defined in the usual Sobolev space and satisfy the mountain-pass geometric. To obtain a ground state solution, the nonlinearities in [18, 25] are assumed to satisfy the monotone condition which is unnecessary in this paper and we believe that it is a partial extension which can reduce the restrictions on the nonlinearity.