Ground state solutions for asymptotically periodic Schrödinger–Poisson systems involving Hartree-type nonlinearities

Abstract

We use the non-Nehari manifold method to deal with the system

$$ \textstyle\begin{cases} -\Delta u+V(x)u+\phi u= (\int_{\mathbb{R}^{3}}\frac {Q(y)F(u(y))}{|x-y|^{\mu}}\,dy )Q(x)f(u(x)),\quad x\in\mathbb{R}^{3}, \\ -\Delta\phi=u^{2}, \quad u \in H^{1}(\mathbb{R}^{3}), \end{cases} $$

where $V(x)$ and $Q(x)$ are periodic and asymptotically periodic in x. Under some mild conditions on f, we establish the existence of the Nehari type ground state solutions in two cases: the periodic one and the asymptotically periodic case.

Introduction

In this paper, we are concerned with the existence of ground state solutions for the nonlinear system

$$ \textstyle\begin{cases} -\Delta u+V(x)u+\phi u= (\int_{\mathbb{R}^{3}}\frac {Q(y)F(u(y))}{|x-y|^{\mu }}\,dy )Q(x)f(u(x)),\quad x\in\mathbb{R}^{3}, \\ -\Delta\phi=u^{2},\quad u \in H^{1}(\mathbb{R}^{3}), \end{cases} $$

(1.1)

where $0<\mu<3$, $V,Q:\mathbb{R}^{3}\to\mathbb{R}$, $f:\mathbb{R}\to \mathbb{R}$, and $F(s)=\int_{0}^{s}f(t)\,dt$ satisfy the following assumptions, respectively:

• (V0) $V\in L^{\infty}(\mathbb{R}^{3})$ and $\inf_{x\in\mathbb {R}^{3}}V(x)>0$;

• (Q0) $Q\in L^{\infty}(\mathbb{R}^{3})$ and $\inf_{x\in\mathbb {R}^{3}}Q(x)>0$;

• (F1) There exist $2-\frac{\mu}{3}< q<6-\mu$ and $c_{0}>0$ such that $|f(s)|\le c_{0}(|s|^{1-\frac{\mu}{3}}+|s|^{q-1})$.

Consider the Sobolev space $H^{1}(\mathbb{R}^{3})$ endowed with the following norm and inner product:

$$ \langle u,v\rangle= \int_{\mathbb{R}^{3}}\bigl[\nabla u\nabla v+V(x)uv\bigr]\,dx, \qquad \lVert u \rVert^{2}= \int_{\mathbb{R}^{3}}\bigl[|\nabla u|^{2} +V(x)u^{2}\bigr] \,dx. $$

In view of $(Q0)$, the norm $\lVert\cdot\rVert$ is equivalent to the standard norm in $H^{1}(\mathbb{R}^{3})$.

It is well known that for any u ∈ $H^{1}({\mathbb{R}}^{3})$, there exists unique $\phi_{u}$ ∈ $\mathcal{D}^{1,2}(\mathbb{R}^{3})$ such that $-\Delta\phi=u^{2}$ by using the Lax–Milgram theorem. Inserting it into the first equation of (1.1), we have

$$ -\Delta u+V(x)u+\phi_{u}(x)u= \biggl( \int_{\mathbb{R}^{3}}\frac {Q(y)F(u(y))}{|x-y|^{\mu}}\,dy \biggr)Q(x)f\bigl(u(x)\bigr), $$

(1.2)

which is variational under our assumptions. Besides, its solution is the critical point of the functional defined in $H^{1}(\mathbb{R}^{3})$ by

$$\begin{aligned} I(u) =&\frac{1}{2} \int_{\mathbb{R}^{3}} \bigl(|\nabla u|^{2}+V(x)u^{2} \bigr) +\frac{1}{4} \int_{\mathbb{R}^{3}} \phi_{u} (x)u^{2}\,dx \\ &{}-\frac{1}{2} \int _{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac{Q(y)F(u(y))}{|x-y|^{\mu}} Q(x)F\bigl(u(x)\bigr)\,dx \,dy. \end{aligned}$$

(1.3)

Under our assumptions and Hardy–Littlewood–Sobolev inequality (see the following part of this paper), we know that $I(u)\in\mathcal {C}^{1}(H^{1}(\mathbb{R}^{3}),\mathbb{R})$. Furthermore, for any $v\in H^{1}(\mathbb{R}^{3})$,

$$\begin{aligned} \bigl\langle I'(u),v\bigr\rangle =& \int_{\mathbb{R}^{3}}\bigl[\nabla u\nabla v+V(x)uv\bigr]\,dx + \int_{\mathbb{R}^{3}} \phi_{u} (x)uv\, dx \\ &{}- \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac{Q(y)F(u(y))}{|x-y|^{\mu}} Q(x)f\bigl(u(x)\bigr)v(x)\,dx\,dy, \end{aligned}$$

(1.4)

and the corresponding Nehari manifold is defined by

$$ \mathcal{N}= \bigl\{ u \in H^{1}\bigl(\mathbb{R}^{3}\bigr) \setminus \{0\}:\bigl\langle I'(u),u\bigr\rangle =0 \bigr\} . $$

(1.5)

Therefore, if $u\in H^{1}(\mathbb{R}^{3})$ is a critical point of (1.3), then the pair $(u,\phi_{u})\in H^{1}(\mathbb{R}^{3})\times \mathcal{D}^{1,2}(\mathbb{R}^{3})$ is a solution of (1.1). So we just say $u \in H^{1}(\mathbb{R}^{3})$ is a weak solution of (1.1) in many cases for simplicity.

When ϕ is absent, System (1.1) will reduce to the generalized Choquard equation:

$$ -\Delta u+V(x)u= \biggl( \int_{\mathbb{R}^{3}}\frac{Q(y)F(u(y))}{|x-y|^{\mu}}\,dy \biggr)Q(x)f\bigl(u(x)\bigr), \quad u\in H^{1}\bigl(\mathbb{R}^{3}\bigr), $$

(1.6)

where $V(x)$ is an external potential and F is a primitive function of f. System (1.6) can be described as an approximation to Hartree–Fock theory of a one component plasma and arises in various branches of mathematical physics, see [1, 2]. It was also called Schrödinger–Newton equation when $V(x)=Q(x)\equiv1$ and $f(s)=s$. Zhang [3] proved the existence and multiplicity of solutions for (1.6). By variational method, Alves and Yang [4] established a new concentration behavior of nontrivial solutions for quasilinear Choquard equations. We point out that the generalized Hartree-type nonlinearity $(\int_{\mathbb{R}^{3}}\frac {Q(y)F(u(y))}{|x-y|^{\mu}}\,dy )Q(x)f(u(x))$ was widely applied in many physical and biological models. For example, Lu [5] obtained ground state solutions of a Kirchhoff-type equation with a Hartree-type nonlinearity.

On the other hand, when $\mu\to3$, System (1.1) will transform to the Schrödinger–Poisson system

$$ \textstyle\begin{cases} -\Delta u+V(x)u+\phi(x)u=g(x,u),\quad x\in\mathbb{R}^{3}, \\ -\Delta\phi=u^{2}, \quad u\in\mathbb{R}^{3}, \end{cases} $$

(1.7)

with $g(x,u)=Q(x)^{2}F(u)f(u)$. System (1.7) is also known as Schrödinger–Maxwell equations. It arises in quantum mechanics which is related to the study of nonlinear stationary Schrödinger equations interacting with the electrostatic field or the Hartree–Fock equation. The nonlinear term $g(x,u)$ is used in the Schrödinger equation to model the interaction among particles or an external nonlinear perturbation. For more details in the physical aspects, we refer readers to [6–8].

In recent years, enormous results have been obtained for System (1.7). When $V(x)\equiv1$ and $g(x,u)=|u|^{p-2}u$, a radial positive solution was obtained for $4< p<6$ in [9, 10]. Later, Ruiz [11] proved the existence of a positive radial solution for $3< p\le4$ by introducing the Nehari–Pohozaev manifold and establishing a key inequality. For more results on the Schrödinger–Maxwell system and related systems, we refer the readers to [12–27]. In [28], Azzollini and Pomponio obtained the existence of a ground state solution for the subcritical cases $3< p<5$ and the critical case $g(x,u)=|u|^{p-2}u+u^{5}$ with $4< p<6$. When $V(x)$ is periodic, that is,

• (V1) $V\in\mathcal{C}(\mathbb{R}^{3},(0,\infty))$ and $V(x)$ is 1-periodic in $x_{1}$, $x_{2}$, and $x_{3}$,

Zhao and Zhao [29] proved the existence of solutions by using the Nehari manifold approach. Sun and Ma [30] proved that System (1.7) has a ground state solution under the following assumption:

• (f1) $\frac{g(x,t)}{|t|^{3}}$ is increasing in t on $\mathbb {R}\setminus \{0\}$ for every $x\in\mathbb{R}^{3}$,

and some other hypotheses. It should be noted that the starting point of their approach is to show that for every $u\in H^{1}(\mathbb {R}^{3})\setminus \{0\}$, under assumption (f1), there exists unique $t(u)>0$ such that $t(u)u\in\mathcal{N}$, which is important in the remaining proof. Using the non-Nehari manifold approach, Chen and Tang [31] weakened (f1) to the following assumption:

• (f2) There exists $\theta_{0}\in(0,1)$ such that

  $$ \biggl[\frac{f(x,\tau)}{\tau^{3}}-\frac{f(x,t\tau)}{(t\tau)^{3}} \biggr]\operatorname{sign}(1-t)+ \theta_{0}V(x)\frac{1-t^{2}}{(t\tau)^{2}}\ge0,\quad \forall x\in \mathbb{R}^{3}, t>0, \tau\neq0. $$

Motivated by the works mentioned above, in this paper, we intend to generalize the results obtained in [29–31] to System (1.1). There are several pivotal difficulties we must overcome in the process. First, due to the competing effect between the two nonlocal terms $\int_{\mathbb{R}^{3}} \phi_{u} (x)u^{2}\,dx$ and $(\int_{\mathbb {R}^{3}}\frac{Q(y)F(u(y))}{|x-y|^{\mu}}\,dy )Q(x)f(u(x))$, the methods dealing with (1.6) become invalid. Secondly, the methods used in [29–31] rely heavily on assumptions (f1) or the weaker case (f2), so the approaches used in [29–31] cannot be applied directly to System (1.1) which involves a Hartree-type nonlinearity. Therefore, some new methods and tricks are required to address the existence of ground state solutions for (1.1). To the best of our knowledge, it seems that there is no work on the existence of ground state solutions of System (1.1). Before stating our results, we introduce some hypotheses on the functions Q and f.

• (Q1) $Q\in\mathcal{C}(\mathbb{R}^{3},(0,\infty))$ and $Q(x)$ is 1-periodic in $x_{1}$, $x_{2}$, and $x_{3}$.

• (F2) $f(s)=o(|s|^{1-\frac{\mu}{3}})$ as $|s|\to0$.

• (F3) $\frac{f(s)}{|s|}$ is nondecreasing in $(-\infty,0)\cup(0,+\infty )$.

• (F4) $\frac{F(s)}{|s|^{\sigma}}\to+\infty$ as $|s|\to+\infty$, where $F(s)=\int_{0}^{s} f(t)\,dt$ and $\sigma=\min\{2,\frac{9-\mu}{4}\}$.

Now we are in a position to present our first result. In the periodic case, we establish the following theorem.

Theorem 1.1

Assume that V, Q, and f satisfy (V1), (Q1), and (F1)–(F4). Then System (1.1) has a solution $\tilde{u}\in H^{1}(\mathbb{R}^{3})$ such that $I(\tilde{u})=\inf_{\mathcal{N}}I>0$.

Under assumptions (V1), (Q1), and (F1)–(F4), we prove that, for every $u\in H^{1}(\mathbb{R}^{3})$, there exists unique $t(u)>0$ such that $t(u)u\in\mathcal{N}$ by establishing a key inequality related to $I(u)$, $I(tu)$, and $\langle I'(u),u\rangle$. Then, using the non-Nehari manifold approach developed by Tang [32, 33] and the concentration compactness principle, we obtain a ground state solution for System (1.1) (see Sects. 2, 3).

In the next part, we consider the asymptotically periodic case. The situation becomes more complex when $V(x)$ and $Q(x)$ are asymptotically periodic due to the loss of $\mathbb{Z}^{3}$-translation invariance of functional I. Consequently, many effective methods applied in periodic problems become invalid. So we shall adopt other methods to overcome the difficulties caused by the dropping of the periodicity of $V(x)$ and $Q(x)$.

First, we define a set as follows:

$$\begin{aligned} \Theta :=& \bigl\{ g(x) \in\mathcal{C}\bigl(\mathbb{R}^{3}\bigr) \cap L^{\infty }\bigl(\mathbb{R}^{3}\bigr)|\mbox{ for every } \varepsilon>0, \mbox{the set } \bigl\{ x\in\mathbb {R}^{3}: \bigl\vert g(x) \bigr\vert \ge\varepsilon \bigr\} \\ &\mbox{has a finite measure} \bigr\} . \end{aligned}$$

To state our results, we make the following assumptions in the asymptotically periodic case:

• (V2) $V(x)=V_{0}(x)+V_{1}(x)$, $V_{0}, V_{1}\in\mathcal{C}(\mathbb{R}^{3}, \mathbb {R})$, $V_{0}(x)$ is 1-periodic in $x_{1}$, $x_{2}$, and $x_{3}$, and $-V_{0}(x)< V_{1}(x)\le0$ for $x\in\mathbb{R}^{3}$, $V_{1}(x) \in\Theta$.

• (Q2) $Q(x)=Q_{0}(x)+Q_{1}(x)$, $Q_{0}, Q_{1}\in\mathcal{C}(\mathbb{R}^{3}, \mathbb {R})$, $Q_{0}(x)$ is 1-periodic in $x_{1}$, $x_{2}$, and $x_{3}$, and $Q_{0}(x)\ge 0$, $Q_{1}(x)\ge0$ for $x\in\mathbb{R}^{3}$, $Q_{1}(x) \in\Theta$.

Then we give our second result as follows.

Theorem 1.2

Assume that V, Q, and f satisfy (V2), (Q2), and (F1)–(F4). Then System (1.1) has a solution $\tilde{u}\in H^{1}(\mathbb{R}^{3})$ such that $I(\tilde{u})=\inf_{\mathcal{N}}I>0$.

Example 1.3

There are indeed functions satisfying (F1)–(F4). A simple example is given by $f(s)=s\ln(1+|s|)$.

The paper is organized as follows. In Sect. 2, we give some notations and preliminaries. In Sect. 3 and Sect. 4, Theorem 1.1 and Theorem 1.2 will be proved, respectively.

In this paper, the norm of $L^{p}(\mathbb{R}^{3})$ is denoted by $\Vert u\rVert_{p}$ for $1\le p<\infty$. We denote the ball centered at x with the radius r by $B_{r}(x)$ and use C to indicate all positive constants in estimates while it does not lead to confusion.

Notations and preliminaries

Lemma 2.1

Under assumption (F2) and (F3), we can obtain:

• (i) $\frac{f(s)}{s}$ is nonincreasing in $(-\infty,0)$, and nondecreasing in $(0,+\infty)$.

• (ii) $f(s)s\ge2F(s)\ge0$, $\forall s\in\mathbb{R}$.

• (iii) $\frac{F(s)}{s^{2}}$ is nonincreasing in $(-\infty,0)$, and nondecreasing in $(0,+\infty)$.

The proof is elementary, so we omit here.

Lemma 2.2

Under assumptions (V0), (Q0), and (F1)–(F3),

$$ I(u)\ge I(tu)+\frac{1-t^{4}}{4}\bigl\langle I'(u),u \bigr\rangle +\frac {(1-t^{2})^{2}}{4}\lVert u\rVert^{2},\quad \forall u\in H^{1}\bigl(\mathbb{R}^{3}\bigr), t\ge0. $$

(2.1)

Proof

Note that

$$ I(u)=\frac{1}{2}\lVert u\rVert^{2}+ \frac{1}{4} \int_{\mathbb{R}^{3}} \phi _{u} (x)u^{2}\,dx- \frac{1}{2} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q(y)F(u(y))}{|x-y|^{\mu}} Q(x)F\bigl(u(x)\bigr)\,dx\,dy $$

(2.2)

and

$$ \bigl\langle I'(u),u\bigr\rangle = \lVert u \rVert^{2} + \int_{\mathbb{R}^{3}} \phi_{u} (x)u^{2}\,dx- \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q(y)F(u(y))}{|x-y|^{\mu}} Q(x)f\bigl(u(x)\bigr)u(x)\,dx\,dy. $$

(2.3)

Thus, from (2.2) and (2.3), one has

$$\begin{aligned} I(u)-I(tu)&=\frac{1-t^{2}}{2}\lVert u\rVert^{2}+\frac{1-t^{4}}{4} \int_{\mathbb{R}^{3}} \phi_{u} (x)u^{2}\,dx \\ &\quad {}-\frac{1}{2} \int_{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{Q(y)F(u(y))}{|x-y|^{\mu}} Q(x)F\bigl(u(x)\bigr)\,dx\,dy \\ &\quad {} +\frac {1}{2} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q(y)F(tu(y))}{|x-y|^{\mu}} Q(x)F\bigl(tu(x)\bigr)\,dx\,dy \\ &=\frac{1-t^{2}}{4}\bigl\langle I'(u),u\bigr\rangle + \frac{(1-t^{2})^{2}}{4}\lVert u\rVert^{2} \\ &\quad {}+ \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \biggl[\frac{1-t^{4}}{4}\frac{Q(y)F(u(y))}{|x-y|^{\mu}} Q(x)f \bigl(u(x)\bigr) \bigl(u(x)\bigr) \\ &\quad {} +\frac{1}{2}\frac{Q(y)F(tu(y))}{|x-y|^{\mu}} Q(x)F\bigl(tu(x)\bigr)- \frac{1}{2}\frac{Q(y)F(u(y))}{|x-y|^{\mu}} Q(x)F\bigl(u(x)\bigr) \biggr]\,dx\,dy. \end{aligned}$$

Define a function

$$\begin{aligned} h(t,u)&= \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \biggl[\frac{1-t^{4}}{4} \frac{Q(y)F(u(y))}{|x-y|^{\mu}} Q(x)f \bigl(u(x)\bigr) \bigl(u(x)\bigr)+\frac{1}{2}\frac {Q(y)F(tu(y))}{|x-y|^{\mu}} Q(x)F \bigl(tu(x)\bigr) \\ &\quad {} -\frac{1}{2}\frac {Q(y)F(u(y))}{|x-y|^{\mu}} Q(x)F\bigl(u(x)\bigr)\biggr]\,dx \,dy, \quad \forall t\ge0, u\in H^{1}\bigl(\mathbb{R}^{3} \bigr). \end{aligned}$$

By (Q0), Lemma 2.1, and elementary computations, we have

$$\begin{aligned} h'(t)&= \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \biggl[\frac {Q(y)F(tu(y))}{|x-y|^{\mu}}Q(x)f\bigl(tu(x)\bigr)u(x) \,dx\,dy \\ &\quad {}-t^{3}\frac {Q(y)F(u(y))}{|x-y|^{\mu}} Q(x)f\bigl(u(x)\bigr)u(x)\biggr]\,dx \,dy \\ &=t^{3} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q(x)Q(y)u^{2}(x)u^{2}(y)}{|x-y|^{\mu}} \biggl[\frac{F(tu(y))}{(t^{2}(u(y))} \biggl( \frac{f(tu(x))}{tu (x)}-\frac{f(u(x))}{u(x)} \biggr) \\ &\quad {} +\frac {f(u(x))}{u(x)} \biggl(\frac{F(tu(y))}{tu^{2}(y)}-\frac {F(u(y))}{u^{2}(y)} \biggr) \biggr]\,dx\,dy \textstyle\begin{cases} \ge0,t\ge1,\\ < 0,0< t< 1, \end{cases}\displaystyle \end{aligned}$$

which yields $h(t)\ge h(1)=0$. Therefore, we have (2.1) holds. □

Corollary 2.3

Under assumptions (V0), (Q0), and (F1)–(F3), for any $u\in \mathcal{N}$,

$$ I(u)=\max_{t\ge0} I(tu). $$

(2.4)

Lemma 2.4

(Hardy–Littlewood–Sobolev inequality [8])

Let $s,r>1$, and $0<\mu<N$ with $\frac{1}{s}+\frac{\mu}{N}+\frac {1}{r}=2$, $f\in L^{s}(\mathbb{R}^{N})$ and $h\in L^{r}(\mathbb{R}^{N})$. Then there exists a sharp constant $C(s,N,\mu,r)$ independent of f, h such that

$$ \int_{\mathbb{R}^{N}} \int_{\mathbb{R}^{N}} \frac{f(x)h(y)}{|x-y|^{\mu}}\le C(s,N,\mu,r)\lVert f \Vert_{s}\lVert h\rVert_{r}. $$

In the sequel, we set $N=3$ and $r=\frac{6}{6-\mu}$ (we take the same value for r in the following part of this paper). It follows from $(F1)$ that $2< rq<2^{*}$. So, for any $u\in H^{1}(\mathbb{R}^{3})$, by an elementary computation, we have

$$ \bigl\lVert F(u) \bigr\rVert _{r}\le C\bigl(\lVert u \rVert_{2}^{2-\frac{\mu}{3}}+\lVert u\rVert_{rq}^{q} \bigr). $$

(2.5)

By (Q0), (2.5), and Hardy–Littlewood–Sobolev inequality, we can obtain

$$ \biggl\vert \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q(y)F(u(y))}{|x-y|^{\mu}} Q(x)F\bigl(u(x)\bigr)\,dx\,dy \biggr\vert \le C \bigl\lVert F(u) \bigr\rVert _{r}^{2}\le C \bigl(\lVert u\rVert_{2}^{4-\frac{2\mu}{3}}+\lVert u\rVert_{rq}^{2q} \bigr). $$

(2.6)

Similarly,

$$ \biggl\vert \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q(y)F(u(y))}{|x-y|^{\mu}} Q(x)f\bigl(u(x)\bigr)u(x)\,dx\,dy \biggr\vert \le C \bigl\lVert F(u) \bigr\rVert _{r}^{2}\le C \bigl(\lVert u\rVert_{2}^{4-\frac{2\mu}{3}}+\lVert u\rVert_{rq}^{2q} \bigr). $$

(2.7)

To show $\mathcal{N}\neq\emptyset$ in our situation, we define a set as follows:

$$ \mathcal{J}:= \textstyle\begin{cases} H^{1}(\mathbb{R}^{3})\setminus \{0\},&\mbox{if }\mu< 1; \\ \{u\in H^{1}(\mathbb{R}^{3}):\int_{\mathbb{R}^{3}}\phi_{u}(x)u^{2}\,dx \\ \quad {}-\int _{\mathbb{R}^{3}}\int_{\mathbb{R}^{3}}\frac{Q(y)F(u(y))}{|x-y|^{\mu}} Q(x)f(u(x))u(x)\,dx\,dy< 0 \},&\mbox{if }1\le\mu< 3. \end{cases} $$

Lemma 2.5

Under assumptions (V0), (Q0), and (F1)–(F4), (i) $\mathcal {N}\subset\mathcal{J}\ne\emptyset$; (ii) for any $u\in\mathcal {J}$, there exists unique $t(u)>0$ such that $t(u)u\in \mathcal {N}$.

Proof

(i) It is easy to see that $\mathcal{J}\neq\emptyset$ if $\mu<1$. Next, we consider that $1\le\mu<3$. From Lemma 2.4 and Sobolev imbedding theorem, there exists $C>0$ such that $\int_{\mathbb{R}^{3}} \phi_{u} (x)u^{2}\,dx\le C\lVert u\rVert^{4}$ for any $u\in H^{1}(\mathbb{R}^{3})$. Fix $v\in H^{1}(\mathbb{R}^{3})$ and $v(x)>0$ for any $x\in\mathbb{R}^{3}$ and set $v_{t} (x)=v(tx)$ for $t>0$. By (Q0), one has

$$\begin{aligned} & \int_{\mathbb{R}^{3}} \phi_{(t^{2}v_{t})} (x) \bigl(t^{2}v_{t} \bigr)^{2}\,dx- \int_{\mathbb {R}^{3}} \int_{\mathbb{R}^{3}}\frac{Q(y)F(t^{2}v_{t}(y))}{|x-y|^{\mu}} Q(x)f\bigl(t^{2}v_{t}(x) \bigr)t^{2}v_{t}(x)\,dx\,dy \\ &\quad =t^{3} \int_{\mathbb{R}^{3}} \phi_{v} (x)v^{2} \,dx-t^{\mu-6} \int_{\mathbb {R}^{3}} \int_{\mathbb{R}^{3}} \frac{Q(t^{-1}y)F(t^{2}v(y))}{|x-y|^{\mu}} Q\bigl(t^{-1}x\bigr)f \bigl(t^{2}v(x)\bigr)t^{2}v(x)\,dx\,dy \\ &\quad \le t^{3} \biggl(C\lVert v\rVert^{4}-Q_{\infty}^{2} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}}\frac{F(t^{2}v(y))}{(t^{2}v(y))^{\frac{9-\mu}{4}}}\frac {f(t^{2}v(x))t^{2}v(x)}{(t^{2}v(x))^{\frac{9-\mu}{4}}} \frac{[v(x)v(y)]^{\frac{9-\mu}{4}}}{|x-y|^{\mu}} \biggr)\,dx\,dy, \end{aligned}$$

(2.8)

where $Q_{\infty}=\inf_{x\in\mathbb{R}^{3}}Q(x)$. Since $1\le\mu<3$, then $\sigma=\frac{9-\mu}{4}$ in (F4), and so

$$ \frac{F(t^{2}v(y)))}{(t^{2}v(y))^{\frac{9-\mu}{4}}}\to+\infty \quad \mbox{and}\quad \frac {f(t^{2}v(x)))t^{2}v(x))}{(t^{2}v(x))^{\frac{9-\mu}{4}}}\to+\infty \quad \mbox{as }t\to +\infty, $$

(2.9)

which, together with (Q0), (2.8), and (2.9), implies

$$\begin{aligned}& \int_{\mathbb{R}^{3}} \phi_{(t^{2}v_{t})} (x) \bigl(t^{2}v_{t} \bigr)^{2}\,dx- \int_{\mathbb {R}^{3}} \int_{\mathbb{R}^{3}}\frac{Q(y)F(t^{2}v_{t}(y))}{|x-y|^{\mu}} Q(x)f\bigl(t^{2}v_{t}(x) \bigr)t^{2}v_{t}(x)\,dx\,dy \\& \quad \to -\infty \quad \mbox{as } t\to+\infty. \end{aligned}$$

Thus taking $u=t^{2}v_{t}$ for t large enough, we have $u\in\mathcal {J}$, and so $\mathcal {J}\ne\varnothing$ in the case $1\le\mu<3$. From (2.3), it is easy to see that $\mathcal {N}\subset\mathcal {J}$.

(ii) First, we prove the existence of $t(u)$. Since $\sigma=2$ in (F4) if $\mu<1$, then through a standard argument, the existence of $t(u)$ can be proved easily. After that, we consider the case $1\le\mu <3$. Let $u\in\mathcal {J}$ be fixed and define a function $g(t)=\langle I'(tu),tu\rangle$ on $[0,+\infty)$. By (2.3), Lemma 2.4, and Sobolev imbedding theorem, one has

$$\begin{aligned} g(t)&=t^{2}\lVert u\rVert^{2} +t^{4} \int_{\mathbb{R}^{3}} \phi_{u} (x)u^{2}\,dx- \int _{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac{Q(y)F(tu(y))}{|x-y|^{\mu}} Q(x)f\bigl(tu(x)\bigr)tu(x)\,dx \,dy \\ &\ge t^{2}\lVert u\rVert^{2}-C\bigl(t^{4-\frac{2\mu}{3}}\lVert u \rVert _{2}^{4-\frac{2\mu}{3}}+t^{2q}\lVert u\rVert_{rq}^{2q} \bigr) \\ &\ge t^{2}\lVert u\rVert^{2}-C\bigl(t^{4-\frac{2\mu}{3}}\lVert u \rVert ^{4-\frac{2\mu}{3}}+t^{2q}\lVert u\rVert^{2q}\bigr). \end{aligned}$$

(2.10)

By (2.3) and Lemma 2.1, we have

$$\begin{aligned} g(t)&=t^{2}\lVert u\rVert^{2}+t^{4} \int_{\mathbb{R}^{3}} \phi_{u} (x)u^{2}\,dx \\ &\quad {}- \int _{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac{Q(y)F(tu(y))}{|x-y|^{\mu}} Q(x)f\bigl(tu(x)\bigr)tu(x)\,dx \,dy \\ &\le t^{2}\lVert u\rVert^{2}+t^{4} \int_{\mathbb{R}^{3}} \phi_{u} (x)u^{2}\,dx \\ &\quad {}-t^{4} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q(x)Q(y)u^{2}(x)u^{2}(y)}{|x-y|^{\mu}}\frac{F(tu(y))}{(tu(y))^{2}} \frac {f(tu(x))}{tu (x)}\,dx\,dy \\ &\le t^{2}\lVert u\rVert^{2}+t^{4} \int_{\mathbb{R}^{3}} \phi_{u} (x)u^{2}\,dx \\ &\quad {}-t^{4} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q(x)Q(y)u^{2}(x)u^{2}(y)}{|x-y|^{\mu}}\frac{F(u(y))}{(u(y))^{2}} \frac {f(u(x))}{u (x)}\,dx\,dy \\ &\le t^{2}\lVert u\rVert^{2}+t^{4}\biggl( \int_{\mathbb{R}^{3}} \phi_{u} (x)u^{2}\,dx \\ &\quad {}- \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}}\frac {Q(y)F(u(y))}{|x-y|^{\mu}} Q(x)f\bigl(u(x)\bigr)u(x)\,dx\,dy \biggr),\quad \forall t\ge1. \end{aligned}$$

(2.11)

It follows from (2.10) and (2.11) that $g(t)>0$ for $t>0$ small, and $g(t)<0$ for t large due to $u\in\mathcal {J}$. Therefore, there exists $t_{0}=t(u)>0$ such that $g(t_{0})=0$ and $t(u)u\in\mathcal {N}$.

Next, we prove that $t(u)$ is unique for any u $\in\mathcal {J}$. For any given $u\in\mathcal {N}$, let $t_{1}, t_{2}>0$ such that $g(t_{1})=g(t_{2})=0$. Jointly with (2.1), we have

$$\begin{aligned} I(t_{1}u)&\ge I(t_{2}u)+\frac{t_{1}^{4}-t_{2}^{4}}{4t_{1}^{4}}\bigl\langle I'(t_{1}u),t_{1}u\bigr\rangle + \frac{(t_{1}^{2}-t_{2}^{2})^{2}}{4t_{1}^{2}}\lVert u\rVert^{2} \\ &=I(t_{2}u)+\frac{(t_{1}^{2}-t_{2}^{2})^{2}}{4t_{1}^{2}}\lVert u\rVert^{2}, \end{aligned}$$

(2.12)

and

$$\begin{aligned} I(t_{2}u)&\ge I(t_{1}u)+\frac{t_{2}^{4}-t_{1}^{4}}{4t_{2}^{4}}\bigl\langle I'(t_{2}u),t_{2}u\bigr\rangle + \frac{(t_{2}^{2}-t_{1}^{2})^{2}}{4t_{2}^{2}}\lVert u\rVert^{2} \\ &=I(t_{1}u)+\frac{(t_{2}^{2}-t_{1}^{2})^{2}}{4t_{2}^{2}}\lVert u\rVert^{2}. \end{aligned}$$

(2.13)

Equations (2.12) and (2.13) imply $t_{1}=t_{2}$. Hence, $t(u)>0$ is unique for any $u\in\mathcal {J}$. □

Lemma 2.6

Under assumptions (V0), (Q0), and (F1)–(F4), then

$$ \inf_{u\in\mathcal {N}} I(u):=c=\inf_{u\in\mathcal{J},u\ne0} \max_{t\ge 0}I(tu)>0. $$

(2.14)

The proof is standard, so we omit here.

Lemma 2.7

Under assumptions (V0), (Q0), and (F1)–(F4), there exist a constant $c_{*}\in(0,c]$ and a sequence $\{u_{n}\}\subset H^{1}({\mathbb {R}^{3}})$ satisfying

$$ I(u_{n})\to c_{*}, \qquad \bigl\lVert I'(u_{n}) \bigr\rVert \bigl(1+ \lVert u_{n} \rVert\bigr) \to0. $$

(2.15)

Proof

It follows from Lemma 2.4 and Sobolev imbedding theorem that

$$\begin{aligned} I(u)&=\frac{1}{2}\lVert u\rVert^{2}+\frac{1}{4} \int_{\mathbb{R}^{3}} \phi _{u} (x)u^{2}\,dx- \frac{1}{2} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q(y)F(u(y))}{|x-y|^{\mu}} Q(x)F\bigl(u(x)\bigr)\,dx\,dy. \\ &\ge\frac{1}{2}\lVert u\rVert^{2}-C\bigl(\lVert u \rVert_{2}^{4-\frac{2\mu }{3}}+\lVert u\rVert_{rq}^{2q} \bigr) \\ &\ge\frac{1}{2}\lVert u\rVert^{2}-C\bigl(\lVert u \rVert^{4-\frac{2\mu }{3}}+\lVert u\rVert^{2q}\bigr). \end{aligned}$$

(2.16)

From (F1), we know that there exist $\delta_{0}>0$ and $\rho_{0} >0$ such that

$$ I(u)\ge\rho_{0},\qquad \lVert u\rVert= \delta_{0}. $$

(2.17)

In view of Lemmas 2.5 and 2.6, we may choose $v_{k}\in \mathcal{N}\subset\mathcal{J}$ such that

$$ c-\frac{1}{k}< I(v_{k})< c+\frac{1}{k}, \quad k\in\mathbb{N}. $$

(2.18)

Using Lemma 2.2 and (2.17), we can obtain that $I(tv_{k})>0$ for small $t>0$ and $I(tv_{k})<0$ for large $t>0$ due to $v_{k}\in\mathcal {N}$. Since $I(0)=0$, then the mountain pass lemma implies that there exists a sequence $\{u_{k,n}\}\subset H^{1}(\mathbb{R}^{3})$ satisfying

$$ I(u_{k,n})\to c_{k},\qquad \bigl\lVert I'(u_{k,n}) \bigr\rVert \bigl(1+ \lVert u_{k,n} \rVert\bigr)\to 0, \quad k\in\mathbb{N}, $$

(2.19)

where $c_{k}\in[\rho_{0},\sup_{t\ge0}I(tv_{k})]$. As a result of Corollary 2.3, we have $I(v_{k})=\sup_{t\ge0}I(tv_{k})$. Then, by (2.18) and (2.19), one has

$$ I(u_{k,n})\to c_{k}\in \biggl[\rho_{0},c+ \frac{1}{k} \biggr),\qquad \bigl\lVert I'(u_{k,n}) \bigr\rVert \bigl(1+ \lVert u_{k,n} \rVert\bigr)\to0,\quad k\in\mathbb{N}. $$

(2.20)

Choose a sequence $\{n_{k}\}\subset\mathbb{N}$ such that

$$ I(u_{k,n_{k}})\to c_{k}\in \biggl[\rho_{0},c+ \frac{1}{k} \biggr),\qquad \bigl\lVert I'(u_{k,n_{k}}) \bigr\rVert \bigl(1+ \lVert u_{k,n_{k}} \rVert\bigr)< \frac{1}{k} , \quad k \in\mathbb{N}. $$

(2.21)

Let $u_{k}=u_{k,n_{k}}$, $k\in\mathbb{N}$. Then, going if necessary to a subsequence, we have

$$ I(u_{n})\to c_{*}\in[\rho_{0},c], \qquad \bigl\lVert I'(u_{n}) \bigr\rVert \bigl(1+ \lVert u_{n} \rVert\bigr) \to0. $$

 □

Lemma 2.8

Under assumptions (V0), (Q0), and (F1)–(F4), any sequence $\{u_{n}\} \subset H^{1}(\mathbb{R}^{3})$ satisfying (2.15) is bounded in $H^{1}(\mathbb{R}^{3})$.

Proof

By Lemma 2.2, one has

$$ c_{*}+o(1)=I(u_{n})-\frac{1}{4}\bigl\langle I'(u_{n}),u_{n} \bigr\rangle \ge\frac {1}{4}\lVert u_{n}\rVert^{2}. $$

This shows that the sequence $\{u_{n}\}$ is bounded in $H^{1}(\mathbb{R}^{3})$. □

Similar to the proof of [31, Lemma 2.7], we can obtain the following lemma.

Lemma 2.9

Under assumptions (V0), (Q0), and (F1)–(F4), if $u_{0}\in\mathcal{N}$ and $I(u_{0})=c$, then $u_{0}$ is a critical point of I.

The period case

In this section, we give the proof of Theorem 1.1.

Proof of Theorem 1.1

Lemma 2.7 implies that there exists a sequence $\{u_{n}\}\subset H^{1}(\mathbb{R}^{3})$ satisfying (2.15), then

$$ I(u_{n})\to c_{*}>0,\qquad \bigl\langle I'(u_{n}),u_{n}\bigr\rangle \to0. $$

(3.1)

By Lemma 2.8, $\{u_{n}\}$ is bounded in $H^{1}(\mathbb{R}^{3})$. Suppose that

$$ \delta:=\varlimsup_{n\to\infty} \sup_{y\in\mathbb{R}^{3}} \int _{B_{1}(y)} \bigl\vert u_{n}(x) \bigr\vert ^{2}\,dx=0, $$

then, by Lion’s concentration compactness principle, $u_{n}\to0$ in $L^{s}(\mathbb{R}^{3})$ for $2< s<6$. For any $\varepsilon >0$, by (F1) and (F2), there exists $C_{\varepsilon }$ such that

$$ \bigl\vert f(u_{n}) \bigr\vert \le \varepsilon \vert u_{n} \vert ^{1-\frac{\mu}{3}}+C_{\varepsilon }\vert u_{n} \vert ^{q-1}. $$

(3.2)

Then, from Lemma 2.5 and (3.2), one has

$$\begin{aligned}& \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac{Q(y)F(u_{n}(y))}{|x-y|^{\mu}} Q(x)\bigl[f\bigl(u_{n}(x) \bigr)u_{n}(x)-F\bigl(u_{n}(x)\bigr)\bigr]\,dx\,dy \\& \quad \le C \bigl\lVert F(u_{n}) \bigr\rVert _{\frac {6}{6-\mu}}^{2} \le C_{1}\varepsilon +o(1). \end{aligned}$$

Fix $\varepsilon =\frac{3c_{*}}{2C_{1}}$, we have

$$ \varlimsup_{n\to\infty} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q(y)F(u_{n}(y))}{|x-y|^{\mu}} Q(x)\bigl[f\bigl(u_{n}(x) \bigr)u_{n}(x)-F\bigl(u_{n}(x)\bigr)\bigr]\,dx\,dy\le \frac {3}{2}c_{*}. $$

(3.3)

By Hardy–Littlewood–Sobolev theorem and Sobolev imbedding theorem, one has

$$\begin{aligned} \varlimsup_{n\to\infty} \int_{\mathbb{R}^{3}} \phi_{u_{n}} (x)u_{n}^{2} \,dx&=\varlimsup_{n\to\infty} \int_{\mathbb{R}^{3}} \int_{\mathbb {R}^{3}}\frac{u_{n}^{2}(x)u_{n}^{2}(y)}{|x-y|^{\mu}}\,dx\,dy \\ &\le C\varlimsup_{n\to\infty}\lVert u_{n}\rVert_{\frac{12}{5}}^{4}=0. \end{aligned}$$

(3.4)

For any $\varepsilon >0$, from (2.2), (2.3), (3.1), (3.3), and (3.4), it follows that

$$\begin{aligned} c_{*}&=I(u_{n})-\frac{1}{2}\bigl\langle I'(u_{n}),u_{n} \bigr\rangle +o(1) \\ &=-\frac{1}{4} \int_{\mathbb{R}^{3}} \phi_{u_{n}} (x)u_{n}^{2} \,dx+\frac {1}{2} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q(y)F(u_{n}(y))}{|x-y|^{\mu}} \\ &\quad {}\times Q(x)\bigl[f\bigl(u_{n}(x)\bigr)u_{n}(x)-F \bigl(u_{n}(x)\bigr)\bigr]\,dx\,dy+o(1) \\ &\le\frac{3}{4}c_{*}+o(1), \end{aligned}$$

which is a contradiction, so $\delta>0$.

Going if necessary to a subsequence, we may assume the existence of $k_{n}\in\mathbb{Z}^{3}$ such that

$$ \int_{B_{1}(k_{n})} \bigl\vert u_{n}(x) \bigr\vert ^{2}\,dx>\frac{\delta}{3}. $$

(3.5)

Let $v_{n}(x)=u_{n}(x+k_{n})$, then

$$ \int_{B_{1}(0)} \bigl\vert v_{n}(x) \bigr\vert ^{2}\,dx>\frac{\delta}{3}. $$

(3.6)

Since $V(x)$, $Q(x)$ are periodic on x, we have

$$ I(v_{n})\to c_{*}\in(0,c], \qquad \bigl\lVert I'(v_{n}) \bigr\rVert \bigl(1+ \lVert v_{n} \rVert\bigr)\to0. $$

Passing to a subsequence, we have $v_{n}\rightharpoonup \tilde{v}$ in $H^{1}(\mathbb {R}^{3})$, $v_{n}\to v_{0}$ in $L_{\mathrm{loc}}^{s}(\mathbb{R}^{3})$, $2\le s<6$, and $v_{n}\to\tilde{v}$ a.e. on $\mathbb{R}^{3}$. Thus, (3.6) implies that $\tilde{v}\ne0$. For any $\varphi\in\mathcal{C}_{0}^{\infty}(\mathbb{R}^{3})$, we have

$$ \bigl\langle I'(\tilde{v}),\varphi\bigr\rangle =\lim _{n\to\infty}\bigl\langle I'(v_{n}),\varphi \bigr\rangle =0. $$

Hence $I'(\tilde{v})=0$. This implies that $\tilde{v}\in\mathcal{N}$ is a nontrivial solution of System (1.1) and $I(\tilde{v})\ge c$. It follows from Lemma 2.1 and Fatou’s lemma that

$$\begin{aligned} c&\ge c_{*} =\lim_{n\to\infty} \biggl[I(v_{n})- \frac{1}{4}\bigl\langle I'(v_{n}),v_{n} \bigr\rangle \biggr] \\ &=\lim_{n\to\infty} \biggl\{ \frac{1}{4}\lVert v_{n} \rVert^{2}+\frac {1}{4} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q(y)F(v_{n}(y))}{|x-y|^{\mu}} Q(x) \bigl[f\bigl(v_{n}(x) \bigr)v_{n}(x)-2F\bigl(v_{n}(x)\bigr) \bigr]\,dx\,dy \biggr\} \\ &\ge\frac{1}{4}\lVert v_{0}\rVert^{2}+ \frac{1}{4} \int_{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{Q(y)F(\tilde{v}(y))}{|x-y|^{\mu}} Q(x) \bigl[f\bigl(\tilde {v}(x) \bigr)\tilde{v}(x)-2F\bigl(\tilde{v}(x)\bigr) \bigr]\,dx\,dy \\ &= I(\tilde{v})-\frac{1}{4}\bigl\langle I'(\tilde{v}), \tilde{v}\bigr\rangle =I(\tilde{v}). \end{aligned}$$

This shows that $I(\tilde{v})\le c$, so $I'(\tilde{v})=0$ and $I(\tilde {v})=c=\inf_{\mathcal{N}}I>0$. □

The asymptotically period case

In this section, we have $V(x)=V_{0}(x)+V_{1}(x)$ and $Q(x)=Q_{0}(x)+Q_{1}(x)$. Define a functional $I_{0}$ as follows:

$$\begin{aligned} I_{0}(u) =&\frac{1}{2} \int_{\mathbb{R}^{3}} \bigl(|\nabla u|^{2}+V_{0}(x)u^{2} \bigr)\,dx+\frac{1}{4} \int_{\mathbb{R}^{3}} \phi_{u} (x)u^{2}\,dx \\ &{}-\frac{1}{2} \int _{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac{Q_{0}(y)F(u(y))}{|x-y|^{\mu}} Q_{0}(x)F\bigl(u(x) \bigr)\,dx\,dy, \end{aligned}$$

(4.1)

where $F(u)=\int_{0}^{u} f(s)\,ds$. Then (V2), (Q2), and (F2) imply that $I_{0}\in\mathcal{C}^{1}(H^{1}(\mathbb{R}^{3}),\mathbb{R})$ and

$$\begin{aligned} \bigl\langle I_{0}'(u),v\bigr\rangle =& \int_{\mathbb{R}^{3}} (\nabla u\nabla v+V_{0}uv )\,dx + \int_{\mathbb{R}^{3}} \phi_{u} (x)uv\, dx \\ &{}- \int_{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{Q_{0}(y)F(u(y))}{|x-y|^{\mu}} Q_{0}(x)f\bigl(u(x) \bigr)v(x)\,dx\,dy. \end{aligned}$$

(4.2)

Through a standard proof, we can obtain the following lemma.

Lemma 4.1

Under assumptions (V2), (Q2), and (F1), if $u_{n}\rightharpoonup 0$, then

$$\begin{aligned}& \lim_{n\to\infty} \int_{\mathbb{R}^{3}}V_{1}(x)u_{n}^{2}\,dx=0 , \lim_{n\to\infty } \int_{\mathbb{R}^{3}}V_{1}(x)u_{n}vdx=0,\quad \forall v\in H^{1}\bigl({\mathbb{R}^{3}}\bigr), \end{aligned}$$

(4.3)

$$\begin{aligned}& \ \lim_{n\to\infty} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q_{1}(y)F(u_{n}(y))}{|x-y|^{\mu}} Q_{0}(x)F \bigl(u_{n}(x)\bigr)\,dx\,dy=0, \end{aligned}$$

(4.4)

$$\begin{aligned}& \lim_{n\to\infty} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q_{1}(y)F(u_{n}(y))}{|x-y|^{\mu}} Q_{1}(x)F \bigl(u_{n}(x)\bigr)\,dx\,dy=0, \end{aligned}$$

(4.5)

$$\begin{aligned}& \lim_{n\to\infty} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q_{1}(y)F(u_{n}(y))}{|x-y|^{\mu}} Q_{0}(x)f \bigl(u_{n}(x)\bigr)v(x)\,dx\,dy=0,\quad \forall v\in H^{1} \bigl({\mathbb{R}^{3}}\bigr), \end{aligned}$$

(4.6)

$$\begin{aligned}& \lim_{n\to\infty} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q_{1}(y)F(u_{n}(y))}{|x-y|^{\mu}} Q_{1}(x)f \bigl(u_{n}(x)\bigr)v(x)\,dx\,dy=0,\quad \forall v\in H^{1} \bigl({\mathbb{R}^{3}}\bigr), \end{aligned}$$

(4.7)

$$\begin{aligned}& \lim_{n\to\infty} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q_{0}(y)F(u_{n}(y))}{|x-y|^{\mu}} Q_{1}(x)f \bigl(u_{n}(x)\bigr)v(x)\,dx\,dy=0,\quad \forall v\in H^{1} \bigl({\mathbb{R}^{3}}\bigr). \end{aligned}$$

(4.8)

Proof of Theorem 1.2

Lemma 2.7 implies the existence of a sequence $\{u_{n}\}\in H^{1}({\mathbb{R}^{3}})$ satisfying (2.15), then

$$ I(u_{n})\to c_{*}, \qquad \bigl\langle I'(u_{n}),u_{n} \bigr\rangle \to0. $$

By Lemma 2.8, $\{u_{n}\}$ is bounded in $H^{1}({\mathbb{R}^{3}})$. Passing to a subsequence, we have $u_{n}\rightharpoonup \tilde{u}$ in $H^{1}({\mathbb {R}^{3}})$, $u_{n}\to\tilde{u}$ in $L_{\mathrm{loc}}^{s}(\mathbb{R}^{3})$ for $2\le s<6$, and $u_{n}\to\tilde{u}$ a.e. on $\mathbb{R}^{3}$. There are two possible cases: (i) $\tilde{u}=0$; (ii) $\tilde{u}\ne0$.

Case (i). In this case, we have $u_{n}\rightharpoonup 0$ in $H^{1}(\mathbb {R}^{3})$, $u_{n}\to0$ in $L_{\mathrm{loc}}^{s}(\mathbb{R}^{3})$, $2\le s<6$, and $u_{n}\to0$ a.e. on $\mathbb{R}^{3}$. Note that

$$\begin{aligned}& \lVert u\rVert^{2}= \int_{\mathbb{R}^{3}} \bigl(|\nabla u|^{2}+V_{0}(x)u^{2} \bigr)\,dx+ \int_{\mathbb{R}^{3}}V_{1}(x)u^{2}\,dx,\quad u\in H^{1}\bigl(\mathbb{R}^{3}\bigr), \end{aligned}$$

(4.9)

$$\begin{aligned}& I_{0}(u)=I(u)-\frac{1}{2} \int_{\mathbb{R}^{3}}V_{1}(x)u^{2}\,dx \\& \hphantom{I_{0}(u)={}}{}+\frac{1}{2} \int _{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}}\frac{F(u(y))F(u(x))}{ |x-y|^{\mu}}\bigl[2Q_{0}(x)Q_{1}(y)+Q_{1}(x)Q_{1}(y) \bigr]\,dx\,dy, \end{aligned}$$

(4.10)

$$\begin{aligned}& \bigl\langle I_{0}'(u),v\bigr\rangle =\bigl\langle I'(u),v\bigr\rangle - \int_{\mathbb {R}^{3}}V_{1}(x)uv\, dx \\& \hphantom{\langle I_{0}'(u),v\rangle={}}{} + \int_{\mathbb{R}^{3}} \int_{\mathbb {R}^{3}}\frac{F(u(y)f(u(x))v(x)}{ |x-y|^{\mu}} \\& \hphantom{\langle I_{0}'(u),v\rangle={}}{}\times\bigl[Q_{1}(y)Q_{0}(x)+Q_{0}(y)Q_{1}(x)+Q_{1}(x)Q_{1}(y) \bigr]\,dx\,dy. \end{aligned}$$

(4.11)

Using (2.15), (4.3)–(4.8), (4.10), and (4.11), one has

$$ I_{0}(u_{n})\to c_{*}, \qquad \bigl\lVert I_{0}'(u_{n}) \bigr\lVert \bigl(1+ \lVert u_{n} \rVert\bigr)\to0. $$

(4.12)

As the proof of (3.5), there exists $k_{n}\in\mathbb{Z}^{3}$, going if necessary to a subsequence, such that

$$ \int_{B_{1}(k_{n})}|u_{n}|^{2}\,dx> \frac{\delta}{2}>0. $$

(4.13)

Define $v_{n}(x)=u_{n}(x+k_{n})$, then

$$ \int_{B_{1}(0)}|v_{n}|^{2}\,dx> \frac{\delta}{2}. $$

(4.14)

Since $V_{0}(x)$, $Q_{0}(x)$ are 1-periodic on x, we have

$$ I_{0}(v_{n})\to c_{*}\in(0,c],\qquad \bigl\lVert I_{0}'(v_{n}) \bigr\rVert \bigl(1+ \lVert v_{n} \rVert\bigr)\to0. $$

Passing to a subsequence, we have $v_{n}\rightharpoonup \tilde{v}$ in $H^{1}(\mathbb {R}^{3})$, $v_{n}\to\tilde{v}$ in $L_{\mathrm{loc}}^{s}(\mathbb{R}^{3})$, $2\le s<6$, and $v_{n}\to\tilde{v}$ a.e. on $\mathbb{R}^{3}$. Thus, (4.14) implies that $\tilde{v}\ne0$. For any $\varphi\in\mathcal{C}_{0}^{\infty}(\mathbb{R}^{3})$, we have

$$ \bigl\langle I_{0}' (\tilde{v}),\varphi\bigr\rangle = \lim_{n\to \infty}\bigl\langle I_{0}'(v_{n}), \varphi\bigr\rangle =0, $$

then $I_{0}'(\tilde{v})=0$. It follows from Lemma 2.1 and Fatou’s lemma that

$$\begin{aligned} c&\ge c_{*}=\lim_{n\to\infty}\biggl[I_{0}(v_{n})- \frac{1}{4}\bigl\langle I_{0}'(v_{n}),v_{n} \bigr\rangle \biggr] \\ &=\lim_{n\to\infty} \biggl\{ \frac{1}{4} \int_{\mathbb{R}^{3}} \bigl[|\nabla v_{n}|^{2}+V_{0}(x)v_{n}^{2} \bigr]\,dx+\frac{1}{4} \int_{\mathbb{R}^{3}} \int_{\mathbb {R}^{3}} \frac{Q_{0}(y)F(v_{n}(y))}{|x-y|^{\mu}} Q_{0}(x) \bigl[f \bigl(v_{n}(x)\bigr)v_{n}(x) \\ &\quad {}-2F\bigl(v_{n}(x)\bigr) \bigr]\,dx\,dy \biggr\} \\ &\ge\frac{1}{4} \int_{\mathbb{R}^{3}} \bigl[|\nabla \tilde{v}|^{2}+V_{0}(x) \tilde {v}^{2} \bigr]\,dx \\ &\quad {}+\frac{1}{4} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}} \frac {Q_{0}(y)F(\tilde{v}(y))}{|x-y|^{\mu}} Q_{0}(x) \bigl[f\bigl( \tilde{v}(x)\bigr)\tilde {v}(x)-2F\bigl(\tilde{v}(x)\bigr) \bigr]\,dx\,dy \\ &=I_{0}(\tilde{v})-\frac{1}{4}\bigl\langle I_{0}'( \tilde{v}),\tilde{v}\bigr\rangle =I_{0}(\tilde{v}). \end{aligned}$$

This means that $I_{0}(\tilde{v})\le c$. Since $I_{0}'(\tilde{v})=0$, it follows from (Q2), (V2) that

$$\begin{aligned} \bigl\langle I'(\tilde{v}),\tilde{v}\bigr\rangle &=\lVert\tilde{v} \rVert^{2}+ \int_{\mathbb{R}^{3}} \phi_{\tilde{v}} (x)\tilde{v}^{2}\,dx- \int_{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{Q(y)F(\tilde{v}(y))}{|x-y|^{\mu}} Q(x)f\bigl(\tilde {v}(x)\bigr) \tilde{v}(x)\,dx\,dy \\ &\le\bigl\langle I_{0}(\tilde{v}),\tilde{v}\bigr\rangle =0, \end{aligned}$$

(4.15)

which means

$$ \int_{\mathbb{R}^{3}} \phi_{\tilde{v}} (x)\tilde{v}^{2}\,dx- \int_{\mathbb {R}^{3}} \int_{\mathbb{R}^{3}}\frac{Q(y)F(\tilde{v}(y))}{|x-y|^{\mu}} Q(x)f\bigl(\tilde{v}(x)\bigr) \tilde{v}(x)\,dx\,dy< 0, $$

and then we have $\tilde{v}\in\mathcal{J}$. According to Lemma 2.5, there exists $t_{0}=t(\tilde{v})>0$ such that $t_{0}\tilde{v}\in \mathcal{N}$, and so $I(t_{0}\tilde{v})\ge c$. From (V2), (Q2), (4.10), and (4.11), we have

$$\begin{aligned} c&\ge I_{0}(\tilde{v}) \\ &=I_{0}(t_{0}\tilde{v})+\frac{1-t_{0}^{4}}{4}\bigl\langle I_{0}'(\tilde{v},\tilde{v})\bigr\rangle + \frac{(1-t^{2})^{2}}{4} \int_{\mathbb {R}^{3}}\bigl[|\nabla \tilde{v}|^{2}+V_{0}(x) \tilde{v}^{2}\bigr]\,dx \\ &\quad {} + \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}}\biggl[\frac{1-t^{4}}{4}\frac{Q(y)F(\tilde {v}(y))}{|x-y|^{\mu}} Q(x)f \bigl(\tilde{v}(x)\bigr)\tilde{v}(x)+\frac {1}{2}\frac{Q(y)F(t\tilde{v}(y))}{|x-y|^{\mu}} Q(x)F \bigl(t\tilde{v}(x)\bigr) \\ &\quad {}-\frac{1}{2}\frac{Q(y)F(\tilde{v}(y))}{|x-y|^{\mu}} Q(x)F\bigl(\tilde {v}(x)\bigr) \biggr]\,dx\,dy \\ &\ge I_{0}(t_{0}\tilde{v}) \\ &=I(t_{0}\tilde{v})-\frac{t_{0}^{2}}{2} \int_{\mathbb {R}^{3}}V_{1}(x)\tilde{v}^{2}\,dx \\ &\quad {}+\frac{1}{2} \int_{\mathbb{R}^{3}} \int_{\mathbb{R}^{3}}\frac{F(t_{0}\tilde{v}(y))F(t_{0}\tilde{v}(x))}{|x-y|^{\mu}} \bigl[2Q_{0}(x)Q_{1}(y)+Q_{1}(x)Q_{1}(y) \bigr]\,dx\,dy \\ &\ge I(t_{0}\tilde{v})\ge c, \end{aligned}$$

which implies $I(t_{0}\tilde{v})=c$. Let $\tilde{u}=t_{0}\tilde{v}$, then $\tilde{u}\in\mathcal{N}$ and $I(\tilde{u})=c$. In view of Lemma 2.9, $I'(\tilde{u})=0$, which means $\tilde{u}\in H^{1}(\mathbb{R}^{3})$ is a solution for System (1.1) with $I(\tilde{u})=\inf_{\mathcal {N}}I>0$.

Case (ii). Following the same process as in the last part of the proof of Theorem 1.1, we can obtain that $I'(\tilde{u})=0$ and $I(\tilde{u})=c=\inf_{\mathcal{N}}I$, which means that $\tilde {u}\in H^{1}(\mathbb{R}^{3})$ is a solution for System (1.1) with $I(\tilde{u})=\inf_{\mathcal{N}}I>0$. □

Conclusion

In this paper, by using the variational methods and non-Nehari manifold methods, the existence of ground state solutions for System (1.1) are established. We consider periodic and asymptotically periodic Schrödinger–Poisson system involving Hartree-type nonlinearities, and generalize some previous results.