Ground states for a coupled Schrödinger system with general nonlinearities

Abstract

We study a coupled Schrödinger system with general nonlinearities. By using variational methods, we prove the existence and asymptotic behaviour of ground state solution for the system with periodic couplings. Moreover, we prove the existence and nonexistence of ground state solution for the system with non-periodic couplings via Nehari manifold method. Especially, the ground state solution with both nontrivial components is obtained, and the sign of nontrivial components is considered.

1 Introduction and main results

We study the existence, nonexistence and asymptotic behaviour of ground state solution of the coupled Schrödinger system

$$ \textstyle\begin{cases} (-\Delta )^{s} u+a_{1}u=f_{1}(u)+b(x) \vert u \vert ^{q-2}u \vert v \vert ^{q}+\lambda v& \text{in } \mathbb{R}^{N}, \\ (-\Delta )^{s} v+a_{2}v=f_{2}(v)+b(x) \vert u \vert ^{q} \vert v \vert ^{q-2}v+\lambda u& \text{in } \mathbb{R}^{N}, \\ \quad u,v\in H^{s}(\mathbb{R}^{N}), \end{cases} $$

(1.1)

where $a_{i}>0$, $i=1,2$, $\lambda \in (-\sqrt{a_{1}a_{2}},0)\cup (0,\sqrt{a _{1}a_{2}})$, $0< s<1$, $N>2s$, $2^{\ast }=\frac{2N}{N-2s}$ and $2<2q<p<2^{\ast }$. $(-\Delta )^{s}$ stands for fractional Laplacian, see [1, 2]. The coupled Schrödinger system arises from Hartree–Fock theory in Bose–Einstein condensates and nonlinear optics, among other physical problems [3, 4].

Solutions with both nontrivial components $(u,v)$, $u,v\neq 0$ are called nontrivial solutions. Solutions with both positive components are called positive solutions $(u,v)$, $u,v>0$. A nontrivial solution is called a ground state solution if its energy is minimum among all nontrivial solutions.

As is well known, there are nonlinear and linear forms of coupling terms for coupled Schrödinger systems. When $\lambda =0$, Eqs. (1.1) reduce to a Schrödinger system with nonlinear couplings. In [5], the authors studied a Schrödinger system with nonlinear couplings

$$ \textstyle\begin{cases} (-\Delta )^{s} u+u=( \vert u \vert ^{2p}+b \vert u \vert ^{p-1} \vert v \vert ^{p+1})u& \text{in } \mathbb{R}^{N}, \\ (-\Delta )^{s} v+a^{2s}v=( \vert v \vert ^{2p}+b \vert v \vert ^{p-1} \vert u \vert ^{p+1})v& \text{in }\mathbb{R}^{N}, \\ \quad u,v\in H^{s}(\mathbb{R}^{N}), \end{cases} $$

(1.2)

where $a>0$ and $2<2p+2<2^{*}$. In the autonomous case, they proved that if $b>0$ is large enough, Eqs. (1.2) have a positive ground state solution with both nontrivial components. Similar systems were also studied in [6–9]. When $b(x)=0$, Eqs. (1.1) reduce to a Schrödinger system with linear couplings. In [10], the authors studied a Schrödinger system with linear couplings. Applying the classical Nehari manifold approach, they proved the existence of ground state solution and multiplicity results. For the other works about linearly coupled system, we refer the readers to [11, 12] and the references therein. When $\lambda b(x) \neq 0$, Eqs. (1.1) are a Schrödinger system with linear and nonlinear couplings. There are few papers concerning this class of system. The authors in [13–15] proved the existence results of (1.1) with $f_{1}(u)=f_{2}(u)=u^{3}$, $q=2$ and $b(x)=b$. To the best of our knowledge, there is almost no research concerning the system with general nonlinearities.

When $\lambda =0$ and $b(x)=0$, Eqs. (1.1) reduce to two scalar equations. The Schrödinger equation with different potentials and nonlinearities is actively studied, see for instance [16–21]. We just mention some results about asymptotic behaviour of ground state solution. Guo and Mederski in [16] studied a Schrödinger equation with sum of periodic and inverse-square potentials as follows:

$$ -\Delta u+\biggl(V(x)-\frac{\mu }{ \vert x \vert ^{2}}\biggr)u=f(x,u), $$

where $V(x)$ is periodic. The superlinear and subcritical term f satisfies a weak monotonicity condition. They proved the existence of ground state solution and the asymptotic behaviour of ground state solution in the limit $\mu \rightarrow 0$. Later in [17, Theorem 1.3], Bieganowski studied the Schrödinger equation

$$ (-\Delta )^{s} u+V(x)u=f(x,u)-K(x) \vert u \vert ^{q-2}u, $$

where $2< q< p<2^{*}$ and the potential functions $V(x)$ and $K(x)$ are $\mathbb{Z}^{N}$-periodic. The author studied the asymptotic behaviour of ground state solution as $K(x)\rightarrow 0$ in $L^{\infty }( \mathbb{R}^{N})$ by using variational methods.

In the presence of general nonlinearities, periodic potentials and nonlinear couplings, we study the asymptotic behaviour of ground state solutions of (1.1) in the limit $b(x)\rightarrow 0$ in $L^{\infty }(\mathbb{R}^{N})$. We assume that

(B):

$0\leq b(x)\in L^{\infty }(\mathbb{R}^{N})$ is $\mathbb{Z}^{N}$-periodic.

The nonlinearities $f_{i},i=1,2$, satisfy:

($F_{1}$):

$f_{i}\in \mathcal{C}^{1}(\mathbb{R})$ and there exist $c_{1},c_{2}>0$ such that

$$ \bigl\vert f'_{i}(u) \bigr\vert \leq c_{1}\bigl(1+ \vert u \vert ^{p-2} \bigr)\quad \text{and}\quad \bigl\vert f_{i}(u) \bigr\vert \leq c_{2}\bigl(1+ \vert u \vert ^{p-1}\bigr)\quad \text{for all } u\in \mathbb{R}. $$

($F_{2}$):

$\lim_{|u|\rightarrow 0^{+}} \frac{f_{i}(u)}{|u|}=0$, $f_{i}(-u)=-f_{i}(u)$ for all $u\in \mathbb{R}$.

($F_{3}$):

$\lim_{|u|\rightarrow +\infty } \frac{F_{i}(u)}{|u|^{2}}\rightarrow +\infty$, where $F_{i}(u)=\int _{0}^{u}f_{i}(s)\,ds$.

($F_{4}$):

$u\mapsto \frac{f_{i}(u)}{|u|^{2q-1}}$ is nondecreasing on $(-\infty ,0)\cup (0,+\infty )$.

To study asymptotic behaviour of ground state solution of (1.1), we introduce the following condition from [17]:

($F_{5}$):

There exist $d>0$ and $2< t\leq p$ such that

$$ f_{i}(u)u-2F_{i}(u)\geq d \vert u \vert ^{t}. $$

We state our main results in what follows.

Theorem 1.1

Suppose that ($F_{1}$)–($F_{4}$) and (B) are satisfied.

• (i) Then Eqs. (1.1) have a ground state solutionω, where

  $$ \omega =(u,v) \textstyle\begin{cases} u>0 \quad \textit{and}\quad v< 0\quad & \textit{as } {-}\sqrt{a _{1}a_{2}}< \lambda < 0, \\ u>0 \quad \textit{and}\quad v>0& \textit{as } 0< \lambda < \sqrt{a_{1}a_{2}}. \end{cases} $$

• (ii) Moreover, ($F_{5}$) holds, every function in the sequence$(b_{n})$satisfies (B) and$b_{n}\rightarrow 0$in$L^{\infty }( \mathbb{R}^{N})$as$n\rightarrow +\infty$. If$(u_{n},v_{n})$is a ground state solution of (1.1) with$b(x)=b_{n}(x)$, then there is a sequence$(z_{n})\subset \mathbb{Z}^{N}$such that

  $$ \bigl(u_{n}(\cdot + z_{n}),v_{n}(\cdot + z_{n})\bigr)\rightarrow (u,v) \quad \textit{strongly in }E, $$

  where$(u,v)$is a ground state solution of (1.1) with$b(x)=0$.

In Theorem 1.1, since we are concerned with (1.1) involving general nonlinearities and nonlinear couplings, moreover $f_{1}$ and $f_{2}$ are independent with each other, the problem becomes complicated in applying variational methods. To prove the existence of ground state solution of (1.1), we find a Palais–Smale sequence on Nehari manifold and use concentration compactness argument to deal with the lack of compactness of the sequence in $\mathbb{R}^{N}$. The proof of (ii) is mainly based on the Nehari manifold method and takes inspiration from [17]. By concentration compactness argument and periodicity of energy functional, we find that there exists a sequence $(z_{n})\subset \mathbb{Z}^{N}$ such that the weak limit of $(u_{n}(\cdot + z_{n}),v_{n}(\cdot + z _{n}))$ is nontrivial and is a ground state solution of (1.1) with $b(x)=0$. Then, a further evaluation of the least energy functional allows us to get the convergence in (ii).

We also study the existence and nonexistence of ground state solution of (1.1) in the presence of non-periodic couplings. In what follows $b(x)$ satisfies:

($B_{1}$):

$0\leq b(x)\in L^{\infty }(\mathbb{R}^{N})$ and $b(x)=b_{ \mathrm{per}}(x)+b_{\mathrm{loc}}(x)$, where $0\leq b_{\mathrm{per}}(x) \in L^{\infty }(\mathbb{R}^{N})$ is $\mathbb{Z}^{N}$-periodic and $b_{\mathrm{loc}}(x)\in L^{\infty }(\mathbb{R}^{N})\cap L^{ \frac{p}{p-2q}}(\mathbb{R}^{N})$ satisfies $\lim_{|x|\rightarrow \infty }b_{\mathrm{loc}}(x)=0$.

Theorem 1.2

Suppose that ($F_{1}$)–($F_{4}$) and ($B_{1}$) are satisfied.

• (i) If$b_{\mathrm{loc}}(x)\geq 0$for a.e. $x\in \mathbb{R}^{N}$and$b_{\mathrm{loc}}(x)>0$on a positive measure set, then (1.1) has a ground state solutionω, where

  $$ \omega =(u,v) \textstyle\begin{cases} u>0 \quad \textit{and}\quad v< 0 & \textit{as } {-}\sqrt{a _{1}a_{2}}< \lambda < 0, \\ u>0\quad \textit{and}\quad v>0& \textit{as } 0< \lambda < \sqrt{a_{1}a_{2}}. \end{cases} $$

• (ii) If$b_{\mathrm{loc}}(x)\leq 0$for a.e. $x\in \mathbb{R}^{N}$and$b_{\mathrm{loc}}(x)<0$on a positive measure set, then (1.1) has no ground state solution.

In Theorem 1.2, $b(x)$ is non-periodic, which brings some difficulties to prove that the weak limit of the obtained PS sequence is nontrivial since the translation of energy functional is not invariant. By comparing its least energy with that in the periodic case, we can deduce that the weak limit is nontrivial. Finally, with concentration compactness argument and direct energy estimation, the existence and nonexistence results are proved under suitable assumptions on the sign of $b_{\mathrm{loc}}(x)$.

The paper is organized in the following way. In Sect. 2 we present several technical results which will be used throughout this paper. In Sect. 3 we study PS sequences on Nehari manifold. We prove Theorem 1.1 in Sect. 4 and Theorem 1.2 in Sect. 5.

2 Preliminaries

We denote the Hilbert space $E:=H^{s}(\mathbb{R}^{N})\times H ^{s}(\mathbb{R}^{N})$ endowed with the norm (see [1]) $\|\omega \|^{2}:=\|(u,v)\|^{2}=\|u\|^{2}+\|v\|^{2}$, where

$$ \Vert u \Vert ^{2}:=[u]_{s}^{2}+a_{i} \vert u \vert _{2}^{2}= \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}}\frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx\,dy+ \int _{\mathbb{R}^{N}}a_{i} \vert u \vert ^{2}\,dx, $$

$|\cdot |_{p}$ stands for the norm of $L^{p}(\mathbb{R}^{N})$ and $|(\cdot ,\cdot )|_{p}=(|\cdot |_{p}^{p}+|\cdot |_{p}^{p})^{ \frac{1}{p}}$ stands for the norm of $L^{p}(\mathbb{R}^{N})\times L ^{p}(\mathbb{R}^{N})$. It is well known that weak solutions of (1.1) are critical points of functional $\mathcal{J}(\omega )= \mathcal{J}(u,v): E\rightarrow \mathbb{R}$

$$ \begin{aligned}[b] \mathcal{J}(\omega ):={}& \mathcal{J}(u,v) \\ ={}&\frac{1}{2} \Vert \omega \Vert ^{2}- \lambda \int _{\mathbb{R}^{N}}uv\,dx- \int _{\mathbb{R}^{N}}F_{1}(u)\,dx- \int _{\mathbb{R}^{N}}F_{2}(v)\,dx \\ &{}-\frac{1}{q} \int _{\mathbb{R}^{N}}b(x) \vert u \vert ^{q} \vert v \vert ^{q}\,dx. \end{aligned} $$

(2.1)

Denote

$$ \mathcal{I}(\omega ):= \int _{\mathbb{R}^{N}}F_{1}(u)\,dx+ \int _{\mathbb{R}^{N}}F_{2}(v)\,dx+\frac{1}{q} \int _{\mathbb{R}^{N}}b(x) \vert u \vert ^{q} \vert v \vert ^{q}\,dx, $$

and Nehari manifold

$$\begin{aligned}& \mathcal{N}:=\bigl\{ \omega \in E\setminus \{0,0\}:\mathcal{J}'( \omega ) \omega =0\bigr\} , \\& c:=\inf \bigl\{ \mathcal{J}(\omega ):\omega \in E\setminus \{0,0\}, \mathcal{J}'(\omega )\omega =0\bigr\} . \end{aligned}$$

Assumptions ($F_{2}$) and ($F_{4}$) imply that

$$ f_{i}(u)u=2q \int _{0}^{u}\frac{f_{i}(u)}{u^{2q-1}}s^{2q-1} \,ds\geq 2q \int _{0}^{u}\frac{f_{i}(s)}{s^{2q-1}}s^{2q-1} \,ds=2qF_{i}(u). $$

(2.2)

The following lemma is standard and follows from ($F_{1}$)–($F_{2}$).

Lemma 2.1

For$\varepsilon >0$, there exists$C_{\varepsilon }>0$such that

$$ \bigl\vert f_{i}(u)u \bigr\vert + \bigl\vert F_{i}(u) \bigr\vert \leq \varepsilon \vert u \vert ^{2}+C_{\varepsilon } \vert u \vert ^{p}. $$

We need several lemmas for our proof.

Lemma 2.2

For$\lambda \in (-\sqrt{a_{1}a_{2}},\sqrt{a_{1}a_{2}})\setminus \{0\}$, there holds

$$ \biggl(1-\frac{ \vert \lambda \vert }{\sqrt{a_{1}a_{2}}}\biggr) \Vert \omega \Vert ^{2}\leq \Vert \omega \Vert ^{2}-2\lambda \int _{\mathbb{R}^{N}}uv\,dx\leq \biggl(1+\frac{ \vert \lambda \vert }{\sqrt{a_{1}a_{2}}}\biggr) \Vert \omega \Vert ^{2}. $$

(2.3)

Proof

Since $\lambda \in (-\sqrt{a_{1}a_{2}},\sqrt{a_{1}a_{2}})\setminus \{0\}$, then $0<\frac{|\lambda |}{\sqrt{a_{1}a_{2}}}<1$ and

$$ \begin{aligned}[b] -2\lambda \int _{\mathbb{R}^{N}} uv\,dx&\geq -2 \vert \lambda \vert \int _{\mathbb{R}^{N}} \vert u \vert \vert v \vert \,dx=-2 \frac{ \vert \lambda \vert }{ \sqrt{a_{1}a_{2}}} \int _{\mathbb{R}^{N}} \sqrt{a_{1}a_{2}} \vert u \vert \vert v \vert \,dx \\ &\geq -\frac{ \vert \lambda \vert }{\sqrt{a_{1}a_{2}}} \int _{\mathbb{R}^{N}}a_{1}u^{2}+a_{2}v^{2} \,dx\geq -\frac{ \vert \lambda \vert }{\sqrt{a _{1}a_{2}}} \Vert \omega \Vert ^{2}. \end{aligned} $$

(2.4)

It follows that $\|\omega \|^{2}-2\lambda \int _{\mathbb{R}^{N}}uv\,dx \geq (1-\frac{|\lambda |}{\sqrt{a_{1}a_{2}}})\|\omega \|^{2}$. The proof of $\|\omega \|^{2}-2\lambda \int _{\mathbb{R}^{N}}uv\,dx\leq (1+\frac{| \lambda |}{\sqrt{a_{1}a_{2}}})\|\omega \|^{2}$ is analogous. □

Lemma 2.3

Suppose that ($F_{1}$)–($F_{4}$) are satisfied and a potential function$b(x)$satisfies (B) or ($B_{1}$), one has$\beta :=\inf_{\omega \in \mathcal{N}}{\|\omega \|}>0$.

Proof

Let $\omega _{n}\in \mathcal{N}$ be such that $\|\omega _{n}\|\rightarrow 0$, then

$$ \begin{aligned} \biggl(1-\frac{ \vert \lambda \vert }{\sqrt{a_{1}a_{2}}}\biggr) \Vert \omega _{n} \Vert ^{2}&\leq \Vert \omega _{n} \Vert ^{2}-2\lambda \int _{\mathbb{R}^{N}}u_{n}v_{n}\,dx \\ &= \int _{\mathbb{R}^{N}}f_{1}(u_{n})u_{n} \,dx+ \int _{\mathbb{R}^{N}}f_{2}(v_{n})v_{n} \,dx+2 \int _{\mathbb{R}^{N}}b(x) \vert u _{n} \vert ^{q} \vert v_{n} \vert ^{q}\,dx, \end{aligned} $$

which implies that

$$ \Vert \omega _{n} \Vert ^{2}\leq C\bigl(\varepsilon \Vert u_{n} \Vert ^{2}+\varepsilon \Vert v _{n} \Vert ^{2}+C_{\varepsilon } \Vert u_{n} \Vert ^{p} +C_{\varepsilon } \Vert v_{n} \Vert ^{p}+ \Vert u_{n} \Vert ^{2q}+ \Vert v_{n} \Vert ^{2q}\bigr) $$

for a constant $C>0$. Let $\varepsilon >0$ be such that $1-\varepsilon C>0$, then

$$ \begin{aligned} 1-\varepsilon C&\leq C\frac{C_{\varepsilon }( \Vert u_{n} \Vert ^{p}+ \Vert v_{n} \Vert ^{p})+ \Vert u_{n} \Vert ^{2q}+ \Vert v_{n} \Vert ^{2q}}{ \Vert \omega _{n} \Vert ^{2}} \\ &\leq C\bigl[C_{\varepsilon }{\bigl( \Vert u_{n} \Vert ^{p-2}+ \Vert v_{n} \Vert ^{p-2}}\bigr)+ \Vert u_{n} \Vert ^{2q-2}+ \Vert v_{n} \Vert ^{2q-2}\bigr]\rightarrow 0. \end{aligned} $$

It is a contradiction. Hence $\inf_{\omega \in \mathcal{N}} {\|\omega \|}>0$. □

Lemma 2.4

Suppose that ($F_{1}$)–($F_{4}$) are satisfied, and a potential function$b(x)$satisfies (B) or ($B_{1}$), then:

($A_{1}$):

There exists$r>0$such that$a:=\inf_{\|\omega \|=r} \mathcal{J}(\omega )>\mathcal{J}(0)=0$;

($A_{2}$):

For any$\omega \in E\setminus \{(0,0)\}$, there exists$t>0$such that$\mathcal{J}(t\omega )<0$;

($A_{3}$):

For$t\in (0,\infty )\setminus \{1\}$and$\omega \in \mathcal{N}$, there holds

$$ \varphi (t):=\frac{t^{2}-1}{2}\mathcal{I}'(\omega )\omega - \mathcal{I}(t\omega )+\mathcal{I}(\omega )< 0; $$

($A_{4}$):

For any$\omega \in E\setminus \{(0,0)\}$, there exists a unique number$t>0$such that$t\omega \in \mathcal{N}$and$\mathcal{J}(t\omega )=\max_{r\geq 0}\mathcal{J}(r \omega )$.

Proof

($A_{1}$) Applying the fractional Sobolev embedding theorem [1] and Lemma 2.1, there exists $C>0$ such that

$$ \int _{\mathbb{R}^{N}}F_{i}(u)\,dx\leq C\bigl(\varepsilon \Vert u \Vert ^{2}+C_{\varepsilon } \Vert u \Vert ^{p}\bigr). $$

Hölder’s inequality implies that

$$ \int _{\mathbb{R}^{N}}b(x) \vert u \vert ^{q} \vert v \vert ^{q}\,dx\leq \vert b \vert _{\infty } \vert u \vert _{2q} ^{q} \vert v \vert _{2q}^{q} \leq C\bigl( \Vert u \Vert ^{2q}+ \Vert v \Vert ^{2q}\bigr). $$

Let $r,C_{1}>0$ for $\|\omega \|\leq r$ and r be sufficiently small, we have

$$ \begin{aligned} &\int _{\mathbb{R}^{N}}F_{1}(u)\,dx+ \int _{\mathbb{R}^{N}}F_{2}(v)\,dx+ \frac{1}{q} \int _{\mathbb{R}^{N}}b(x) \vert u \vert ^{q} \vert v \vert ^{q}\,dx \\ &\quad \leq C(\varepsilon _{1}+\varepsilon _{2}) \Vert \omega \Vert ^{2}+C(C _{\varepsilon _{1}}+C_{\varepsilon _{2}}) \Vert \omega \Vert ^{p}+2C \Vert \omega \Vert ^{2q} \\ &\quad \leq \frac{1}{4}\biggl(1-\frac{ \vert \lambda \vert }{\sqrt{a_{1}a_{2}}}\biggr) \Vert \omega \Vert ^{2}\leq \frac{1}{4}\biggl(1-\frac{ \vert \lambda \vert }{\sqrt{a_{1}a _{2}}} \biggr)r^{2}. \end{aligned} $$

For $\|\omega \|=r$, it suffices to show that

$$ \mathcal{J}(\omega )\geq \frac{1}{4}\biggl(1-\frac{ \vert \lambda \vert }{\sqrt{a _{1}a_{2}}} \biggr)r^{2}>0. $$

($A_{2}$) For any $\omega \in E\setminus \{(0,0)\}$ and $t>0$, by using Fatou’s lemma and ($F_{3}$), we have

$$ \lim_{t\rightarrow +\infty } \int _{\mathbb{R}^{3}}\frac{F _{1}(tu)}{t^{2}u^{2}}u^{2}+ \frac{F_{2}(tv)}{t^{2}v^{2}}v^{2}\,dx\geq \int _{\mathbb{R}^{3}}\lim_{t\rightarrow +\infty }\biggl( \frac{F _{1}(tu)}{t^{2}u^{2}}u^{2}+\frac{F_{2}(tv)}{t^{2}v^{2}}v^{2} \biggr)\,dx\rightarrow +\infty , $$

which implies that

$$ \begin{aligned} \mathcal{J}(t\omega )/t^{2}={}& \frac{1}{2}\biggl( \Vert \omega \Vert ^{2}-2\lambda \int _{\mathbb{R}^{N}}uv\,dx\biggr)- \int _{\mathbb{R}^{N}}\frac{F_{1}(tu)+F_{2}(tv)}{t ^{2}}\,dx \\ &{}-\frac{t^{2q-2}}{q} \int _{\mathbb{R}^{N}}b(x) \vert u \vert ^{q} \vert v \vert ^{q}\,dx \rightarrow -\infty\quad \text{as }t\rightarrow + \infty . \end{aligned} $$

Hence $\mathcal{J}(t\omega )\rightarrow -\infty$ as $t\rightarrow + \infty$.

($A_{3}$) For $\omega \in \mathcal{N}$ and $t>0$, let

$$ \varphi (t)=\frac{t^{2}-1}{2}\mathcal{I}'(\omega )\omega - \mathcal{I}(t \omega )+\mathcal{I}(\omega ), $$

obviously, $\varphi '(t)=t\mathcal{I}'(\omega )\omega -\mathcal{I}'(t \omega )\omega$. It follows from Lemma 2.3 that

$$ \mathcal{I}'(\omega )\omega = \Vert \omega \Vert ^{2}-2\lambda \int _{\mathbb{R}^{N}}uv\,dx\geq \biggl(1-\frac{ \vert \lambda \vert }{ \sqrt{a_{1}a_{2}}}\biggr) \Vert \omega \Vert ^{2}>0. $$

We have

$$ \begin{aligned} t\mathcal{I}'(\omega )\omega - \mathcal{I}'(t\omega )\omega ={}&t\biggl( \int _{\mathbb{R}^{N}}f_{1}(u)u\,dx + \int _{\mathbb{R}^{N}}f_{2}(v)v\,dx+2 \int _{\mathbb{R}^{N}}b(x) \vert u \vert ^{q} \vert v \vert ^{q}\,dx\biggr) \\ &{}-\biggl( \int _{\mathbb{R}^{N}}f_{1}(tu)u\,dx+ \int _{\mathbb{R}^{N}}f_{2}(tv)v\,dx+2 t^{2q-1} \int _{\mathbb{R}^{N}}b(x) \vert u \vert ^{q} \vert v \vert ^{q}\,dx\biggr). \end{aligned} $$

In view of ($F_{4}$), if $t<1$, then

$$ \begin{aligned} \varphi '(t)&=t\mathcal{I}'( \omega )\omega -\mathcal{I}'(t\omega ) \omega \\ &> t^{2q-1}\biggl( \int _{\mathbb{R}^{N}}f_{1}(u)u\,dx+ \int _{\mathbb{R}^{N}}f_{2}(v)v\,dx- \int _{\mathbb{R}^{N}}\frac{f_{1}(tu)u}{t ^{2q-1}}\,dx - \int _{\mathbb{R}^{N}}\frac{f_{2}(tv)v}{t^{2q-1}}\biggr)\,dx\geq 0. \end{aligned} $$

While for $t>1$, we have $\varphi '(t)=t\mathcal{I}'(\omega )\omega - \mathcal{I}'(t\omega )\omega <0$. Hence $\varphi (t)<\varphi (1)=0$ for $t\in (0,+\infty )\setminus \{1\}$.

($A_{4}$) In view of ($A_{1}$) and ($A_{2}$), for any $\omega \in E\setminus \{(0,0)\}$, there exists a maximum point $t_{\mathrm{max}}$ of $t\mapsto \mathcal{J}(t\omega )$ such that $\mathcal{J}'(t_{\mathrm{max}}\omega )\omega =0$ and $t_{\mathrm{max}} \omega \in \mathcal{N}$.

For any $\omega \in \mathcal{N}$ and $t\in (0,+\infty )\setminus \{1\}$, we have

$$ \mathcal{J}(t\omega )=\mathcal{J}(\omega )+\biggl(\mathcal{J}(t\omega )- \mathcal{J}(\omega )-\frac{t^{2}-1}{2}\mathcal{J}'(\omega ) \omega \biggr) = \mathcal{J}(\omega )+\varphi (t)< \mathcal{J}(\omega ). $$

 □

Lemma 2.5

$\mathcal{J}$is coercive on$\mathcal{N}$, i.e. there is a sequence$(\omega _{n})\subset \mathcal{N}$such that$\mathcal{J}(\omega _{n}) \rightarrow +\infty$as$\|\omega _{n}\|\rightarrow +\infty$.

Proof

Let $(\omega _{n})\subset \mathcal{N}$ be a sequence such that $\|\omega _{n}\|\rightarrow +\infty$ as $n\rightarrow +\infty$. From (2.2), we find

$$ \begin{aligned} \mathcal{J}(\omega _{n})&=\mathcal{J}( \omega _{n})-\frac{1}{2q} \mathcal{J}'(\omega _{n})\omega _{n}\geq \biggl(\frac{1}{2}- \frac{1}{2q}\biggr) \biggl( \Vert \omega _{n} \Vert ^{2}-2\lambda \int _{\mathbb{R}^{N}}u_{n}v_{n}\,dx\biggr) \\ &\geq \biggl(\frac{1}{2}-\frac{1}{2q}\biggr) \biggl(1- \frac{ \vert \lambda \vert }{\sqrt{a _{1}a_{2}}}\biggr) \Vert \omega _{n} \Vert ^{2}\rightarrow +\infty . \end{aligned} $$

 □

The Nehari manifold $\mathcal{N}$ has the following properties.

Proposition 2.6

• (i) $\mathcal{N}\subset E$is a$\mathcal{C}^{1}$-manifold;

• (ii) ωis a nonzero free critical point of$\mathcal{J}$if and only ifωis a critical point of$\mathcal{J}$constrained on$\mathcal{N}$;

• (iii) If$(\omega _{n})$is a$(PS)$sequence for$\mathcal{J}|_{ \mathcal{N}}$, then$\omega _{n}$is a$(PS)$sequence for$\mathcal{J}$.

Proof

(i) For $\omega \in \mathcal{N}$, we denote

$$ \begin{aligned} \xi (u,v):={}&\mathcal{J}'(u,v) (u,v) \\ ={}& \bigl\Vert (u,v) \bigr\Vert ^{2}-2\lambda \int uv\,dx- \int f_{1}(u)u\,dx \\ &{}- \int f_{2}(v)v\,dx-2 \int b(x) \vert u \vert ^{q} \vert v \vert ^{q}\,dx. \end{aligned} $$

Let $\varphi _{i}(s):=\frac{f_{i}(s)}{s^{2q-1}}$ for $s>0$. In view of ($F_{4}$), we have $\frac{d\varphi _{i}(s)}{ds}\geq 0$, i.e. $f_{i}'(s)s^{2q-1}-(2q-1)f_{i}(s)s^{2q-2}\geq 0$ for $s>0$, which implies

$$ f_{i}(s)s-f_{i}'(s)s^{2} \leq -(2q-2)f_{i}(s)s \quad \text{for } s>0. $$

Assume $s<0$, then $-s>0$ and $-f_{i}(-s)s-f_{i}'(-s)s^{2}\leq (2q-2)f _{i}(-s)s$ for $s<0$, in view of ($F_{2}$), we find

$$ f_{i}(s)s-f_{i}'(s)s^{2} \leq -(2q-2)f_{i}(s)s \quad \text{for } s< 0. $$

It is clear that

$$ \begin{aligned} \xi '(u,v) (u,v)={}&2 \bigl\Vert (u,v) \bigr\Vert ^{2}-4\lambda \int uv\,dx- \int f_{1}(u)u+f'_{1}(u)u ^{2}\,dx \\ &{}- \int f_{2}(v)v+f'_{2}(v)v^{2} \,dx-4q \int b(x) \vert u \vert ^{q} \vert v \vert ^{q}\,dx \\ ={}& \int f_{1}(u)u-f'_{1}(u)u^{2} \,dx+ \int f_{2}(v)v-f'_{2}(v)v ^{2}\,dx \\ &{}-(2q-2)2 \int b(x) \vert u \vert ^{q} \vert v \vert ^{q}\,dx \\ \leq {}&{-}(2q-2) \biggl( \int f_{1}(u)u+f_{2}(v)v\,dx+2 \int b(x) \vert u \vert ^{q} \vert v \vert ^{q}\,dx\biggr) \\ ={}&{-}(2q-2) \biggl( \bigl\Vert (u,v) \bigr\Vert ^{2}-2\lambda \int uv\,dx\biggr). \end{aligned} $$

It follows from Lemma 2.3 that

$$ \bigl\Vert (u,v) \bigr\Vert ^{2}-2\lambda \int uv\,dx\geq \biggl(1-\frac{ \vert \lambda \vert }{\sqrt{a _{1}a_{2}}}\biggr) \Vert \omega \Vert ^{2}>0, $$

then

$$ \xi '(u,v) (u,v)\leq -(2q-2) \biggl( \bigl\Vert (u,v) \bigr\Vert ^{2}-2\lambda \int uv\,dx\biggr)< 0. $$

(2.5)

Hence $\mathcal{N}\subset E$ is a $\mathcal{C}^{1}$-manifold.

(ii) If $\omega \neq (0,0)$ is a critical point of $\mathcal{J}$, then $\mathcal{J}'(\omega )=0$ and $\omega \in \mathcal{N}$. If $\omega \in \mathcal{N}$ is a critical point of $\mathcal{J}$ on $\mathcal{N}$, by applying the Lagrange multiplier theorem, one has $\mathcal{J}'(\omega )=\delta \xi '(\omega )$ and $\mathcal{J}'(\omega )\omega =\delta \xi '(\omega )\omega$ for $\delta \in \mathbb{R}$. From (2.5) we deduce that $\delta =0$ and $\mathcal{J}'(\omega )=0$.

(iii) Let $(\omega _{n})\subset \mathcal{N}$ be a $(PS)$ sequence of $\mathcal{J|_{N}}$, then

$$ \mathcal{J}(\omega _{n})\geq \biggl(\frac{1}{2}- \frac{1}{2q}\biggr) \biggl( \Vert \omega _{n} \Vert ^{2}-2\lambda \int u_{n}v_{n}\,dx\biggr)\geq \biggl( \frac{1}{2}-\frac{1}{2q}\biggr) \biggl(1-\frac{ \vert \lambda \vert }{\sqrt{a_{1}a_{2}}} \biggr) \Vert \omega _{n} \Vert ^{2}, $$

which implies $(\omega _{n})$ is bounded in E. For some $\delta _{n} \in \mathbb{R}$, we have

$$ \circ (1)=\mathcal{J}'|_{\mathcal{N}}(\omega _{n})=\mathcal{J}'(\omega _{n})-\delta _{n}\xi '(\omega _{n}), $$

(2.6)

thus $\delta _{n}\xi '(\omega _{n})\omega _{n}+\circ (1)=\mathcal{J}'( \omega _{n})\omega _{n}=0$. From (2.5) we deduce that $\delta _{n}\rightarrow 0$. In view of (2.6), we get $\mathcal{J}'( \omega _{n})\rightarrow 0$. □

3 Palais–Smale sequences on Nehari manifold

In this section, $f_{i}$ satisfies ($F_{1}$)–($F_{4}$), a potential function $b(x)$ satisfies (B) or ($B_{1}$).

Lemma 3.1

There exists a bounded sequence$(u_{n},v_{n})\subset \mathcal{N}$such that$\mathcal{J}(u_{n},v_{n})\rightarrow c$and$\mathcal{J}'(u_{n},v _{n})\rightarrow 0$as$n\rightarrow +\infty$.

Proof

It follows from Lemma 2.3 and Lemma 2.5 that $\mathcal{J}$ is bounded from below on $\mathcal{N}$. By using Ekeland’s variational principle [22], there exists a sequence $(u_{n},v_{n})\subset \mathcal{N}$ such that

$$ \begin{aligned} &\mathcal{J}(u_{n},v_{n}) \leq \inf_{\mathcal{N}} \mathcal{J}(u,v)+\frac{1}{n}, \\ &\mathcal{J}(u,v)\geq \mathcal{J}(u_{n},v_{n})- \frac{1}{n} \bigl\Vert (u_{n}-u,v_{n}-v) \bigr\Vert \quad \text{for any }(u,v)\in \mathcal{N}. \end{aligned} $$

(3.1)

Hence $\mathcal{J}(u_{n},v_{n})\rightarrow \inf_{\mathcal{N}}\mathcal{J}(u,v)=c$ as $n\rightarrow +\infty$. It follows that

$$ \begin{aligned}[b] c+\frac{1}{n}&\geq \mathcal{J}(u_{n},v_{n})\geq \biggl( \frac{1}{2}- \frac{1}{2q}\biggr) \biggl( \Vert \omega _{n} \Vert ^{2}-2\lambda \int u_{n}v_{n}\,dx\biggr) \\ &\geq \biggl(\frac{1}{2}-\frac{1}{2q}\biggr) \biggl(1- \frac{ \vert \lambda \vert }{\sqrt{a_{1}a _{2}}}\biggr) \Vert \omega _{n} \Vert ^{2}, \end{aligned} $$

(3.2)

and

$$ \Vert \omega _{n} \Vert ^{2}\leq C. $$

(3.3)

For a fixed $(y,z)\in E$ and $\|(y,z)\|\leq 1$, we denote

$$ G_{n}(s,t)= \mathcal{J}'(u_{n}+sy+tu_{n}, v_{n}+sz+tv_{n}) (u_{n}+sy+tu _{n}, v_{n}+sz+tv_{n}). $$

(3.4)

Obviously, $G_{n}(0,0)=\mathcal{J}'(u_{n}, v_{n})(u_{n}, v_{n})=0$. In view of (2.5), we have

$$ \frac{\partial G_{n}}{\partial t}(0,0)=\xi '(u_{n},v_{n}) (u_{n},v_{n})< 0. $$

By implicit function theorem, there exist $C^{1}$ functions $t_{n}(s):(-\delta _{n},\delta _{n})\rightarrow \mathbb{R}$ such that $t_{n}(0)=0$ and

$$ {G_{n}\bigl(s,t_{n}(s)\bigr)=0 \quad \text{for } s\in (-\delta _{n},\delta _{n}).} $$

(3.5)

Differentiating $G_{n}(s,t_{n}(s))$ in s at $s=0$, we have

$$ \frac{\partial G_{n}}{\partial s}(0,0)+ \frac{\partial G_{n}}{\partial t}(0,0)t_{n}'(0)=0. $$

(3.6)

Combining Lemma 2.3 and (2.5), we get

$$ \begin{aligned}[b] \biggl\vert \frac{\partial G_{n}}{\partial t}(0,0) \biggr\vert &= \bigl\vert \xi '(u_{n},v_{n}) (u_{n},v _{n}) \bigr\vert \\ &\geq (2q-2) \biggl( \bigl\Vert (u_{n},v_{n}) \bigr\Vert ^{2}-2\lambda \int u_{n}v_{n}\,dx\biggr) \\ &\geq (2q-2) \biggl(1-\frac{ \vert \lambda \vert }{\sqrt{a_{1}a_{2}}}\biggr) \Vert \omega _{n} \Vert ^{2} \geq (2q-2) \biggl(1- \frac{ \vert \lambda \vert }{\sqrt{a_{1}a_{2}}}\biggr)\beta ^{2}. \end{aligned} $$

(3.7)

It is clear that

$$ \begin{aligned}[b] \biggl\vert \frac{\partial G_{n}}{\partial s}(0,0) \biggr\vert &= \bigl\vert \xi '(u_{n},v_{n}) (y,z) \bigr\vert \\ &\leq 2 \bigl\vert \bigl\langle (u_{n},v_{n}),(y,z) \bigr\rangle \bigr\vert +2 \biggl\vert \lambda \int u_{n}z\,dx \biggr\vert +2 \biggl\vert \lambda \int v_{n}y\,dx \biggr\vert \\ &\quad {}+ \biggl\vert \int f_{1}(u_{n})y\,dx \biggr\vert + \biggl\vert \int f'_{1}(u_{n})u_{n}y \,dx \biggr\vert \\ &\quad {}+ \biggl\vert \int f_{2}(v _{n})z\,dx \biggr\vert + \biggl\vert \int f'_{2}(v_{n})v_{n}z \,dx \biggr\vert \\ &\quad {}+2q \int b(x) \vert u_{n} \vert ^{q-1} \vert v_{n} \vert ^{q} \vert y \vert \,dx+2q \int b(x) \vert u_{n} \vert ^{q} \vert v _{n} \vert ^{q-1} \vert z \vert \,dx. \end{aligned} $$

(3.8)

By using Hölder’s inequality, embedding theorem and (3.3), we find

$$ 2 \bigl\vert \bigl\langle (u_{n},v_{n}),(y,z)\bigr\rangle \bigr\vert +2 \biggl\vert \lambda \int u_{n}z\,dx \biggr\vert +2 \biggl\vert \lambda \int v_{n}y\,dx \biggr\vert \leq C_{1}. $$

(3.9)

In view of ($F_{1}$), Lemma 2.1 and (3.3), we have

$$ \begin{aligned}[b] &\biggl\vert \int f_{1}(u_{n})y\,dx \biggr\vert + \biggl\vert \int f'_{1}(u_{n})u_{n}y \,dx \biggr\vert \\ &\quad {}+ \biggl\vert \int f_{2}(v_{n})z\,dx \biggr\vert + \biggl\vert \int f'_{2}(v_{n})v_{n}z \,dx \biggr\vert \leq C_{2}. \end{aligned} $$

(3.10)

Moreover, we deduce that

$$ \begin{aligned}[b] &2q \int b(x) \vert u_{n} \vert ^{q-1} \vert v_{n} \vert ^{q} \vert y \vert \,dx+2q \int b(x) \vert u_{n} \vert ^{q} \vert v _{n} \vert ^{q-1} \vert z \vert \,dx \\ &\quad \leq 2q \vert b \vert _{\infty } \vert u_{n} \vert ^{q-1}_{2q} \vert v_{n} \vert _{2q}^{q} \vert y \vert _{2q}+2q \vert b \vert _{ \infty } \vert u_{n} \vert ^{q}_{2q} \vert v_{n} \vert _{2q}^{q-1} \vert z \vert _{2q}\leq C_{3}. \end{aligned} $$

(3.11)

It follows from (3.8)–(3.11) that

$$ \biggl\vert \frac{\partial G_{n}}{\partial s}(0,0) \biggr\vert \leq C_{4}. $$

(3.12)

Combining (3.6), (3.7) and (3.12), we get

$$ \bigl\vert t_{n}'(0) \bigr\vert \leq C_{5}. $$

(3.13)

Denote

$$ \begin{aligned} &(\bar{y},\bar{z})_{n,s}=s(y,z)+t_{n}(s) (u_{n},v_{n}), \\ &(y,z)_{n,s}=(u_{n},v_{n})+(\bar{y}, \bar{z})_{n,s}. \end{aligned} $$

(3.14)

From (3.4) and (3.5), we find $(y,z)_{n,s}\in \mathcal{N}$ for $s\in (-\delta _{n},\delta _{n})$. It follows from (3.1) that

$$ \bigl\vert \mathcal{J}(y,z)_{n,s}- \mathcal{J}(u_{n},v_{n}) \bigr\vert \leq \frac{1}{n} \bigl\Vert ( \bar{y},\bar{z})_{n,s} \bigr\Vert . $$

(3.15)

Applying Taylor’s expansion on the left-hand side of (3.15), we get

$$ \begin{aligned}[b] \mathcal{J}(y,z)_{n,s}- \mathcal{J}(u_{n},v_{n})&=\mathcal{J}'(u_{n},v _{n}) (\bar{y},\bar{z})_{n,s}+r(n,s) \\ &=s\mathcal{J}'(u_{n},v_{n}) (y,z)+r(n,s), \end{aligned} $$

(3.16)

where $r(n,s)=o(\|(\bar{y},\bar{z})_{n,s}\|)$ as $|s|\rightarrow 0$. Combining (3.3), (3.13), (3.14) and $t_{n}(0)=0$, we find

$$ \limsup_{ \vert s \vert \rightarrow 0} \frac{ \Vert (\bar{y},\bar{z})_{n,s} \Vert }{ \vert s \vert }\leq C_{6}, $$

(3.17)

where $C_{6}$ is independent of n. It follows that $r(n,s)=o(|s|)$ as $|s|\rightarrow 0$. From (3.15), (3.16) and (3.17), we get

$$ \bigl\vert \mathcal{J}'(u_{n},v_{n}) (y,z) \bigr\vert \leq \frac{C_{6}}{n}. $$

(3.18)

Hence $\mathcal{J}'(u_{n},v_{n})\rightarrow 0$ as $n\rightarrow + \infty$. □

From (iii) of Proposition 2.6 and Lemma 3.1, we get that $(\omega _{n})$ is bounded in E and $\mathcal{J}'(\omega _{n})\rightarrow 0$. Hence there exists a subsequence of $(\omega _{n})$ such that $(u_{n},v_{n})\rightharpoonup (u_{0},v_{0})$ in E. Then we have the following result.

Lemma 3.2

Suppose$\omega _{n}\rightharpoonup \omega _{0}$inEand$\mathcal{J}'(\omega _{n})\rightarrow 0$, then$\mathcal{J}'(\omega _{0})=0$.

Proof

For any $\phi =(\varphi ,\psi )$, φ, $\psi \in C_{0} ^{\infty }(\mathbb{R}^{N})$, we have

$$ \begin{aligned}[b] \mathcal{J}'(\omega _{n})\phi &=\bigl\langle (u_{n},v_{n}),( \varphi ,\psi ) \bigr\rangle -\lambda \int _{\mathbb{R}^{N}}u_{n}\psi \,dx-\lambda \int _{\mathbb{R}^{N}}v_{n}\varphi \,dx \\ &\quad {}- \int _{\mathbb{R}^{N}}f_{1}(u_{n})\varphi \,dx- \int _{\mathbb{R}^{N}}f_{2}(v_{n})\psi \,dx- \int _{\mathbb{R}^{N}}b(x) \vert u _{n} \vert ^{q-2}u_{n} \vert v_{n} \vert ^{q}\varphi \,dx \\ &\quad {}- \int _{\mathbb{R}^{N}}b(x) \vert v_{n} \vert ^{q-2}v_{n} \vert u_{n} \vert ^{q} \psi \,dx. \end{aligned} $$

(3.19)

Up to a subsequence, we have

$$ \begin{aligned} &(u_{n},v_{n})\rightarrow (u_{0},v_{0})\quad \text{in } L_{\mathrm{loc}} ^{t}\bigl(\mathbb{R}^{N}\bigr)\times L_{\mathrm{loc}}^{t}\bigl(\mathbb{R}^{N}\bigr) \text{ for }1\leq t< 2^{*}, \\ &(u_{n},v_{n})\rightarrow (u_{0},v_{0})\quad \text{for a.e. } x\in \mathbb{R}^{N}. \end{aligned} $$

The weak convergence $\omega _{n}\rightharpoonup \omega _{0}$ implies that $\langle (u_{n},v_{n}),(\varphi ,\psi )\rangle \rightarrow \langle (u _{0},v_{0}),(\varphi ,\psi )\rangle$, $\int _{\mathbb{R}^{N}}u_{n} \psi \,dx\rightarrow \int _{\mathbb{R}^{N}}u_{0}\psi \,dx$ and $\int _{\mathbb{R}^{N}}v_{n}\varphi \,dx\rightarrow \int _{\mathbb{R}^{N}}v _{0}\varphi \,dx$.

Let $K\subset \mathbb{R}^{N}$ be a compact set containing supports of φ, ψ, then $(u_{n},v_{n})\rightarrow (u_{0},v _{0})$ in $L^{t}(K)\times L^{t}(K)$ for $1\leq t<2^{*}$. By [23, Theorem 4.9], there exist $l_{K}(x)\in L^{2q}(K)$ and $m_{K}(x)\in L^{2q}(K)$ such that $|u_{n}(x)|\leq l_{K}(x)$ and $|v_{n}(x)|\leq m_{K}(x)$ for a.e. $x\in K$. Let $h_{K}(x):=l_{K}(x)+m _{K}(x)$ for $x\in K$, then $h_{K}(x)\in L^{2q}(K)$ and

$$ \bigl\vert u_{n}(x) \bigr\vert , \bigl\vert v_{n}(x) \bigr\vert \leq h_{K}(x) \quad \text{for a.e. }x \in K. $$

Hence $b(x)|u_{n}|^{q-2}u_{n}|v_{n}|^{q}\varphi \leq b(x)h_{K}^{2q-1}| \varphi |$ for a.e. $x\in K$, and

$$ \int _{K}b(x)h_{K}^{2q-1} \vert \varphi \vert \,dx\leq \vert b \vert _{\infty } \vert h_{K}\chi _{K} \vert _{2q}^{2q-1} \vert \varphi \vert _{2q}. $$

Applying Lebesgue’s dominated convergence theorem, we deduce that

$$ \int _{K}b(x) \vert u_{n} \vert ^{q-2}u_{n} \vert v_{n} \vert ^{q}\varphi \,dx\rightarrow \int _{K}b(x) \vert u_{0} \vert ^{q-2}u_{0} \vert v_{0} \vert ^{q}\varphi \,dx. $$

By similar arguments as above and Lemma 2.1, we deduce

$$ \int _{K}f_{1}(u_{n})\varphi \,dx \rightarrow \int _{K}f_{1}(u_{0})\varphi \,dx\quad \text{and}\quad \int _{K}f_{2}(v_{n})\psi \,dx \rightarrow \int _{K}f_{2}(v _{0})\psi \,dx. $$

It follows from (3.19) that

$$ \mathcal{J}'(u_{n},v_{n}) (\varphi , \psi )\rightarrow \mathcal{J}'(u _{0},v_{0}) (\varphi ,\psi ). $$

Hence $\mathcal{J}'(u_{0},v_{0})=0$. □

We introduce the vanishing lemma from [24].

Lemma 3.3

([24, Lemma 2.4])

Assume that$\{u_{k}\}$is a bounded sequence in$H^{s}(\mathbb{R}^{N})$, which satisfies

$$ \sup_{z\in \mathbb{R}^{N}} \int _{B(z,1)} \vert u_{k} \vert ^{2}\,dx \rightarrow 0. $$

(3.20)

Then$u_{k}\rightarrow 0$strongly in$L^{r}(\mathbb{R}^{N})$for every$2< r<\frac{2N}{N-2s}$.

Lemma 3.4

Assume that$\{\omega _{n}\}$is a PS sequence constrained on$\mathcal{N}$, which satisfies

$$ \sup_{z\in \mathbb{R}^{N}} \int _{B(z,1)} \vert \omega _{n} \vert ^{2}\,dx \rightarrow 0, $$

(3.21)

then$\|\omega _{n}\|\rightarrow 0$.

Proof

Combining Lemma 3.3 and (3.21), we get $u_{n},v_{n} \rightarrow 0$ in $L^{r}(\mathbb{R}^{N})$ for $2< r<2^{*}$. For $\omega _{n}\subset \mathcal{N}$, we have

$$ \begin{aligned}[b] \biggl(1-\frac{ \vert \lambda \vert }{\sqrt{a_{1}a_{2}}} \biggr) \Vert \omega _{n} \Vert ^{2}&\leq \Vert \omega _{n} \Vert ^{2}-2\lambda \int _{\mathbb{R}^{N}}u_{n}v_{n}\,dx \\ &= \int _{\mathbb{R}^{N}}f_{1}(u_{n})u_{n}+f_{2}(v_{n})v _{n}\,dx+2 \int _{\mathbb{R}^{N}}b(x) \vert u_{n} \vert ^{q} \vert v_{n} \vert ^{q}\,dx. \end{aligned} $$

(3.22)

It is clear that

$$ \biggl\vert \int _{\mathbb{R}^{N}}f_{1}(u_{n})u_{n} \,dx \biggr\vert \leq \varepsilon \vert u_{n} \vert ^{2} _{2}+C_{\varepsilon } \vert u_{n} \vert _{p}^{p}. $$

Let $\varepsilon \rightarrow 0$, we have $|\int _{\mathbb{R}^{N}}f_{1}(u _{n})u_{n}\,dx|\rightarrow 0$. Moreover,

$$ \int _{\mathbb{R}^{N}}b(x) \vert u_{n} \vert ^{q} \vert v_{n} \vert ^{q}\,dx\leq \vert b \vert _{\infty } \vert u _{n} \vert _{2q}^{q} \vert v_{n} \vert _{2q}^{q}\rightarrow 0. $$

It follows from (3.22) that $\|\omega _{n}\|\rightarrow 0$. □

4 Ground states of a Schrödinger system with periodic couplings

We prove (i) and (ii) of Theorem 1.1 in Sects. 4.1–4.2, respectively.

4.1 Existence

Step 1: We find$(u_{0},v_{0})\in E$such that$\mathcal{J}'(u_{0},v _{0})=0$.

In view of Lemma 3.1, there exists a bounded $(PS)_{c}$-sequence of $\mathcal{J}$ constrained on $\mathcal{N}$, i.e. a sequence ${\omega _{n}}\subset \mathcal{N}$ such that $\mathcal{J}( \omega _{n})\rightarrow c$ and $(\mathcal{J}|_{\mathcal{N}})'(\omega _{n})\rightarrow 0$. It follows from (iii) of Proposition 2.6 that $\mathcal{J}'(\omega _{n})\rightarrow 0$. In view of Lemma 3.2, up to a subsequence, then

$$\begin{aligned}& (u_{n},v_{n}) \rightharpoonup (u_{0},v_{0})\quad \text{in }E, \\& (u_{n},v_{n})\rightarrow (u_{0},v_{0}) \quad \text{in }L_{\mathrm{loc}}^{t}\bigl( \mathbb{R}^{N} \bigr)\times L_{\mathrm{loc}}^{t}\bigl(\mathbb{R}^{N} \bigr) \text{ for }1 \leq t< 2^{*}, \\& (u_{n},v_{n})\rightarrow (u_{0},v_{0})\quad \text{for a.e. }x \in \mathbb{R}^{N}, \end{aligned}$$

and $\mathcal{J}'(u_{0},v_{0})=0$.

Step 2: We check whether$(u_{0},v_{0})\neq (0,0)$.

Suppose

$$ \sup_{z\in \mathbb{R}^{N}} \int _{B(z,1)} \vert \omega _{n} \vert ^{2}\,dx \rightarrow 0. $$

It follows from Lemma 3.4 that $\|(u_{n},v_{n})\|\rightarrow 0$. We get a contradiction with respect to Lemma 2.3. By Lions’ lemma [25] there exists $(y_{n})\subset \mathbb{R}^{N}$ such that

$$ \liminf_{n\rightarrow \infty } \int _{B(y_{n},1)} \vert u _{n} \vert ^{2}\,dx>\delta \quad \text{or}\quad \liminf_{n\rightarrow \infty } \int _{B(y_{n},1)} \vert v _{n} \vert ^{2}\,dx>\delta . $$

We assume, without loss of generality, that

$$ \liminf_{n\rightarrow \infty } \int _{B(y_{n},1)} \vert u _{n} \vert ^{2}\,dx>\delta . $$

For each $y_{n}\in \mathbb{R}^{N}$, we will find $z_{n}\in \mathbb{Z} ^{N}$ such that $B(y_{n},1)\subset B(z_{n},1+\sqrt{N})$, then

$$ \liminf_{n\rightarrow \infty } \int _{B(z_{n},1+\sqrt{N})} \vert u_{n} \vert ^{2}\,dx\geq \liminf_{n\rightarrow \infty } \int _{B(y_{n},1)} \vert u_{n} \vert ^{2}\,dx>\delta . $$

Since $\mathcal{J}$ and $\mathcal{N}$ are invariant under translations of the form $\omega \mapsto \omega (\cdot -k)$ with $k\in \mathbb{Z} ^{N}$, we may assume that $(z_{n})$ is bounded in $\mathbb{Z}^{N}$. It is clear that $u_{0}\neq 0$ by $u_{n}\rightarrow u_{0}$ in $L_{ \mathrm{loc}}^{2}(\mathbb{R}^{N})$. Hence $\omega _{0}=(u_{0},v_{0}) \neq (0,0)$, $(u_{0},v_{0})\in \mathcal{N}$ and $\mathcal{J}(u_{0},v _{0})\geq c$.

Step 3: We find$(u',v')$such that$\mathcal{J}'(u',v')=0$and$\mathcal{J}(u',v')=c$, where$u'>0$and$v'<0$as$\lambda \in (-\sqrt{a_{1}a_{2}},0)$,$u'>0$and$v'>0$as$\lambda \in (0,\sqrt{a_{1}a_{2}})$.

Applying Fatou’s lemma, we get

$$ \begin{aligned} c&=\liminf_{n\rightarrow \infty } \mathcal{J}(u_{n},v _{n}) \\ &=\liminf_{n\rightarrow \infty } \int \frac{1}{2}f _{1}(u_{n})u_{n}-F_{1}(u_{n}) \\ &\quad {}+\frac{1}{2}f_{2}(v_{n})v_{n}-F_{2}(v_{n}) \,dx+\biggl(1- \frac{1}{q}\biggr) \int b(x) \vert u_{n} \vert ^{q} \vert v_{n} \vert ^{q}\,dx \\ &\geq \int \frac{1}{2}f_{1}(u_{0})u_{0}-F_{1}(u_{0})+ \frac{1}{2}f_{2}(v_{0})v_{0}-F_{2}(v_{0}) \,dx \\ &\quad {}+\biggl(1-\frac{1}{q}\biggr) \int b(x) \vert u_{0} \vert ^{q} \vert v_{0} \vert ^{q}\,dx \\ &=\mathcal{J}(u_{0},v_{0}). \end{aligned} $$

From the above computations, we find that $\mathcal{J}(u_{0},v_{0})=c$. Hence $(u_{0},v_{0})\neq (0,0)$ is a ground state solution of (1.1).

Case 1. $\lambda \in (-\sqrt{a_{1}a_{2}},0)$.

It is clear that $\|(|u_{0}|,-|v_{0}|)\|\leq \|(u_{0},v_{0}) \|$. By ($A_{4}$) of Lemma 2.4, there exists $t>0$ such that $(t|u_{0}|,-t|v_{0}|)\in \mathcal{N}$ and $\mathcal{J}(t|u_{0}|,-t|v _{0}|)\geq c$, then

$$ c\leq \mathcal{J}\bigl(t \vert u_{0} \vert ,-t \vert v_{0} \vert \bigr)\leq \mathcal{J}(tu_{0},tv_{0}) \leq \mathcal{J}(u_{0},v_{0})=c. $$

Let $(u',v'):=(t|u_{0}|,-t|v_{0}|)$, $u'\geq 0$ and $v'\leq 0$, we get that $(u',v')$ is a ground state solution of (1.1) by (ii) of Proposition 2.6. It follows from (1.1) that

$$ (-\Delta )^{s} u'+a_{1}u'=f_{1} \bigl(u'\bigr)+b(x) \bigl\vert u' \bigr\vert ^{q-2}u' \bigl\vert v' \bigr\vert ^{q}+\lambda v'\geq 0 $$

and

$$ (-\Delta )^{s} v'+a_{2}v'=f_{2} \bigl(v'\bigr)+b(x) \bigl\vert u' \bigr\vert ^{q} \bigl\vert v' \bigr\vert ^{q-2}v'+ \lambda u'\leq 0. $$

In view of (1.1), if $u'=0$, then $v'=0$. Hence $u',v'\neq 0$ by $(u',v')\neq (0,0)$. Applying the strong maximum principle [26] to each equality of (1.1), we get that $(u',v')$, $u'>0$ and $v'<0$ is a ground state solution of (1.1).

Case 2. $\lambda \in (0,\sqrt{a_{1}a_{2}})$.

There exists $t'>0$ such that $(t'|u_{0}|,t'|v_{0}|)\in \mathcal{N}$ and $\mathcal{J}(t'|u_{0}|,t'|v_{0}|)\geq c$. We deduce that

$$ c\leq \mathcal{J}\bigl(t' \vert u_{0} \vert ,t' \vert v_{0} \vert \bigr)\leq \mathcal{J} \bigl(t'u_{0},t'v _{0} \bigr)\leq \mathcal{J}(u_{0},v_{0})=c. $$

By similar arguments in Case 1, we get that $(u',v')$, $u'>0$ and $v'>0$ is a ground state solution of (1.1). This completes the proof of (i) of Theorem 1.1.

4.2 Asymptotic behaviour of ground states as $b_{n}\rightarrow 0$ in $L^{\infty }(\mathbb{R}^{N})$

Denote

$$ G_{i}(u):=\frac{1}{2}f_{i}(u)u-F_{i}(u),\quad i=1,2. $$

From (2.2), it suffices to show that $G_{i}(u)\geq 0$. The following version of Brezis–Lieb lemma [16] is crucial to proving the asymptotic behaviour of ground states.

Lemma 4.1

(Brezis–Lieb lemma)

Assume that ($F_{1}$)–($F_{4}$) are satisfied, let$\{u_{n}\}$be a bounded sequence such that$u_{n}\rightharpoonup u$weakly in$H^{s}(\mathbb{R}^{N})$. Then

$$ \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{N}}\bigl[G _{i}(u_{n})-G_{i}(u_{n}-u) \bigr]\,dx= \int _{\mathbb{R}^{N}}G_{i}(u)\,dx,\quad i=1,2. $$

(4.1)

Proof

It is clear that

$$ \begin{aligned}[b] \int _{\mathbb{R}^{N}}\bigl[G_{i}(u_{n})-G_{i}(u_{n}-u) \bigr]\,dx&= \int _{\mathbb{R}^{N}} \int _{0}^{1}\frac{d}{dt}G_{i}(u_{n}-u+tu)\,dt\,dx \\ &= \int _{0}^{1} \int _{\mathbb{R}^{N}}g_{i}(u_{n}-u+tu)u\,dx \,dt, \end{aligned} $$

(4.2)

where $g_{i}(u):=\frac{d}{du}G_{i}(u)$, and $g_{i}(u)=\frac{1}{2}f _{i}'(u)u-\frac{1}{2}f_{i}(u)$. From ($F_{1}$), we find

$$ \bigl\vert f_{i}'(u_{n}-u+tu) (u_{n}-u+tu) \bigr\vert \leq c_{1}\bigl( \vert u_{n}-u+tu \vert + \vert u_{n}-u+tu \vert ^{p-1}\bigr). $$

Since $(u_{n}-u+tu)$ is bounded in $H^{s}(\mathbb{R}^{N})$, by using Hölder’s inequality, ($F_{1}$) and Lemma 2.1, we get $\int _{\mathbb{R}^{N}}g_{i}(u_{n}-u+tu)u\,dx$ is bounded. For every $\varepsilon >0$, there is $\sigma >0$ such that

$$ \int _{\varOmega } \bigl\vert g_{i}(u_{n}-u+tu)u \bigr\vert \,dx< \varepsilon $$

for any $n\in \mathbb{N}$ and every measurable subset $\varOmega \subset \mathbb{R}^{N}$ such that $|\varOmega |<\sigma$. Thus $(g_{i}(u_{n}-u+tu)u)$ is uniformly integrable. Moreover, for any $\varepsilon >0$, there exists a measurable subset $\varOmega \subset \mathbb{R}^{N}$ of finite measure $|\varOmega |<+\infty$ such that, for any $n\geq 1$,

$$ \int _{\mathbb{R}^{N}\setminus \varOmega } \bigl\vert g_{i}(u_{n}-u+tu)u \bigr\vert \,dx< \varepsilon . $$

Hence $(g_{i}(u_{n}-u+tu)u)$ is tight over $\mathbb{R}^{N}$. Since $g_{i}(u_{n}-u+tu)u\rightarrow g_{i}(tu)u$ a.e. in $\mathbb{R}^{N}$, in view of the Vitali convergence theorem, $g_{i}(tu)u$ is integrable and

$$ \int _{\mathbb{R}^{N}}g_{i}(u_{n}-u+tu)u\,dx \rightarrow \int _{\mathbb{R} ^{N}}g_{i}(tu)u\,dx \quad \text{as }n \rightarrow +\infty . $$

From (4.2), we deduce

$$ \begin{aligned} \int _{\mathbb{R}^{N}}\bigl[G_{i}(u_{n})-G_{i}(u_{n}-u) \bigr]\,dx&\rightarrow \int _{0}^{1} \int _{\mathbb{R}^{N}}g_{i}(tu)u\,dx\,dt \\ &= \int _{\mathbb{R}^{N}}G_{i}(u)\,dx \quad \text{as } n \rightarrow +\infty . \end{aligned} $$

This completes the proof of Lemma 4.1. □

We denote that $\mathcal{J}_{n}$ is the corresponding functional of (1.1) with $b(x)=b_{n}(x)$, $\mathcal{J}_{0}$ is the corresponding functional of (1.1) with $b(x)=0$. $\mathcal{N} _{n}$ and $\mathcal{N}_{0}$ are well defined in a similar way. Denote

$$ c_{n}:=\inf_{\mathcal{N}_{n}}\mathcal{J}_{n} \quad \text{and}\quad c_{0}:=\inf_{\mathcal{N}_{0}} \mathcal{J}_{0}. $$

From (i) of Theorem 1.1, there exist $\omega _{n}\in \mathcal{N}_{n}$ such that $\mathcal{J}_{n}(\omega _{n})=c_{n}$ and $\omega _{0}\in \mathcal{N}_{0}$ such that $\mathcal{J}_{0}(\omega _{0})=c _{0}$. We need several lemmas for the proof.

Lemma 4.2

Suppose that ($F_{1}$)–($F_{4}$) and (B) are satisfied, then$\omega _{n}$is bounded inE. Moreover, one has

$$ \lim_{n\rightarrow \infty }c_{n}=\lim_{n\rightarrow \infty } \inf_{\mathcal{N}_{n}} \mathcal{J}_{n}=\inf _{\mathcal{N}_{0}}\mathcal{J} _{0}=c_{0}. $$

Proof

Let $t_{n}>0$ be such that $t_{n}\omega _{n}\in \mathcal{N}_{0}$, we have

$$ \begin{aligned}[b] c_{n}&= \mathcal{J}_{n}(\omega _{n})\geq \mathcal{J}_{n}(t_{n} \omega _{n})= \mathcal{J}_{0}(t_{n}\omega _{n})-\frac{t_{n}^{2q}}{q} \int _{\mathbb{R} ^{N}}b_{n}(x) \vert u_{n} \vert ^{q} \vert v_{n} \vert ^{q}\,dx \\ &\geq c_{0}-\frac{t_{n}^{2q}}{q} \int _{\mathbb{R}^{N}}b_{n}(x) \vert u_{n} \vert ^{q} \vert v _{n} \vert ^{q}\,dx. \end{aligned} $$

(4.3)

Let $t_{n}'>0$ be such that $t_{n}'\omega _{0}\in \mathcal{N}_{n}$, then

$$ \begin{aligned}[b] c_{0}&= \mathcal{J}_{0}(\omega _{0})\geq \mathcal{J}_{0} \bigl(t_{n}'\omega _{0}\bigr)= \mathcal{J}_{n}\bigl(t_{n}'\omega _{0}\bigr)+\frac{t_{n}^{\prime 2q}}{q} \int _{\mathbb{R}^{N}}b_{n}(x) \vert u_{0} \vert ^{q} \vert v_{0} \vert ^{q}\,dx \\ &\geq c_{n}+\frac{t_{n}^{\prime 2q}}{q} \int _{\mathbb{R}^{N}}b_{n}(x) \vert u_{0} \vert ^{q} \vert v _{0} \vert ^{q}\,dx. \end{aligned} $$

(4.4)

Combining (4.3) and (4.4), we have

$$ \begin{aligned}[b] c_{n}&\leq c_{0}-\frac{t_{n}^{\prime 2q}}{q} \int _{\mathbb{R}^{N}}b_{n}(x) \vert u _{0} \vert ^{q} \vert v_{0} \vert ^{q}\,dx \\ &\leq c_{0}\leq c_{n}+\frac{t_{n}^{2q}}{q} \int _{\mathbb{R}^{N}}b_{n}(x) \vert u _{n} \vert ^{q} \vert v_{n} \vert ^{q}\,dx. \end{aligned} $$

(4.5)

Since $t_{n}\omega _{n}\in \mathcal{N}_{0}$, we have

$$ \begin{aligned} &\Vert t_{n}\omega _{n} \Vert ^{2}-2\lambda \int _{\mathbb{R}^{N}} t_{n}^{2} u _{n}v_{n}\,dx- \int _{\mathbb{R}^{N}}f_{1}(t_{n}u_{n})t_{n}u_{n}+f_{2}(t _{n}v_{n})t_{n}v_{n}\,dx \\ &\quad {}-2 \int _{\mathbb{R}^{N}}b_{n}(x) \vert t_{n}u_{n} \vert ^{q} \vert t_{n}v_{n} \vert ^{q}\,dx=0. \end{aligned} $$

Suppose $t_{n}\rightarrow +\infty$, in view of ($F_{3}$) and (2.2), we get that

$$ \begin{aligned} 0&= \Vert \omega _{n} \Vert ^{2}-2\lambda \int _{\mathbb{R}^{N}} u_{n}v_{n}\,dx- \int _{\mathbb{R}^{N}}\frac{f_{1}(t_{n}u_{n})t_{n}u_{n}+f_{2}(t_{n}v _{n})t_{n}v_{n}}{t^{2}_{n}}\,dx \\ &\quad {}-2t_{n}^{2q-2} \int _{\mathbb{R}^{N}}b_{n}(x) \vert u_{n} \vert ^{q} \vert v_{n} \vert ^{q}\,dx \\ &\leq \biggl(1+\frac{ \vert \lambda \vert }{\sqrt{a_{1}a_{2}}}\biggr) \Vert \omega _{n} \Vert ^{2}- \int _{\mathbb{R}^{N}}\frac{2qF_{1}(t_{n}u_{n})+2qF_{2}(t_{n}v_{n})}{t _{n}^{2}}\,dx \\ &\quad {}-2t_{n}^{2q-2} \int _{\mathbb{R}^{N}}b_{n}(x) \vert u_{n} \vert ^{q} \vert v_{n} \vert ^{q}\,dx \rightarrow -\infty . \end{aligned} $$

It is a contradiction. Hence $(t_{n})$ is bounded. It follows from (2.2) and (4.4) that

$$ \begin{aligned}[b] c_{0}&\geq c_{n}=\mathcal{J}_{n}(\omega _{n})= \mathcal{J}_{n}(\omega _{n})-\frac{1}{2q} \mathcal{J}'_{n}(\omega _{n})\omega _{n} \\ &\geq \biggl(\frac{1}{2}-\frac{1}{2q}\biggr) \biggl( \Vert \omega _{n} \Vert ^{2}-2\lambda \int _{\mathbb{R}^{N}}u_{n}v_{n}\,dx\biggr) \geq \biggl(\frac{1}{2}-\frac{1}{2q}\biggr) \biggl(1- \frac{ \vert \lambda \vert }{\sqrt{a_{1}a_{2}}}\biggr) \Vert \omega _{n} \Vert ^{2}. \end{aligned} $$

(4.6)

Hence $\omega _{n}$ is bounded in E, then

$$ \int _{\mathbb{R}^{N}}b_{n}(x) \vert u_{n} \vert ^{q} \vert v_{n} \vert ^{q}\,dx \leq \vert b_{n} \vert _{ \infty } \vert u_{n} \vert ^{q}_{2q} \vert v_{n} \vert ^{q}_{2q}\rightarrow 0 \quad \text{as }n \rightarrow +\infty . $$

In view of (4.5), we deduce that $c_{n}\rightarrow c_{0}$ as $n\rightarrow +\infty$. This completes the proof of Lemma 4.2. □

Lemma 4.3

For each ground state solution$\omega _{n}$of$\mathcal{J}_{n}$, there exist$\omega \neq (0,0)$and$(z_{n})\subset \mathbb{Z}^{N}$such that$\omega _{n}(\cdot +z_{n})\rightharpoonup \omega$inE. Moreover, ωis a ground state solution of$\mathcal{J}_{0}$, i.e. $\mathcal{J}_{0}'(\omega )=0$and$\mathcal{J}_{0}(\omega )=c_{0}$.

Proof

In view of Lemma 4.2, $\omega _{n}$ is bounded in E. Suppose

$$ \sup_{z\in \mathbb{R}^{N}} \int _{B(z,1)} \vert \omega _{n} \vert ^{2}\,dx \rightarrow 0. $$

Applying similar arguments in Lemma 3.4, we get $\|\omega _{n} \|\rightarrow 0$. Since

$$ \Vert \omega _{n} \Vert ^{2}-2\lambda \int _{\mathbb{R}^{N}}u_{n}v_{n}\,dx\leq \biggl(1+\frac{ \vert \lambda \vert }{\sqrt{a_{1}a_{2}}}\biggr) \Vert \omega _{n} \Vert ^{2}\rightarrow 0, $$

we have

$$ \begin{aligned} \limsup_{n\rightarrow +\infty } \mathcal{J}_{n}(\omega _{n})&=\limsup _{n\rightarrow +\infty }\biggl[\frac{1}{2}\biggl( \Vert \omega _{n} \Vert ^{2}-2\lambda \int _{\mathbb{R}^{N}}u_{n}v_{n}\,dx\biggr) \\ &\quad {}- \int _{\mathbb{R}^{N}}F_{1}(u_{n})+F_{2}(v_{n}) \,dx-\frac{1}{q} \int _{\mathbb{R}^{N}}b_{n}(x) \vert u_{n} \vert ^{q} \vert v_{n} \vert ^{q}\,dx \biggr] \\ &=\limsup_{n\rightarrow +\infty }\biggl[- \int _{\mathbb{R} ^{N}}F_{1}(u_{n})+F_{2}(v_{n}) \,dx\biggr]\leq 0. \end{aligned} $$

On the other hand, from ($A_{1}$) and ($A_{4}$) in Lemma 2.4, we have

$$ \mathcal{J}_{n}(\omega _{n})\geq \mathcal{J}_{n} \biggl(r\cdot \frac{\omega _{n}}{ \Vert \omega _{n} \Vert }\biggr)\geq a>0. $$

It is a contradiction. By Lions’ lemma [25] there exists $(y_{n})\subset \mathbb{R}^{N}$ such that

$$ \liminf_{n\rightarrow \infty } \int _{B(y_{n},1)} \vert u _{n} \vert ^{2}\,dx>\delta\quad \text{or}\quad \liminf_{n\rightarrow \infty } \int _{B(y_{n},1)} \vert v _{n} \vert ^{2}\,dx>\delta . $$

We assume, without loss of generality, that

$$ \liminf_{n\rightarrow \infty } \int _{B(y_{n},1)} \vert u _{n} \vert ^{2}\,dx>\delta . $$

For each $y_{n}\in \mathbb{R}^{N}$, we will find $z_{n}\in \mathbb{Z} ^{N}$ such that $B(y_{n},1)\subset B(z_{n},1+\sqrt{N})$, then

$$ \liminf_{n\rightarrow \infty } \int _{B(z_{n},1+\sqrt{N})} \vert u_{n} \vert ^{2}\,dx\geq \liminf_{n\rightarrow \infty } \int _{B(y_{n},1)} \vert u_{n} \vert ^{2}\,dx>\delta . $$

Let $\bar{\omega }_{n}:=\omega _{n}(\cdot +z_{n})$, $\bar{u}_{n}:=u _{n}(\cdot +z_{n})$ and $\bar{v}_{n}:=v_{n}(\cdot +z_{n})$, up to a subsequence, there exists $\omega \in E$ such that

$$ \begin{aligned} &\bar{\omega }_{n}\rightharpoonup \omega\quad \text{in } E, \\ &\bar{\omega }_{n}\rightarrow \omega \quad \text{in } L_{\mathrm{loc}}^{t}\bigl(\mathbb{R}^{N}\bigr)\times L_{\mathrm{loc}}^{t}\bigl( \mathbb{R}^{N}\bigr) \text{ for } t\in \bigl[1,2^{\ast }\bigr), \\ &\bar{\omega }_{n}\rightarrow \omega \quad \mbox{a.e. } x\in \mathbb{R}^{N}. \end{aligned} $$

We have

$$ \liminf_{n\rightarrow \infty } \int _{B(0,1+ \sqrt{N})} \vert \bar{u}_{n} \vert ^{2}\,dx>\delta . $$

Hence $u\neq 0$ and $\omega \neq (0,0)$.

For any $\phi =(\varphi ,\psi )$, $\varphi , \psi \in C_{0} ^{\infty }(\mathbb{R}^{N})$, it is clear that

$$ \begin{aligned}[b] 0&=\mathcal{J}_{n}'( \omega _{n})\phi (\cdot -z_{n}) \\ &=\mathcal{J}_{0}'( \bar{ \omega }_{n})\phi - \int _{\mathbb{R}^{N}}b_{n}(x+z_{n}) \vert \bar{u}_{n} \vert ^{q-2} \bar{u}_{n} \vert \bar{v}_{n} \vert ^{q}\varphi \,dx \\ &\quad {}- \int _{\mathbb{R}^{N}}b_{n}(x+z_{n}) \vert \bar{u}_{n} \vert ^{q} \vert \bar{v}_{n} \vert ^{q-2}\bar{v}_{n}\psi \,dx. \end{aligned} $$

(4.7)

By using Hölder’s inequality, we deduce that

$$ \int _{\mathbb{R}^{N}}b_{n}(x+z_{n}) \vert \bar{u}_{n} \vert ^{q-2}\bar{u}_{n} \vert \bar{v}_{n} \vert ^{q}\varphi \,dx\leq \vert b_{n} \vert _{\infty } \vert \bar{u}_{n} \vert _{2q} ^{q-1} \vert \bar{v}_{n} \vert _{2q}^{q} \vert \varphi \vert _{2q}\rightarrow 0. $$

(4.8)

Combining (4.7) and (4.8), we find that $\mathcal{J}'_{0}(\bar{ \omega }_{n})\phi \rightarrow 0$. It follows from Lemma 3.2 that $\mathcal{J}'_{0}(\bar{\omega }_{n})\phi \rightarrow \mathcal{J}'_{0}( \omega )\phi$ and $\mathcal{J}'_{0}(\omega )=0$.

Since $c_{n}\rightarrow c_{0}$, using Fatou’s lemma, we have

$$ \begin{aligned}[b] c_{0}&=\liminf _{n\rightarrow +\infty }\mathcal{J} _{n}(\omega _{n}) \\ &= \liminf_{n\rightarrow +\infty }\biggl[ \int _{\mathbb{R}^{N}}\frac{1}{2}f_{1}( \bar{u}_{n})\bar{u}_{n}-F_{1}( \bar{u}_{n}) \\ &\quad {}+\frac{1}{2}f_{2}(\bar{v}_{n}) \bar{v}_{n}-F_{2}(\bar{v} _{n})\,dx+ \biggl(1-\frac{1}{q}\biggr) \int _{\mathbb{R}^{N}}b_{n}(x) \vert \bar{u}_{n} \vert ^{q} \vert \bar{v}_{n} \vert ^{q}\,dx\biggr] \\ &\geq \liminf_{n\rightarrow +\infty }\biggl[ \int _{\mathbb{R}^{N}}\frac{1}{2}f_{1}( \bar{u}_{n})\bar{u}_{n}-F_{1}( \bar{u}_{n})+\frac{1}{2}f_{2}( \bar{v}_{n})\bar{v}_{n}-F_{2}(\bar{v} _{n})\,dx\biggr] \\ &\geq \int _{\mathbb{R}^{N}}\frac{1}{2}f_{1}(u)u-F_{1}(u)+ \frac{1}{2}f_{2}(v)v-F_{2}(v)\,dx \\ &=\mathcal{J}_{0}(\omega )\geq c_{0}. \end{aligned} $$

(4.9)

Thus ω is a ground state solution of $\mathcal{J}_{0}$. This completes the proof of Lemma 4.3. □

Proof of (ii) of Theorem 1.1

We find that

$$ \begin{aligned}[b] &\biggl(1-\frac{ \vert \lambda \vert }{\sqrt{a_{1}a_{2}}} \biggr) \Vert \bar{\omega }_{n}-\omega \Vert ^{2} \\ &\quad \leq \Vert \bar{\omega }_{n}-\omega \Vert ^{2}-2 \lambda \int _{\mathbb{R}^{N}}(\bar{u}_{n}-u) (\bar{v}_{n}-v) \,dx \\ &\quad =\mathcal{J}_{n}'(\bar{\omega }_{n}) ( \bar{\omega }_{n}- \omega )-\langle \omega ,\bar{\omega }_{n}-\omega \rangle +\lambda \int _{\mathbb{R}^{N}}u(\bar{v}_{n}-v)\,dx \\ &\qquad {}+\lambda \int _{\mathbb{R}^{N}}v(\bar{u}_{n}-u)\,dx+ \int _{\mathbb{R}^{N}}f_{1}(\bar{u}_{n}) ( \bar{u}_{n}-u)+f_{2}(\bar{v} _{n}) ( \bar{v}_{n}-v)\,dx \\ &\qquad {}+ \int _{\mathbb{R}^{N}}b_{n}(x) \vert \bar{u}_{n} \vert ^{q-2}\bar{u} _{n} \vert \bar{v}_{n} \vert ^{q}(\bar{u}_{n}-u)\,dx \\ &\qquad {}+ \int _{\mathbb{R}^{N}}b_{n}(x) \vert \bar{v}_{n} \vert ^{q-2}\bar{v} _{n} \vert \bar{u}_{n} \vert ^{q}(\bar{v}_{n}-v)\,dx. \end{aligned} $$

(4.10)

Since $\bar{\omega }_{n}\rightharpoonup \omega$ in E, we have $\langle \omega ,\bar{\omega }_{n}-\omega \rangle \rightarrow 0$, $\lambda \int _{\mathbb{R}^{N}}u(\bar{v}_{n}-v)\,dx\rightarrow 0$ and $\lambda \int _{\mathbb{R}^{N}}v(\bar{u}_{n}-u)\,dx\rightarrow 0$. It is suffices to show that

$$ \mathcal{J}_{n}'(\bar{ \omega }_{n}) (\bar{\omega }_{n}-\omega )= \mathcal{J}_{n}'(\omega _{n})\omega _{n}-\mathcal{J}_{n}'(\omega _{n}) \omega (\cdot -z_{n})=0. $$

From Lemma 4.1, we get

$$ \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{N}}\bigl[G _{1}(\bar{u}_{n})-G_{1}( \bar{u}_{n}-u)+G_{2}(\bar{v}_{n})-G_{2}( \bar{v}_{n}-v)\bigr]\,dx= \int _{\mathbb{R}^{N}}G_{1}(u)+G_{2}(v)\,dx. $$

(4.11)

In view of (4.9), we have

$$ c_{0}=\lim_{n\rightarrow \infty } \int _{\mathbb{R} ^{N}}G_{1}(\bar{u}_{n})+G_{2}( \bar{v}_{n})\,dx= \int _{\mathbb{R}^{N}}G _{1}(u)+G_{2}(v)\,dx. $$

(4.12)

It follows from (4.11) and (4.12) that

$$ \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{N}}G _{1}(\bar{u}_{n}-u)+G_{2}( \bar{v}_{n}-v)\,dx=0. $$

Since $G_{i}\geq 0$, $i=1,2$, then

$$ \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{N}}G _{1}(\bar{u}_{n}-u) \,dx=0 \quad \text{and}\quad \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{N}}G _{2}(\bar{v}_{n}-v) \,dx=0. $$

From ($F_{5}$), we deduce that

$$ \vert \bar{u}_{n}-u \vert ^{t}_{t}= \int _{\mathbb{R}^{N}} \vert \bar{u}_{n}-u \vert ^{t}\,dx \leq \frac{2}{d} \int _{\mathbb{R}^{N}}G_{1}(\bar{u}_{n}-u)\,dx \rightarrow 0, $$

(4.13)

and $|\bar{v}_{n}-v|^{t}_{t}\rightarrow 0$. By fractional embedding theorem [1], we get that $\bar{u}_{n}$ and $\bar{v}_{n}$ are bounded in $L^{r}(\mathbb{R}^{N})$ for $2\leq r\leq 2^{*}$. Using (4.13) and the interpolation inequality, we get $\bar{u}_{n} \rightarrow u$ and $\bar{v}_{n}\rightarrow v$ in $L^{r}(\mathbb{R} ^{N})$ for $2< r<2^{*}$. For any $\varepsilon >0$, there exists $C_{\varepsilon }>0$ such that

$$ \biggl\vert \int _{\mathbb{R}^{N}}f_{1}(\bar{u}_{n}) ( \bar{u}_{n}-u)\,dx \biggr\vert \leq \varepsilon \vert \bar{u}_{n} \vert _{2} \vert \bar{u}_{n}-u \vert _{2}+C_{\varepsilon } \vert \bar{u}_{n} \vert _{p} ^{p-1} \vert \bar{u}_{n}-u \vert _{p}. $$

Let $\varepsilon \rightarrow 0$, we have

$$ \biggl\vert \int _{\mathbb{R}^{N}}f_{1}(\bar{u}_{n}) ( \bar{u}_{n}-u)\,dx \biggr\vert \rightarrow 0. $$

Moreover,

$$ \biggl\vert \int _{\mathbb{R}^{N}}b_{n}(x) \vert \bar{u}_{n} \vert ^{q-2}\bar{u}_{n} \vert \bar{v} _{n} \vert ^{q}(\bar{u}_{n}-u)\,dx \biggr\vert \leq \vert b_{n} \vert _{\infty } \vert \bar{u}_{n} \vert _{2q} ^{q-1} \vert \bar{v}_{n} \vert _{2q}^{q} \vert \bar{u}_{n}-u \vert _{2q}\rightarrow 0. $$

It follows from (4.10) that $\|\bar{\omega }_{n}-\omega \| \rightarrow 0$. This completes the proof of (ii) of Theorem 1.1. □

5 Ground states of a Schrödinger system with non-periodic couplings

We prove (i) and (ii) of Theorem 1.2 in Sects. 5.1–5.2, respectively. We denote that $\mathcal{J}_{\mathrm{per}}$ is the corresponding functional of (1.1) with $b(x)=b_{ \mathrm{per}}(x)$. $\mathcal{N}_{\mathrm{per}}$ and $c_{\mathrm{per}}$ are well defined in a similar way.

5.1 Existence

We need the following lemma.

Lemma 5.1

Assume that$b_{\mathrm{loc}}(x)\geq 0$for a.e. $x\in \mathbb{R}^{N}$and$b_{\mathrm{loc}}(x)>0$on a positive measure set, then$c< c_{\mathrm{per}}$.

Proof

From (i) of Theorem 1.1, we find a critical point $\omega '$ of $\mathcal{J}_{\mathrm{per}}$, where

$$ \omega '=\bigl(u',v'\bigr) \textstyle\begin{cases} u'>0 \quad \text{and}\quad v'< 0 &\text{as } {-}\sqrt{a _{1}a_{2}}< \lambda < 0, \\ u'>0\quad \text{and}\quad v'>0& \text{as } 0< \lambda < \sqrt{a_{1}a_{2}}, \end{cases} $$

$\mathcal{J}_{\mathrm{per}}(u',v')= c_{\mathrm{per}}$ and $\mathcal{J}'_{\mathrm{per}}(u',v')= 0$. We get that $\int b_{ \mathrm{loc}}(x)|u'|^{q}|v'|^{q}\,dx>0$. Let $t>0$ be such that $t(u',v')\in \mathcal{N}$, then

$$ \begin{aligned}[b] c&\leq \mathcal{J} \bigl(tu',tv'\bigr)=\mathcal{J}_{\mathrm{per}} \bigl(tu',tv'\bigr)-\frac{t ^{2q}}{q} \int b_{\mathrm{loc}}(x) \bigl\vert u' \bigr\vert ^{q} \bigl\vert v' \bigr\vert ^{q}\,dx \\ &< \mathcal{J}_{\mathrm{per}}\bigl(tu',tv'\bigr) \leq \mathcal{J}_{\mathrm{per}}\bigl(u',v'\bigr)=c _{\mathrm{per}}. \end{aligned} $$

(5.1)

 □

Proof of (i) of Theorem 1.2

We divide the proof into three steps. Step 1 and Step 3 are similar with those in Sect. 4.1, we omit them here. By similar arguments as Step 1 in Sect. 4.1, we find $(u_{n},v_{n})\rightharpoonup (u_{0},v_{0})$ in E and $\mathcal{J}'(u_{0},v_{0})=0$.

Step 2: We check whether$(u_{0},v_{0})\neq (0,0)$.

Similarly, from Step 2 in Sect. 4.1, there exists $z_{n}\in \mathbb{Z}^{N}$ such that $B(y_{n},1)\subset B(z_{n},1+ \sqrt{N})$ and

$$ \liminf_{n\rightarrow \infty } \int _{B(z_{n},1+\sqrt{N})} \vert u_{n} \vert ^{2}\,dx\geq \liminf_{n\rightarrow \infty } \int _{B(y_{n},1)} \vert u_{n} \vert ^{2}\,dx>\delta . $$

(5.2)

We claim that $(z_{n})$ is bounded, and hence $u_{0}\neq 0$, $(u_{0},v_{0})\in \mathcal{N}$ and $\mathcal{J}(u_{0},v_{0})\geq c$.

We check the claim. Suppose that $(z_{n})$ is unbounded, then we can choose a subsequence of $(z_{n})$ such that $|z_{n}|\rightarrow \infty$ as $n\rightarrow \infty$. Let $\bar{u}_{n}:=u_{n}(\cdot +z _{n})$, $\bar{v}_{n}:=v_{n}(\cdot +z_{n})$, up to a subsequence, then

$$\begin{aligned}& (\bar{u}_{n},\bar{v}_{n}) \rightharpoonup (\bar{u},\bar{v})\quad \text{in }E, \\& (\bar{u}_{n},\bar{v}_{n})\rightarrow (\bar{u},\bar{v}) \quad \text{in }L _{\mathrm{loc}}^{t}\bigl(\mathbb{R}^{N} \bigr)\times L_{\mathrm{loc}}^{t}\bigl( \mathbb{R}^{N} \bigr) \text{ for }1\leq t< 2^{*}, \\& (\bar{u}_{n},\bar{v}_{n})\rightarrow (\bar{u}, \bar{v})\quad \text{for a.e. }x \in \mathbb{R}^{N}. \end{aligned}$$

We deduce that

$$ \liminf_{n\rightarrow \infty } \int _{B(0,1+ \sqrt{N})} \vert \bar{u}_{n} \vert ^{2}\,dx>\delta $$

by (5.2). We find that $\bar{u}\neq 0$ by $\bar{u}_{n}\rightarrow \bar{u}$ in $L_{\mathrm{loc}}^{2}(\mathbb{R}^{N})$, thus $\bar{\omega }=(\bar{u},\bar{v})\neq (0,0)$. For any $\phi =(\varphi ,\psi )$, $\varphi , \psi \in C_{0}^{\infty }(\mathbb{R}^{N})$, we have

$$ \begin{aligned}[b] 0&\leftarrow \mathcal{J}'(\omega _{n})\phi (\cdot -z_{n}) \\ &=\mathcal{J} _{\mathrm{per}}'(\bar{\omega }_{n})\phi - \int _{\mathbb{R}^{N}}b_{ \mathrm{loc}}(x+z_{n}) \vert \bar{u}_{n} \vert ^{q-2}\bar{u}_{n} \vert \bar{v}_{n} \vert ^{q} \varphi \,dx \\ &\quad {}- \int _{\mathbb{R}^{N}}b_{\mathrm{loc}}(x+z_{n}) \vert \bar{u} _{n} \vert ^{q} \vert \bar{v}_{n} \vert ^{q-2}\bar{v}_{n}\psi \,dx. \end{aligned} $$

(5.3)

Let $K\subset \mathbb{R}^{N}$ be a compact set containing supports of φ, ψ, then

$$ \begin{aligned}[b] &\int _{K}b_{\mathrm{loc}}(x+z_{n}) \vert \bar{u}_{n} \vert ^{q-2}\bar{u}_{n} \vert \bar{v}_{n} \vert ^{q}\varphi \,dx \\ &\quad \leq \vert \bar{u}_{n} \vert _{p}^{q-1} \vert \bar{v}_{n} \vert _{p}^{q} \vert \varphi \vert _{p}\biggl( \int _{K} \bigl\vert b_{\mathrm{loc}}(x+z_{n}) \bigr\vert ^{\frac{p}{p-2q}}\,dx\biggr)^{ \frac{p-2q}{p}}\rightarrow 0 \end{aligned} $$

(5.4)

as $|z_{n}|\rightarrow \infty$. Combining (5.3) and (5.4), we get $\mathcal{J}'_{\mathrm{per}}(\bar{\omega }_{n}) \phi \rightarrow 0$. It follows from Lemma 3.2 that $\mathcal{J}'_{\mathrm{per}}(\bar{\omega }_{n})\phi \rightarrow \mathcal{J}'_{\mathrm{per}}(\bar{\omega })\phi$ and $\mathcal{J}'_{ \mathrm{per}}(\bar{\omega })=0$. Hence $(\bar{u},\bar{v})\in \mathcal{N}_{\mathrm{per}}$ and $\mathcal{J}_{\mathrm{per}}(\bar{u}, \bar{v})\geq c_{\mathrm{per}}$. It is clear that

$$ \begin{aligned} a_{1}\bar{u}_{n}^{2}+a_{2} \bar{v}_{n}^{2}-2\lambda \bar{u}_{n} \bar{v}_{n}&\geq a_{1}\bar{u}_{n}^{2}+a_{2} \bar{v}_{n}^{2}-2 \vert \lambda \vert \vert \bar{u}_{n} \vert \vert \bar{v}_{n} \vert \\ &\geq a_{1}\bar{u}_{n}^{2}+a_{2} \bar{v}_{n}^{2}-\frac{ \vert \lambda \vert }{\sqrt{a_{1}a_{2}}} \bigl(a_{1}\bar{u}_{n}^{2}+a_{2} \bar{v}_{n} ^{2}\bigr) \\ &=\biggl(1-\frac{ \vert \lambda \vert }{\sqrt{a_{1}a_{2}}}\biggr) \bigl(a_{1}\bar{u} _{n}^{2}+a_{2}\bar{v}_{n}^{2} \bigr)\geq 0. \end{aligned} $$

Applying Fatou’s lemma, we find

$$ \begin{aligned}[b] c&=\liminf_{n\rightarrow \infty } \mathcal{J}(\omega _{n}) \\ &=\liminf_{n\rightarrow \infty }\biggl[ \biggl(\frac{1}{2}- \frac{1}{2q}\biggr) \biggl([u_{n}]_{s}^{2}+[v_{n}]_{s}^{2}+ \int a_{1} \vert u_{n} \vert ^{2}+a _{2} \vert v_{n} \vert ^{2} -2\lambda u_{n}v_{n}\,dx\biggr) \\ &\quad {}+ \int \frac{1}{2q}f_{1}(u_{n})u _{n}-F_{1}(u_{n})+\frac{1}{2q}f_{2}(v_{n})v_{n}-F_{2}(v_{n}) \,dx\biggr] \\ &=\liminf_{n\rightarrow \infty }\biggl[\biggl( \frac{1}{2}- \frac{1}{2q}\biggr) \biggl([\bar{u}_{n}]_{s}^{2}+[ \bar{v}_{n}]_{s}^{2}+ \int a_{1} \vert \bar{u}_{n} \vert ^{2}+a_{2} \vert \bar{v}_{n} \vert ^{2}-2\lambda \bar{u} _{n}\bar{v}_{n}\,dx \biggr) \\ &\quad {}+ \int \frac{1}{2q}f_{1}(\bar{u}_{n}) \bar{u}_{n}-F_{1}( \bar{u}_{n})+ \frac{1}{2q}f_{2}(\bar{v}_{n}) \bar{v}_{n}-F_{2}(\bar{v} _{n})\,dx\biggr] \\ &\geq \biggl(\frac{1}{2}-\frac{1}{2q}\biggr) \biggl([ \bar{u}]_{s}^{2}+[ \bar{v}]_{s}^{2}+ \int a_{1} \vert \bar{u} \vert ^{2}+a_{2} \vert \bar{v} \vert ^{2}-2\lambda \bar{u}\bar{v}\,dx\biggr) \\ &\quad {}+ \int \frac{1}{2q}f_{1}(\bar{u}) \bar{u}-F_{1}(\bar{u})+ \frac{1}{2q}f_{2}( \bar{v})\bar{v}-F_{2}(\bar{v})\,dx \\ &=\mathcal{J}_{\mathrm{per}}(\bar{u},\bar{v})\geq c_{ \mathrm{per}}. \end{aligned} $$

(5.5)

We get a contradiction with Lemma 5.1. Hence $(z_{n})$ is bounded. □

5.2 Nonexistence

Suppose by contradiction that there exists a ground state solution of (1.1), i.e. $\omega _{0}=(u_{0},v_{0})\neq (0,0)$ such that $\mathcal{J}(u_{0},v_{0})=c$ and $\mathcal{J}'(u_{0},v_{0})=0$. By using similar arguments as Step 3 in Sect. 4.1, we find a critical point of $\mathcal{J}$, where

$$ \omega '=\bigl(u',v'\bigr) \textstyle\begin{cases} u'>0 \quad \text{and}\quad v'< 0& \text{as } {-}\sqrt{a _{1}a_{2}}< \lambda < 0, \\ u'>0 \quad \text{and}\quad v'>0 &\text{as } 0< \lambda < \sqrt{a_{1}a_{2}}, \end{cases} $$

$\mathcal{J}(u',v')=c$ and $\mathcal{J}'(u',v')=0$.

Lemma 5.2

Assume that$b_{\mathrm{loc}}(x)\leq 0$for a.e. $x\in \mathbb{R}^{N}$and$b_{\mathrm{loc}}(x)<0$on a positive measure set, then$c>c_{\mathrm{per}}$.

Proof

It is clear that $\int b_{\mathrm{loc}}(x)|u'|^{q}|v'|^{q}\,dx<0$. Let $t>0$ be such that $t(u',v')\in \mathcal{N}_{\mathrm{per}}$, then

$$ \begin{aligned}[b] c_{\mathrm{per}}&\leq \mathcal{J}_{\mathrm{per}}\bigl(tu',tv'\bigr)= \mathcal{J}\bigl(tu',tv'\bigr)+\frac{t ^{2q}}{q} \int b_{\mathrm{loc}}(x) \bigl\vert u' \bigr\vert ^{q} \bigl\vert v' \bigr\vert ^{q}\,dx \\ &< \mathcal{J}\bigl(tu',tv'\bigr)\leq \mathcal{J} \bigl(u',v'\bigr)=c. \end{aligned} $$

(5.6)

 □

Let $\omega \in \mathcal{N}_{\mathrm{per}}$ be a ground state solution of $\mathcal{J}_{\mathrm{per}}$, i.e. $\mathcal{J}_{ \mathrm{per}}(u,v)=c_{\mathrm{per}}$ and $\mathcal{J}'_{\mathrm{per}}(u,v)=0$. Denote that $\bar{\omega }:= \omega (\cdot -y)$ for $y\in \mathbb{Z}^{N}$, we find that $\bar{ \omega }\in \mathcal{N}_{\mathrm{per}}$. There exists $t>0$ such that $t\bar{\omega }\in \mathcal{N}$. For any $y\in \mathbb{Z}^{N}$, we have

$$ \begin{aligned}[b] c_{\mathrm{per}}&= \mathcal{J}_{\mathrm{per}}(\omega )=\mathcal{J}_{ \mathrm{per}}(\bar{\omega }) \geq \mathcal{J}_{\mathrm{per}}(t\bar{ \omega })=\mathcal{J}(t\bar{\omega }) + \frac{t^{2q}}{q} \int _{\mathbb{R}^{N}}b_{\mathrm{loc}}(x) \vert \bar{u} \vert ^{q} \vert \bar{v} \vert ^{q}\,dx \\ &\geq c+\frac{t^{2q}}{q} \int _{\mathbb{R}^{N}}b_{ \mathrm{loc}}(x) \vert \bar{u} \vert ^{q} \vert \bar{v} \vert ^{q}\,dx. \end{aligned} $$

(5.7)

Obviously, $\int _{\mathbb{R}^{N}}b_{\mathrm{loc}}(x)t^{2q}|\bar{u}|^{q}| \bar{v}|^{q}\,dx=\int _{\mathbb{R}^{N}}b_{\mathrm{loc}}(x+y)t^{2q}|u|^{q}|v|^{q}\,dx$. Since $\mathcal{J}_{\mathrm{per}}$ is coercive on $\mathcal{N}_{ \mathrm{per}}$ and $\mathcal{J}_{\mathrm{per}}(t\omega )=\mathcal{J} _{\mathrm{per}}(t\bar{\omega })\leq c_{\mathrm{per}}$, we find that tω is bounded in E. Furthermore, u, v are bounded in $L^{2q}(\mathbb{R}^{N})$ and $L^{p}(\mathbb{R}^{N})$ by embedding theorem. For any $0<\varepsilon <1$, we choose $R(\varepsilon )>0$ such that $\int _{B_{R}^{c}}|u|^{2q}\,dx<\varepsilon ^{2}$, and choose $y(\varepsilon )>0$ such that $\int _{B_{R}}|b_{\mathrm{loc}}(x+y)|^{ \frac{p}{p-2q}}\,dx<\varepsilon ^{\frac{p}{p-2q}}$, then there exist $C_{1},C_{2},C_{3}>0$ such that

$$ \begin{aligned}[b] &\int _{\mathbb{R}^{N}}b_{\mathrm{loc}}(x+y)t^{2q} \vert u \vert ^{q} \vert v \vert ^{q}\,dx \\ &\quad \leq \biggl\vert \int _{B_{R}}b_{\mathrm{loc}}(x+y)t^{2q} \vert u \vert ^{q} \vert v \vert ^{q}\,dx \biggr\vert + \biggl\vert \int _{B_{R}^{c}}b_{\mathrm{loc}}(x+y)t^{2q} \vert u \vert ^{q} \vert v \vert ^{q}\,dx \biggr\vert \\ &\quad \leq t^{2q} \vert u \vert ^{q}_{p} \vert v \vert ^{q}_{p}\biggl( \int _{B_{R}} \bigl\vert b_{ \mathrm{loc}}(x+y) \bigr\vert ^{\frac{p}{p-2q}}\,dx\biggr)^{\frac{p-2q}{p}} +t^{2q} \vert b _{\mathrm{loc}} \vert _{\infty } \vert v \vert _{2q}^{q}\biggl( \int _{B_{R}^{c}} \vert u \vert ^{2q}\,dx \biggr)^{ \frac{1}{2}} \\ &\quad \leq C_{1}\varepsilon +C_{2}\varepsilon \leq C_{3}\varepsilon , \end{aligned} $$

(5.8)

where ε is arbitrary. In view of (5.7), let $|y|$ be sufficiently large, we get $c_{\mathrm{per}}\geq c$. It is contradictory to Lemma 5.2. This completes the proof of (ii) of Theorem 1.2.