Nontrivial solutions for Klein–Gordon–Maxwell systems with sign-changing potentials

Abstract

This paper is concerned with the nonlinear Klein–Gordon–Maxwell systems. Unlike all known results in the literature, the Schrödinger operator $-\Delta +V$ is allowed to be indefinite and the weaker superlinear conditions are imposed instead of the common 4-superlinear conditions on f. By combining a local linking argument and Morse theory, we obtain that the system admits a nontrivial solution.

1 Introduction

In this paper, we investigate the existence of nontrivial solution to the following Klein–Gordon–Maxwell systems with sign-changing potentials:

$$ \textstyle\begin{cases} -\Delta u+V(x)u-(2\omega +\phi )\phi u=f(x,u), &x\in \mathbb{R}^{3}, \\ -\Delta \phi =(\omega +\phi )u^{2}, &x\in \mathbb{R}^{3}, \end{cases} $$

(1.1)

where ω is a positive constant, $f\in C(\mathbb{R}^{3}\times \mathbb{R})$ is a nonlinearity, $V\in C(\mathbb{R}^{3})$ is a potential, and V changes sign.

It is known to all that the Klein–Gordon–Maxwell system was proposed by Benci and Fortunato [1], they considered the following electrostatic nonlinear Klein–Gordon–Maxwell system:

$$ \textstyle\begin{cases} -\Delta u+ (m_{0}-(\omega +\phi ^{2}) )u= \vert u \vert ^{p-2}u, &x\in \mathbb{R}^{3}, \\ -\Delta \phi =(\omega +\phi )u^{2}, &x\in \mathbb{R}^{3}, \end{cases} $$

(1.2)

where $m_{0}$, ω are real positive constants, and obtained infinitely many solitary wave solutions of (1.2) when $m_{0}>\omega$, $p\in (4,6)$. Such a system is a model describing solitary waves for the nonlinear stationary Klein–Gordon equation in the three-dimensional space interacting with the electromagnetic field; see [2, 3].

System (1.2) was introduced by D’Aprile and Mugnai in [4], where the authors proved there just the existence of a trivial solution for $p\in (0,2]$ or $p\geq 6$ by applying a Pohozaev-type argument. In [5], system (1.2) was shown to admit infinitely many radial solutions for any $0<\omega <m_{0}\sqrt{\frac{p-2}{2}}$ and $p\in (2,4)$. Azzollini and Pomponio in [6] investigated the existence range of $(m_{0},\omega )$ for $p\in (2,4)$ by using the Nehari method. Via the minimization method of [6] and the indirect method developed by Struwe [7] and Jeanjean [8], a weaker condition on (1.2) was proposed by Azzollini, Pisani, and Pomponio in [9], they established the nontrivial radial solution of (1.2). Recently, the results in [9] were improved by Chen and Tang [10].

For the system (1.1) with nonconstant potential V, it seems that the first result is due to He [11], where he assumed that the potential V is positive and coercive, and showed the existence of high-energy solutions by a variant fountain theorem and symmetric mountain pass theorem. Furthermore, in [12] Li and Tang improved and complemented the results in [11]. In [13], Cunha proved that system (1.1) with $f(x,u)=f(u)$ has a ground state solution, assuming that the potential V is positive and periodic, the nonlinearity f satisfies 4-superlinear and the weaker Ambrosetti–Rabinowitz (A-R) condition.

When V is sign-changing, Ding and Li [14] proved that system (1.1) has infinitely many solutions under a variant 4-superlinear condition. Later, improving the results in [11, 12, 14] by some new trick and symmetric mountain pass theorem, Chen and Tang [15] obtained infinitely many solutions under a very general nonlinear term. As we know, when V is sign-changing, the existing literature only deals with the existence of infinitely many solutions of system (1.1). In this case, it is necessary that $f(x,t)$ is odd as a function of t.

The main object of the present paper is to complement the results obtained in [11–15]. We emphasize that in all these papers the authors only considered the case where the energy functional of system (1.1) is even. In [11–15], one may apply the symmetric mountain pass theorem (see [16]) which can be applied to the Schrödinger operator $-\Delta +V$ when it is not positive definite. But, without the symmetry hypothesis on the energy functional, the approaches used in [11–15] are no longer applicable for (1.1) when the potential V is sign-changing. Besides, by all accounts, the mountain pass theorem is no longer suited to this situation. We attempt to take advantage of the linking theorem to solve the semilinear problems. Unfortunately, as pointed out in [17], the energy functional of Klein–Gordon–Maxwell system (1.1) would not enjoy the general linking geometry due to the nonlocal term; see [17] for details.

Motivated by the above works, the question about the existence of solution for (1.1) with a sign-changing potential has not been studied in the past and our paper is a first attempt to prove the existence of a solution without the symmetry hypothesis.

We assume that V and f enjoy the following hypotheses:

$(V_{1})$ $V\in C(\mathbb{R}^{3},\mathbb{R})$, $\inf_{x\in \mathbb{R}^{3}}V(x)>-\infty$ and $\mu (V^{-1}(-\infty ,M])<\infty$ for all $M>0$, where μ is the Lebesgue measure on $\mathbb{R}^{3}$.

$(F_{1})$ $f\in C(\mathbb{R}^{3}\times \mathbb{R},\mathbb{R})$ and there exist constants $C_{0}>0$ and $p\in (2,6)$ such that

$$ \bigl\vert f(x,t) \bigr\vert \leq C_{0}\bigl(1+ \vert t \vert ^{p-1}\bigr),\quad \text{for all } (x,t)\in \mathbb{R}^{3} \times \mathbb{R}; $$

$(F_{2})$ $f(x,t)=o(|t|)$ as $t\to 0$, uniformly in $x\in \mathbb{R}^{3}$;

$(F_{3})$ $\lim_{|t|\to \infty}\frac{F(x,t)}{|t|^{2}}=+\infty$ uniformly in $x\in \mathbb{R}^{3}$, and there exists $r_{0}>0$ such that $F(x,t)\geq 0$, for all $x\in \mathbb{R}^{3}$, $|t|\geq r_{0}$;

$(F_{4})$ there exists $\mu >2$ and $\theta >0$ such that $f(x,t)t-\mu F(x,t)+\theta t^{2}\geq 0$, for all $(x,t)\in \mathbb{R}^{3}\times \mathbb{R}$.

What is new is the potential V is allowed to be sign-changing. To overcome this difficulty, the authors need to give some modifications and definitions as follows.

Under hypotheses $(V_{1})$, $(F_{1})$, $(F_{3})$, and $(F_{4})$, we can choose $\alpha >0$ such that

$$\begin{aligned}& \widetilde{V}(x):=V(x)+\alpha \geq 1,\quad \text{for all } x\in \mathbb{R}^{3}, \end{aligned}$$

(1.3)

$$\begin{aligned}& f(x,t)t+\alpha t^{2}\geq 0, \quad \text{for all } (x,t)\in \mathbb{R}^{3} \times \mathbb{R}, \end{aligned}$$

(1.4)

and

$$ 2F(x,t)+\alpha t^{2}\geq 0,\quad \text{for all } (x,t)\in \mathbb{R}^{3} \times \mathbb{R}. $$

(1.5)

In our problem, the work space E is defined by

$$ E:= \biggl\{ u\in H^{1}\bigl(\mathbb{R}^{3}\bigr): \int _{\mathbb{R}^{3}} \bigl( \vert \nabla u \vert ^{2}+V(x)u^{2} \bigr)\,\text{d}x< +\infty \biggr\} . $$

Thus, E is a Hilbert space with its norm is $\|u\|= (\int _{\mathbb{R}^{3}} (|\nabla u|^{2}+ \widetilde{V}(x)u^{2} )\,\text{d}x )^{1/2}$.

If $(V_{1})$ holds, by the compact embedding theorem established by Bartsch–Wang [18], the embedding $E\hookrightarrow L^{s}(\mathbb{R}^{3})$ is compact for any $s\in [2,6)$. Applying the spectral theory of self-adjoint compact operators, denote $-\infty <\lambda _{1}\leq \lambda _{2}\leq \cdots \leq \lambda _{n} \leq \cdots$ as the complete sequence of eigenvalues of

$$ -\Delta u +V(x)u=\lambda u,\quad u\in E. $$

(1.6)

Each eigenvalue is repeated according to its multiplicity, and let $e_{1},e_{2},\ldots$ be the corresponding orthonormal eigenfunctions in $L^{2}(\mathbb{R}^{3})$.

Inspired by [15, 19, 20], by combining a local linking argument of Li and Willem [21] and infinite-dimensional Morse theorem [22], we will obtain our main result as follows:

Theorem 1.1

Assume $(V_{1})$, $(F_{1})$–$(F_{4})$ hold. If 0 is not an eigenvalue of (1.6), then problem (1.1) possesses a nontrivial solution.

Remark 1.1

It is interesting to note that unlike many works in Klein–Gordon–Maxwell systems, in our paper, the potential V is allowed to be sign-changing. The method we used in this paper also works for other nonlocal problems, which include the Schrödinger–Kirchhoff equation [23] and the Schrödinger–Poisson system [24].

Notation

Throughout this paper, we assume that the usual norm in $L^{s}$-space is $\|u\|_{s}= (\int _{\mathbb{R}^{3}}|u|^{s}\,\textit{d}x )^{1/s}$; $\|u\|_{D^{1,2}}= (\int _{\mathbb{R}^{3}}|\nabla u|^{2}\,\textit{d}x )^{1/2}$ denotes the usual norm in $D^{1,2}$-space; $C, C_{1}, C_{2},\dots$ denote different positive constants.

2 Proof of Theorem 1.1

System (1.1) has a variational structure, its weak solutions $(u,\phi )\in E\times D^{1,2}(\mathbb{R}^{3})$ are critical points of the functional given by

$$ I(u,\phi )=\frac{1}{2} \int _{\mathbb{R}^{3}} \bigl( \vert \nabla u \vert ^{2}+V(x)u^{2}- \vert \nabla \phi \vert ^{2}-(2\omega +\phi )\phi u^{2} \bigr)\,\text{d}x- \int _{ \mathbb{R}^{3}}F(x,u)\,\text{d}x. $$

(2.1)

But $I(u,\phi )$ may be strongly ill-behaved in E, i.e., unbounded from below and above on infinite-dimensional spaces. The next step is the usual reduction argument to obtain a single variable functional.

Lemma 2.1

([11])

For any fixed $u\in H^{1}(\mathbb{R}^{3})$, there exists a unique $\phi =\phi _{u}\in D^{1,2}(\mathbb{R}^{3})$ which solves equation

$$ -\Delta \phi +u^{2}\phi =-\omega u^{2}. $$

Moreover, the map $\Phi :u\in H^{1}(\mathbb{R}^{3})\longmapsto \Phi [u]:=\phi _{u}\in D^{1,2}( \mathbb{R}^{3})$ is continuously differentiable, and

(1) $-\omega \leq \phi _{u}\leq 0$ on the set $\{x:u(x)\neq 0\}$;

(2) $\|\phi _{u}\|_{D^{1,2}}\leq C_{1}\|u\|^{2}$ and $\int _{\mathbb{R}^{3}}|\phi _{u}|u^{2}\,\textit{d}x\leq C_{2}\|u\|_{ \frac{12}{5}}^{4}\leq C_{3}\|u\|^{4}$.

Lemma 2.2

([6])

If $u_{n}\rightharpoonup u$ in $H^{1}(\mathbb{R}^{3})$, then, up to subsequences, $\phi _{u_{n}}\rightharpoonup \phi _{u}$ in $D^{1,2}(\mathbb{R}^{3})$. Moreover, $\Phi '(u_{n})\to \Phi '(u)$ in the sense of distributions, where Φ is defined in Lemma 2.1.

Using Lemma 2.1, the functional $J(u)=I(u,\phi _{u})$ has the form

$$ J(u)=\frac{1}{2} \int _{\mathbb{R}^{3}} \bigl( \vert \nabla u \vert ^{2}+V(x)u^{2} \bigr)\,\text{d}x- \int _{\mathbb{R}^{3}} \biggl(\frac{1}{2}\omega \phi _{u} u^{2}+F(x,u) \biggr)\,\text{d}x. $$

(2.2)

In view of [11], under $(V_{1})$ and $(F_{1})$, we have $J(u)\in C^{1}(E,\mathbb{R})$ and we will look for its critical points. It is well known that if u is a critical point for $J(u)$ with $\phi =\phi _{u}$, then $(u,\phi )\in E\times D^{1,2}(\mathbb{R}^{3})$ is a critical point for $I(u,\phi )$.

We are ready to prove Theorem 1.1. Since 0 is not an eigenvalue of (1.6), we can assume that there exists an integer $d\geq 0$ such that $0\in (\lambda _{d},\lambda _{d+1})$ where we set $\lambda _{0}=-\infty$. For $d\geq 1$, we denote $E^{-}=\operatorname{span}\{e_{1},\dots ,e_{d}\} \text{and} E^{+}=(E^{-})^{ \perp}$. In particular, if $d=0$, we set $E^{-}=\{0\}$ and $E^{+}=E$. Then $E^{-}$ and $E^{+}$ are the negative and positive spaces of the quadratic form $B(u)=\frac{1}{2}\int _{\mathbb{R}^{3}} (|\nabla u|^{2}+V(x)u^{2} )\,\text{d}x$, respectively. Furthermore, there exits a constant $\eta >0$ such that

$$ \pm B(u)\geq \eta \Vert u \Vert ^{2},\quad u\in E^{\pm}. $$

(2.3)

To find critical points of the functionals with indefinite quadratic part, a natural idea is to apply the linking theorem. Fortunately, as in [19] on Schrödinger–Poisson equations and [20] on quasilinear Schrödinger equations, we observe that our functional J has a local linking at 0. Hence, by employing infinite-dimensional Morse theory [22], we can prove the existence of one nontrivial solution for problem (1.1). Thus we recall some concepts and results about Morse theory and local linking theorem.

Let E be a real Banach space, J be a $C^{1}$-functional on E, and u an isolated critical point of J with $J(u)=c$. Define

$$ J^{c}:=\bigl\{ u\in E:J(u)\leq c\bigr\} . $$

Next we recall that the qth critical group of J at an isolated critical point u is defined by $C_{q}(J,u):=H_{q}(J^{c},J^{c}\setminus \{0\})$, $q\in \mathbb{N}=\{0,1,2, \dots \}$, where $H_{q}(\cdot ,\cdot )$ is called the singular homology with integer coefficients.

If J satisfies the (PS) condition or Cerami condition and $a<\inf_{u\in \mathcal{K}} J(u)$, where $\mathcal{K}$ is defined by $\mathcal{K}:=\{u\in E: J'(u)=0\}$, then the critical groups of J at infinity are defined by

$$ C_{q}(J,\infty ):=H_{q}\bigl(E,J^{a}\bigr). $$

Definition 2.1

Let $\{u_{n}\}\subset E$ be a Cerami sequence, that is,

$$ \bigl\{ J(u_{n})\bigr\} \quad \text{is bounded and } \bigl(1+ \Vert u_{n} \Vert \bigr)J'(u_{n}) \to 0. $$

(2.4)

Then J is said to satisfy the Cerami condition if any Cerami sequence has a convergent subsequence in E.

In Morse theory, we need to show that J satisfies the Cerami condition. Fortunately, by a standard argument as in [15], we easily prove the following lemma.

Lemma 2.3

Assume that $(V_{1})$, $(F_{1})$, $(F_{3})$, and $(F_{4})$ hold. Then J satisfies the Cerami condition.

Proof

Let $\{u_{n}\}\subset E$ be a Cerami sequence. First, we prove that $\{\|u_{n}\|\}$ is bounded. On the contrary suppose that $\|u_{n}\|\rightarrow \infty$. Consider $v_{n} = u_{n}/\|u_{n}\|$, then $\|v_{n}\|=1$. Passing to a subsequence, we know that $v_{n}\rightharpoonup v$ in E, then since the embedding $E\hookrightarrow L^{s}(\mathbb{R}^{3})$ is compact for any $s\in [2,6)$, we have $v_{n}\rightarrow v$ in $L^{s}(\mathbb{R}^{3})$ for $2\leq s < 6$, and $v_{n}\rightarrow v$ a.e. on $\mathbb{R}^{3}$. Noticing that $-\omega \leq \phi _{u_{n}}\leq 0$ and $(F_{4})$ holds, we have

$$ \begin{aligned} c+o(1)={}&J(u_{n})- \frac{1}{\mu}\bigl\langle J'(u_{n}),u_{n} \bigr\rangle \\ ={}& \biggl(\frac{1}{2}-\frac{1}{\mu} \biggr) \Vert u_{n} \Vert ^{2}+ \int _{ \mathbb{R}^{3}} \biggl[ \biggl(\frac{2}{\mu}-\frac{1}{2} \biggr)\omega \phi _{u_{n}}+\frac{1}{\mu}\phi ^{2}_{u_{n}}- \biggl(\frac{1}{2}- \frac{1}{\mu} \biggr)\alpha \biggr]u^{2}_{n} \,\text{d}x \\ & {}+ \int _{\mathbb{R}^{3}} \biggl[\frac{1}{\mu}f(x,u_{n})u_{n}-F(x,u_{n}) \biggr]\,\text{d}x \\ \geq{}& \biggl(\frac{1}{2}-\frac{1}{\mu} \biggr) \Vert u_{n} \Vert ^{2}- \int _{ \mathbb{R}^{3}} \biggl[\frac{2}{\mu}\omega ^{2} + \biggl(\frac{1}{2}- \frac{1}{\mu} \biggr)\alpha +\frac{\theta}{\mu} \biggr]u^{2}_{n} \,\text{d}x. \end{aligned} $$

(2.5)

Multiplying (2.5) by $1/\|u_{n}\|^{2}$, it follows from $v_{n}\rightarrow v$ in $L^{2}(\mathbb{R}^{3})$ that

$$ \begin{aligned} \biggl[\frac{2}{\mu}\omega ^{2}+ \biggl(\frac{1}{2}-\frac{1}{\mu} \biggr)\alpha + \frac{\theta}{\mu} \biggr] \Vert v \Vert ^{2}_{2}= \biggl[ \frac{2}{\mu}\omega ^{2}+ \biggl(\frac{1}{2}-\frac{1}{\mu} \biggr) \alpha +\frac{\theta}{\mu} \biggr]\lim_{n\rightarrow \infty} \Vert v_{n} \Vert ^{2}_{2}>0, \end{aligned} $$

(2.6)

where $\mu > 2$. Then one deduces that $v\neq 0$. For a.e. $x \in \mathbb{R}^{3}$ such that $v(x)\neq 0$, we have $\lim_{n\rightarrow \infty}|u_{n}(x)|=\infty$. Hence, it follows from $(F_{1})$, $(F_{3})$, (1.3), (1.5), (2.4), (2.6), and Fatou’s lemma that

$$ \begin{aligned} 0={}&\lim_{n\rightarrow \infty}\frac{c+o(1)}{ \Vert u_{n} \Vert ^{2}}=\lim _{n \rightarrow \infty}\frac{J(u_{n})}{ \Vert u_{n} \Vert ^{2}} \\ \leq{}& \frac{1}{2}-\liminf_{n\rightarrow \infty} \frac{\omega}{2 \Vert u_{n} \Vert ^{2}} \int _{R^{3}}\phi _{u_{n}}u^{2}_{n} \,\text{d}x \\ &{}- \liminf_{n\rightarrow \infty} \int _{R^{3}} \frac{2F(x,u_{n})+\alpha \vert u_{n} \vert ^{2}}{2 \vert u_{n} \vert ^{2}} \vert v_{n} \vert ^{2} \,\text{d}x \\ \leq{}& \frac{1}{2}+\liminf_{n\rightarrow \infty} \frac{\omega ^{2} \Vert u_{n} \Vert ^{2}_{2}}{2 \Vert u_{n} \Vert ^{2}}- \int _{R^{3}} \liminf_{n\rightarrow \infty} \frac{2F(x,u_{n})+\alpha \vert u_{n} \vert ^{2}}{2 \vert u_{n} \vert ^{2}} \vert v_{n} \vert ^{2} \,\text{d}x=-\infty , \end{aligned} $$

(2.7)

which is a contradiction. Hence, ${u_{n}}$ is bounded in E. And then, we have $u_{n} \rightharpoonup \overline{u}$ in E, $u_{n} \rightarrow \overline{u}$ in $L^{s}(\mathbb{R}^{3})$ for $2\leq s < 6$ and $u_{n} \rightarrow \overline{u}$ a.e. on $\mathbb{R}^{3}$. With the help of Lemma 2.2, a standard argument (see [14], Lemma 3.1) shows that $u_{n}\rightarrow \overline{u}$ in E up to a subsequence. □

2.1 Critical groups at zero

Proposition 2.1

([21])

Suppose $J\in C^{1}(E,\mathbb{R})$ has a local linking at zero with respect to the decomposition $E=E^{-}\oplus E^{+}$, i.e., for some $\varepsilon >0$,

$$ \begin{aligned} J(u)\leq 0\quad \textit{for } u\in E^{-}\cap B_{\varepsilon}, \quad \textit{and}\quad J(u)>0\quad \textit{for } u\in \bigl(E^{+} \setminus \{0\}\bigr)\cap B_{\varepsilon}, \end{aligned} $$

(2.8)

where $B_{\varepsilon}=\{u\in E: \|u\|<\varepsilon \}$. If $d=\dim E^{-}<\infty$, then $C_{d}(J,0)\neq 0$.

Lemma 2.4

Under assumptions $(V_{1})$, $(F_{1})$, and $(F_{2})$, the functional J has a local linking at zero with respect to decomposition $E=E^{-}\oplus E^{+}$, where $E^{-}$, $E^{+}$ are as in Sect. 2 and $d=\dim E^{-}$.

Proof

From $(F_{1})$ and $(F_{2})$, for all $\varepsilon >0$, there exists $C_{\varepsilon}>0$ such that

$$ \bigl\vert F(x,t) \bigr\vert \leq \varepsilon t^{2}+C_{\varepsilon} \vert t \vert ^{p}. $$

(2.9)

Hence, we get

$$ \biggl\vert \int _{\mathbb{R}^{3}}F(x,u)\,\text{d}x \biggr\vert \leq o\bigl( \Vert u \Vert ^{2}\bigr) \quad \text{as } \Vert u \Vert \to 0. $$

Using this and Lemma 2.1, as $\|u\|\to 0$, we obtain

$$ \begin{aligned} J(u)&=\frac{1}{2} \int _{\mathbb{R}^{3}} \bigl( \vert \nabla u \vert ^{2}+V(x)u^{2} \bigr)\,\text{d}x- \int _{\mathbb{R}^{3}} \biggl(\frac{1}{2}\omega \phi _{u} u^{2}+F(x,u) \biggr)\,\text{d}x \\ &=\frac{1}{2} \int _{\mathbb{R}^{3}} \bigl( \vert \nabla u \vert ^{2}+V(x)u^{2} \bigr)\,\text{d}x+o\bigl( \Vert u \Vert ^{2}\bigr). \end{aligned} $$

(2.10)

From this and (2.3), this implies that J has a local linking property at zero. □

2.2 Critical groups at infinity

Lemma 2.5

Let f satisfy $(F_{1})$, $(F_{3})$, and $(F_{4})$. Then $C_{q}(J,\infty )\cong 0$, for any $q\in \mathbb{N}=\{0,1,2,\dots \}$.

Proof

Let $S^{\infty}$ be the unit sphere in E. First, we need prove that

$$ J(sh)\to -\infty ,\quad \text{as } s\to +\infty , \text{for any } h\in S^{ \infty}. $$

(2.11)

Due to $(F_{3})$, Lemma 2.1(1), and (1.5), we deduce

$$\begin{aligned} J(sh) =&\frac{1}{2} \int _{\mathbb{R}^{3}} \bigl( \bigl\vert \nabla (sh) \bigr\vert ^{2}+ \widetilde{V}(x) (sh)^{2} \bigr)\,\text{d}x- \frac{1}{2} \int _{\mathbb{R}^{3}} \omega \phi _{sh}(sh)^{2} \,\text{d}x \\ &{} -\frac{1}{2} \int _{\mathbb{R}^{3}} \bigl(2F(x,sh)+\alpha (sh)^{2} \bigr) \,\text{d}x \\ \leq& s^{2} \biggl(\frac{1}{2}+\frac{1}{2}\omega ^{2} \int _{\mathbb{R}^{3}}h^{2} \,\text{d}x-\frac{1}{2s^{2}} \int _{\mathbb{R}^{3}} \bigl(2F(x,sh)+ \alpha (sh)^{2} \bigr) \,\text{d}x \biggr) \\ \to& -\infty , \quad \text{as } s\to +\infty . \end{aligned}$$

In what follows, we will prove the following claim.

Claim

There exists $A>0$ such that if $J(u)\leq -A$, then

$$ \frac{\text{d}}{\text{d}t}\bigg|_{t=1}J(tu)< 0. $$

If the claim is not true, there exists a sequence $\{u_{n}\}\subset E$ such that

$$ J(u_{n})\leq -n \quad \text{and}\quad \bigl\langle J'(u_{n}),u_{n}\bigr\rangle = \frac{\text{d}}{\text{d}t}\bigg|_{t=1}J(tu_{n})\geq 0. $$

(2.12)

From (2.12), $(F_{4})$, and the fact that $-\omega \leq \phi _{u_{n}}\leq 0$, we deduce

$$ \begin{aligned} 0\geq{}& J(u_{n})- \frac{1}{\mu}\bigl\langle J'(u_{n}),u_{n} \bigr\rangle \\ ={}& \biggl(\frac{1}{2}-\frac{1}{\mu} \biggr) \int _{\mathbb{R}^{3}} \bigl( \vert \nabla u_{n} \vert ^{2}+\widetilde{V}(x) \vert u_{n} \vert ^{2} \bigr)\,\text{d}x \\ &{} + \int _{\mathbb{R}^{3}} \biggl( \biggl(\frac{2}{\mu}-\frac{1}{2} \biggr)\omega \phi _{u_{n}}+\frac{1}{\mu}\phi _{u_{n}}^{2}- \biggl( \frac{1}{2}-\frac{1}{\mu} \biggr)\alpha \biggr)u_{n}^{2} \,\text{d}x \\ &{} + \int _{\mathbb{R}^{3}} \biggl(\frac{1}{\mu}f(x,u_{n})u_{n}-F(x,u_{n}) \biggr)\,\text{d}x \\ \geq{}& \biggl(\frac{1}{2}-\frac{1}{\mu} \biggr) \int _{\mathbb{R}^{3}} \bigl( \vert \nabla u_{n} \vert ^{2}+\widetilde{V}(x) \vert u_{n} \vert ^{2} \bigr)\,\text{d}x \\ &{}- \int _{ \mathbb{R}^{3}} \biggl(\frac{2}{\mu}\omega ^{2}+ \biggl(\frac{1}{2}- \frac{1}{\mu} \biggr)\alpha +\frac{\theta}{\mu} \biggr)u_{n}^{2}\,\text{d}x. \end{aligned} $$

(2.13)

It is easy to know that if $\{u_{n}\}$ is a bounded sequence in E, then $\{J(u_{n})\}$ is also bounded. Then, since $J(u_{n})\leq -n$, we must have

$$ \int _{\mathbb{R}^{3}} \bigl( \vert \nabla u_{n} \vert ^{2}+\widetilde{V}(x) \vert u_{n} \vert ^{2} \bigr)\,\text{d}x\to +\infty , \quad \text{as } n\to +\infty . $$

Set $v_{n}=\frac{u_{n}}{\|u_{n}\|}$, then $\{v_{n}\}$ is a bounded sequence in E. Up to subsequence, we may assume that

$$ v_{n}\rightharpoonup v\quad \text{in } E,\qquad v_{n} \to v \quad \text{in } L^{2}\bigl( \mathbb{R}^{3}\bigr),\qquad v_{n}\to v \quad \text{a.e. in } \mathbb{R}^{3}. $$

(2.14)

Multiplying both sides of (2.13) by $\|u_{n}\|^{-2}$, we obtain

$$ \int _{\mathbb{R}^{3}} \biggl(\frac{2}{\mu}\omega ^{2}+ \biggl( \frac{1}{2}-\frac{1}{\mu} \biggr)\alpha +\frac{\theta}{\mu} \biggr)v_{n}^{2} \,\text{d}x\geq \frac{1}{2}- \frac{1}{\mu}. $$

(2.15)

Then since $\mu >2$, (2.14) and (2.15) imply that $v\neq 0$. From this, $(F_{3})$, $(F_{4})$, and (1.4), we know

$$ \frac{1}{ \Vert u_{n} \Vert ^{2}} \int _{\mathbb{R}^{3}} \bigl(f(x,u_{n})u_{n}+ \alpha u_{n}^{2} \bigr)\,\text{d}x\to +\infty ,\quad \text{as } n\to + \infty . $$

(2.16)

From the assumption $\langle J'(u_{n}),u_{n}\rangle \geq 0$ and (2.16), we have

$$ \begin{aligned} 0\leq{}& \bigl\langle J'(u_{n}),u_{n}\bigr\rangle \\ ={}& \int _{\mathbb{R}^{3}} \bigl( \vert \nabla u_{n} \vert ^{2}+\widetilde{V}(x)u_{n}^{2} \bigr)\,\text{d}x- \int _{\mathbb{R}^{3}}(2\omega +\phi _{u})\phi _{u}u_{n}^{2} \,\text{d}x \\ &{}- \int _{\mathbb{R}^{3}} \bigl(f(x,u_{n})u_{n}+\alpha u_{n}^{2} \bigr)\,\text{d}x \\ \leq{}& \Vert u_{n} \Vert ^{2} \biggl(1+ \frac{\int _{\mathbb{R}^{3}}2\omega ^{2}u_{n}^{2}\,\text{d}x}{ \Vert u_{n} \Vert ^{2}}- \frac{1}{ \Vert u_{n} \Vert ^{2}} \int _{\mathbb{R}^{3}} \bigl(f(x,u_{n})u_{n}+ \alpha u_{n}^{2} \bigr)\,\text{d}x \biggr) \\ \to{}& -\infty ,\quad \text{as } n \to \infty . \end{aligned} $$

(2.17)

This is impossible, so the conclusion of the claim is true.

From the claim and (2.11), for $B>A$ large enough and $u\in S^{\infty}$, there exists a unique $T:=T(u)>0$ such that

$$ J\bigl(T(u)u\bigr)=-B. $$

Based on the implicit function theorem, we get that

$$T\text{ is a continuous function from }S^{\infty}\text{ to }\mathbb{R}. $$

So the deformation retract $\eta :[0,1]\times (E\setminus B^{\infty})\to E$ defined by

$$ \eta (s,u)=(1-s)u+sT(u)u $$

satisfies $\eta (0,u)=u$, $\eta (1,u)\in J^{-B}$, where $B^{\infty}=\{u\in E:\|u\|\leq 1\}$. It follows that

$$ C_{q}(J,\infty )=H^{q}\bigl(E,J^{-B}\bigr)\cong H^{q}\bigl(E,E\setminus B^{\infty}\bigr) \cong 0,\quad \text{for all } q\in \mathbb{N}. $$

 □

To find the nontrivial critical point, we will apply the following proposition.

Proposition 2.2

([22])

Suppose $J\in C^{1}(E,\mathbb{R})$ satisfies the Cerami condition and for some $k\in \mathbb{N}$, $C_{k}(J,0)\neq C_{k}(J,\infty )$. Then J has a nontrivial critical point.

Proof of Theorem 1.1

First, from Lemma 2.3, we known that J satisfies the Cerami condition. On the one hand, due to Lemma 2.4, J has a local linking at zero with respect to the decomposition $E=E^{-}\oplus E^{+}$, then by Proposition 2.1, for $d=\dim E^{-1}$, we have $C_{d}(J,0)\neq 0$. On the other hand, Lemma 2.5 says that for all $q\in \mathbb{N}$, $C_{q}(J,\infty )=0$. Hence, using Proposition 2.2, we prove that J has a nontrivial critical point u. Then $(u,\phi _{u})$ is a nontrivial solution of system (1.1). □