Multiplicity of small negative-energy solutions for a class of semilinear elliptic systems

Abstract

This paper is concerned with the following semilinear elliptic systems:

$$\left \{ \textstyle\begin{array}{@{}l} -\Delta u+V(x)u=H(x)F_{u}(x, u, v), \quad x\in\mathbb{R}^{N},\\ -\Delta v+V(x)v=H(x)F_{v}(x, u, v), \quad x\in\mathbb{R}^{N},\\ u(x)\rightarrow0,\qquad v(x)\rightarrow0\quad \mbox{as } |x|\rightarrow\infty, \end{array}\displaystyle \right . $$

where $V(x)$, $H(x)$ are nonnegative continuous functions. Under some appropriate assumptions on $V(x)$, $H(x)$, and $F(x, u, v)$, we prove the existence of infinitely many small negative-energy solutions by using the fountain theorem established by Zou. Recent results from the literature are extended.

Introduction

In this paper, we consider the existence and multiplicity of solutions to the following semilinear elliptic systems:

$$ \left \{ \textstyle\begin{array}{@{}l} -\Delta u+V(x)u=H(x)F_{u}(x, u, v),\quad x\in\mathbb{R}^{N},\\ -\Delta v+V(x)v=H(x)F_{v}(x, u, v),\quad x\in\mathbb{R}^{N},\\ u(x)\rightarrow0, \qquad v(x)\rightarrow0\quad \mbox{as } |x|\rightarrow\infty, \end{array}\displaystyle \right . $$

(1.1)

where $V(x)$, $H(x)$ are nonnegative continuous functions, we assume that the functions $V(x)$, $H(x)$, and $F(x,u,v)$ satisfy the following hypotheses:

(H₁):

$V\in C(\mathbb{R}^{N},\mathbb {R} )$ satisfies $\inf_{x\in\mathbb{R}^{N}}V(x)\geq a_{0}>0$, where $a_{0}>0$ is a constant. Moreover, for any $M>0$, $\operatorname{meas} \{x\in\mathbb{R}^{N}: V(x)\leq M\}<\infty$, where meas denotes the Lebesgue measure in $\mathbb{R}^{N}$.

(H₂):

$F\in C^{1}(\mathbb {R}^{N}\times\mathbb {R}^{2}, \mathbb {R})$, $|F_{u}(x,u,v)|\leq c(|(u,v)|+|(u,v)|^{p-1})$, and $|F_{v}(x,u,v)|\leq c(|(u,v)|+|(u,v)|^{q-1})$ for some $1< p, q<2$, where c is a positive constant, and $|(u,v)|=(u^{2}+v^{2})^{\frac{1}{2}}$.

(H₃):

$F(x, 0,0)=0$, $F(x, u, v)\geq0$ for all $(x,u,v)\in\mathbb {R}^{N}\times\mathbb {R}^{2}$, and for some $1<\mu <2$, there exists $c_{1}>0$ such that $F(x, u, v)\geq c_{1}|(u,v)|^{\mu }$.

(H₄):

$H(x)\geq0$ and $H(x)\in L^{\frac{2}{2-p}}(\mathbb {R}^{N}, \mathbb{R})\cap L^{\frac{2}{2-q}}(\mathbb {R}^{N}, \mathbb{R}) \cap L^{\frac{2}{2-\mu}}(\mathbb {R}^{N}, \mathbb{R})\cap L^{\infty }(\mathbb {R}^{N}, \mathbb{R})$.

(H₅):

$F(x,u,v)=F(x,-u,-v)$ for all $(x,u,v)\in\mathbb {R}^{N}\times \mathbb {R}^{2}$.

When Ω is a bounded domain of $\mathbb{R}^{N}$, the problem

$$ \left \{ \textstyle\begin{array}{@{}l} -\Delta u=\lambda(a(x)u+b(x)v)+F_{u}(x, u, v) \quad\mbox{in }\Omega,\\ -\Delta v=\lambda(b(x)u+c(x)v)+F_{v}(x, u, v) \quad\mbox{in } \Omega,\\ u(x)=v(x)= 0 \quad\mbox{on }\partial\Omega, \end{array}\displaystyle \right . $$

(1.2)

which is related to reaction-diffusion systems that appear in chemical and biological phenomena, including the steady and unsteady state situation (see [1–4]), has been extensively investigated in recent years. For the results on existence, multiple solutions, and positive solutions to problem (1.2), we refer the readers to [1, 4–9] and the references therein. Qu and Tang [5] obtained the existence and multiplicity of weak solutions of problem (1.2) by using the Ekeland variational principle, the mountain pass theorem, and the saddle point theorem in critical point theory, and by applying the local linking theorem and the saddle point theorem some new existence theorems of weak solutions were obtained by Duan et al. [6]. In [7], by using Morse theory the multiplicity of solutions was obtained for cooperative elliptic systems at resonance. Costa and Magalhães [8, 9] researched subquadratic perturbation problems of semiliner elliptic systems by minimax methods.

Recently, the problems in the whole space $\mathbb {R}^{N}$ were considered in some works. For example, see [10–15] and the references therein. Cao and Tang [10] studied the following Schrödinger systems:

$$ \left \{ \textstyle\begin{array}{@{}ll} -\Delta u+V(x)u=F_{u}(x, u, v), \quad x\in\mathbb{R}^{N},\\ -\Delta v+V(x)v=F_{v}(x, u, v), \quad x\in\mathbb{R}^{N}. \end{array}\displaystyle \right . $$

(1.3)

Under suitable assumptions on $F(x,u,v)$, they obtained the existence of infinitely many solutions characterized by the number of nodes of each component. When $N\geq3$, $V(x)=0$, $F_{u}(x, u, v)=p(x)f(v)$, and $F_{v}(x, u, v)=q(x)g(u)$, Zhang et al. [11] obtained the existence and nonexistence of entire solutions to (1.3). Wu [12] obtained five new critical point theorems on the product spaces and studied three existence theorems for the sequence of high-energy solutions to problem (1.3), whereas Zhou et al. [13] established the existence of high-energy solutions to (1.3) under some conditions that are weaker than those in [12], which unify and sharply improve the recent results in [14].

Inspired by all these facts, the aim of this paper is to study the multiplicity of small negative-energy solutions to problem (1.1) via variational methods, which have been widely used to study Schrödinger equations; see [16–23] and the references therein. To the best of our knowledge, there has been few works concerning this case up to now.

Now, we state our main results.

Theorem 1.1

Suppose that conditions (H₁)-(H₅) hold. Then problem (1.1) possesses infinitely many solutions $\{(u_{k},v_{k})\}$ satisfying

$$\begin{aligned} &\frac{1}{2} \int_{\mathbb{R}^{N}}\bigl(|\nabla u_{k}|^{2}+V(x)u_{k}^{2} \bigr)\,dx+ \frac{1}{2} \int_{\mathbb{R}^{N}}\bigl(|\nabla v_{k}|^{2}+V(x)v_{k}^{2} \bigr)\,dx\\ &\quad{}- \int_{\mathbb{R}^{N}}H(x)F(x, u_{k}, v_{k})\,dx \rightarrow0^{-} \quad \textit{as } k\rightarrow\infty. \end{aligned}$$

The remainder of this paper is as follows. In Section 2, we present some preliminary results. In Section 3, we give a proof of the main result.

Variational setting and preliminaries

In this section, we outline the variational framework for problem (1.1) and give some preliminary lemmas.

Let

$$H^{1}\bigl(\mathbb{R}^{N}\bigr)=\bigl\{ u\in L^{2} \bigl(\mathbb{R}^{N}\bigr): \nabla u \in L^{2}\bigl( \mathbb{R}^{N}\bigr) \bigr\} $$

with the norm

$$\|u\|_{H^{1}}=\biggl( \int_{\mathbb{R}^{N}}\bigl(|\nabla u|^{2}+|u|^{2}\bigr) \,dx\biggr)^{\frac{1}{2}}. $$

Let

$$X=\biggl\{ u\in H^{1}\bigl(\mathbb{R}^{N}\bigr)\Big| \int_{\mathbb{R}^{N}}\bigl(|\nabla u|^{2}+V(x)u^{2} \bigr)\,dx< +\infty\biggr\} $$

with the inner product and norm

$$\langle u,v\rangle_{X}= \int_{\mathbb{R}^{N}}\bigl(\nabla u\nabla v+V(x)uv\bigr)\,dx,\quad \|u \|_{X}=\langle u, u\rangle_{X}^{\frac{1}{2}}. $$

As usual, for $1\leq p<+\infty$, we let

$$\|u\|_{p}=\biggl( \int_{\mathbb{R}^{N}}\bigl|u(x)\bigr|^{p}\,dx\biggr)^{\frac{1}{p}},\quad u\in L^{p}\bigl(\mathbb{R}^{N}\bigr), $$

and

$$\|u\|_{\infty}=\operatorname{ess} \sup_{x\in\mathbb{R}^{N}}\bigl|u(x)\bigr|,\quad u\in L^{\infty}\bigl(\mathbb{R}^{N}\bigr). $$

Then $E=X\times X$ is a Hilbert space with the inner product

$$\bigl\langle (u,v),(\varphi,\psi)\bigr\rangle =\langle u,\varphi \rangle_{X}+\langle v,\psi\rangle_{X},\quad (u,v), (\varphi,\psi) \in X\times X, $$

and the norm

$$\bigl\| (u,v)\bigr\| ^{2}=\bigl\langle (u,v),(u,v)\bigr\rangle =\|u \|_{X}^{2}+\|v\| _{X}^{2},\quad (u,v), ( \varphi,\psi)\in X\times X. $$

Define the functional I on E by

$$ I(u,v)=\frac{1}{2}\bigl\| (u,v)\bigr\| ^{2}- \int_{\mathbb{R}^{N}}H(x)F(x, u, v)\,dx. $$

(2.1)

Then a weak solution of system (1.1) is a critical point of I if I is continuously differentiable on E.

Moreover, we have the following compactness lemma.

Lemma 2.1

Under assumption (H₁), the embedding $E\hookrightarrow L^{r}(\mathbb{R}^{N})\times L^{r}(\mathbb{R}^{N})$ is continuous for $2\leq r\leq2^{*}$, and $E\hookrightarrow L^{r}(\mathbb {R}^{N})\times L^{r}(\mathbb{R}^{N})$ is compact for $2\leq r<2^{*}$.

Proof

By Lemma 3.4 in [24] we know that, under assumption (H₁), the embedding $X\hookrightarrow L^{r}(\mathbb{R}^{N})$ is continuous for $r\in[2, 2^{*}]$ and that $X\hookrightarrow L^{r}(\mathbb{R}^{N})$ is compact for $r\in[2, 2^{*})$, that is, there exist constants $C_{r}>0$ such that $\|u\|_{r}\leq C_{r}\|u\|_{X}$, $\forall u\in X$, and for any bounded sequence $\{u_{n}\}\subset X$, there exists a subsequence of $\{u_{n}\}$ such that $u_{n}\rightarrow u_{0}$ in $L^{r}(\mathbb{R}^{N})$, $r\in [2, 2^{*})$. Therefore, for any $(u,v)\in E$, there exists $C>0$ such that

$$\bigl\| (u,v)\bigr\| _{r}^{r}\leq C\bigl(\|u\|_{r}^{r}+ \|u\|_{r}^{r}\bigr)\leq C\bigl(\|u\|_{X}^{r}+ \|u\|_{X}^{r}\bigr)\leq C\bigl\| (u,v)\bigr\| ^{r}, $$

that is, $\|(u,v)\|_{r}\leq C\|(u,v)\|$, so that $E\hookrightarrow L^{r}(\mathbb{R}^{N})\times L^{r}(\mathbb{R}^{N})$ is continuous for $2\leq r\leq2^{*}$. On the other hand, suppose that $\{(u_{n},v_{n})\}\subset E$ are bounded, that is, $\{u_{n}\}$ and $\{v_{n}\}$ are bounded in X, then there exist subsequences $\{u_{n}\}$ and $\{v_{n}\}$ such that

$$u_{n}\rightarrow u_{0},\qquad v_{n}\rightarrow v_{0} \quad\mbox{in } L^{r}\bigl(\mathbb{R}^{N}\bigr), r\in\bigl[2, 2^{*}\bigr). $$

Therefore,

$$0\leq\|u_{n}-u_{0}\|_{r}^{r}+ \|v_{n}-v_{0}\|_{r}^{r}\leq \bigl\| (u_{n},v_{n})-(u_{0},v_{0}) \bigr\| _{r}^{r}\leq C \bigl(\|u_{n}-u_{0} \|_{r}^{r}+\|v_{n}-v_{0} \|_{r}^{r}\bigr)\rightarrow0 $$

as $n\rightarrow\infty$, that is,

$$(u_{n},v_{n})\rightarrow(u_{0},v_{0})\quad \mbox{in }L^{r}\bigl(\mathbb{R}^{N}\bigr)\times L^{r} \bigl(\mathbb{R}^{N}\bigr), r\in\bigl[2, 2^{*}\bigr), $$

so that $E\hookrightarrow L^{r}(\mathbb{R}^{N})\times L^{r}(\mathbb{R}^{N})$ is compact for $r\in[2, 2^{*})$. The proof is complete. □

Lemma 2.2

If assumptions (H₁)-(H₂) hold, then $I\in C^{1}(E, R)$,

$$ \begin{aligned}[b] \bigl\langle I'(u,v), (\varphi,\psi)\bigr\rangle ={}& \int_{\mathbb{R}^{N}}\nabla u\nabla\varphi \,dx + \int_{\mathbb{R}^{N}}V(x)u\varphi \,dx- \int_{\mathbb {R}^{N}}H(x)F_{u}(x,u,v)\varphi \,dx\\ &{} + \int_{\mathbb{R}^{N}}\nabla v\nabla\psi \,dx + \int_{\mathbb{R}^{N}}V(x)v\psi \,dx- \int_{\mathbb {R}^{N}}H(x)F_{v}(x,u,v)\psi \,dx, \end{aligned} $$

(2.2)

and $\Psi': E\rightarrow E^{*}$ is compact, where $\Psi(u,v)=\int _{\mathbb{R}^{N}}H(x)F(x,u,v)\,dx$.

Proof

The proof is similar to that of Lemma 3.1 in [12]; we omit it. □

To complete the proof of our theorem, the following theorem will be needed in our argument. Let E be a Banach space with norm $\|\cdot\|$ and $E=\overline {\bigoplus_{j\in N}X_{j}}$ with $\operatorname{dim} X_{j}<\infty$ for any $j\in N$. Set $Y_{k}=\bigoplus_{j=0}^{k}X_{j}$, $Z_{k}=\overline{\bigoplus_{j=k+1}^{\infty}X_{j}}$. Consider the $C^{1}$ functional

$$\varphi_{\lambda}(u)=A(u)-\lambda B(u),\quad \lambda\in[1,2], $$

where $A, B: E\rightarrow\mathbb{R}$ are two functionals.

Theorem 2.1

([25], Theorem 2.1)

Suppose that the functional $\varphi_{\lambda}(u)$ satisfies:

(C₁):

$\varphi_{\lambda}(u)$ maps bounded sets to bounded sets uniformly for $\lambda\in[1,2]$. Furthermore, $\varphi_{\lambda}(-u)=\varphi_{\lambda}(u)$ for all $(\lambda, u)\in [1,2]\times E$.

(C₂):

$B(u)\geq0$; $B(u)\rightarrow\infty$ as $\|u\|\rightarrow \infty$ on any finite-dimensional subspace of E.

(C₃):

There exists $\rho_{k}>r_{k}>0$ such that

$$\begin{aligned}& a_{k}(\lambda) :=\inf_{u\in Z_{k}, \|u\|=\rho_{k}}\varphi_{\lambda}(u) \geq 0>b_{k}(\lambda):=\max_{u\in Y_{k}, \|u\|=r_{k}}\varphi _{\lambda}(u),\quad \lambda\in[1,2],\\& d_{k}(\lambda):=\inf_{u\in Z_{k}, \|u\|\leq\rho_{k}}\varphi _{\lambda}(u)\rightarrow0\quad \textit{as }k\rightarrow \infty\textit{ uniformly for }\lambda \in[1,2]. \end{aligned}$$

Then there exist $\lambda_{n}\rightarrow1$ and $u(\lambda_{n})\in Y_{n}$ such that

$$\varphi_{\lambda_{n}}'|_{Y_{n}}\bigl(u( \lambda_{n})\bigr)=0,\qquad \varphi_{\lambda _{n}}\bigl(u( \lambda_{n})\bigr) \rightarrow c_{k}\in\bigl[d_{k}(2),b_{k}(1) \bigr] \quad\textit{as }n\rightarrow\infty. $$

In particular, if $\{u(\lambda_{n})\}$ has a convergent subsequence for every k, then $\varphi_{1}$ has many nontrivial critical points $\{u_{k}\}\subset E\setminus\{0\}$ satisfying $\varphi _{1}(u_{k})\rightarrow0^{-}$ as $n\rightarrow\infty$.

Proof of the main result

In order to apply Theorem 2.1 to prove our main result, we define A, B, and $\varphi_{\lambda}$ on our working space E by

$$ A(u,v)=\frac{1}{2}\bigl\| (u,v)\bigr\| ^{2},\qquad B(u,v)= \int_{\mathbb{R}^{N}}H(x)F(x, u, v)\,dx, $$

(3.1)

and

$$ \varphi_{\lambda}(u,v)=A(u,v)-\lambda B(u,v)=\frac{1}{2}\bigl\| (u,v)\bigr\| ^{2}-\lambda \int_{\mathbb{R}^{N}}H(x)F(x, u, v)\,dx $$

(3.2)

for all $(u,v)\in E$ and $\lambda\in[1,2]$. Obviously, $\varphi _{\lambda}(u,v)\in C^{1}(E, R)$ for all $\lambda\in[1,2]$. We choose a completely orthonormal basis $\{e_{j}: j\in N\}$ of X and let $X_{j}=\operatorname{span}\{e_{j}\}$ for all $j \in N$. Then $Y_{k}=\operatorname{span} \{e_{1},\ldots,e_{k}\}$, $Z_{k}=Y_{k}^{\bot }$, and $E=(Y_{k}\times Y_{k})\oplus(Z_{k}\times Z_{k})$. Note that $\varphi_{1}=I$, where I is defined in (2.1).

Lemma 3.1

Suppose that conditions (H₁), (H₃), and (H₄) hold. Then $B(u,v)\geq0$. Furthermore, $B(u,v)\rightarrow\infty$ as $\|(u,v)\|\rightarrow\infty$ on any finite-dimensional subspace of E.

Proof

Evidently, by (H₃) and (H₄), $B(u,v)=\int _{\mathbb{R}^{N}}H(x)F(x, u, v)\,dx\geq0$, Now we claim that for any finite-dimensional subspace $\widetilde{E}\subset E$, there exists $\varepsilon>0$ such that

$$ \operatorname{meas}\bigl\{ x\in\mathbb{R}^{N}: H(x)\bigl|(u,v)\bigr|^{\mu}\geq \varepsilon \bigl\| (u,v)\bigr\| ^{\mu}\bigr\} \geq\varepsilon, \quad\forall(u,v)\in \widetilde{E}. $$

(3.3)

Arguing by contradiction, we assume that there exists a sequence $\{ (u_{n},v_{n})\}_{n\in\mathbb{N}}\subset\widetilde{E}\setminus\{(0,0)\}$ such that

$$ \operatorname{meas}\biggl\{ x\in\mathbb{R}^{N}: H(x)\bigl|(u_{n},v_{n})\bigr|^{\mu} \geq\frac {1}{n}\bigl\| (u_{n},v_{n})\bigr\| ^{\mu}\biggr\} < \frac{1}{n}. $$

(3.4)

Set $(s_{n},w_{n})=\frac{(u_{n},v_{n})}{\|(u_{n},v_{n})\|}\subset \widetilde{E}\setminus\{(0,0)\}$, then $\|(s_{n},w_{n})\|=1$ for all $n\in\mathbb{N}$, and

$$ \operatorname{meas} \biggl\{ x\in\mathbb{R}^{N}: H(x)\bigl|(s_{n},w_{n})\bigr|^{\mu} \geq\frac {1}{n}\biggr\} < \frac{1}{n}. $$

(3.5)

Since $\operatorname{dim} \widetilde{E}<\infty$, it follows from the compactness of the unit sphere of Ẽ that there exists a subsequence, say $\{(s_{n},w_{n})\}$, such that $(s_{n},w_{n})\rightarrow(s_{0},w_{0})$ in Ẽ. It is easy to verify that $\|(s_{0},w_{0})\|=1$. In view of the equivalence of the norms on the finite-dimensional space Ẽ, we have $(s_{n},w_{n})\rightarrow(s_{0},w_{0})$ in $L^{2}(\mathbb{R}^{N})$, that is,

$$ \int_{\mathbb{R}^{N}}\bigl|(s_{n},w_{n})-(s_{0},w_{0})\bigr|^{2} \,dx\rightarrow0 \quad\mbox{as } n\rightarrow\infty. $$

(3.6)

By (3.6) and the Hölder inequality we have

$$\begin{aligned} &\int_{\mathbb{R}^{N}}H(x)\bigl|(s_{n},w_{n})- (s_{0},w_{0})\bigr|^{\mu}\,dx \\ &\quad\leq\bigl\| H(x) \bigr\| _{{\frac{2}{2-\mu}}}\biggl( \int_{\mathbb {R}^{N}}\bigl|(s_{n},w_{n})- (s_{0},w_{0})\bigr|^{2}\,dx\biggr)^{\frac{\mu}{2}} \rightarrow0 \quad\mbox{as } n\rightarrow\infty. \end{aligned}$$

(3.7)

Therefore, there exist $\xi_{1}, \xi_{2}>0$ such that

$$ \operatorname{meas} \bigl\{ x\in\mathbb{R}^{N}: H(x)\bigl|(s_{0},w_{0})\bigr|^{\mu} \geq\xi _{1}\bigr\} \geq\xi_{2}. $$

(3.8)

Otherwise, we get

$$ \operatorname{meas} \biggl\{ x\in\mathbb{R}^{N}: H(x)\bigl|(s_{0},w_{0})\bigr|^{\mu} \geq\frac {1}{n}\biggr\} =0, $$

(3.9)

which implies that

$$0\leq \int_{\mathbb{R}^{N}}H(x)\bigl|(s_{0},w_{0})\bigr|^{\mu+2} \,dx \leq\frac{\|(s_{0},w_{0})\|^{2}_{{2}}}{n}\leq\frac {C^{2}\|(s_{0},w_{0})\|^{2}}{n}=\frac{C^{2}}{n}\rightarrow0 \quad\mbox{as } n\rightarrow\infty. $$

Hence, $(s_{0},w_{0})=0$, which contradicts with $\|(s_{0},w_{0})\|=1$. Therefore, (3.8) holds.

Now let

$$\begin{aligned}& \Omega_{0}=\operatorname{meas} \bigl\{ x\in\mathbb{R}^{N}: H(x)\bigl|(s_{0},w_{0})\bigr|^{\mu }\geq\xi_{1}\bigr\} , \\& \Omega_{n}=\operatorname{meas} \biggl\{ x\in\mathbb{R}^{N}: H(x)\bigl|(s_{0},w_{0})\bigr|^{\mu }< \frac{1}{n}\biggr\} , \\& \Omega_{n}^{c}=\mathbb{R}^{N}\setminus \Omega_{n}=\operatorname{meas} \biggl\{ x\in \mathbb{R}^{N}: H(x)\bigl|(s_{0},w_{0})\bigr|^{\mu}\geq\frac{1}{n} \biggr\} . \end{aligned}$$

Then by (3.5) and (3.8) we get

$$\begin{aligned}[b] \operatorname{meas} (\Omega_{n}\cap \Omega_{0}) &=\operatorname{meas} \bigl(\Omega_{0}\setminus \bigl(\Omega_{n}^{c}\cap\Omega_{0}\bigr)\bigr) \\ &\geq\operatorname{meas} (\Omega_{0})-\operatorname{meas} \bigl( \Omega_{n}^{c}\cap\Omega _{0}\bigr) \\ &\geq\xi_{2}-\frac{1}{n} \end{aligned} $$

for all positive integer n. Let n be large enough such that $\xi _{2}-\frac{1}{n}\geq\frac{\xi_{2}}{2}$ and $\frac{\xi_{1}}{2^{\mu}}-\frac{1}{n}\geq\frac{\xi_{1}}{2^{\mu+1}}$. Then we have

$$\begin{aligned}[b] &\int_{\mathbb{R}^{N}}H(x)\bigl|(s_{n},w_{n})-(s_{0},w_{0})\bigr|^{\mu} \,dx \\ &\quad\geq \int_{\Omega_{n}\cap\Omega _{0}}H(x)\bigl|(s_{n},w_{n})-(s_{0},w_{0})\bigr|^{\mu} \,dx\\ &\quad\geq\frac{1}{2^{\mu}} \int_{\Omega_{n}\cap\Omega _{0}}H(x)\bigl|(s_{0},w_{0})\bigr|^{\mu} \,dx- \int_{\Omega_{n}\cap\Omega _{0}}H(x)\bigl|(s_{n},w_{n})\bigr|^{\mu} \,dx \\ &\quad\geq\biggl(\frac{\xi_{1}}{2^{\mu}}-\frac{1}{n}\biggr) \operatorname{meas} ( \Omega_{n}\cap \Omega_{0})>0, \end{aligned} $$

which is a contradiction with (3.7). Therefore, (3.3) holds. For the ε given in (3.3), let

$$\Omega_{(u,v)}=\operatorname{meas} \bigl\{ x\in\mathbb{R}^{N}: H(x)\bigl|(u,v)\bigr|^{\mu}\geq \varepsilon\bigl\| (u,v)\bigr\| ^{\mu}\bigr\} ,\quad \forall(u,v)\in\widetilde{E}\setminus\bigl\{ (0,0)\bigr\} . $$

Then by (3.3)

$$ \operatorname{meas} (\Omega_{(u,v)})\geq\varepsilon,\quad \forall(u,v)\in \widetilde{E}\setminus\bigl\{ (0,0)\bigr\} . $$

(3.10)

Combining (H₃) and (3.10), we have

$$B(u,v)\geq \int_{\mathbb{R}^{N}}c_{1}H(x)\bigl|(u,v)\bigr|^{\mu}\,dx\geq \int_{\Omega _{(u,v)}}\varepsilon c_{1}\bigl\| (u,v)\bigr\| ^{\mu} \,dx \geq\varepsilon^{2} c_{1}\bigl\| (u,v)\bigr\| ^{\mu}, $$

which implies that $B(u,v)\rightarrow\infty$ as $\|(u,v)\|\rightarrow \infty$ on any finite-dimensional space of E. The proof is completed. □

Lemma 3.2

Suppose that (H₁)-(H₂) and (H₄) are satisfied. Then there exists a sequence $\rho_{k}\rightarrow0^{+}$ as $k\rightarrow\infty$ such that $a_{k}(\lambda) :=\inf_{(u,v)\in Z_{k}\times Z_{k}, \|(u,v)\|=\rho_{k}}\varphi _{\lambda}(u,v)\geq0$ and $d_{k}(\lambda):=\inf_{(u,v)\in Z_{k}\times Z_{k}, \|(u,v)\|\leq\rho_{k}} \varphi_{\lambda }(u,v)\rightarrow0$ as $k\rightarrow\infty$ uniformly for $\lambda\in [1,2]$, where $Z_{k}=\overline{\bigoplus_{j=k+1}^{\infty }X_{j}}=\overline{\operatorname{span}\{e_{k},\ldots\}}$ for all $k\in N$.

Proof

Let

$$ \alpha_{k}(r):=\sup_{(u,v)\in Z_{k}\times Z_{k}, \|(u,v)\|=1}\bigl\| (u,v)\bigr\| _{r},\quad \forall k\in N, $$

(3.11)

where $\|(u,v)\|_{r}=(\int_{\mathbb {R}^{N}}|(u,v)|^{r})^{\frac{1}{r}}$. Then $\alpha_{k}(r)\rightarrow0$ as $k\rightarrow\infty$. Indeed, $\alpha_{k}(r)$ is convergent since $\alpha_{k}(r)$ are decreasing in k and $\alpha_{k}(r)\geq0$. Furthermore, for any k, there exists $(u_{k},v_{k})\in Z_{k}\times Z_{k}$ such that $\|(u_{k},v_{k})\|=1$ and $\|(u_{k},v_{k})\|_{r}\geq\frac{\alpha_{k}(r)}{2}$.

For any $\varphi\in X$, $\varphi=\sum_{n=1}^{\infty}a_{n}e_{n}$, we get

$$\bigl|\langle u_{k},\varphi\rangle_{X}\bigr|= \Biggl| \Biggl\langle u_{k},\sum_{n=k+1}^{\infty}a_{n}e_{n} \Biggr\rangle _{X} \Biggr|\leq\|u_{k}\|_{X} \Biggl\| \sum _{n=k+1} ^{\infty}a_{n}e_{n} \Biggr\| _{X}\leq \Biggl\| \sum_{n=k+1} ^{\infty}a_{n}e_{n} \Biggr\| _{X}\rightarrow0 $$

as $k\rightarrow\infty$, which implies that $u_{k}\rightharpoonup0$ in X. Since the embedding $X\hookrightarrow L^{r}(\mathbb {R}^{N})$ is compact, $u_{k}\rightarrow0$ in $L^{r}(\mathbb {R}^{N})$ for $r\in[2, 2^{*})$. The same argument implies that $v_{k}\rightarrow0$ in $L^{r}(\mathbb {R}^{N})$ for $r\in[2, 2^{*})$. Consequently,

$$\bigl\| (u_{k},v_{k})\bigr\| _{r}^{r} \leq2^{\frac {r}{2}}\bigl(\|u_{k}\|_{r}^{r}+ \|v_{k}\|_{r}^{r}\bigr)\rightarrow \quad\mbox{as } k \rightarrow\infty, $$

that is,

$$ \alpha_{k}(r)\rightarrow0 \quad\mbox{as } k\rightarrow\infty. $$

(3.12)

By (H₂) we have

$$ \begin{aligned}[b] \bigl|F(x,u,v)\bigr| &=\bigl|F(x,u,v)-F(x,0,0)\bigr|\\ &\leq \int_{0}^{1}\bigl|F_{u}(x,tu,tv)\bigr||u|\,dt+ \int _{0}^{1}\bigl|F_{v}(x,tu,tv)\bigr||v|\,dt\\ &\leq C\bigl(\bigl|(u,v)\bigr|^{2}+\bigl|(u,v)\bigr|^{p}+\bigl|(u,v)\bigr|^{q} \bigr). \end{aligned} $$

(3.13)

Therefore, by (3.2), (3.11)-(3.13), and the Hölder inequality we get

$$\begin{aligned} \varphi_{\lambda}(u,v) &=\frac{1}{2}\bigl\| (u,v) \bigr\| ^{2}-\lambda \int_{\mathbb{R}^{N}}H(x)F(x, u, v)\,dx \\ &\geq\frac{1}{2}\bigl\| (u,v)\bigr\| ^{2}-2C\bigl(\bigl\| (u,v) \bigr\| _{2}^{2}+\bigl\| H(x)\bigr\| _{\frac {2}{2-p}}\bigl\| (u,v)\bigr\| _{2}^{p}+ \bigl\| H(x)\bigr\| _{\frac{2}{2-q}}\bigl\| (u,v)\bigr\| _{2}^{q}\bigr) \\ &\geq\frac{1}{2}\bigl\| (u,v)\bigr\| ^{2}-2C\bigl(\alpha_{k}^{2}(2) \bigl\| (u,v)\bigr\| ^{2}+\alpha _{k}^{p}(2)\bigl\| (u,v) \bigr\| ^{p}+\alpha_{k}^{q}(2)\bigl\| (u,v)\bigr\| ^{q} \bigr). \end{aligned}$$

(3.14)

By (3.13) there exist a positive integer $k_{1}$ such that

$$ 2C\alpha_{k}^{2}(2)\leq\frac{1}{8},\quad \forall k\geq k_{1}. $$

(3.15)

Then, by (3.14), we have

$$ \varphi_{\lambda}(u,v)\geq\frac{3}{8}\bigl\| (u,v)\bigr\| ^{2}-2C \bigl(\alpha _{k}^{p}(2)\bigl\| (u,v)\bigr\| ^{p}+ \alpha_{k}^{q}(2)\bigl\| (u,v)\bigr\| ^{q} \bigr). $$

(3.16)

Let

$$ \rho_{k}=\max\bigl\{ \bigl(16C\alpha_{k}^{p}(2) \bigr)^{\frac{1}{2-p}}, \bigl(16C\alpha _{k}^{q}(2) \bigr)^{\frac{1}{2-q}}\bigr\} . $$

(3.17)

Obviously, $\rho_{k}\rightarrow0$ as $k\rightarrow\infty$ since $p, q\in(1,2)$. By (3.16) and (3.17) direct computation shows that

$$a_{k}(\lambda) :=\inf_{(u,v)\in Z_{k}\times Z_{k}, \|(u,v)\|= \rho_{k}}\varphi_{\lambda}(u,v) \geq\frac{\rho_{k}^{2}}{8}>0,\quad \forall k\geq k_{1}. $$

Moreover, by (3.14), for any $(u,v)\in Z_{k}\times Z_{k}$ with $\|(u,v)\|= \rho_{k}$, we have

$$\varphi_{\lambda}(u,v)\geq-2C\bigl(\alpha_{k}^{2}(2) \bigl\| (u,v)\bigr\| ^{2}+\alpha _{k}^{p}(2)\bigl\| (u,v) \bigr\| ^{p}+\alpha_{k}^{q}(2)\bigl\| (u,v)\bigr\| ^{q} \bigr). $$

Therefore,

$$0\geq\inf_{(u,v)\in Z_{k}\times Z_{k}, \|(u,v)\|\leq\rho _{k}} \varphi_{\lambda}(u,v)\geq-2C\bigl( \alpha_{k}^{2}(2)\bigl\| (u,v)\bigr\| ^{2}+\alpha _{k}^{p}(2)\bigl\| (u,v)\bigr\| ^{p}+\alpha_{k}^{q}(2) \bigl\| (u,v)\bigr\| ^{q}\bigr). $$

Since $\alpha_{k}(2)\rightarrow0$ as $k\rightarrow\infty$, we have

$$d_{k}(\lambda):=\inf_{(u,v)\in Z_{k}\times Z_{k}, \|(u,v)\|\leq \rho_{k}}\varphi_{\lambda}(u,v) \rightarrow0 \quad\mbox{as } k\rightarrow \infty \mbox{ uniformly for } \lambda\in[1,2]. $$

The proof is completed. □

Lemma 3.3

Suppose that (H₁)-(H₄) hold. Then for the positive integer $k_{1}$ and the sequence $\{\rho_{k}\}$ obtained in Lemma  3.2, for all $k\geq k_{1}$, there exist $0< r_{k}<\rho_{k}$ such that

$$b_{k}(\lambda):=\max_{(u,v)\in Y_{k}\times Y_{k}, \|(u,v)\|=r_{k}}\varphi_{\lambda}(u,v)< 0\quad \textit{for all } \lambda\in [1,2], \forall k\geq k_{1}, $$

where $Y_{k}=\bigoplus_{j=1}^{k}X_{j}=\operatorname{span}\{e_{1},\ldots e_{k}\}$ for all $k\in N$.

Proof

Note that $Y_{k}\times Y_{k}$ is a finite-dimensional subspace of E. Then by (3.3) there exists a constant $\varepsilon_{k}$ such that

$$ \operatorname{meas} \bigl(\Omega_{(u,v)}^{k}\bigr)\geq \varepsilon_{k}, \quad\forall(u,v)\in Y_{k}\times Y_{k}\setminus\bigl\{ (0,0)\bigr\} , $$

(3.18)

where

$$\Omega_{(u,v)}^{k}=\operatorname{meas} \bigl\{ x\in \mathbb{R}^{N}: H(x)\bigl|(u,v)\bigr|^{\mu }\geq\varepsilon_{k} \bigl\| (u,v)\bigr\| ^{\mu}\bigr\} ,\quad \forall(u,v)\in Y_{k}\times Y_{k}\setminus\bigl\{ (0,0)\bigr\} . $$

Combining (3.2), (H₃), (H₄), and (3.18), for any $k\in N$ and $\lambda\in[1,2]$, we have

$$ \begin{aligned}[b] \varphi_{\lambda}(u,v) &=\frac{1}{2}\bigl\| (u,v) \bigr\| ^{2}-\lambda \int_{\mathbb{R}^{N}}H(x)F(x, u, v)\,dx \\ &\leq\frac{1}{2}\bigl\| (u,v)\bigr\| ^{2}-c_{1} \int_{\mathbb{R}^{N}}H(x)\bigl|(u,v)\bigr|^{\mu }\,dx \\ &\leq\frac{1}{2}\bigl\| (u,v)\bigr\| ^{2}-c_{1} \int_{\Omega _{(u,v)}^{k}}H(x)\bigl|(u,v)\bigr|^{\mu}\,dx \\ &\leq\frac{1}{2}\bigl\| (u,v)\bigr\| ^{2}-c_{1} \varepsilon_{k}\bigl\| (u,v)\bigr\| ^{\mu} \operatorname{meas} \bigl( \Omega_{(u,v)}^{k}\bigr) \\ &\leq\frac{1}{2}\bigl\| (u,v)\bigr\| ^{2}-c_{1} \varepsilon_{k}^{2}\bigl\| (u,v)\bigr\| ^{\mu}. \end{aligned} $$

(3.19)

For $\|(u,v)\|=r_{k}<\rho_{k}$ small enough, we have

$$b_{k}(\lambda):=\max_{(u,v)\in Y_{k}\times Y_{k}, \|(u,v)\|=r_{k}}\varphi_{\lambda}(u,v)< 0 \quad\mbox{for all } \lambda\in [1,2], \forall k\geq k_{1}, $$

since $\mu\in(1,2)$. The proof is completed. □

Now we give the proof of Theorem 1.1.

Proof

Obviously, condition (C₁) in Theorem 2.1 holds. By Lemmas 3.1-3.3 conditions (C₂) and (C₃) in Theorem 2.1 are also satisfied. Furthermore, by Theorem 2.1, there exist $\lambda_{n}\rightarrow1$ and $(u(\lambda_{n}),v(\lambda _{n}))\in Y_{n}\times Y_{n}$ such that

$$ \begin{aligned} &\varphi_{\lambda_{n}}'|_{Y_{n}\times Y_{n}}\bigl(u( \lambda_{n}),v(\lambda_{n})\bigr)=0, \\ &\varphi_{\lambda_{n}} \bigl(u(\lambda_{n}),v(\lambda_{n})\bigr) \rightarrow c_{k}\in\bigl[d_{k}(2),b_{k}(1)\bigr] \quad\mbox{as } n\rightarrow\infty . \end{aligned} $$

(3.20)

For simplicity, in what follows, we always set $(u_{n},v_{n})=(u(\lambda_{n}),v(\lambda_{n}))$ for all $n\in\mathbb {N}$.

Now we claim that the sequence $\{(u_{n},v_{n})\}$ obtained in (3.19) is bounded in E. Indeed, by (H₂), (H₄), (3.2), (3.20), and the Hölder inequality we have

$$ \begin{aligned}[b]\bigl\| (u_{n},v_{n})\bigr\| ^{2} &\leq2 \varphi_{\lambda_{n}}\bigl((u_{n},v_{n})\bigr)+2 \lambda_{n} \int_{\mathbb {R}^{N}}H(x)F(x,u_{n},v_{n})\,dx \\ &\leq C_{0}+C\bigl(\bigl\| (u_{n},v_{n}) \bigr\| ^{2}+\bigl\| (u,v)\bigr\| _{p}^{p}+\bigl\| (u,v) \bigr\| _{q}^{q}\bigr) \end{aligned} $$

(3.21)

for some $C_{0}>0$. Since $p, q\in(1,2)$, (3.21) implies that $\{(u_{n},v_{n})\}$ is bounded in E.

Finally, we show that $\{(u_{n},v_{n})\}$ possesses a strong convergent sequence in E. Indeed, since $\{(u_{n},v_{n})\}$ is bounded, there exists $(u_{0},v_{0})\in E$ such that

$$\begin{aligned}& (u_{n},v_{n})\rightharpoonup(u_{0},v_{0}) \quad\mbox{in } E,\\& (u_{n},v_{n})\rightarrow(u_{0},v_{0}) \quad\mbox{in } L^{p}\bigl(\mathbb {R}^{N}\bigr)\times L^{p}\bigl(\mathbb {R}^{N}\bigr), p\in\bigl[2,2^{*}\bigr),\\& (u_{n},v_{n})\rightarrow(u_{0},v_{0}) \quad\mbox{a.e. on } \mathbb {R}^{N}. \end{aligned}$$

By (2.2) we easily get

$$ \begin{aligned}[b] \bigl\| (u_{n},v_{n})-(u_{0},v_{0}) \bigr\| ^{2} ={}&\bigl\langle \varphi_{\lambda_{n}}'(u_{n},v_{n})- \varphi _{1}'(u_{0},v_{0}),(u_{n},v_{n})-(u_{0},v_{0}) \bigr\rangle \\ &{} + \int_{\mathbb {R}^{N}}H(x) \bigl(\lambda _{n}F_{u}(x,u_{n},v_{n})-F_{u}(x,u_{0},v_{0}) \bigr) (u_{n}-u_{0})\,dx \\ &{} + \int_{\mathbb {R}^{N}}H(x) \bigl(\lambda _{n}F_{v}(x,u_{n},v_{n})-F_{v}(x,u_{0},v_{0}) \bigr) (v_{n}-v_{0})\,dx. \end{aligned} $$

(3.22)

Clearly,

$$ \bigl\langle \varphi_{\lambda_{n}}'(u_{n},v_{n})- \varphi _{1}'(u_{0},v_{0}),(u_{n},v_{n})-(u_{0},v_{0}) \bigr\rangle \rightarrow0. $$

(3.23)

Denote

$$\begin{aligned}& M:= \int_{\mathbb {R}^{N}}H(x) \bigl(\lambda _{n}F_{u}(x,u_{n},v_{n})-F_{u}(x,u_{0},v_{0}) \bigr) (u_{n}-u_{0})\,dx,\\& N:= \int_{\mathbb {R}^{N}}H(x) \bigl(\lambda _{n}F_{v}(x,u_{n},v_{n})-F_{v}(x,u_{0},v_{0}) \bigr) (v_{n}-v_{0})\,dx. \end{aligned}$$

Then by (H₂), (H₄), and the Hölder and Minkowski inequalities we have

$$\begin{aligned} M \leq{}& C\|u_{n}-u_{0}\|_{2} \biggl( \int_{\mathbb {R}^{N}}H^{2}(x) \bigl(2\bigl|(u_{n},v_{n})\bigr| +2\bigl|(u_{n},v_{n})\bigr|^{p-1}+2\bigl|(u_{n},v_{n})\bigr|^{q-1} \\ &{} +\bigl|(u_{0},v_{0})\bigr|+\bigl|(u_{0},v_{0})\bigr|^{p-1}+\bigl|(u_{0},v_{0})\bigr|^{q-1} \bigr)^{2}\,dx \biggr)^{\frac{1}{2}} \\ \leq{}& C\|u_{n}-u_{0}\|_{2} \biggl( \int_{\mathbb {R}^{N}}H^{2}(x) \bigl(2\bigl|(u_{n},v_{n})\bigr|+\bigl|(u_{0},v_{0})\bigr| \bigr)^{2}+H^{2}(x) \bigl(2\bigl|(u_{n},v_{n})\bigr|^{p-1} \\ &{} +\bigl|(u_{0},v_{0})\bigr|^{p-1}\bigr)^{2}+H^{2}(x) \bigl(2\bigl|(u_{n},v_{n})\bigr|^{q-1}+\bigl|(u_{0},v_{0})\bigr|^{q-1} \bigr)^{2}\,dx \biggr)^{\frac {1}{2}} \\ \leq{}& C\|u_{n}-u_{0}\|_{2} \biggl( \int_{\mathbb {R}^{N}}H^{2}(x) \bigl(4\bigl|(u_{n},v_{n})\bigr|^{2} +\bigl|(u_{0},v_{0})\bigr|^{2} \bigr)+H^{2}(x) \bigl(4\bigl|(u_{n},v_{n})\bigr|^{2p-2} \\ &{} +\bigl|(u_{0},v_{0})\bigr|^{2p-2}\bigr)+H^{2}(x) \bigl(4\bigl|(u_{n},v_{n})\bigr|^{2q-2}+\bigl|(u_{0},v_{0})\bigr|^{2q-2} \bigr)\,dx \biggr)^{\frac{1}{2}} \\ \leq{}& C\|u_{n}-u_{0}\|_{2} \biggl[ \biggl( \int_{\mathbb {R}^{N}}4H^{2}(x)\bigl|(u_{n},v_{n})\bigr|^{2} \,dx \biggr)^{\frac{1}{2}}+ \biggl( \int _{\mathbb {R}^{N}}H^{2}(x)\bigl|(u_{0},v_{0})\bigr|^{2} \,dx \biggr)^{\frac{1}{2}} \\ &{} + \biggl( \int_{\mathbb {R}^{N}}4H^{2}(x)\bigl|(u_{n},v_{n})\bigr|^{2p-2} \biggr)^{\frac{1}{2}}+ \biggl( \int_{\mathbb {R}^{N}}H^{2}(x)\bigl|(u_{0},v_{0})\bigr|^{2p-2} \,dx \biggr)^{\frac{1}{2}} \\ &{} + \biggl( \int_{\mathbb {R}^{N}}4H^{2}(x)\bigl|(u_{n},v_{n})\bigr|^{2q-2} \,dx \biggr)^{\frac{1}{2}}\,dx+ \biggl( \int_{\mathbb {R}^{N}}H^{2}(x)\bigl|(u_{0},v_{0})\bigr|^{2q-2} \,dx \biggr)^{\frac{1}{2}} \biggr] \\ \leq{}& C\|u_{n}-u_{0}\|_{2} \bigl[2\|H \|_{\infty }\bigl\| (u_{n},v_{n})\bigr\| _{2}^{2}+ \|H\|_{\infty}\bigl\| (u_{0},v_{0})\bigr\| _{2}^{2} +2\|H\|_{\frac{2}{2-p}}\bigl\| (u_{n},v_{n})\bigr\| _{2}^{p-1} \\ &{} +\|H\|_{\frac{2}{2-p}}\bigl\| (u_{0},v_{0})\bigr\| _{2}^{p-1}+2 \|H\|_{\frac {2}{2-q}}\bigl\| (u_{n},v_{n})\bigr\| _{2}^{q-1}+ \|H\|_{\frac{2}{2-q}}\bigl\| (u_{0},v_{0})\bigr\| _{2}^{q-1} \bigr] \\ \leq{}& C\|u_{n}-u_{0}\|_{2} \bigl( \bigl\| (u_{n},v_{n})\bigr\| _{2}^{2}+ \bigl\| (u_{0},v_{0})\bigr\| _{2}^{2}+ \bigl\| (u_{n},v_{n})\bigr\| _{2}^{p-1} + \bigl\| (u_{0},v_{0})\bigr\| _{2}^{p-1} \\ &{} +\bigl\| (u_{n},v_{n})\bigr\| _{2}^{q-1}+ \bigl\| (u_{0},v_{0})\bigr\| _{2}^{q-1} \bigr). \end{aligned}$$

(3.24)

Since $(u_{n},v_{n})\rightarrow(u_{0},v_{0})$ in $L^{p}(\mathbb {R}^{N})\times L^{p}(\mathbb {R}^{N})$, for any $p\in[2,2^{*})$, we obtain

$$ \int_{\mathbb {R}^{N}}H(x) \bigl(\lambda _{n}F_{u}(x,u_{n},v_{n})-F_{u}(x,u_{0},v_{0}) \bigr) (u_{n}-u_{0})\,dx\rightarrow 0 \quad\mbox{as } n\rightarrow \infty. $$

(3.25)

Similarly, we can also obtain

$$ \int_{\mathbb {R}^{N}}H(x) \bigl(\lambda _{n}F_{v}(x,u_{n},v_{n})-F_{v}(x,u_{0},v_{0}) \bigr) (v_{n}-v_{0})\,dx\rightarrow 0 \quad\mbox{as } n\rightarrow \infty. $$

(3.26)

Therefore, by (3.22)-(3.26) we get $\|(u_{n},v_{n})-(u_{0},v_{0})\| \rightarrow0$ as $n\rightarrow\infty$.

Now from the last assertion of Theorem 2.1 we know that $I=\varphi_{1}$ has infinitely many nontrivial critical points. Therefore, system (1.1) possesses infinitely many small negative-energy solutions. The proof is completed. □