On G-invariant solutions of a singular biharmonic elliptic system involving multiple critical exponents in $R^{N}$

Abstract

In this work, a biharmonic elliptic system is investigated in $\mathbb{R}^{N}$, which involves singular potentials and multiple critical exponents. By the Rellich inequality and the symmetric criticality principle, the existence and multiplicity of G-invariant solutions to the system are established. To our best knowledge, our results are new even in the scalar cases.

Introduction

In this article, we study the singular fourth-order elliptic problem:

$$ \textstyle\begin{cases} \Delta^{2} u=\mu\frac{u}{ \vert x \vert ^{4}}+Q(x) \sum_{i=1}^{m}\frac{\varsigma_{i}\alpha_{i}}{2^{\ast\ast}} \vert u \vert ^{\alpha_{i}-2}u \vert v \vert ^{\beta_{i}}+\sigma h(x) \vert u \vert ^{q-2}u, &\text{in } \mathbb{R}^{N},\\ \Delta^{2} v=\mu\frac{v}{ \vert x \vert ^{4}}+Q(x)\sum_{i=1}^{m} \frac{\varsigma_{i}\beta_{i}}{2^{\ast\ast}} \vert u \vert ^{\alpha_{i}} \vert v \vert ^{\beta_{i}-2}v+\sigma h(x) \vert v \vert ^{q-2}v, &\text{in } \mathbb{R}^{N},\\ \int_{\mathbb{R}^{N}} ( \vert \Delta u \vert ^{2}+ \vert \Delta v \vert ^{2} )\,dx< +\infty \quad\text{and}\quad u, v\not\equiv0, &\text{in } \mathbb{R}^{N}, \end{cases} $$

(1.1)

where $\Delta^{2}$ denotes the biharmonic operator, $N\geq5$, $\sigma\geq0$, $\mu\in[0, \overline{\mu})$ with $\overline{\mu}\triangleq\frac{1}{16}N^{2}(N-4)^{2}$, $q\in(1, 2)$, $\varsigma_{i}\in(0, +\infty)$, and $\alpha_{i}$, $\beta_{i}>1$ satisfy $\alpha_{i}+\beta_{i}=2^{\ast\ast}\ (i=1, \ldots, m; 1\leq m\in\mathbb{N})$, $2^{\ast\ast} \triangleq\frac{2N}{N-4}$ is the critical Sobolev exponent; $Q(x)$ and $h(x)$ are G-invariant functions such that $Q(x)\in\mathscr {C}(\mathbb{R}^{N})\cap L^{\infty}(\mathbb{R}^{N})$ and $h(x)\in L^{\theta}(\mathbb{R}^{N})$ with $\theta\triangleq 2^{\ast\ast}/(2^{\ast\ast}-q)$ (see Sect. 2 for details).

There have been by now a large number of papers concerning the existence, nonexistence as well as qualitative properties of nontrivial solutions to critical elliptic problems of second order. With no hope of being complete, we would like to mention some of them [1–4]. In most of these papers, the authors deal with the elliptic problems involving singular potentials and critical exponents. For instance, Deng and Jin in [4] handled the following singular equation:

$$ -\Delta u=\mu\frac{u}{ \vert x \vert ^{2}}+Q(x) \vert x \vert ^{-s}u^{2^{\ast}(s)-1} \quad\text{and}\quad u>0 \quad\text{in } \mathbb{R}^{N}, $$

(1.2)

where $N> 2$, $\mu\in[0, \frac{1}{4}(N-2)^{2})$, $s\in[0, 2)$, $2^{\ast}(s)=\frac{2(N-s)}{N-2}$, and $2^{\ast}(0)=2^{\ast}\triangleq\frac{2N}{N-2}$, and Q is G-invariant with respect to a subgroup G of $O(\mathbb{N})$. By applying analytic techniques and critical point theory, several results on the existence and multiplicity of G-invariant solutions to (1.2) were obtained. Subsequently, Waliullah [5] extended the results in [4] to the weighted polyharmonic elliptic equations. In particular, Waliullah considered the following semilinear partial differential equation:

$$ (-\Delta)^{k} u=Q(x) \vert u \vert ^{2_{(k)}^{\ast}-2}u \quad\text{in } \mathbb{R}^{N}, $$

(1.3)

where $k>1$, $N>2k$, $2_{(k)}^{\ast}=\frac{2N}{N-2k}$, and Q is G-invariant. By employing the minimizing sequence and the concentration–compactness method, the author attained the existence of nontrivial G-invariant solution to (1.3). Borrowing ideas from [4, 5], Deng and Huang [6–8] recently established a few valuable results for the scalar elliptic problems in a bounded G-invariant domain. Moreover, let us also mention that when $\mu=0$ and the right-hand side nonlinearity term $\vert x \vert ^{-s}u^{2^{\ast}(s)-1}$ in (1.2) is substituted by $u^{q-1}$ with $1< q\leq2^{\ast}$, there have been a variety of remarkable results on G-invariant solutions in [9–11]. Furthermore, for other results about this aspect, see [12] with singular Lane–Emden–Fowler equations, [13] with singular p-Laplacian equations, [14] with biharmonic operators and [15] with $p(x)$-biharmonic operators [16], and monograph [17] with generalized Lane–Emden–Fowler equations or Gierer–Meinhardt systems involving singular nonlinearity.

For the systems of singular elliptic equations involving critical exponents, a wide range of works concerning the solutions structures have been presented in recent years. For example, Cai and Kang [18] studied the following elliptic system with multiple critical terms:

$$ \textstyle\begin{cases} \mathcal{L}_{\mu}u=\frac{\varsigma_{1}\alpha_{1}}{2^{\ast}} \vert u \vert ^{\alpha_{1}-2}u \vert v \vert ^{\beta_{1}}+ \frac{\varsigma_{2}\alpha_{2}}{2^{\ast}} \vert u \vert ^{\alpha _{2}-2}u \vert v \vert ^{\beta_{2}} +a_{1} \vert u \vert ^{q_{1}-2}u+a_{2}v, &\text{in } \Omega,\\ \mathcal{L}_{\mu}v=\frac{\varsigma_{1}\beta_{1}}{2^{\ast}} \vert u \vert ^{\alpha_{1}} \vert v \vert ^{\beta_{1}-2}v +\frac{\varsigma_{2}\beta_{2}}{2^{\ast}} \vert u \vert ^{\alpha_{2}} \vert v \vert ^{\beta_{2}-2}v +a_{2}u+a_{3} \vert v \vert ^{q_{2}-2}v, &\text{in } \Omega,\\ u=v=0, &\text{on } \partial\Omega, \end{cases} $$

(1.4)

where $N\geq3$, $\Omega\subset\mathbb{R}^{N}$ is a smooth bounded domain such that $0\in\Omega$, $\mathcal{L}_{\mu}=-\Delta-\mu \vert x \vert ^{-2}$, $\mu<\frac{1}{4}(N-2)^{2}$, $a_{j}\in\mathbb{R}\ (j=1, 2, 3)$, $\varsigma_{i}\in(0, +\infty)$, $q_{i}\in[2, 2^{\ast})$, and $\alpha_{i}$, $\beta_{i}>1$ fulfill $\alpha_{i}+\beta_{i}=2^{\ast}\ (i=1, 2)$. By a variational minimax method combined with a delicate analysis of Palais–Smale sequences, the authors proved the existence of positive solutions to (1.4). Very recently, Nyamoradi and Hsu [19] investigated the following quasilinear elliptic system involving multiple critical exponents:

$$ \textstyle\begin{cases} -\operatorname{div}( \vert x \vert ^{-ap} \vert \nabla u \vert ^{p-2}\nabla u)=\sum_{i=1}^{m}\frac{\varsigma_{i}\alpha_{i} \vert u \vert ^{\alpha_{i}-2}u \vert v \vert ^{\beta_{i}}}{p^{\ast}(a, b) \vert x \vert ^{bp^{\ast}(a, b)}} +\sum_{i=1}^{m}\frac{ \lambda_{i}f_{i}(x) }{ \vert x \vert ^{\beta}} \vert u \vert ^{q-2}u, &\text{in } \Omega,\\ -\operatorname{div}( \vert x \vert ^{-ap} \vert \nabla v \vert ^{p-2}\nabla v)=\sum_{i=1}^{m}\frac{ \varsigma_{i}\beta_{i} \vert u \vert ^{\alpha_{i}} \vert v \vert ^{\beta_{i}-2}v}{p^{\ast}(a, b) \vert x \vert ^{bp^{\ast}(a, b)}} +\sum_{i=1}^{m}\frac{ \mu_{i}f_{i}(x) }{ \vert x \vert ^{\beta}} \vert v \vert ^{q-2}v, &\text{in } \Omega,\\ u=v=0, &\text{on } \partial\Omega, \end{cases} $$

(1.5)

where $0\in\Omega$ is a smooth bounded domain in $\mathbb{R}^{N}$, $1< p< N$, $0\leq a<\frac{N-p}{p}$, $a\leq b< a+1$, $0<\varsigma_{i}, \lambda_{i}, \mu_{i}<+\infty$, $\alpha_{i}$, $\beta_{i}>1$, $\alpha_{i}+\beta_{i}=p^{\ast}(a, b)=\frac{Np}{N-p(a+1-b)}$ for $i=1,\ldots, m$. By employing the analytic techniques of Nehari manifold, the authors established the existence and multiplicity of positive solutions to (1.5) under certain appropriate hypotheses on the parameters q, β, $\lambda_{i}$, $\mu_{i}$ and the weighted functions $f_{i}(x)\ (i=1, \ldots, m)$. Other results relating to second-order elliptic systems can be found in [20–23] and the references therein. For the systems of fourth-order elliptic equations, we would like to refer the reader to the papers [24–26] for the elliptic problems related to nonlinearities with critical growth.

Nevertheless, elliptic systems involving the G-invariant solutions have seldom been studied; we only find a handful of results in [27–30]. To the best of our knowledge, there are few results on G-invariant solutions for the singular fourth-order elliptic problem (1.1) even in the scalar cases $\sigma=0$, $0<\mu<\overline{\mu}$, $m=1$, and $u=v$. Therefore, it is necessary for us to investigate (1.1) thoroughly. Let $\overline{Q}>0$ be a constant. This work is dedicated to seeking the G-invariant solutions for both the cases of $\sigma=0$, $Q(x)\not\equiv\overline{Q}$ and $\sigma>0$, $Q(x)\equiv\overline{Q}$ in (1.1). Our arguments are mainly based upon the symmetric criticality principle due to Palais [31] and variational methods.

The rest of this article is schemed as follows. The variational framework and the main results of this paper are presented in Sect. 2. The proofs of G-invariant solutions for the cases $\sigma=0$ and $Q(x)\not\equiv\overline{Q}$ are detailed in Sect. 3, while the multiplicity results for the cases $\sigma>0$ and $Q(x)\equiv\overline{Q}$ are proved in Sect. 4.

Preliminaries and main results

Let $\mathscr{D}^{2, 2}(\mathbb{R}^{N})$ denote the completion of $\mathscr{C}_{0}^{\infty}(\mathbb{R}^{N})$ under the norm $(\int_{\mathbb{R}^{N}} \vert \Delta u \vert ^{2}\,dx)^{1/2}$, associated with the inner product given by $\langle u, \varphi\rangle=\int_{\mathbb{R}^{N}}\Delta u\Delta\varphi \,dx$. Recall the well-known Rellich inequality [32]

$$ \int_{\mathbb{R}^{N}} \vert \Delta u \vert ^{2}\,dx\geq \overline{\mu} \int_{\mathbb{R}^{N}}\frac{u^{2}}{ \vert x \vert ^{4}}\,dx, \quad\forall u\in \mathscr{D}^{2, 2} \bigl(\mathbb{R}^{N} \bigr), $$

(2.1)

where $N\geq5$, $\overline{\mu}= \frac{1}{16}N^{2}(N-4)^{2}$. We now employ the following norm in $\mathscr{D}^{2, 2}(\mathbb{R}^{N})$:

$$\Vert u \Vert _{\mu}\triangleq \biggl[ \int_{\mathbb {R}^{N}} \bigl( \vert \Delta u \vert ^{2}-\mu \vert x \vert ^{-4}u^{2} \bigr)\,dx \biggr]^{\frac{1}{2}},\quad 0\leq\mu< \overline{\mu}. $$

Thanks to the Rellich inequality (2.1), we find that the above norm $\Vert \cdot \Vert _{\mu}$ is equivalent to the usual norm $(\int_{\mathbb{R}^{N}} \vert \Delta\cdot \vert ^{2}\,dx)^{1/2}$. Besides, we define the product space $(\mathscr{D}^{2, 2}(\mathbb{R}^{N}))^{2}$ endowed with the norm

$$ \bigl\Vert (u, v) \bigr\Vert _{\mu}= \bigl( \Vert u \Vert _{\mu }^{2}+ \Vert v \Vert _{\mu}^{2} \bigr)^{\frac{1}{2}}, \quad\forall(u, v)\in \bigl(\mathscr{D}^{2, 2} \bigl( \mathbb{R}^{N} \bigr) \bigr)^{2}. $$

(2.2)

As usual, we denote by G any closed subgroup of $O(\mathbb{N})$, the group of orthogonal linear transformations. Let $G_{x}=\{gx; g\in G\}$ be the orbit of $x\in\mathbb{R}^{N}$; $\vert G_{x} \vert$ denote the number of elements in $G_{x}$ and $\vert G_{0} \vert = \vert G_{\infty} \vert =1$. Denote $\vert G \vert =\inf_{x\in\mathbb{R}^{N}\backslash\{0\}} \vert G_{x} \vert$. Note that $\vert G \vert$ may be +∞. We call Ω a G-invariant subset of $\mathbb{R}^{N}$, if $x\in\Omega$, then $gx\in\Omega$ for all $g\in G$. A function $f: \mathbb{R}^{N}\mapsto\mathbb{R}$ is called G-invariant if $f(gx)=f(x)$ for every $g\in G$ and $x\in\mathbb{R}^{N}$. In particular, an $O(\mathbb{N})$-invariant function is called radial.

The natural functional space to frame the analysis of (1.1) by variational methods is the Hilbert space $(\mathscr{D}_{ G}^{2, 2}(\mathbb{R}^{N}))^{2}$, which is the subspace of $(\mathscr{D}^{2, 2}(\mathbb{R}^{N}))^{2}$ consisting of all G-invariant functions. This work is devoted to the study of the following systems:

$$\bigl(\mathscr{P}^{Q}_{\sigma} \bigr) \textstyle\begin{cases} \Delta^{2} u=\mu\frac{u}{ \vert x \vert ^{4}}+Q(x) \sum_{i=1}^{m}\frac{\varsigma_{i}\alpha_{i}}{2^{\ast\ast}} \vert u \vert ^{\alpha_{i}-2}u \vert v \vert ^{\beta_{i}}+\sigma h(x) \vert u \vert ^{q-2}u, &\text{in } \mathbb{R}^{N},\\ \Delta^{2} v=\mu\frac{v}{ \vert x \vert ^{4}}+Q(x)\sum_{i=1}^{m} \frac{\varsigma_{i}\beta_{i}}{2^{\ast\ast}} \vert u \vert ^{\alpha_{i}} \vert v \vert ^{\beta_{i}-2}v+\sigma h(x) \vert v \vert ^{q-2}v, &\text{in } \mathbb{R}^{N},\\ (u, v)\in (\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}) )^{2}\quad\text{and}\quad u, v\not\equiv 0, &\text{in } \mathbb{R}^{N}. \end{cases} $$

To clearly describe the results of this paper, several notations should be presented:

$$\begin{aligned} & \mathcal{A}_{\mu}\triangleq\inf_{u\in\mathscr{D}^{2, 2}(\mathbb{R}^{N})\backslash\{0\}} \frac{\int_{\mathbb{R}^{N}} ( \vert \Delta u \vert ^{2}-\mu\frac{u^{2}}{ \vert x \vert ^{4}} )\,dx}{ (\int_{\mathbb{R}^{N}} \vert u \vert ^{2^{\ast\ast}}\,dx ) ^{\frac{2}{2^{\ast\ast}}}}, \end{aligned}$$

(2.3)

$$\begin{aligned} & y_{\epsilon}(x)\triangleq C\epsilon^{-\Lambda_{0}}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr), \end{aligned}$$

(2.4)

where $\epsilon>0$, $\Lambda_{0}\triangleq\frac{N-4}{2}$, and the constant $C=C(N, \mu)>0$, depending only on N and μ. From [26, 33], we mention that $y_{\epsilon}(x)$ satisfies the following equations:

$$ \int_{\mathbb{R}^{N}} \biggl( \vert \Delta y_{\epsilon} \vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{ \vert x \vert ^{4}} \biggr)\,dx=1 $$

(2.5)

and

$$\int_{\mathbb{R}^{N}}y_{\epsilon}^{2^{\ast\ast}-1}\varphi \,dx= \mathcal{A}_{\mu}^{-\frac{2^{\ast\ast}}{2}} \int_{\mathbb {R}^{N}} \biggl(\Delta y_{\epsilon}\Delta\varphi-\mu \frac{y_{\epsilon}\varphi }{ \vert x \vert ^{4}} \biggr)\,dx $$

for all $\varphi\in\mathscr{D}^{2, 2}(\mathbb{R}^{N})$. Hence, we obtain (let $\varphi=y_{\epsilon}$)

$$ \int_{\mathbb{R}^{N}}y_{\epsilon}^{2^{\ast\ast}} \,dx= \mathcal{A}_{\mu}^{-\frac{2^{\ast\ast}}{2}}. $$

(2.6)

According to [26, Lemma 2.1] and [33, Theorem 2], we remark that the function $U_{\mu}(x)$ in (2.4) is positive, radial symmetric, radially decreasing, and solves

$$\textstyle\begin{cases} \Delta^{2}u=\mu\frac{u}{ \vert x \vert ^{4}}+ u^{2^{\ast\ast}-1}, &\text{in } \mathbb{R}^{N}\backslash\{0\}, \\ u\in\mathscr{D}^{2, 2}(\mathbb{R}^{N})\quad \text{and}\quad u>0, &\text{in } \mathbb{R}^{N}\backslash\{0\}. \end{cases} $$

By setting $r= \vert x \vert$, there holds that

$$\begin{aligned} &U_{\mu}(r)=O_{1} \bigl(r^{-l_{1}(\mu)} \bigr), \quad\text{as } r\rightarrow0, \end{aligned}$$

(2.7)

$$\begin{aligned} &U_{\mu}(r)=O_{1} \bigl(r^{-l_{2}(\mu)} \bigr),\qquad U_{\mu}^{\prime}(r)=O_{1} \bigl(r^{-l_{2}(\mu)-1} \bigr),\quad \text{as } r\rightarrow+\infty, \end{aligned}$$

(2.8)

where $O_{1}(r^{t})$ $(r\rightarrow r_{0})$ means that there exist constants $C_{1}$, $C_{2}>0$ such that $C_{1}r^{t}\leq O_{1}(r^{t})\leq C_{2}r^{t}$ as $r\rightarrow r_{0}$, $l_{1}(\mu)\triangleq\Lambda_{0}\vartheta(\mu)$, $l_{2}(\mu)\triangleq\Lambda_{0}(2-\vartheta(\mu))$, $\Lambda_{0}=\frac{N-4}{2}$, and $\vartheta(\mu): [0, \overline{\mu}]\mapsto[0, 1]$ is defined as

$$\vartheta(\mu)\triangleq 1-\frac{\sqrt{N^{2}-4N+8-4\sqrt{(N-2)^{2}+\mu}}}{N-4}. $$

This implies $\vartheta(0)=0$, $\vartheta(\overline{\mu})=1$ and

$$ 0\leq l_{1}(\mu)< \Lambda_{0}< l_{2}( \mu)\leq2\Lambda_{0},\quad \forall\mu\in[0, \overline{\mu}). $$

(2.9)

Moreover, there exist positive constants $C_{3}=C_{3}(N, \mu)$ and $C_{4}=C_{4}(N, \mu)$ such that

$$ 0< C_{3}\leq U_{\mu}(x) \bigl( \vert x \vert ^{\frac{l_{1}(\mu)}{\Lambda_{0}}} + \vert x \vert ^{\frac{l_{2}(\mu)}{\Lambda_{0}}} \bigr)^{\Lambda_{0}} \leq C_{4}, \quad\forall x\in\mathbb{R}^{N}\backslash\{0\}. $$

(2.10)

The following hypotheses are needed.

• (q.1) $Q(x)$ is G-invariant.

• (q.2) $Q(x)\in\mathscr{C}(\mathbb{R}^{N})\cap L^{\infty}(\mathbb{R}^{N})$, and $Q_{+}(x)\not\equiv0$, where $Q_{+}(x)=\max\{0, Q(x)\}$.

• (h.1) $h(x)$ is G-invariant.

• (h.2) $h(x)$ is a nonnegative function in $\mathbb{R}^{N}$ such that

  $$0< \Vert h \Vert _{\theta}\triangleq \biggl( \int_{\mathbb{R}^{N}} h^{\theta}(x)\,dx \biggr)^{\frac{1}{\theta}}< +\infty \quad\text{with } \theta=\frac{2^{\ast\ast}}{2^{\ast\ast}-q}. $$

The main results of this work can be stated in the following.

Theorem 2.1

Assume that (q.1) and (q.2) hold. If

$$ \int_{\mathbb{R}^{N}}Q(x)y_{\epsilon}^{2^{\ast\ast}}\,dx\geq \max \bigl\{ \vert G \vert ^{\frac{2-2^{\ast\ast}}{2}} \mathcal{A}_{0}^{-\frac{2^{\ast\ast}}{2}} \Vert Q_{+} \Vert _{\infty}, \mathcal{A}_{\mu}^{-\frac{2^{\ast\ast}}{2}}Q_{+}(0), \mathcal{A}_{\mu}^{-\frac{2^{\ast\ast}}{2}}Q_{+}(\infty) \bigr\} >0 $$

(2.11)

for certain $\epsilon>0$, where $Q_{+}(\infty)=\limsup_{ \vert x \vert \rightarrow\infty}Q_{+}(x)$, then problem $(\mathscr{P}^{Q}_{0})$ possesses at least one nontrivial solution in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$.

Corollary 2.1

Assume that (q.1) and (q.2) hold. Then we have the following statements.

• (1) Problem $(\mathscr{P}^{Q}_{0})$ admits at least one nontrivial solution if

  $$Q(0)>0, \quad Q(0)\geq\max \bigl\{ \vert G \vert ^{\frac{2-2^{\ast\ast}}{2}} ( \mathcal{A}_{0}/\mathcal{A}_{\mu} )^{-\frac{2^{\ast\ast}}{2}} \Vert Q_{+} \Vert _{\infty}, Q_{+}(\infty) \bigr\} , $$

  and either (i) $Q(x)\geq Q(0)+\xi_{0} \vert x \vert ^{2^{\ast\ast}(l_{2}(\mu)-\Lambda_{0})}$ for some $\xi_{0}>0$ and $\vert x \vert$ small, or (ii) $\vert Q(x)-Q(0) \vert \leq \xi_{1} \vert x \vert ^{\varsigma}$ for some constants $\xi_{1}>0$, $\varsigma>2^{\ast\ast}(l_{2}(\mu)-\Lambda_{0})>0$ and $\vert x \vert$ small and

  $$ \int_{\mathbb{R}^{N}} \bigl(Q(x)-Q(0) \bigr) \vert x \vert ^{-2^{\ast\ast}l_{2}(\mu)}\,dx>0. $$

  (2.12)

• (2) Problem $(\mathscr{P}^{Q}_{0})$ has at least one nontrivial solution if $\lim_{ \vert x \vert \rightarrow\infty}Q(x)=Q(\infty)$ exists and is positive,

  $$Q(\infty)\geq\max \bigl\{ \vert G \vert ^{\frac{2-2^{\ast\ast}}{2}} ( \mathcal{A}_{0}/\mathcal{A}_{\mu} )^{-\frac{2^{\ast\ast}}{2}} \Vert Q_{+} \Vert _{\infty}, Q_{+}(0) \bigr\} , $$

  and either (i) $Q(x)\geq Q(\infty)+\xi_{2} \vert x \vert ^{-2^{\ast\ast}(\Lambda_{0}-l_{1}(\mu))}$ for certain $\xi_{2}>0$ and large $\vert x \vert$, or (ii) $\vert Q(x)-Q(\infty) \vert \leq\xi_{3} \vert x \vert ^{-\kappa}$ for some constants $\xi_{3}>0$, $\kappa>2^{\ast\ast}(\Lambda_{0}-l_{1}(\mu))>0$ and large $\vert x \vert$ and

  $$ \int_{\mathbb{R}^{N}} \bigl(Q(x)-Q(\infty) \bigr) \vert x \vert ^{-2^{\ast\ast}l_{1}(\mu)}\,dx>0. $$

  (2.13)

• (3) If $Q(x)\geq Q(\infty)=Q(0)>0$ on $\mathbb{R}^{N}$ and

  $$Q(\infty)=Q(0)\geq \vert G \vert ^{\frac{2-2^{\ast\ast}}{2}} (\mathcal{A}_{0}/ \mathcal{A}_{\mu} )^{-\frac{2^{\ast\ast}}{2}} \Vert Q_{+} \Vert _{\infty}, $$

  then problem $(\mathscr{P}^{Q}_{0})$ possesses at least one nontrivial solution.

Remark 2.1

Conditions (q.1) and (q.2) are essentially introduced in [9]. According to (q.2), we only presume that $Q(x)$ is bounded and continuous on $\mathbb{R}^{N}$. Hence, the above results do not require the continuity of $Q(x)$ at infinity.

Theorem 2.2

Assume that $\vert G \vert =+\infty$ and $Q_{+}(0)=Q_{+}(\infty)=0$. Then there exist infinitely many G-invariant solutions to problem $(\mathscr{P}^{Q}_{0})$.

Corollary 2.2

If Q is a radial function such that $Q_{+}(0)=Q_{+}(\infty)=0$, then there exist infinitely many radial solutions to problem $(\mathscr{P}^{Q}_{0})$.

Theorem 2.3

Let $\overline{Q}>0$ be a constant. Assume that $Q(x)\equiv\overline{Q}$ and (h.1), (h.2) hold. Then there exists $\sigma^{\ast}>0$ such that, for any $\sigma\in(0, \sigma^{\ast})$, problem $(\mathscr{P}^{\overline{Q}}_{\sigma})$ possesses at least two nontrivial solutions in $(\mathscr{D}_{ G}^{2, 2}(\mathbb{R}^{N}))^{2}$.

Remark 2.2

The main results of this paper extend and complement those of [4, 5, 26, 29, 30]. Even in the scalar cases $\sigma=0$, $0<\mu<\overline{\mu}$, $m=1$, and $u=v$, the above results in the whole space are new.

Throughout this paper, we denote various positive constants as $C_{i}\ (i=1, 2, \ldots)$ or C. The dual space of $(\mathscr {D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ ($(\mathscr{D}^{2, 2}(\mathbb{R}^{N}))^{2}$, resp.) is denoted by $(\mathscr{D}_{ G}^{-2, 2}(\mathbb{R}^{N}))^{2}$ ($(\mathscr{D}^{-2, 2}(\mathbb{R}^{N}))^{2}$, resp.). The ball of center x and radius r is denoted by $B_{r}(x)$. $o_{n}(1)$ is a generic infinitesimal value as $n\rightarrow\infty$. For any $\epsilon>0$, $t\in\mathbb{R}$, $O(\epsilon^{t})$ denotes the quantity satisfying $\vert O(\epsilon^{t}) \vert /\epsilon^{t}\leq C$, and $O_{1}(\epsilon^{t})$ $(\epsilon\rightarrow\epsilon_{0})$ means that there exist constants $C_{1}$, $C_{2}>0$ such that $C_{1}\epsilon^{t}\leq O_{1}(\epsilon^{t})\leq C_{2}\epsilon^{t}$ as $\epsilon\rightarrow\epsilon_{0}$. In a Banach space X, we denote by ‘→’ and ‘⇀’ strong and weak convergence, respectively. A functional $\mathcal{F}\in \mathscr{C}^{1}(X, \mathbb{R})$ is called to satisfy the $(PS)_{c}$ condition if each sequence $\{w_{n}\}$ in X satisfying $\mathcal{F}(w_{n})\rightarrow c$ in $\mathbb{R}$, $\mathcal{F}^{\prime}(w_{n})\rightarrow0$ in $X^{\ast}$ contains a strongly convergent subsequence.

Existence and multiplicity results for problem $(\mathscr {P}^{Q}_{0})$

The energy functional corresponding to problem $(\mathscr{P}^{Q}_{0})$ is defined on $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ by

$$ \mathcal{F}(u, v)=\frac{1}{2} \bigl\Vert (u, v) \bigr\Vert _{\mu}^{2} -\frac{1}{2^{\ast\ast}} \int_{\mathbb{R}^{N}} Q(x)\sum_{i=1}^{m} \varsigma_{i} \vert u \vert ^{\alpha_{i}} \vert v \vert ^{\beta_{i}}\,dx. $$

(3.1)

It follows from (q.2) and the Rellich inequality (2.1) that $\mathcal{F}$ is a well-defined $\mathscr{C}^{1}$ functional on $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$. Then the critical points of $\mathcal{F}$ correspond to weak solutions of problem $(\mathscr{P}^{Q}_{0})$. According to the principle of symmetric criticality (see Lemma 3.1), any critical point of $\mathcal{F}$ in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ is also a solution of $(\mathscr{P}^{Q}_{0})$ in $(\mathscr {D}^{2, 2}(\mathbb{R}^{N}))^{2}$. This means that $(u, v)\in (\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ satisfies $(\mathscr{P}^{Q}_{0})$ if and only if, for any $(\varphi_{1}, \varphi_{2})\in(\mathscr{D}^{2, 2}(\mathbb{R}^{N}))^{2}$,

$$\begin{aligned} & \bigl\langle \mathcal{F}^{\prime}(u, v), ( \varphi_{1}, \varphi_{2}) \bigr\rangle \\ &\quad= \int_{\mathbb{R}^{N}} \biggl(\Delta u\Delta\varphi_{1}+\Delta v \Delta\varphi_{2}-\mu\frac{u\varphi_{1} +v\varphi_{2}}{ \vert x \vert ^{4}} \biggr)\,dx \\ &\qquad-\frac{1}{2^{\ast\ast}} \int_{\mathbb{R}^{N}}Q(x) \Biggl(\varphi_{1}\sum _{i=1}^{m} \varsigma_{i} \alpha_{i} \vert u \vert ^{\alpha_{i}-2}u \vert v \vert ^{\beta_{i}} +\varphi_{2}\sum_{i=1}^{m} \varsigma_{i}\beta_{i} \vert u \vert ^{\alpha_{i}} \vert v \vert ^{\beta_{i}-2}v \Biggr)\,dx=0. \end{aligned}$$

(3.2)

Lemma 3.1

If $Q(x)$ is a G-invariant function, then $\mathcal{F}^{\prime}(u, v)=0$ in $(\mathscr{D}_{G}^{-2, 2}(\mathbb{R}^{N}))^{2}$ implies $\mathcal{F}^{\prime}(u, v)=0$ in $(\mathscr{D}^{-2, 2}(\mathbb{R}^{N}))^{2}$.

Proof

The proof is similar to that of [9, Lemma 1] and is omitted here. □

For $\mu\in[0, \overline{\mu})$, $\varsigma_{i}\in(0, +\infty)$, $\alpha_{i}$, $\beta_{i}>1$, and $\alpha_{i}+\beta_{i}=2^{\ast\ast}\ (i=1, \ldots, m)$, we define

$$\begin{aligned} &\mathcal{A}_{\mu, m}\triangleq\inf_{(u, v)\in (\mathscr{D}^{2, 2}(\mathbb{R}^{N})\backslash\{0\})^{2}} \frac{\int_{\mathbb{R}^{N}} ( \vert \Delta u \vert ^{2}+ \vert \Delta v \vert ^{2}-\mu \frac{u^{2}+v^{2}}{ \vert x \vert ^{4}} )\,dx}{ (\int_{\mathbb{R}^{N}}\sum_{i=1}^{m}\varsigma_{i} \vert u \vert ^{\alpha_{i}} \vert v \vert ^{\beta_{i}}\,dx )^{\frac{2}{2^{\ast\ast}}}}, \end{aligned}$$

(3.3)

$$\begin{aligned} & \mathscr{B}(\tau)\triangleq\frac{1+\tau^{2}}{ (\sum_{i=1}^{m}\varsigma_{i}\tau^{\beta_{i}} )^{\frac{2}{2^{\ast\ast}}}},\quad \tau\geq0, \end{aligned}$$

(3.4)

$$\begin{aligned} &\mathscr{B}(\tau_{\min})\triangleq\min _{\tau\geq 0}\mathscr{B}(\tau)>0, \end{aligned}$$

(3.5)

where $\tau_{\min}>0$ is a minimal point of $\mathscr{B}(\tau)$ and hence a root of the equation

$$ \sum_{i=1}^{m} \varsigma_{i}\tau^{\beta_{i}-1} \bigl(\alpha_{i}\tau ^{2}-\beta_{i} \bigr) =0,\quad \tau\geq0. $$

(3.6)

Lemma 3.2

Let $y_{\epsilon}(x)$ be the minimizer of $\mathcal{A}_{\mu}$ defined in (2.4), $\mu\in[0, \overline{\mu})$, $\varsigma_{i}\in(0, +\infty)$, $\alpha_{i}$, $\beta_{i}>1$, and $\alpha_{i}+\beta_{i}=2^{\ast\ast}\ (i=1, \ldots, m)$. Then we have the following statements.

• (i) $\mathcal{A}_{\mu, m}=\mathscr{B}(\tau_{\min})\mathcal{A}_{\mu}$;

• (ii) $\mathcal{A}_{\mu, m}$ has the minimizer $(y_{\epsilon}(x), \tau_{\min}y_{\epsilon}(x))$ for all $\epsilon>0$.

Proof

The proof is a repeat of that in [19, Theorem 2.2] (see also [21, Theorem 5]) and hence is omitted here. □

To find conditions under which the Palais–Smale condition holds, we need the following concentration compactness principle due to Lions [34].

Lemma 3.3

Let $\{(u_{n}, v_{n})\}$ be a weakly convergent sequence to $(u, v)$ in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ such that $\vert \Delta u_{n} \vert ^{2}\rightharpoonup\eta^{(1)}$, $\vert \Delta v_{n} \vert ^{2}\rightharpoonup\eta^{(2)}$, $\vert u_{n} \vert ^{\alpha_{i}} \vert v_{n} \vert ^{\beta_{i}} \rightharpoonup\nu^{(i)}\ (i=1, \ldots, m)$, $\vert x \vert ^{-4} \vert u_{n} \vert ^{2}\rightharpoonup\gamma^{(1)}$, and $\vert x \vert ^{-4} \vert v_{n} \vert ^{2}\rightharpoonup\gamma^{(2)}$ in the sense of measures. Then there exists some at most countable set $\mathscr{J}$, $\{\eta_{j}^{(1)}\geq 0\}_{j\in\mathscr{J}\cup\{0\}}$, $\{\eta_{j}^{(2)}\geq 0\}_{j\in\mathscr{J}\cup\{0\}}$, $\{\nu_{j}^{(i)}\geq 0\}_{j\in\mathscr{J}\cup\{0\}}$, $\gamma_{0}^{(1)}\geq0$, $\gamma_{0}^{(2)}\geq0$, $\{x_{j}\}_{j\in\mathscr{J}}\subset\mathbb{R}^{N}\backslash\{0\}$ such that

• (a) $\eta^{(1)}\geq \vert \Delta u \vert ^{2}+ \sum_{j\in\mathscr {J}}\eta_{j}^{(1)}\delta_{x_{j}}+\eta_{0}^{(1)}\delta_{0}$, $\eta^{(2)}\geq \vert \Delta v \vert ^{2}+ \sum_{j\in\mathscr {J}}\eta_{j}^{(2)}\delta_{x_{j}}+\eta_{0}^{(2)}\delta_{0}$,

• (b) $\nu^{(i)}= \vert u \vert ^{\alpha_{i}} \vert v \vert ^{\beta_{i}}+ \sum_{j\in\mathscr {J}}\nu_{j}^{(i)}\delta_{x_{j}}+\nu_{0}^{(i)}\delta_{0}$, $i=1, \ldots, m$,

• (c) $\gamma^{(1)}= \vert x \vert ^{-4} \vert u \vert ^{2}+\gamma_{0}^{(1)}\delta_{0}$, $\gamma^{(2)}= \vert x \vert ^{-4} \vert v \vert ^{2}+\gamma_{0}^{(2)}\delta_{0}$,

• (d) $\mathcal{A}_{0, m} (\sum_{i=1}^{m}\varsigma_{i}\nu_{j}^{(i)} )^{\frac{2}{2^{\ast\ast}}}\leq\eta_{j}^{(1)}+\eta_{j}^{(2)}$,

• (e) $\mathcal{A}_{\mu, m} (\sum_{i=1}^{m}\varsigma_{i}\nu_{0}^{(i)} )^{\frac{2}{2^{\ast\ast}}}\leq\eta_{0}^{(1)}+\eta_{0}^{(2)} -\mu(\gamma_{0}^{(1)}+\gamma_{0}^{(2)})$,

where $\delta_{x_{j}}$, $j\in\mathscr{J}\cup \{0 \}$, is a Dirac mass of 1 concentrated at $x_{j}\in\mathbb{R}^{N}$.

To establish the existence results for problem $(\mathscr{P}^{Q}_{0})$, we need the following local $(PS)_{c}$ condition, which is indispensable for the proof of Theorem 2.1.

Lemma 3.4

Assume that (q.1) and (q.2) hold. Then the $(PS)_{c}$ condition in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ holds for $\mathcal{F}$ if

$$ c< c_{0}^{\ast}\triangleq\frac{2}{N} \min \bigl\{ \vert G \vert \mathcal{A}_{0, m}^{\frac{N}{4}} \Vert Q_{+} \Vert _{\infty}^{1-\frac{N}{4}}, \mathcal{A}_{\mu, m}^{\frac{N}{4}}Q_{+}(0)^{1-\frac{N}{4}}, \mathcal{A}_{\mu, m}^{\frac{N}{4}}Q_{+}(\infty)^{1-\frac{N}{4}} \bigr\} . $$

(3.7)

Proof

We follow closely the arguments in [9, Proposition 2]. It is trivial to check that the $(PS)_{c}$ sequence $\{(u_{n}, v_{n})\}$ of $\mathcal{F}$ is bounded in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$. Then we may assume that $(u_{n}, v_{n})\rightharpoonup(u, v)$ in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$. In view of Lemma 3.3, there exist measures $\eta^{(1)}$, $\eta^{(2)}$, $\nu^{(i)}\ (i=1, \ldots, m)$, $\gamma^{(1)}$, and $\gamma^{(2)}$ such that relations (a)–(e) of this lemma hold. We begin by considering the concentration at the point $x_{j}\in\mathbb{R}^{N}\backslash\{0\}$, $j\in\mathscr{J}$. For $\epsilon>0$ small, we define the cut-off function $\psi_{x_{j}}^{\epsilon}(x)\in\mathscr {C}_{0}^{\infty}(\mathbb{R}^{N})$ such that $0\leq\psi_{x_{j}}^{\epsilon}(x)\leq1$, $\psi_{x_{j}}^{\epsilon}(x)=1$ in $B_{\epsilon}(x_{j})$, $\psi_{x_{j}}^{\epsilon}(x)=0$ on $\mathbb{R}^{N}\backslash B_{2\epsilon}(x_{j})$, $\vert \nabla\psi_{x_{j}}^{\epsilon} \vert \leq 2/\epsilon$, and $\vert \Delta\psi_{x_{j}}^{\epsilon} \vert \leq 2/\epsilon^{2}$ on $\mathbb{R}^{N}$. Then, by Lemma 3.1, $\lim_{n\rightarrow\infty}\langle\mathcal{F}^{\prime}(u_{n}, v_{n}), (u_{n}\psi_{x_{j}}^{\epsilon}, v_{n}\psi_{x_{j}}^{\epsilon})\rangle=0$; hence, combining (3.2), the Hölder inequality, and the Sobolev inequality, we derive

$$\begin{aligned} & \int_{\mathbb{R}^{N}} \psi_{x_{j}}^{\epsilon} \Biggl\{ d \eta^{(1)}+d\eta^{(2)} -\mu \bigl(d\gamma^{(1)}+d \gamma^{(2)} \bigr) -Q(x) \sum_{i=1}^{m} \frac{\varsigma_{i}}{2^{\ast\ast}} (\alpha_{i}+\beta_{i} )\,d \nu^{(i)} \Biggr\} \\ &\quad\leq\overline{\lim_{n\rightarrow\infty}} \int_{\mathbb{R}^{N}} \bigl\{ 2 \bigl\vert \Delta u_{n} \bigl\langle \nabla u_{n}, \nabla\psi_{x_{j}}^{\epsilon} \bigr\rangle +\Delta v_{n} \bigl\langle \nabla v_{n}, \nabla \psi_{x_{j}}^{\epsilon} \bigr\rangle \bigr\vert + \bigl\vert (u_{n}\Delta u_{n}+v_{n}\Delta v_{n} )\Delta\psi_{x_{j}}^{\epsilon} \bigr\vert \bigr\} \,dx \\ &\quad\leq\sup_{n\geq 1} \biggl( \int_{\mathbb{R}^{N}} \vert \Delta u_{n} \vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggl[2\overline{\lim _{n\rightarrow \infty}} \biggl( \int_{\mathbb{R}^{N}} \vert \nabla u_{n} \vert ^{2} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \\ &\qquad{} +\overline{\lim _{n\rightarrow\infty}} \biggl( \int_{\mathbb{R}^{N}} \vert u_{n} \vert ^{2} \bigl\vert \Delta\psi_{x_{j}} ^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggr] \\ &\qquad{} +\sup_{n\geq1} \biggl( \int_{\mathbb{R}^{N}} \vert \Delta v_{n} \vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggl[2\overline{\lim _{n\rightarrow \infty}} \biggl( \int_{\mathbb{R}^{N}} \vert \nabla v_{n} \vert ^{2} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \\ &\qquad{} +\overline{\lim _{n\rightarrow\infty}} \biggl( \int_{\mathbb{R}^{N}} \vert v_{n} \vert ^{2} \bigl\vert \Delta\psi_{x_{j}} ^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggr] \\ &\quad\leq C \biggl\{ \biggl( \int_{\mathbb{R}^{N}} \vert \nabla u \vert ^{2} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac {1}{2}}+ \biggl( \int_{\mathbb{R}^{N}} \vert u \vert ^{2} \bigl\vert \Delta \psi_{x_{j}}^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac {1}{2}}+ \biggl( \int_{\mathbb{R}^{N}} \vert v \vert ^{2} \bigl\vert \Delta \psi_{x_{j}}^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \\ &\qquad{} + \biggl( \int_{\mathbb{R}^{N}} \vert \nabla v \vert ^{2} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggr\} \leq C \biggl\{ \biggl( \int_{B_{2\epsilon}(x_{j})} \vert \nabla u \vert ^{\frac{2N}{N-2}}\,dx \biggr)^{\frac{N-2}{2N}} \biggl( \int_{\mathbb{R}^{N}} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{N}\,dx \biggr)^{\frac{1}{N}} \\ &\qquad{} + \biggl( \int_{B_{2\epsilon}(x_{j})} \vert u \vert ^{2^{\ast\ast}}\,dx \biggr)^{\frac{1}{2^{\ast\ast}}} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert \Delta\psi_{x_{j}}^{\epsilon} \bigr\vert ^{\frac{N}{2}} \biggr) ^{\frac{2}{N}} \\ &\qquad{}+ \biggl( \int_{B_{2\epsilon}(x_{j})} \vert v \vert ^{2^{\ast\ast}}\,dx \biggr)^{\frac{1}{2^{\ast\ast}}} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert \Delta\psi_{x_{j}}^{\epsilon} \bigr\vert ^{\frac{N}{2}} \biggr)^{\frac{2}{N}} \\ &\qquad{} + \biggl( \int_{B_{2\epsilon}(x_{j})} \vert \nabla v \vert ^{\frac{2N}{N-2}}\,dx \biggr)^{\frac{N-2}{2N}} \biggl( \int_{\mathbb{R}^{N}} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{N}\,dx \biggr)^{\frac{1}{N}} \biggr\} \leq C \biggl\{ \biggl( \int_{B_{2\epsilon}(x_{j})} \vert \nabla u \vert ^{\frac{2N}{N-2}}\,dx \biggr)^{\frac{N-2}{2N}} \\ &\qquad{} + \biggl( \int_{B_{2\epsilon}(x_{j})} \vert \Delta u \vert ^{2}\,dx \biggr)^{\frac{1}{2}}+ \biggl( \int_{B_{2\epsilon}(x_{j})} \vert \Delta v \vert ^{2}\,dx \biggr)^{\frac{1}{2}}+ \biggl( \int_{B_{2\epsilon}(x_{j})} \vert \nabla v \vert ^{\frac{2N}{N-2}}\,dx \biggr)^{\frac{N-2}{2N}} \biggr\} . \end{aligned}$$

(3.8)

As $\epsilon\rightarrow0$, it follows from (3.8) and Lemma 3.3 that

$$ Q(x_{j})\sum_{i=1}^{m} \varsigma_{i}\nu_{j}^{(i)}\geq \eta_{j}^{(1)}+\eta_{j}^{(2)}. $$

(3.9)

This means that the concentration of the measures $\nu^{(i)}\ (i=1, \ldots, m)$ cannot occur at points where $Q(x_{j})\leq0$. By virtue of (3.9) and (d) of Lemma 3.3, we conclude that either (i) $\nu_{j}^{(i)}=0\ (i=1, \ldots, m)$ or (ii) $\sum_{i=1}^{m}\varsigma_{i}\nu_{j}^{(i)}\geq (\mathcal{A}_{0, m}/ \Vert Q_{+} \Vert _{\infty})^{N/4}$. Let us now study the possibility of concentration at $x=0$ and at ∞. By the argument similar to that of $x_{j}\in \mathbb{R}^{N}\backslash\{0\}$, we find $\eta_{0}^{(1)}+\eta_{0}^{(2)} -\mu(\gamma_{0}^{(1)}+\gamma_{0}^{(2)}) -Q(0)\sum_{i=1}^{m}\varsigma_{i}\nu_{0}^{(i)} \leq0$. Together with (e) of Lemma 3.3, it follows that either (iii) $\nu_{0}^{(i)} =0\ (i=1, \ldots, m)$ or (iv) $\sum_{i=1}^{m}\varsigma_{i}\nu_{0}^{(i)}\geq (\mathcal{A}_{\mu, m}/Q_{+}(0))^{N/4}$. To discuss the concentration at infinity of the sequence $\{(u_{n}, v_{n})\}$, we define the following quantities:

• (1) $\eta_{\infty}^{(1)}=\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}}\int_{ \vert x \vert >R} \vert \Delta u_{n} \vert ^{2}\,dx$, $\eta_{\infty}^{(2)}=\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}}\int_{ \vert x \vert >R} \vert \Delta v_{n} \vert ^{2}\,dx$,

• (2) $\nu_{\infty}^{(i)}= \lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}}\int_{ \vert x \vert >R} \vert u_{n} \vert ^{\alpha_{i}} \vert v_{n} \vert ^{\beta_{i}}\,dx$, $i=1, \ldots, m$,

• (3) $\gamma_{\infty}^{(1)}=\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}}\int_{ \vert x \vert >R} \vert x \vert ^{-4} \vert u_{n} \vert ^{2}\,dx$, $\gamma_{\infty}^{(2)}=\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}}\int_{ \vert x \vert >R} \vert x \vert ^{-4} \vert v_{n} \vert ^{2}\,dx$.

It is obvious that $\eta_{\infty}^{(1)}$, $\eta_{\infty}^{(2)}$, $\nu_{\infty}^{(i)}\ (i=1, \ldots, m)$, $\gamma_{\infty}^{(1)}$, and $\gamma_{\infty}^{(2)}$ defined by (1)–(3) exist and are finite. For $R>1$, let $\psi_{R}(x)\in\mathscr {C}^{\infty}(\mathbb{R}^{N})$ be a function such that $0\leq\psi_{R}(x)\leq1$, $\psi_{R}(x)=1$ for $\vert x \vert >R+1$, $\psi_{R}(x)=0$ for $\vert x \vert < R$, $\vert \nabla\psi_{R} \vert \leq2/R$, and $\vert \Delta\psi_{R} \vert \leq2/R^{2}$. Because the sequence $\{(u_{n}\psi_{R}, v_{n}\psi_{R})\}$ is bounded in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$, we deduce from (3.2) and the fact that $\alpha_{i}+\beta_{i}=2^{\ast\ast}\ (i=1, \ldots, m)$ that

$$\begin{aligned} 0={}&\lim_{n\rightarrow\infty} \bigl\langle \mathcal{F}^{\prime}(u_{n}, v_{n}), (u_{n}\psi_{R}, v_{n} \psi_{R}) \bigr\rangle \\ ={}&\lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \Biggl\{ \Biggl( \vert \Delta u_{n} \vert ^{2}+ \vert \Delta v_{n} \vert ^{2}- \mu\frac{ \vert u_{n} \vert ^{2} + \vert v_{n} \vert ^{2}}{ \vert x \vert ^{4}}-Q(x)\sum_{i=1}^{m} \varsigma_{i} \vert u_{n} \vert ^{\alpha_{i}} \vert v_{n} \vert ^{\beta_{i}} \Biggr)\psi_{R} \\ &{} + \bigl(2\Delta u_{n}\langle\nabla u_{n}, \nabla \psi_{R}\rangle+u_{n}\Delta u_{n}\Delta \psi_{R}+2\Delta v_{n}\langle\nabla v_{n}, \nabla\psi_{R}\rangle+v_{n}\Delta v_{n}\Delta \psi_{R} \bigr) \Biggr\} \,dx. \end{aligned}$$

(3.10)

Furthermore, by utilizing the Hölder inequality and the Sobolev inequality, we obtain

$$\begin{aligned} &\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}} \int_{\mathbb{R}^{N}} \bigl(2 \bigl\vert \Delta u_{n}\langle \nabla u_{n}, \nabla \psi_{R}\rangle \bigr\vert + \vert u_{n}\Delta u_{n}\Delta\psi_{R} \vert \bigr)\,dx \\ &\quad\leq\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}} \biggl( \int_{\mathbb {R}^{N}} \vert \Delta u_{n} \vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggl[2 \biggl( \int_{\mathbb{R}^{N}} \vert \nabla u_{n} \vert ^{2} \vert \nabla\psi_{R} \vert ^{2}\,dx \biggr)^{\frac{1}{2}}\\ &\qquad{}+ \biggl( \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2} \vert \Delta\psi_{R} \vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggr] \\ &\quad\leq C\lim_{R\rightarrow \infty} \biggl\{ \biggl( \int_{R< \vert x \vert < R+1} \vert \nabla u \vert ^{2} \vert \nabla\psi_{R} \vert ^{2}\,dx \biggr)^{\frac{1}{2}}+ \biggl( \int_{R< \vert x \vert < R+1} \vert u \vert ^{2} \vert \Delta \psi_{R} \vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggr\} \\ &\quad\leq C\lim_{R\rightarrow \infty} \biggl\{ \biggl( \int_{R< \vert x \vert < R+1} \vert \nabla u \vert ^{\frac{2N}{N-2}}\,dx \biggr)^{\frac{N-2}{2N}}+ \biggl( \int_{R< \vert x \vert < R+1} \vert \Delta u \vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggr\} =0. \end{aligned}$$

Similarly, we have $\lim_{R\rightarrow \infty}\overline{\lim_{n\rightarrow\infty}}\int_{\mathbb{R}^{N}} (2 \vert \Delta v_{n}\langle\nabla v_{n}, \nabla \psi_{R}\rangle \vert + \vert v_{n}\Delta v_{n}\Delta\psi_{R} \vert )\,dx=0$. Consequently, it follows from (3.10) and definitions (1)–(3) of the quantities $\eta_{\infty}^{(1)}$, $\eta_{\infty}^{(2)}$, $\nu_{\infty}^{(i)}\ (i=1, \ldots, m)$, $\gamma_{\infty}^{(1)}$, and $\gamma_{\infty}^{(2)}$ that

$$ Q_{+}(\infty)\sum_{i=1}^{m} \varsigma_{i}\nu_{\infty}^{(i)} \geq \eta_{\infty}^{(1)}+\eta_{\infty}^{(2)} -\mu \bigl( \gamma_{\infty}^{(1)}+\gamma_{\infty}^{(2)} \bigr). $$

(3.11)

Moreover, in view of (3.3), we find $\mathcal{A}_{\mu, m}(\sum_{i=1}^{m} \varsigma_{i}\nu_{\infty}^{(i)})^{\frac{2}{2^{\ast\ast}}} \leq\eta_{\infty}^{(1)}+\eta_{\infty}^{(2)} -\mu(\gamma_{\infty}^{(1)}+\gamma_{\infty}^{(2)})$. This, combined with (3.11), implies that either (v) $\nu_{\infty}^{(i)}=0\ (i=1, \ldots, m)$ or (vi) $\sum_{i=1}^{m}\varsigma_{i}\nu_{\infty}^{(i)} \geq(\mathcal{A}_{\mu, m}/Q_{+}(\infty))^{N/4}$. In the following, we claim that (ii), (iv), and (vi) cannot occur. For every continuous nonnegative function ψ such that $0\leq\psi(x)\leq 1$ on $\mathbb{R}^{N}$, we find

$$\begin{aligned} c&=\lim_{n\rightarrow\infty} \biggl(\mathcal{F}(u_{n}, v_{n}) -\frac{1}{2^{\ast\ast}} \bigl\langle \mathcal{F}^{\prime}(u_{n}, v_{n}), (u_{n}, v_{n}) \bigr\rangle \biggr) \\ &=\frac{2}{N} \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \biggl( \vert \Delta u_{n} \vert ^{2}+ \vert \Delta v_{n} \vert ^{2}-\mu \frac{ \vert u_{n} \vert ^{2}+ \vert v_{n} \vert ^{2}}{ \vert x \vert ^{4}} \biggr)\,dx \\ &\geq\frac{2}{N}\overline{\lim_{n\rightarrow\infty}} \int_{\mathbb{R}^{N}} \biggl( \vert \Delta u_{n} \vert ^{2}+ \vert \Delta v_{n} \vert ^{2}-\mu \frac{ \vert u_{n} \vert ^{2}+ \vert v_{n} \vert ^{2}}{ \vert x \vert ^{4}} \biggr)\psi(x)\,dx. \end{aligned}$$

Note that the measures $\nu^{(i)}\ (i=1, \ldots, m)$ are bounded and G-invariant. This means that if (ii) holds, then the set $\mathscr{J}$ must be finite. Moreover, if $x_{j}\neq0$ is a singular point of $\nu^{(i)}\ (i=1, \ldots, m)$, so is $gx_{j}$ for each $g\in G$, and the mass of $\nu^{(i)}\ (i=1, \ldots, m)$ concentrated at $gx_{j}$ is the same for every $g\in G$. Assuming that (ii) occurs for some $j\in\mathscr{J}$ with $x_{j}\neq0$, we choose ψ with compact support so that $\psi(gx_{j})=1$ for every $g\in G$, and we derive

$$\begin{aligned} c&\geq\frac{2}{N} \vert G \vert \bigl(\eta_{j}^{(1)}+ \eta_{j}^{(2)} \bigr)\geq \frac{2}{N} \vert G \vert \mathcal{A}_{0, m} \Biggl(\sum_{i=1}^{m} \varsigma_{i}\nu_{j}^{(i)} \Biggr) ^{\frac{2}{2^{\ast\ast}}} \\ &\geq\frac{2}{N} \vert G \vert \mathcal{A}_{0, m} \bigl( \mathcal{A}_{0, m}/ \Vert Q_{+} \Vert _{\infty} \bigr)^{\frac{2}{2^{\ast \ast}-2}} =\frac{2}{N} \vert G \vert \mathcal{A}_{0, m}^{\frac{N}{4}} \Vert Q_{+} \Vert _{\infty}^{1-\frac{N}{4}}, \end{aligned}$$

which is impossible. Similarly, assuming that (iv) holds for $x=0$, we take ψ with compact support so that $\psi(0)=1$, and we have

$$\begin{aligned} c&\geq\frac{2}{N} \bigl(\eta_{0}^{(1)}+ \eta_{0}^{(2)} -\mu\gamma_{0}^{(1)}-\mu \gamma_{0}^{(2)} \bigr)\geq \frac{2}{N} \mathcal{A}_{\mu, m} \Biggl(\sum_{i=1}^{m} \varsigma_{i}\nu_{0}^{(i)} \Biggr) ^{\frac{2}{2^{\ast\ast}}} \\ &\geq\frac{2}{N}\mathcal{A}_{\mu, m} \bigl(\mathcal{A}_{\mu, m}/Q_{+}(0) \bigr)^{\frac{2}{2^{\ast\ast}-2}} =\frac{2}{N}\mathcal{A}_{\mu, m}^{\frac{N}{4}}Q_{+}(0)^{1-\frac{N}{4}}, \end{aligned}$$

a contradiction to (3.7). Finally, if (vi) occurs, we choose $\psi=\psi_{R}$ to obtain

$$\begin{aligned} c&\geq\frac{2}{N} \bigl(\eta_{\infty}^{(1)}+ \eta_{\infty}^{(2)} -\mu\gamma_{\infty}^{(1)}-\mu \gamma_{\infty}^{(2)} \bigr)\geq \frac{2}{N} \mathcal{A}_{\mu, m} \Biggl(\sum_{i=1}^{m} \varsigma_{i}\nu_{\infty}^{(i)} \Biggr) ^{\frac{2}{2^{\ast\ast}}} \\ &\geq\frac{2}{N}\mathcal{A}_{\mu, m} \bigl(\mathcal{A}_{\mu, m}/Q_{+}( \infty) \bigr)^{\frac{2}{2^{\ast\ast}-2}} =\frac{2}{N}\mathcal{A}_{\mu, m}^{\frac{N}{4}}Q_{+}( \infty)^{1-\frac{N}{4}}, \end{aligned}$$

which contradicts (3.7). Hence, $\nu_{j}^{(i)}=0\ (i=1, \ldots, m)$ for all $j\in\mathscr{J}\cup\{0, \infty\}$, and this yields

$$\lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \sum_{i=1}^{m} \varsigma_{i} \vert u_{n} \vert ^{\alpha_{i}} \vert v_{n} \vert ^{\beta_{i}}\,dx = \int_{\mathbb{R}^{N}}\sum_{i=1}^{m} \varsigma_{i} \vert u \vert ^{\alpha _{i}} \vert v \vert ^{\beta_{i}} \,dx. $$

Finally, taking into account $\lim_{n\rightarrow\infty}\langle \mathcal{F}^{\prime}(u_{n}, v_{n})-\mathcal{F}^{\prime}(u, v), (u_{n}-u, v_{n}-v)\rangle=0$, we naturally deduce $(u_{n}, v_{n})\rightarrow(u, v)$ as $n\rightarrow\infty$ in $(\mathscr {D}^{2, 2}(\mathbb{R}^{N}))^{2}$. □

Thanks to Lemma 3.4, we immediately obtain the following result.

Corollary 3.1

If $\vert G \vert =+\infty$ and $Q_{+}(0)=Q_{+}(\infty)=0$, then the functional $\mathcal{F}$ satisfies the $(PS)_{c}$ condition for every $c\in\mathbb{R}$.

Proof of Theorem 2.1

Let $y_{\epsilon}$ be the extremal function satisfying (2.4)–(2.10). We now choose $\epsilon>0$ such that (2.11) is fulfilled. It is clear from (q.2), (3.1), and (3.2) that there exist constants $\alpha_{0}>0$ and $\rho>0$ such that $\mathcal{F}(u, v)\geq\alpha_{0}$ for all $\Vert (u, v) \Vert _{\mu}=\rho$. Moreover, if we set $u=y_{\epsilon}$, $v=\tau_{\min}y_{\epsilon}$, and

$$\begin{aligned} \Phi(t)={}&\mathcal{F}(ty_{\epsilon}, t\tau_{\min}y_{\epsilon})= \frac{t^{2}}{2} \bigl(1+\tau_{\min}^{2} \bigr) \int_{\mathbb{R}^{N}} \biggl( \vert \Delta y_{\epsilon} \vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{ \vert x \vert ^{4}} \biggr)\,dx\\ &{} -\frac{t^{2^{\ast\ast}}}{2^{\ast\ast}} \sum _{i=1}^{m}\varsigma_{i} \tau_{\min}^{\beta_{i}} \int_{\mathbb{R}^{N}} Q(x)y_{\epsilon}^{2^{\ast\ast}}\,dx \end{aligned}$$

with $t\geq0$, then $\max_{t\geq0}\Phi(t)$ is attained for some finite $\overline{t}>0$ with $\Phi^{\prime}(\overline{t})=0$. This yields

$$ \max_{t\geq0}\Phi(t)=\mathcal{F}( \overline{t}y_{\epsilon}, \overline{t}\tau_{\min}y_{\epsilon})= \frac{2}{N} \biggl\{ \frac{ (1+\tau_{\min}^{2} ) \int_{\mathbb{R}^{N}} ( \vert \Delta y_{\epsilon} \vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{ \vert x \vert ^{4}} )\,dx}{ (\sum_{i=1}^{m}\varsigma_{i}\tau_{\min}^{\beta_{i}} \int_{\mathbb{R}^{N}}Q(x)y_{\epsilon}^{2^{\ast\ast}}\,dx ) ^{\frac{2}{2^{\ast\ast}}}} \biggr\} ^{\frac{2^{\ast\ast}}{2^{\ast\ast}-2}}. $$

(3.12)

Besides, because $\mathcal{F}(ty_{\epsilon}, t\tau_{\min}y_{\epsilon})\rightarrow-\infty$ as $t\rightarrow+\infty$, there exists $t_{0}>0$ such that $\Vert (t_{0}y_{\epsilon}, t_{0}\tau_{\min}y_{\epsilon }) \Vert _{\mu}>\rho$ and $\mathcal{F}(t_{0}y_{\epsilon}, t_{0}\tau_{\min}y_{\epsilon})<0$. Now, we define

$$ c_{0}=\inf_{\gamma\in\Gamma}\max _{t\in[0, 1]}\mathcal{F} \bigl(\gamma(t) \bigr), $$

(3.13)

where $\Gamma=\{\gamma\in\mathscr{C}([0, 1], (\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}); \gamma(0)=(0, 0), \mathcal{F}(\gamma(1))<0, \Vert \gamma(1) \Vert _{\mu }>\rho\}$. It follows directly from (2.5), (2.11), (3.4), (3.5), (3.7), (3.12), (3.13), and Lemma 3.2 that

$$\begin{aligned} c_{0}&\leq\mathcal{F}(\overline{t}y_{\epsilon}, \overline{t} \tau_{\min}y_{\epsilon})=\frac{2}{N} \biggl\{ \frac{ (1+\tau_{\min}^{2} ) \int_{\mathbb{R}^{N}} ( \vert \Delta y_{\epsilon} \vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{ \vert x \vert ^{4}} )\,dx}{ (\sum_{i=1}^{m}\varsigma_{i}\tau_{\min}^{\beta_{i}} \int_{\mathbb{R}^{N}}Q(x)y_{\epsilon}^{2^{\ast\ast}}\,dx ) ^{\frac{2}{2^{\ast\ast}}}} \biggr\} ^{\frac{2^{\ast\ast}}{2^{\ast\ast}-2}} \\ &\leq\frac{2}{N} \biggl\{ \frac{\mathscr{B}(\tau_{\min})\int_{\mathbb{R}^{N}} ( \vert \Delta y_{\epsilon} \vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{ \vert x \vert ^{4}} )\,dx}{ (\max \{ \vert G \vert ^{\frac{2-2^{\ast\ast}}{2}} \mathcal{A}_{0}^{-\frac{2^{\ast\ast}}{2}} \Vert Q_{+} \Vert _{\infty}, \mathcal{A}_{\mu}^{-\frac{2^{\ast\ast}}{2}}Q_{+}(0), \mathcal{A}_{\mu}^{-\frac{2^{\ast\ast}}{2}}Q_{+}(\infty) \} ) ^{\frac{2}{2^{\ast\ast}}}} \biggr\} ^{\frac{2^{\ast\ast}}{2^{\ast\ast}-2}} \\ &=\frac{2}{N}\min \bigl\{ \vert G \vert \mathcal{A}_{0, m}^{\frac{N}{4}} \Vert Q_{+} \Vert _{\infty}^{1-\frac{N}{4}}, \mathcal{A}_{\mu, m}^{\frac{N}{4}}Q_{+}(0)^{1-\frac{N}{4}}, \mathcal{A}_{\mu, m}^{\frac{N}{4}}Q_{+}(\infty)^{1-\frac{N}{4}} \bigr\} =c_{0}^{\ast}. \end{aligned}$$

If $c_{0}< c_{0}^{\ast}$, then the $(PS)_{c}$ condition holds by Lemma 3.4. Thus we arrive at the conclusion by the mountain pass theorem in [35]. If $c_{0}=c_{0}^{\ast}$, then $\gamma(t)=(tt_{0}y_{\epsilon}, tt_{0}\tau_{\min}y_{\epsilon})$, with $0\leq t\leq1$, is a path in Γ such that $\max_{t\in [0, 1]}\mathcal{F}(\gamma(t))=c_{0}$. Hence, either $\Phi^{\prime}(\overline{t})=0$ and we are done, or γ can be deformed to a path $\widetilde{\gamma}\in\Gamma$ with $\max_{t\in[0, 1]}\mathcal{F}(\widetilde{\gamma}(t))< c_{0}$ and we have a contradiction. Thus we conclude from Lemma 3.1 that there exists a nontrivial G-invariant solution $(u_{0}, v_{0})\in(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N})\backslash\{0\})^{2}$ to problem $(\mathscr {P}_{0}^{Q})$ and the results follow. □

Proof of Corollary 2.1

In view of (2.6) and Theorem 2.1, it is sufficient to prove that

$$ \int_{\mathbb{R}^{N}} \bigl(Q(x)-\widetilde{Q} \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx\geq0 $$

(3.14)

for some $\epsilon>0$, where $\widetilde{Q}=\max\{ \vert G \vert ^{\frac{2-2^{\ast\ast}}{2}} (\mathcal{A}_{0}/\mathcal{A}_{\mu})^{-\frac{2^{\ast\ast}}{2}} \Vert Q_{+} \Vert _{\infty}, Q_{+}(0), Q_{+}(\infty)\}$.

Part (1), case (i). By virtue of (3.14), we need to show that

$$ \epsilon^{-2^{\ast\ast}l_{2}(\mu)} \int_{\mathbb{R}^{N}} \bigl(Q(x)-Q(0) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx\geq0 $$

(3.15)

for certain $\epsilon>0$. By the hypothesis, we choose $\varrho_{0}>0$ so that $Q(x)\geq Q(0)+\xi_{0} \vert x \vert ^{2^{\ast\ast}(l_{2}(\mu)-\Lambda_{0})}$ for $\vert x \vert \leq\varrho_{0}$. It follows from $2^{\ast\ast}\Lambda_{0}=N$ and (2.8) that

$$\begin{aligned} &\epsilon^{-2^{\ast\ast}l_{2}(\mu)} \int_{ \vert x \vert \leq\varrho_{0}} \bigl(Q(x)-Q(0) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \\ &\quad\geq\xi_{0} \int_{ \vert x \vert \leq\varrho_{0}}\epsilon^{-2^{\ast\ast }l_{2}(\mu)} \vert x \vert ^{2^{\ast\ast}(l_{2}(\mu)-\Lambda_{0})} U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \\ &\quad=\xi_{0} \int_{ \vert x \vert \leq\varrho_{0}} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon } \biggr) ^{l_{2}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}} \vert x \vert ^{-N}\,dx\rightarrow+\infty \end{aligned}$$

(3.16)

as $\epsilon\rightarrow0$. On the other hand, for any $\epsilon>0$, we deduce from (2.8), (2.9), and the fact that $2^{\ast\ast}l_{2}(\mu)>N$ that

$$\begin{aligned} &\biggl\vert \epsilon^{-2^{\ast\ast}l_{2}(\mu)} \int_{ \vert x \vert > \varrho_{0}} \bigl(Q(x)-Q(0) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \biggr\vert \\ &\quad\leq \int_{ \vert x \vert > \varrho_{0}}\frac{ \vert Q(x)-Q(0) \vert }{ \vert x \vert ^{2^{\ast\ast}l_{2}(\mu)}} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{2}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}}\,dx \\ &\quad \leq C \int_{ \vert x \vert > \varrho_{0}}\frac{1}{ \vert x \vert ^{2^{\ast\ast}l_{2}(\mu)}}\,dx\leq \overline{C}_{1} \end{aligned}$$

(3.17)

for some constant $\overline{C}_{1}>0$ independent of ϵ. Combining (3.16) and (3.17), we obtain (3.15) for ϵ sufficiently small.

Part (1), case (ii). By the hypothesis, we choose $\varrho_{1}>0$ so that $\vert Q(x)-Q(0) \vert \leq\xi_{1} \vert x \vert ^{\varsigma}$ for $\vert x \vert \leq\varrho_{1}$. Taking into account $\varsigma>2^{\ast\ast}(l_{2}(\mu)-\Lambda_{0})>0$, $N-1+\varsigma-2^{\ast\ast}l_{2}(\mu)>-1$ and $N-1-2^{\ast\ast}l_{2}(\mu)<-1$, we derive

$$\begin{aligned} &\epsilon^{-2^{\ast\ast}l_{2}(\mu)} \int_{\mathbb{R}^{N}} \bigl\vert Q(x)-Q(0) \bigr\vert U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \\ &\quad= \int_{\mathbb{R}^{N}}\frac{ \vert Q(x)-Q(0) \vert }{ \vert x \vert ^{2^{\ast\ast }l_{2}(\mu)}} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{2}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}}\,dx \\ &\quad\leq C \int_{\mathbb{R}^{N}}\frac{ \vert Q(x)-Q(0) \vert }{ \vert x \vert ^{2^{\ast\ast }l_{2}(\mu)}}\,dx \\ &\quad\leq C \biggl(\xi_{1} \int_{ \vert x \vert \leq\varrho_{1}} \vert x \vert ^{\varsigma-2^{\ast\ast}l_{2}(\mu)}\,dx + \int_{ \vert x \vert >\varrho_{1}} \bigl\vert Q(x)-Q(0) \bigr\vert \vert x \vert ^{-2^{\ast\ast}l_{2}(\mu )}\,dx \biggr) \\ &\quad\leq C \biggl( \int_{0}^{\varrho_{1}} r^{N-1+\varsigma-2^{\ast\ast}l_{2}(\mu)}\,dr + \int_{\varrho_{1}}^{+\infty}r^{N-1-2^{\ast\ast}l_{2}(\mu)}\,dr \biggr) < +\infty. \end{aligned}$$

Thus, by (2.8), (2.12), and the Lebesgue dominated convergence theorem, we have

$$\begin{aligned} & \lim_{\epsilon\rightarrow0} \int_{\mathbb{R}^{N}}\epsilon^{-2^{\ast\ast}l_{2}(\mu)} \bigl(Q(x)-Q(0) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \\ &\quad =\lim_{\epsilon\rightarrow 0} \int_{\mathbb{R}^{N}} \bigl(Q(x)-Q(0) \bigr) \vert x \vert ^{-2^{\ast\ast }l_{2}(\mu)} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{2}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}}\,dx \\ &\quad =C \int_{\mathbb{R}^{N}} \bigl(Q(x)-Q(0) \bigr) \vert x \vert ^{-2^{\ast\ast }l_{2}(\mu)}\,dx>0. \end{aligned}$$

Hence (3.15) holds for ϵ small enough.

Part (2), case (i). According to (3.14), we need to prove that

$$ \epsilon^{-2^{\ast\ast}l_{1}(\mu)} \int_{\mathbb{R}^{N}} \bigl(Q(x)-Q(\infty) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx\geq0 $$

(3.18)

for certain $\epsilon>0$. By the assumption, we take $\varrho_{2}>0$ such that $Q(x)\geq Q(\infty)+ \xi_{2} \vert x \vert ^{-2^{\ast\ast}(\Lambda_{0}-l_{1}(\mu))}$ for all $\vert x \vert \geq\varrho_{2}$. It follows from (2.7) that

$$\begin{aligned} &\epsilon^{-2^{\ast\ast}l_{1}(\mu)} \int_{ \vert x \vert \geq \varrho_{2}} \bigl(Q(x)-Q(\infty) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \\ &\quad = \int_{ \vert x \vert \geq \varrho_{2}} \bigl(Q(x)-Q(\infty) \bigr) \vert x \vert ^{-2^{\ast\ast}l_{1}(\mu)} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{1}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}}\,dx \\ &\quad \geq\xi_{2} \int_{ \vert x \vert \geq\varrho_{2}} \vert x \vert ^{-N} \biggl[ \biggl( \frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{1}(\mu)}U_{\mu} \biggl( \frac{ \vert x \vert }{\epsilon} \biggr) \biggr] ^{2^{\ast\ast}}\,dx\rightarrow+\infty \end{aligned}$$

as $\epsilon\rightarrow+\infty$. On the other hand, for any $\epsilon>0$, we conclude from (2.7), (q.2), and the fact that $N-1-2^{\ast\ast}l_{1}(\mu)>-1$ that

$$\begin{aligned} &\biggl\vert \int_{ \vert x \vert \leq \varrho_{2}}\epsilon^{-2^{\ast\ast}l_{1}(\mu)} \bigl(Q(x)-Q(\infty ) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \biggr\vert \\ &\quad \leq \int_{ \vert x \vert \leq \varrho_{2}}\frac{ \vert Q(x)-Q(\infty) \vert }{ \vert x \vert ^{2^{\ast\ast}l_{1}(\mu)}} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{1}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}}\,dx \\ &\quad\leq C \int_{ \vert x \vert \leq \varrho_{2}}\frac{ \vert Q(x)-Q(\infty) \vert }{ \vert x \vert ^{2^{\ast\ast}l_{1}(\mu )}}\,dx\leq C \int_{0}^{\varrho_{2}}r^{N-1-2^{\ast\ast}l_{1}(\mu)}\,dr\leq \overline{C}_{2} \end{aligned}$$

for some constant $\overline{C}_{2}>0$ independent of $\epsilon>0$. By putting these two estimates together, we obtain (3.18) for $\epsilon>0$ large enough.

Part (2), case (ii). By the assumption, we take $\varrho_{3}>0$ such that $\vert Q(x)-Q(\infty) \vert \leq \xi_{3} \vert x \vert ^{-\kappa}$ for all $\vert x \vert \geq\varrho_{3}$. Taking into account $\kappa>2^{\ast\ast}(\Lambda_{0}-l_{1}(\mu))>0$, $N-1-\kappa-2^{\ast\ast}l_{1}(\mu)<-1$ and $N-1-2^{\ast\ast}l_{1}(\mu)>-1$, we find

$$\begin{aligned} &\epsilon^{-2^{\ast\ast}l_{1}(\mu)} \int_{\mathbb{R}^{N}} \bigl\vert Q(x)-Q(\infty) \bigr\vert U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \\ &\quad = \int_{\mathbb{R}^{N}}\frac{ \vert Q(x)-Q(\infty) \vert }{ \vert x \vert ^{2^{\ast\ast }l_{1}(\mu)}} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{1}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}}\,dx\\ &\quad \leq C \int_{\mathbb{R}^{N}}\frac{ \vert Q(x)-Q(\infty) \vert }{ \vert x \vert ^{2^{\ast\ast }l_{1}(\mu)}}\,dx \\ &\quad \leq C \biggl(\xi_{3} \int_{ \vert x \vert \geq \varrho_{3}} \vert x \vert ^{-\kappa-2^{\ast\ast}l_{1}(\mu)}\,dx + \int_{ \vert x \vert \leq \varrho_{3}} \bigl\vert Q(x)-Q(\infty) \bigr\vert \vert x \vert ^{-2^{\ast\ast}l_{1}(\mu)}\,dx \biggr) \\ &\quad \leq C \biggl( \int_{ \varrho_{3}}^{+\infty}r^{N-1-\kappa-2^{\ast\ast}l_{1}(\mu)}\,dr + \int_{0}^{\varrho_{3}}r^{N-1-2^{\ast\ast}l_{1}(\mu)}\,dr \biggr)< +\infty. \end{aligned}$$

Therefore, by (2.7), (2.13), and the Lebesgue dominated convergence theorem, we obtain

$$\begin{aligned} & \lim_{\epsilon\rightarrow+\infty} \int_{\mathbb{R}^{N}}\epsilon^{-2^{\ast\ast}l_{1}(\mu)} \bigl(Q(x)-Q(\infty) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \\ &\quad =\lim_{\epsilon\rightarrow +\infty} \int_{\mathbb{R}^{N}}\frac{Q(x)-Q(\infty)}{ \vert x \vert ^{2^{\ast \ast}l_{1}(\mu)}} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{1}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}}\,dx \\ &\quad =C \int_{\mathbb{R}^{N}} \bigl(Q(x)-Q(\infty) \bigr) \vert x \vert ^{-2^{\ast\ast }l_{1}(\mu)}\,dx>0. \end{aligned}$$

Thus (3.18) holds for $\epsilon>0$ enough large. Similar to the above, we find that part (3) follows. □

To prove Theorem 2.2, we need the following symmetric mountain pass theorem (see [36] or [37, Theorem 9.12]).

Lemma 3.5

Let X be an infinite dimensional Banach space, and let $\mathcal{F}\in\mathscr{C}^{1}(X, \mathbb{R})$ be an even functional satisfying the $(PS)_{c}$ condition for each c and $\mathcal{F}(0)=0$. Furthermore, one supposes that:

• (i) there exist constants $\widetilde{\alpha}>0$ and $\rho>0$ such that $\mathcal{F}(w)\geq\widetilde{\alpha}$ for all $\Vert w \Vert =\rho$;

• (ii) there exists an increasing sequence of subspaces $\{X_{k}\}$ of X, with $\dim X_{k}=k$, such that for every k one can find a constant $R_{k}>0$ such that $\mathcal{F}(w)\leq0$ for all $w\in X_{k}$ with $\Vert w \Vert \geq R_{k}$.

Then $\mathcal{F}$ possesses a sequence of critical values $\{c_{k}\}$ tending to ∞ as $k\rightarrow\infty$.

Proof of Theorem 2.2

We follow closely the arguments in [9, Theorem 3] (see also [38, Theorem 3]). By virtue of Lemma 3.5 with $X=(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ and $w=(u, v)\in X$, we easily see from (q.2), (2.2), (3.1), and (3.3) that

$$\mathcal{F}(u, v)\geq\frac{1}{2} \bigl\Vert (u, v) \bigr\Vert _{\mu}^{2}-\frac{1}{2^{\ast\ast}} \Vert Q \Vert _{\infty} \mathcal{A}_{\mu, m}^{-\frac{2^{\ast\ast}}{2}} \bigl\Vert (u, v) \bigr\Vert _{\mu}^{2^{\ast\ast}}. $$

Thanks to $2^{\ast\ast}>2$, there exist constants $\widetilde{\alpha}>0$ and $\rho>0$ such that $\mathcal{F}(u, v)\geq\widetilde{\alpha}$ for any $(u, v)$ with $\Vert (u, v) \Vert _{\mu}=\rho$. To find an appropriate sequence of finite dimensional subspaces of $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$, we set $\Omega=\{x\in\mathbb{R}^{N}; Q(x)>0\}$. The set Ω is G-invariant, and we can define $(\mathscr{D}_{G}^{2, 2}(\Omega))^{2}$, which is the subspace of G-invariant functions of $(\mathscr{D}^{2, 2}(\Omega))^{2}$. Extending functions in $(\mathscr{D}_{G}^{2, 2}(\Omega))^{2}$ by 0 outside Ω, we can presume that $(\mathscr{D}_{ G}^{2, 2}(\Omega))^{2}\subset(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$. Let $\{X_{k}\}$ be an increasing sequence of subspaces of $(\mathscr{D}_{G}^{2, 2}(\Omega))^{2}$ with $\dim X_{k}=k$ for every k. As in [38, Theorem 3], we define $\varphi_{1, k}$, …, $\varphi_{k, k}\in\mathscr {C}_{0}^{\infty}(\mathbb{R}^{N})$ such that $0\leq\varphi_{i, k}\leq1$, $\operatorname{supp}(\varphi_{i, k})\cap\operatorname{supp}(\varphi_{j, k})=\emptyset$, $i\neq j$, and

$$\bigl\vert \operatorname{supp}(\varphi_{i, k})\cap\Omega \bigr\vert >0,\qquad \bigl\vert \operatorname{supp}(\varphi_{j, k})\cap\Omega \bigr\vert >0, \quad\forall i, j\in\{1, \ldots, k\}. $$

Taking $e_{i, k}=(a\varphi_{i, k}, b\varphi_{i, k})\in X_{k}$, $i=1, \ldots, k$, and $X_{k}=\operatorname{span}\{e_{1, k}, \ldots, e_{k, k}\}$, where a and b are two positive constants, we conclude from the construction of $X_{k}$ that $\dim X_{k}=k$ for every k. Therefore, there exists a constant $\epsilon(k)>0$ such that

$$\begin{aligned} &\int_{\Omega}Q(x) \bigl(\varsigma_{1} \vert \tilde{u} \vert ^{\alpha_{1}} \vert \tilde{v} \vert ^{\beta_{1}} + \cdots+\varsigma_{m} \vert \tilde{u} \vert ^{\alpha_{m}} \vert \tilde{v} \vert ^{\beta _{m}} \bigr)\,dx \\ &\quad= \int_{\Omega}Q(x) \Biggl( \varsigma_{1} \Biggl\vert \sum_{i=1}^{k}at_{i, k} \varphi_{i, k} \Biggr\vert ^{\alpha_{1}} \Biggl\vert \sum_{i=1}^{k}bt_{i, k} \varphi_{i, k} \Biggr\vert ^{\beta_{1}}+\cdots \\ &\qquad{}+ \varsigma_{m} \Biggl\vert \sum_{i=1}^{k}at_{i, k} \varphi_{i, k} \Biggr\vert ^{\alpha_{m}} \Biggl\vert \sum _{i=1}^{k}bt_{i, k} \varphi_{i, k} \Biggr\vert ^{\beta_{m}} \Biggr)\,dx \geq\epsilon(k) \end{aligned}$$

for all $(\tilde{u}, \tilde{v})=\sum_{i=1}^{k}t_{i, k}e_{i, k}\in X_{k}$, with $\Vert (\tilde{u}, \tilde{v}) \Vert _{\mu}=1$. Hence, if $(u, v)\in X_{k}\backslash\{(0, 0)\}$, then we write $(u, v)=t(\tilde{u}, \tilde{v})$, with $t= \Vert (u, v) \Vert _{\mu}$ and $\Vert (\tilde{u}, \tilde{v}) \Vert _{\mu}=1$. Therefore, we derive

$$\mathcal{F}(u, v)=\frac{1}{2}t^{2} -\frac{1}{2^{\ast\ast}}t^{2^{\ast\ast}} \int_{\Omega}Q(x) \sum_{i=1}^{m} \varsigma_{i} \vert \tilde{u} \vert ^{\alpha_{i}} \vert \tilde {v} \vert ^{\beta_{i}} \,dx\leq\frac{1}{2}t^{2} - \frac{\epsilon(k)}{2^{\ast\ast}}t^{2^{\ast\ast}}\leq0 $$

for $t>0$ sufficiently large. By Corollary 3.1 and Lemma 3.5, we conclude that there exists a sequence of critical values $c_{k}\rightarrow\infty$ as $k\rightarrow\infty$ and the results follow. □

Proof of Corollary 2.2

Because $Q(x)$ is radial, we know that the corresponding group $G=O(\mathbb{N})$ and $\vert G \vert =+\infty$. By Corollary 3.1, $\mathcal{F}$ satisfies the $(PS)_{c}$ condition for every $c\in\mathbb{R}$. Therefore, we deduce from Theorem 2.2 that the results follow. □

Multiplicity results for problem $(\mathscr{P}_{\sigma }^{\overline{Q}})$

The purpose of this section is to investigate problem $(\mathscr {P}_{\sigma}^{\overline{Q}})$ and prove Theorem 2.3; here we always presume that $\sigma>0$ and $Q(x)\equiv\overline{Q}>0$ is a constant. The corresponding energy functional of problem $(\mathscr{P}_{\sigma}^{\overline{Q}})$ is defined on $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ by

$$ \mathscr{E}_{\sigma}(u, v)=\frac{1}{2} \bigl\Vert (u, v) \bigr\Vert _{\mu}^{2} -\frac{\overline{Q}}{2^{\ast\ast}} \int_{\mathbb{R}^{N}} \sum_{i=1}^{m} \varsigma_{i} \vert u \vert ^{\alpha_{i}} \vert v \vert ^{\beta_{i}}\,dx -\frac{\sigma}{q} \int_{\mathbb{R}^{N}}h(x) \bigl( \vert u \vert ^{q} + \vert v \vert ^{q} \bigr)\,dx, $$

(4.1)

where $1< q<2$. In view of (h.2), (2.3), and the Hölder inequality, we find

$$\begin{aligned} & \int_{\mathbb{R}^{N}}h(x) \bigl( \vert u \vert ^{q} + \vert v \vert ^{q} \bigr)\,dx \\ &\quad \leq \biggl( \int_{\mathbb{R}^{N}} h^{\theta}(x)\,dx \biggr)^{\frac{1}{\theta}} \biggl\{ \biggl( \int_{\mathbb{R}^{N}} \vert u \vert ^{2^{\ast\ast}}\,dx \biggr)^{\frac{q}{2^{\ast\ast}}} + \biggl( \int_{\mathbb{R}^{N}} \vert v \vert ^{2^{\ast\ast}}\,dx \biggr)^{\frac{q}{2^{\ast\ast}}} \biggr\} \\ &\quad\leq\mathcal{A}_{\mu}^{-\frac{q}{2}} \Vert h \Vert _{\theta} \bigl( \Vert u \Vert _{\mu}^{q}+ \Vert v \Vert _{\mu}^{q} \bigr)\leq C \Vert h \Vert _{\theta} \bigl\Vert (u, v) \bigr\Vert _{\mu}^{q}. \end{aligned}$$

(4.2)

It follows from (4.1) and (4.2) that $\mathscr{E}_{\sigma}\in\mathscr{C}^{1}((\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}, \mathbb{R})$ and there exists a one-to-one correspondence between the weak solutions of $(\mathscr {P}_{\sigma}^{\overline{Q}})$ and the critical points of $\mathscr{E}_{\sigma}$. We now observe that an analogously symmetric criticality principle of Lemma 3.1 clearly holds. Consequently, the weak solutions of problem $(\mathscr {P}_{\sigma}^{\overline{Q}})$ are exactly the critical points of the functional $\mathscr{E}_{\sigma}$.

Lemma 4.1

Assume that (h.1) and (h.2) hold. Then there exists a positive constant M depending only on N, q, $\mathcal{A}_{\mu}$, and $\Vert h \Vert _{\theta}$, such that any bounded sequence $\{(u_{n}, v_{n})\}\subset(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ satisfying

$$\begin{aligned} \begin{aligned} &\mathscr{E}_{\sigma}(u_{n}, v_{v}) \rightarrow c< \frac{2}{N} \overline{Q}^{1-\frac{N}{4}} \mathcal{A}_{\mu, m}^{\frac{N}{4}}-M \sigma^{\frac{2}{2-q}}, \\ & \mathscr{E}_{\sigma}^{\prime}(u_{n}, v_{v})\rightarrow0\quad (n\rightarrow\infty) \end{aligned} \end{aligned}$$

(4.3)

contains a convergent subsequence.

Proof

By the hypothesis, $\{(u_{n}, v_{n})\}$ is bounded in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$. Hence we obtain a subsequence, still denoted by $\{(u_{n}, v_{n})\}$, satisfying $(u_{n}, v_{n})\rightharpoonup(u, v)$ in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$, $(u_{n}, v_{n})\rightarrow(u, v)$ a.e. in $\mathbb{R}^{N}$ and $(u_{n}, v_{n})\rightarrow(u, v)$ in $(L_{\operatorname{loc}}^{r}(\mathbb{R}^{N}))^{2}$ for all $r\in[1, 2^{\ast\ast})$. By virtue of (h.2), the Hölder inequality and the Lebesgue dominated theorem, we derive

$$ \lim_{n\rightarrow \infty} \int_{\mathbb{R}^{N}}h(x) \bigl( \vert u_{n} \vert ^{q} + \vert v_{n} \vert ^{q} \bigr)\,dx = \int_{\mathbb{R}^{N}}h(x) \bigl( \vert u \vert ^{q} + \vert v \vert ^{q} \bigr)\,dx. $$

(4.4)

Applying the standard argument, we easily check from (4.4) that $(u, v)$ is a critical point of $\mathscr{E}_{\sigma}$. Further, in view of (h.2), (4.1), (4.2), and the Hölder inequality, by direct calculation, we obtain

$$\begin{aligned} \mathscr{E}_{\sigma}(u, v)&=\mathscr{E}_{\sigma}(u, v)-\frac{1}{2^{\ast\ast}} \bigl\langle \mathscr{E}_{\sigma}^{\prime}(u, v), (u, v) \bigr\rangle \\ &=\frac{2}{N} \bigl\Vert (u, v) \bigr\Vert _{\mu}^{2}- \frac{\sigma}{2^{\ast\ast}q} \bigl(2^{\ast\ast}-q \bigr) \int_{\mathbb{R}^{N}}h(x) \bigl( \vert u \vert ^{q} + \vert v \vert ^{q} \bigr)\,dx \\ &\geq\frac{2}{N} \bigl( \Vert u \Vert _{\mu}^{2}+ \Vert v \Vert _{\mu}^{2} \bigr) -\frac{\sigma}{2^{\ast\ast}q} \bigl(2^{\ast\ast}-q \bigr) \mathcal{A}_{\mu}^{-\frac{q}{2}} \Vert h \Vert _{\theta} \bigl( \Vert u \Vert _{\mu}^{q}+ \Vert v \Vert _{\mu}^{q} \bigr) \\ &\geq-(2-q) \biggl(\frac{qN}{4} \biggr)^{\frac{q}{2-q}} \biggl( \frac{2^{\ast\ast}-q}{2^{\ast\ast}q}\mathcal{A}_{\mu }^{-\frac{q}{2}} \Vert h \Vert _{\theta} \biggr)^{\frac{2}{2-q}} \sigma ^{\frac{2}{2-q}} \triangleq-M \sigma^{\frac{2}{2-q}}, \end{aligned}$$

(4.5)

where $M=(2-q)(\frac{qN}{4})^{\frac{q}{2-q}} (\frac{2^{\ast\ast}-q}{2^{\ast\ast}q}\mathcal{A}_{\mu}^{-\frac{q}{2}} \Vert h \Vert _{\theta})^{\frac{2}{2-q}}$ is a positive constant. We now set $\widetilde{u}_{n}=u_{n}-u$ and $\widetilde{v}_{n}=v_{n}-v$. Then, by the Brezis–Lieb lemma [39] and arguing as in [40, Lemma 2.1], we have

$$\begin{aligned} &\bigl\Vert (\widetilde{u}_{n}, \widetilde{v}_{n}) \bigr\Vert _{\mu }^{2}= \bigl\Vert (u_{n}, v_{n}) \bigr\Vert _{\mu}^{2}- \bigl\Vert (u, v) \bigr\Vert _{\mu }^{2}+o_{n}(1), \end{aligned}$$

(4.6)

$$\begin{aligned} & \int_{\mathbb{R}^{N}} \vert \widetilde{u}_{n} \vert ^{\alpha_{i}} \vert \widetilde{v}_{n} \vert ^{\beta_{i}}\,dx = \int_{\mathbb{R}^{N}} \vert u_{n} \vert ^{\alpha_{i}} \vert v_{n} \vert ^{\beta_{i}}\,dx- \int_{\mathbb{R}^{N}} \vert u \vert ^{\alpha_{i}} \vert v \vert ^{\beta_{i}}\,dx+o_{n}(1),\quad i=1, \ldots, m. \end{aligned}$$

(4.7)

Taking into account $\mathscr{E}_{\sigma}(u_{n}, v_{n})=c+o_{n}(1)$ and $\mathscr{E}_{\sigma}^{\prime}(u_{n}, v_{n})=o_{n}(1)$, we conclude from (4.1), (4.4), (4.6), and (4.7) that

$$\begin{aligned} c+o_{n}(1)={}&\mathscr{E}_{\sigma}(u_{n}, v_{n})=\mathscr{E}_{\sigma}(u, v)+ \frac{1}{2} \bigl\Vert (\widetilde{u}_{n}, \widetilde{v}_{n}) \bigr\Vert _{\mu}^{2} \\ &{} -\frac{\overline{Q}}{2^{\ast\ast}} \int_{\mathbb{R}^{N}} \sum_{i=1}^{m} \varsigma_{i} \vert \widetilde{u}_{n} \vert ^{\alpha_{i}} \vert \widetilde{v}_{n} \vert ^{\beta_{i}}\,dx+o_{n}(1) \end{aligned}$$

(4.8)

and

$$ \bigl\Vert (\widetilde{u}_{n}, \widetilde{v}_{n}) \bigr\Vert _{\mu}^{2} - \overline{Q} \int_{\mathbb{R}^{N}}\sum_{i=1}^{m} \varsigma_{i} \vert \widetilde{u}_{n} \vert ^{\alpha_{i}} \vert \widetilde{v}_{n} \vert ^{\beta_{i}}\,dx=o_{n}(1). $$

(4.9)

As a result, for a subsequence $\{(\widetilde{u}_{n}, \widetilde{v}_{n})\}$, we find

$$\bigl\Vert (\widetilde{u}_{n}, \widetilde{v}_{n}) \bigr\Vert _{\mu}^{2} \rightarrow\overline{\xi}\geq0,\qquad \overline{Q} \int_{\mathbb{R}^{N}} \sum_{i=1}^{m} \varsigma_{i} \vert \widetilde{u}_{n} \vert ^{\alpha_{i}} \vert \widetilde{v}_{n} \vert ^{\beta _{i}}\,dx \rightarrow \overline{\xi} $$

as $n\rightarrow\infty$. It follows from (3.3) that $\mathcal{A}_{\mu, m}(\overline{\xi}/\overline{Q})^{\frac{2}{2^{\ast\ast}}}\leq \overline{\xi}$. This yields either $\overline{\xi}=0$ or $\overline{\xi}\geq\overline{Q}^{1-\frac{N}{4}}\mathcal{A}_{\mu, m}^{\frac{N}{4}}$. If $\overline{\xi}\geq \overline{Q}^{1-\frac{N}{4}}\mathcal{A}_{\mu, m}^{\frac{N}{4}}$, then we deduce from (4.5), (4.8), and (4.9) that

$$c=\mathscr{E}_{\sigma}(u, v)+ \biggl(\frac{1}{2}- \frac{1}{2^{\ast\ast}} \biggr)\overline{\xi} \geq\frac{2}{N} \overline{Q}^{1-\frac{N}{4}}\mathcal{A}_{\mu, m}^{\frac{N}{4}}-M \sigma^{\frac{2}{2-q}}, $$

which contradicts (4.3). Therefore, we obtain $\Vert (\widetilde{u}_{n}, \widetilde{v}_{n}) \Vert _{\mu }^{2}\rightarrow 0$ as $n\rightarrow+\infty$, and hence, $(u_{n}, v_{n})\rightarrow (u, v)$ in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$. The conclusion of this lemma follows. □

Lemma 4.2

Assume that (h.1) and (h.2) hold. Then there exists $\sigma_{1}^{\ast}>0$ such that for any $\sigma\in(0, \sigma_{1}^{\ast})$ the following geometric conditions for $\mathscr{E}_{\sigma}(u, v)$ hold:

• (i) $\mathscr{E}_{\sigma}(0, 0)=0$; there exist constants $\overline{\alpha}>0$ and $\rho>0$ such that $\mathscr{E}_{\sigma}(u, v)\geq\overline{\alpha}$ for all $\Vert (u, v) \Vert _{\mu}=\rho$;

• (ii) there exists $(e_{u}, e_{v})\in (\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ such that $\Vert (e_{u}, e_{v}) \Vert _{\mu}>\rho$ and $\mathscr{E}_{\sigma}(e_{u}, e_{v})< 0$.

Proof

In view of (h.2), (3.3), (4.1), (4.2), and the Hölder inequality, by direct computation, we derive

$$\begin{aligned} \mathscr{E}_{\sigma}(u, v)&\geq\frac{1}{2} \bigl\Vert (u, v) \bigr\Vert _{\mu}^{2}-\frac{\overline{Q}}{2^{\ast\ast }} \mathcal{A}_{\mu, m}^{-\frac{2^{\ast\ast}}{2}} \bigl\Vert (u, v) \bigr\Vert _{\mu}^{2^{\ast\ast}}-\frac{\sigma}{q}C \Vert h \Vert _{\theta} \bigl\Vert (u, v) \bigr\Vert _{\mu}^{q} \\ &\geq \biggl(\frac{1}{2}-\varsigma_{0} \biggr) \bigl\Vert (u, v) \bigr\Vert _{\mu}^{2}-\frac{\overline{Q}}{2^{\ast\ast }} \mathcal{A}_{\mu, m}^{-\frac{2^{\ast\ast}}{2}} \bigl\Vert (u, v) \bigr\Vert _{\mu}^{2^{\ast\ast}}-C(\varsigma_{0})\sigma^{\frac {2}{2-q}} \end{aligned}$$

(4.10)

for any $\varsigma_{0}\in(0, \frac{1}{2})$, where $C(\varsigma_{0})=(\frac{2}{q}-1)\varsigma_{0} [C \Vert h \Vert _{\theta}/(2\varsigma_{0})]^{2/(2-q)}$ is a positive constant. It follows from the last inequality in (4.10) that there exist constants $\overline{\alpha}>0$, $\rho>0$, and $\sigma_{1}^{\ast}>0$ such that $\mathscr{E}_{\sigma}(u, v)\geq\overline{\alpha}>0$ for all $\Vert (u, v) \Vert _{\mu}=\rho$, $\varsigma_{0}\in(0, \frac{1}{2})$ and $\sigma\in(0, \sigma_{1}^{\ast})$. This yields (i).

On the other hand, taking into account $\int_{\mathbb{R}^{N}}h(x)( \vert u \vert ^{q}+ \vert v \vert ^{q})\,dx\geq0$, we deduce from (4.1) that there exists $(\check{u}, \check{v})\in(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N})\backslash\{0\})^{2}$ such that $\mathscr {E}_{\sigma}(t\check{u}, t\check{v})\rightarrow-\infty$ as $t\rightarrow+\infty$. Therefore, we can choose $(e_{u}, e_{v})=(T\check{u}, T\check{v})$ ($T>0$ large enough) such that $\Vert (e_{u}, e_{v}) \Vert _{\mu}>\rho$ and $\mathscr {E}_{\sigma}(e_{u}, e_{v})< 0$. Thus (ii) follows. □

Lemma 4.3

Assume that (h.1) and (h.2) hold. Then there exists $\sigma_{2}^{\ast}>0$ such that

$$ \sup_{t\geq0}\mathscr{E}_{\sigma} (ty_{\epsilon}, t\tau_{\min}y_{\epsilon} ) < \frac{2}{N} \overline{Q}^{1-\frac{N}{4}}\mathcal{A}_{\mu, m}^{\frac{N}{4}}-M \sigma^{\frac{2}{2-q}} $$

(4.11)

for any $\sigma\in(0, \sigma_{2}^{\ast})$ and small $\epsilon>0$, where $M>0$ is given in Lemma 4.1 and $\tau_{\min}>0$ satisfies (3.4)–(3.6).

Proof

Similar to the proof in Alves [41, Theorem 3], we define the functions

$$\begin{aligned} \Psi(t)={}&\mathscr{E}_{\sigma} (ty_{\epsilon}, t \tau_{\min}y_{\epsilon} )= \frac{t^{2}}{2} \bigl(1+ \tau_{\min}^{2} \bigr) \int_{\mathbb {R}^{N}} \biggl( \vert \Delta y_{\epsilon} \vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{ \vert x \vert ^{4}} \biggr)\,dx \\ &{}-\frac{t^{2^{\ast\ast}}}{2^{\ast\ast}} \Biggl(\sum_{i=1}^{m} \varsigma_{i} \tau_{\min}^{\beta_{i}} \Biggr)\overline{Q} \int_{\mathbb{R}^{N}}y_{\epsilon}^{2^{\ast\ast}}\,dx- \frac{ \sigma}{q}t^{q} \bigl(1+\tau_{\min}^{q} \bigr) \int_{\mathbb{R}^{N}}h(x) y_{\epsilon}^{q}\,dx \end{aligned}$$

(4.12)

and

$$ \widetilde{\Psi}(t)=\frac{t^{2}}{2} \bigl(1+ \tau_{\min}^{2} \bigr) \int_{\mathbb{R}^{N}} \biggl( \vert \Delta y_{\epsilon} \vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{ \vert x \vert ^{4}} \biggr)\,dx -\frac{t^{2^{\ast\ast}}}{2^{\ast\ast}} \Biggl(\sum _{i=1}^{m}\varsigma_{i} \tau_{\min}^{\beta_{i}} \Biggr)\overline{Q} \int_{\mathbb{R}^{N}}y_{\epsilon}^{2^{\ast\ast}}\,dx $$

(4.13)

with $t\geq0$. Note that $\widetilde{\Psi}(0)=0$, $\widetilde{\Psi}(t)>0$ for $t\rightarrow0^{+}$, and $\lim_{t\rightarrow+\infty}\widetilde{\Psi}(t)=-\infty$. Hence, $\sup_{t\geq0}\widetilde{\Psi}(t)$ can be achieved at some finite $t_{\epsilon}^{0}>0$ at which $\widetilde{\Psi}^{\prime}(t)$ becomes zero. In view of (2.5), (2.6), (3.4)–(3.6), (4.13), and Lemma 3.2, by simple arithmetic, we derive

$$\begin{aligned} \sup_{t\geq 0}\widetilde{\Psi}(t)=\widetilde{\Psi} \bigl(t_{\epsilon}^{0} \bigr)={}& \biggl(\frac{1}{2}- \frac{1}{2^{\ast\ast}} \biggr) \biggl\{ \frac{ (1+\tau_{\min}^{2} ) \int_{\mathbb{R}^{N}} ( \vert \Delta y_{\epsilon} \vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{ \vert x \vert ^{4}} )\,dx}{ [ (\sum_{i=1}^{m}\varsigma_{i} \tau_{\min}^{\beta_{i}} )\overline{Q} \int_{\mathbb{R}^{N}}y_{\epsilon}^{2^{\ast\ast}}\,dx ]^{\frac {2}{2^{\ast\ast}}}} \biggr\} ^{\frac{2^{\ast\ast}}{2^{\ast\ast}-2}} \\ ={}&\frac{2}{N}\overline{Q}^{1-\frac{N}{4}} \bigl(\mathscr{B}( \tau_{\min})\mathcal{A}_{\mu} \bigr) ^{\frac{N}{4}}= \frac{2}{N}\overline{Q}^{1-\frac{N}{4}}\mathcal {A}_{\mu, m}^{\frac{N}{4}}. \end{aligned}$$

(4.14)

Let $\overline{\sigma}>0$ be such that

$$\frac{2}{N}\overline{Q}^{1-\frac{N}{4}}\mathcal{A}_{\mu, m}^{\frac{N}{4}}-M \sigma^{\frac{2}{2-q}}>0,\quad \forall\sigma\in(0, \overline{\sigma}). $$

On the one hand, by virtue of (h.1), (h.2), (2.5), and (4.12), we conclude that

$$\Psi(t)=\mathscr{E}_{\sigma} (ty_{\epsilon}, t\tau_{\min}y_{\epsilon} )\leq\frac{t^{2}}{2} \bigl(1+\tau _{\min}^{2} \bigr),\quad \forall t\geq0, \sigma>0, $$

and there exists $T_{0}\in(0, 1)$ independent of ϵ such that

$$ \sup_{0\leq t\leq T_{0}}\Psi(t)\leq\frac{T_{0}^{2}}{2} \bigl(1+\tau_{\min}^{2} \bigr) < \frac{2}{N} \overline{Q}^{1-\frac{N}{4}}\mathcal{A}_{\mu, m}^{\frac{N}{4}}-M \sigma^{\frac{2}{2-q}},\quad \forall\sigma\in(0, \overline{\sigma}). $$

(4.15)

On the other hand, it follows from (4.12), (4.13), and (4.14) that

$$\begin{aligned} \sup_{t\geq T_{0}}\Psi(t)&\leq\sup _{t\geq 0} \widetilde{\Psi}(t)-\frac{\sigma}{q}T_{0}^{q} \bigl(1+ \tau_{\min }^{q} \bigr) \int_{\mathbb{R}^{N}} h(x)y_{\epsilon}^{q}\,dx \\ &=\frac{2}{N}\overline{Q}^{1-\frac{N}{4}}\mathcal{A}_{\mu, m}^{\frac{N}{4}}- \frac{\sigma}{q}T_{0}^{q} \bigl(1+\tau_{\min }^{q} \bigr) \int_{\mathbb{R}^{N}}h(x)y_{\epsilon}^{q}\,dx. \end{aligned}$$

(4.16)

Now, taking $\sigma>0$ such that $-\frac{\sigma}{q}T_{0}^{q}(1+\tau_{\min}^{q}) \int_{\mathbb{R}^{N}}h(x)y_{\epsilon}^{q}\,dx <-M\sigma^{\frac{2}{2-q}}$, namely

$$0< \sigma< \biggl[\frac{T_{0}^{q}}{qM} \bigl(1+\tau_{\min}^{q} \bigr) \int_{\mathbb{R}^{N}}h(x) y_{\epsilon}^{q}\,dx \biggr]^{\frac{2-q}{q}}\triangleq\widetilde{\sigma}, $$

we find from (4.16) that

$$ \sup_{t\geq T_{0}}\Psi(t)< \frac{2}{N} \overline{Q}^{1-\frac{N}{4}}\mathcal {A}_{\mu, m}^{\frac{N}{4}}-M \sigma^{\frac{2}{2-q}}, \quad\forall\sigma\in(0, \widetilde{\sigma}). $$

(4.17)

Choosing $\sigma_{2}^{\ast}=\min\{\overline{\sigma},\widetilde{\sigma}\}$, we deduce from (4.15) and (4.17) that

$$\sup_{t\geq0}\Psi(t) < \frac{2}{N}\overline{Q}^{1-\frac{N}{4}} \mathcal{A}_{\mu, m}^{\frac{N}{4}}-M\sigma^{\frac{2}{2-q}}, \quad\forall\sigma \in \bigl(0, \sigma_{2}^{\ast} \bigr), $$

which implies (4.11). Hence the results of this lemma follow. □

Proof of Theorem 2.3

Taking $\rho>0$ and $\sigma^{\ast}=\min\{\sigma_{1}^{\ast}, \sigma_{2}^{\ast}\}$, for $0<\sigma<\sigma^{\ast}$, given in the proofs of Lemmas 4.2 and 4.3, we define

$$c_{1}\triangleq\inf_{\overline{\mathcal{B}} _{\rho}(0)}\mathscr{E}_{\sigma}(u, v), $$

where $\overline{\mathcal{B}}_{\rho}(0)=\{(u, v)\in(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}; \Vert (u, v) \Vert _{\mu}\leq\rho\}$. It is easy to see that the metric space $\overline{\mathcal{B}}_{\rho}(0)$ is complete. According to the Ekeland variational principle [42], we deduce that there exists a sequence $\{(u_{n}, v_{n})\}\subset\overline{\mathcal{B}}_{\rho}(0)$ such that $\mathscr{E}_{\sigma}(u_{n}, v_{n})\rightarrow c_{1}$ and $\mathscr{E}_{\sigma}^{\prime}(u_{n}, v_{n})\rightarrow0$ as $n\rightarrow\infty$.

Let $\varphi_{0}$, $\psi_{0}\in\mathscr {C}_{0}^{\infty}(\mathbb{R}^{N})$ be the G-invariant functions such that $\varphi_{0}$, $\psi_{0}> 0$. It follows from (h.1) and (h.2) that $\int_{\mathbb{R}^{N}}h(x)( \varphi_{0}^{q}+\psi_{0}^{q})\,dx>0$. In view of $1< q<2<2^{\ast\ast}$, we find that there exists $\tilde{t}_{0}=\tilde{t}_{0}(\varphi_{0}, \psi_{0})>0$ sufficiently small such that

$$\begin{aligned} \mathscr{E}_{\sigma}(\tilde{t}_{0}\varphi_{0}, \tilde{t}_{0}\psi_{0})={}&\frac{\tilde{t}_{0}^{2}}{2} \bigl\Vert ( \varphi_{0}, \psi_{0}) \bigr\Vert _{\mu}^{2}\\ &{}- \frac{\overline{Q}}{2^{\ast\ast}} \tilde{t}_{0}^{2^{\ast\ast}} \int_{\mathbb{R}^{N}}\sum_{i=1}^{m} \varsigma_{i}\varphi_{0}^{\alpha_{i}} \psi_{0}^{\beta_{i}}\,dx-\frac{\sigma}{q}\tilde{t}_{0}^{q} \int _{\mathbb{R}^{N}}h(x) \bigl( \varphi_{0}^{q}+ \psi_{0}^{q} \bigr)\,dx< 0. \end{aligned}$$

This yields

$$c_{1}< 0< \frac{2}{N}\overline{Q}^{1-\frac{N}{4}} \mathcal{A}_{\mu, m}^{\frac{N}{4}}-M\sigma^{\frac{2}{2-q}}, \quad \forall\sigma \in \bigl(0, \sigma^{\ast} \bigr). $$

By virtue of Lemma 4.1, $\mathscr{E}_{\sigma}$ admits a nontrivial critical point $(u_{1}, v_{1})$ with $\mathscr{E}_{\sigma}(u_{1}, v_{1})=c_{1}<0$. Applying the principle of symmetric criticality, we obtain that $(u_{1}, v_{1})$ is a nontrivial G-invariant solution of problem $(\mathscr{P}_{\sigma}^{\overline{Q}})$.

Furthermore, we now define

$$c_{2}\triangleq\inf_{\gamma\in\Gamma}\max_{t\in[0, 1]} \mathscr{E}_{\sigma} \bigl(\gamma(t) \bigr), $$

where $\Gamma=\{\gamma\in\mathscr{C}([0, 1], (\mathscr{D}_{ G}^{2, 2}(\mathbb{R}^{N}))^{2}); \gamma(0)=(0, 0), \gamma(1)=(e_{u}, e_{v})\}$. It follows from Lemmas 4.2 and 4.3 that

$$0< \overline{\alpha}\leq c_{2}< \frac{2}{N}\overline{Q}^{1-\frac{N}{4}} \mathcal{A}_{\mu, m}^{\frac{N}{4}}-M\sigma^{\frac{2}{2-q}},\quad \forall\sigma \in \bigl(0, \sigma^{\ast} \bigr). $$

This, combined with the mountain pass theorem, implies that $c_{2}$ is another nonzero critical value of $\mathscr{E}_{\sigma}$. Similar to the above arguments, problem $(\mathscr{P}_{\sigma}^{\overline{Q}})$ possesses another nontrivial G-invariant solution $(u_{2}, v_{2})$ with $\mathscr{E}_{\sigma}(u_{2}, v_{2})=c_{2}>0$. □