On G-invariant solutions of a singular biharmonic elliptic system involving multiple critical exponents in $R^{N}$

In this work, a biharmonic elliptic system is investigated in $\mathbb{R}^{N}$, which involves singular potentials and multiple critical exponents. By the Rellich inequality and the symmetric criticality principle, the existence and multiplicity of G-invariant solutions to the system are established. To our best knowledge, our results are new even in the scalar cases.

In this article, we study the singular fourth-order elliptic problem:

where $\Delta^{2}$ denotes the biharmonic operator, $N\geq5$, $\sigma\geq0$, $\mu\in[0, \overline{\mu})$ with $\overline{\mu}\triangleq\frac{1}{16}N^{2}(N-4)^{2}$, $q\in(1, 2)$, $\varsigma_{i}\in(0, +\infty)$, and $\alpha_{i}$, $\beta_{i}>1$ satisfy $\alpha_{i}+\beta_{i}=2^{\ast\ast}\ (i=1, \ldots, m; 1\leq m\in\mathbb{N})$, $2^{\ast\ast} \triangleq\frac{2N}{N-4}$ is the critical Sobolev exponent; $Q(x)$ and $h(x)$ are G-invariant functions such that $Q(x)\in\mathscr {C}(\mathbb{R}^{N})\cap L^{\infty}(\mathbb{R}^{N})$ and $h(x)\in L^{\theta}(\mathbb{R}^{N})$ with $\theta\triangleq 2^{\ast\ast}/(2^{\ast\ast}-q)$ (see Sect. 2 for details).

There have been by now a large number of papers concerning the existence, nonexistence as well as qualitative properties of nontrivial solutions to critical elliptic problems of second order. With no hope of being complete, we would like to mention some of them [1–4]. In most of these papers, the authors deal with the elliptic problems involving singular potentials and critical exponents. For instance, Deng and Jin in [4] handled the following singular equation:

where $N> 2$, $\mu\in[0, \frac{1}{4}(N-2)^{2})$, $s\in[0, 2)$, $2^{\ast}(s)=\frac{2(N-s)}{N-2}$, and $2^{\ast}(0)=2^{\ast}\triangleq\frac{2N}{N-2}$, and Q is G-invariant with respect to a subgroup G of $O(\mathbb{N})$. By applying analytic techniques and critical point theory, several results on the existence and multiplicity of G-invariant solutions to (1.2) were obtained. Subsequently, Waliullah [5] extended the results in [4] to the weighted polyharmonic elliptic equations. In particular, Waliullah considered the following semilinear partial differential equation:

where $k>1$, $N>2k$, $2_{(k)}^{\ast}=\frac{2N}{N-2k}$, and Q is G-invariant. By employing the minimizing sequence and the concentration–compactness method, the author attained the existence of nontrivial G-invariant solution to (1.3). Borrowing ideas from [4, 5], Deng and Huang [6–8] recently established a few valuable results for the scalar elliptic problems in a bounded G-invariant domain. Moreover, let us also mention that when $\mu=0$ and the right-hand side nonlinearity term $\vert x \vert ^{-s}u^{2^{\ast}(s)-1}$ in (1.2) is substituted by $u^{q-1}$ with $1< q\leq2^{\ast}$, there have been a variety of remarkable results on G-invariant solutions in [9–11]. Furthermore, for other results about this aspect, see [12] with singular Lane–Emden–Fowler equations, [13] with singular p-Laplacian equations, [14] with biharmonic operators and [15] with $p(x)$-biharmonic operators [16], and monograph [17] with generalized Lane–Emden–Fowler equations or Gierer–Meinhardt systems involving singular nonlinearity.

For the systems of singular elliptic equations involving critical exponents, a wide range of works concerning the solutions structures have been presented in recent years. For example, Cai and Kang [18] studied the following elliptic system with multiple critical terms:

where $N\geq3$, $\Omega\subset\mathbb{R}^{N}$ is a smooth bounded domain such that $0\in\Omega$, $\mathcal{L}_{\mu}=-\Delta-\mu \vert x \vert ^{-2}$, $\mu<\frac{1}{4}(N-2)^{2}$, $a_{j}\in\mathbb{R}\ (j=1, 2, 3)$, $\varsigma_{i}\in(0, +\infty)$, $q_{i}\in[2, 2^{\ast})$, and $\alpha_{i}$, $\beta_{i}>1$ fulfill $\alpha_{i}+\beta_{i}=2^{\ast}\ (i=1, 2)$. By a variational minimax method combined with a delicate analysis of Palais–Smale sequences, the authors proved the existence of positive solutions to (1.4). Very recently, Nyamoradi and Hsu [19] investigated the following quasilinear elliptic system involving multiple critical exponents:

where $0\in\Omega$ is a smooth bounded domain in $\mathbb{R}^{N}$, $1< p< N$, $0\leq a<\frac{N-p}{p}$, $a\leq b< a+1$, $0<\varsigma_{i}, \lambda_{i}, \mu_{i}<+\infty$, $\alpha_{i}$, $\beta_{i}>1$, $\alpha_{i}+\beta_{i}=p^{\ast}(a, b)=\frac{Np}{N-p(a+1-b)}$ for $i=1,\ldots, m$. By employing the analytic techniques of Nehari manifold, the authors established the existence and multiplicity of positive solutions to (1.5) under certain appropriate hypotheses on the parameters q, β, $\lambda_{i}$, $\mu_{i}$ and the weighted functions $f_{i}(x)\ (i=1, \ldots, m)$. Other results relating to second-order elliptic systems can be found in [20–23] and the references therein. For the systems of fourth-order elliptic equations, we would like to refer the reader to the papers [24–26] for the elliptic problems related to nonlinearities with critical growth.

Nevertheless, elliptic systems involving the G-invariant solutions have seldom been studied; we only find a handful of results in [27–30]. To the best of our knowledge, there are few results on G-invariant solutions for the singular fourth-order elliptic problem (1.1) even in the scalar cases $\sigma=0$, $0<\mu<\overline{\mu}$, $m=1$, and $u=v$. Therefore, it is necessary for us to investigate (1.1) thoroughly. Let $\overline{Q}>0$ be a constant. This work is dedicated to seeking the G-invariant solutions for both the cases of $\sigma=0$, $Q(x)\not\equiv\overline{Q}$ and $\sigma>0$, $Q(x)\equiv\overline{Q}$ in (1.1). Our arguments are mainly based upon the symmetric criticality principle due to Palais [31] and variational methods.

The rest of this article is schemed as follows. The variational framework and the main results of this paper are presented in Sect. 2. The proofs of G-invariant solutions for the cases $\sigma=0$ and $Q(x)\not\equiv\overline{Q}$ are detailed in Sect. 3, while the multiplicity results for the cases $\sigma>0$ and $Q(x)\equiv\overline{Q}$ are proved in Sect. 4.

Let $\mathscr{D}^{2, 2}(\mathbb{R}^{N})$ denote the completion of $\mathscr{C}_{0}^{\infty}(\mathbb{R}^{N})$ under the norm $(\int_{\mathbb{R}^{N}} \vert \Delta u \vert ^{2}\,dx)^{1/2}$, associated with the inner product given by $\langle u, \varphi\rangle=\int_{\mathbb{R}^{N}}\Delta u\Delta\varphi \,dx$. Recall the well-known Rellich inequality [32]

where $N\geq5$, $\overline{\mu}= \frac{1}{16}N^{2}(N-4)^{2}$. We now employ the following norm in $\mathscr{D}^{2, 2}(\mathbb{R}^{N})$:

Thanks to the Rellich inequality (2.1), we find that the above norm $\Vert \cdot \Vert _{\mu}$ is equivalent to the usual norm $(\int_{\mathbb{R}^{N}} \vert \Delta\cdot \vert ^{2}\,dx)^{1/2}$. Besides, we define the product space $(\mathscr{D}^{2, 2}(\mathbb{R}^{N}))^{2}$ endowed with the norm

As usual, we denote by G any closed subgroup of $O(\mathbb{N})$, the group of orthogonal linear transformations. Let $G_{x}=\{gx; g\in G\}$ be the orbit of $x\in\mathbb{R}^{N}$; $\vert G_{x} \vert $ denote the number of elements in $G_{x}$ and $\vert G_{0} \vert = \vert G_{\infty} \vert =1$. Denote $\vert G \vert =\inf_{x\in\mathbb{R}^{N}\backslash\{0\}} \vert G_{x} \vert $. Note that $\vert G \vert $ may be +∞. We call Ω a G-invariant subset of $\mathbb{R}^{N}$, if $x\in\Omega$, then $gx\in\Omega$ for all $g\in G$. A function $f: \mathbb{R}^{N}\mapsto\mathbb{R}$ is called G-invariant if $f(gx)=f(x)$ for every $g\in G$ and $x\in\mathbb{R}^{N}$. In particular, an $O(\mathbb{N})$-invariant function is called radial.

The natural functional space to frame the analysis of (1.1) by variational methods is the Hilbert space $(\mathscr{D}_{ G}^{2, 2}(\mathbb{R}^{N}))^{2}$, which is the subspace of $(\mathscr{D}^{2, 2}(\mathbb{R}^{N}))^{2}$ consisting of all G-invariant functions. This work is devoted to the study of the following systems:

To clearly describe the results of this paper, several notations should be presented:

where $\epsilon>0$, $\Lambda_{0}\triangleq\frac{N-4}{2}$, and the constant $C=C(N, \mu)>0$, depending only on N and μ. From [26, 33], we mention that $y_{\epsilon}(x)$ satisfies the following equations:

and

for all $\varphi\in\mathscr{D}^{2, 2}(\mathbb{R}^{N})$. Hence, we obtain (let $\varphi=y_{\epsilon}$)

According to [26, Lemma 2.1] and [33, Theorem 2], we remark that the function $U_{\mu}(x)$ in (2.4) is positive, radial symmetric, radially decreasing, and solves

By setting $r= \vert x \vert $, there holds that

where $O_{1}(r^{t})$ $(r\rightarrow r_{0})$ means that there exist constants $C_{1}$, $C_{2}>0$ such that $C_{1}r^{t}\leq O_{1}(r^{t})\leq C_{2}r^{t}$ as $r\rightarrow r_{0}$, $l_{1}(\mu)\triangleq\Lambda_{0}\vartheta(\mu)$, $l_{2}(\mu)\triangleq\Lambda_{0}(2-\vartheta(\mu))$, $\Lambda_{0}=\frac{N-4}{2}$, and $\vartheta(\mu): [0, \overline{\mu}]\mapsto[0, 1]$ is defined as

This implies $\vartheta(0)=0$, $\vartheta(\overline{\mu})=1$ and

Moreover, there exist positive constants $C_{3}=C_{3}(N, \mu)$ and $C_{4}=C_{4}(N, \mu)$ such that

The following hypotheses are needed.

1. (q.1)

   $Q(x)$ is G-invariant.

2. (q.2)

   $Q(x)\in\mathscr{C}(\mathbb{R}^{N})\cap L^{\infty}(\mathbb{R}^{N})$, and $Q_{+}(x)\not\equiv0$, where $Q_{+}(x)=\max\{0, Q(x)\}$.

3. (h.1)

   $h(x)$ is G-invariant.

4. (h.2)

   $h(x)$ is a nonnegative function in $\mathbb{R}^{N}$ such that

   $$0< \Vert h \Vert _{\theta}\triangleq \biggl( \int_{\mathbb{R}^{N}} h^{\theta}(x)\,dx \biggr)^{\frac{1}{\theta}}< +\infty \quad\text{with } \theta=\frac{2^{\ast\ast}}{2^{\ast\ast}-q}. $$

The main results of this work can be stated in the following.

Theorem 2.1

Assume that (q.1) and (q.2) hold. If

for certain $\epsilon>0$, where $Q_{+}(\infty)=\limsup_{ \vert x \vert \rightarrow\infty}Q_{+}(x)$, then problem $(\mathscr{P}^{Q}_{0})$ possesses at least one nontrivial solution in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$.

Corollary 2.1

Assume that (q.1) and (q.2) hold. Then we have the following statements.

1. (1)

   Problem $(\mathscr{P}^{Q}_{0})$ admits at least one nontrivial solution if

   $$Q(0)>0, \quad Q(0)\geq\max \bigl\{ \vert G \vert ^{\frac{2-2^{\ast\ast}}{2}} ( \mathcal{A}_{0}/\mathcal{A}_{\mu} )^{-\frac{2^{\ast\ast}}{2}} \Vert Q_{+} \Vert _{\infty}, Q_{+}(\infty) \bigr\} , $$

   and either (i) $Q(x)\geq Q(0)+\xi_{0} \vert x \vert ^{2^{\ast\ast}(l_{2}(\mu)-\Lambda_{0})}$ for some $\xi_{0}>0$ and $\vert x \vert $ small, or (ii) $\vert Q(x)-Q(0) \vert \leq \xi_{1} \vert x \vert ^{\varsigma}$ for some constants $\xi_{1}>0$, $\varsigma>2^{\ast\ast}(l_{2}(\mu)-\Lambda_{0})>0$ and $\vert x \vert $ small and

   $$ \int_{\mathbb{R}^{N}} \bigl(Q(x)-Q(0) \bigr) \vert x \vert ^{-2^{\ast\ast}l_{2}(\mu)}\,dx>0. $$

   (2.12)
2. (2)

   Problem $(\mathscr{P}^{Q}_{0})$ has at least one nontrivial solution if $\lim_{ \vert x \vert \rightarrow\infty}Q(x)=Q(\infty)$ exists and is positive,

   $$Q(\infty)\geq\max \bigl\{ \vert G \vert ^{\frac{2-2^{\ast\ast}}{2}} ( \mathcal{A}_{0}/\mathcal{A}_{\mu} )^{-\frac{2^{\ast\ast}}{2}} \Vert Q_{+} \Vert _{\infty}, Q_{+}(0) \bigr\} , $$

   and either (i) $Q(x)\geq Q(\infty)+\xi_{2} \vert x \vert ^{-2^{\ast\ast}(\Lambda_{0}-l_{1}(\mu))}$ for certain $\xi_{2}>0$ and large $\vert x \vert $, or (ii) $\vert Q(x)-Q(\infty) \vert \leq\xi_{3} \vert x \vert ^{-\kappa}$ for some constants $\xi_{3}>0$, $\kappa>2^{\ast\ast}(\Lambda_{0}-l_{1}(\mu))>0$ and large $\vert x \vert $ and

   $$ \int_{\mathbb{R}^{N}} \bigl(Q(x)-Q(\infty) \bigr) \vert x \vert ^{-2^{\ast\ast}l_{1}(\mu)}\,dx>0. $$

   (2.13)
3. (3)

   If $Q(x)\geq Q(\infty)=Q(0)>0$ on $\mathbb{R}^{N}$ and

   $$Q(\infty)=Q(0)\geq \vert G \vert ^{\frac{2-2^{\ast\ast}}{2}} (\mathcal{A}_{0}/ \mathcal{A}_{\mu} )^{-\frac{2^{\ast\ast}}{2}} \Vert Q_{+} \Vert _{\infty}, $$

   then problem $(\mathscr{P}^{Q}_{0})$ possesses at least one nontrivial solution.

Remark 2.1

Conditions (q.1) and (q.2) are essentially introduced in [9]. According to (q.2), we only presume that $Q(x)$ is bounded and continuous on $\mathbb{R}^{N}$. Hence, the above results do not require the continuity of $Q(x)$ at infinity.

Theorem 2.2

Assume that $\vert G \vert =+\infty$ and $Q_{+}(0)=Q_{+}(\infty)=0$. Then there exist infinitely many G-invariant solutions to problem $(\mathscr{P}^{Q}_{0})$.

Corollary 2.2

If Q is a radial function such that $Q_{+}(0)=Q_{+}(\infty)=0$, then there exist infinitely many radial solutions to problem $(\mathscr{P}^{Q}_{0})$.

Theorem 2.3

Let $\overline{Q}>0$ be a constant. Assume that $Q(x)\equiv\overline{Q}$ and (h.1), (h.2) hold. Then there exists $\sigma^{\ast}>0$ such that, for any $\sigma\in(0, \sigma^{\ast})$, problem $(\mathscr{P}^{\overline{Q}}_{\sigma})$ possesses at least two nontrivial solutions in $(\mathscr{D}_{ G}^{2, 2}(\mathbb{R}^{N}))^{2}$.

Remark 2.2

The main results of this paper extend and complement those of [4, 5, 26, 29, 30]. Even in the scalar cases $\sigma=0$, $0<\mu<\overline{\mu}$, $m=1$, and $u=v$, the above results in the whole space are new.

Throughout this paper, we denote various positive constants as $C_{i}\ (i=1, 2, \ldots)$ or C. The dual space of $(\mathscr {D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ ($(\mathscr{D}^{2, 2}(\mathbb{R}^{N}))^{2}$, resp.) is denoted by $(\mathscr{D}_{ G}^{-2, 2}(\mathbb{R}^{N}))^{2}$ ($(\mathscr{D}^{-2, 2}(\mathbb{R}^{N}))^{2}$, resp.). The ball of center x and radius r is denoted by $B_{r}(x)$. $o_{n}(1)$ is a generic infinitesimal value as $n\rightarrow\infty$. For any $\epsilon>0$, $t\in\mathbb{R}$, $O(\epsilon^{t})$ denotes the quantity satisfying $\vert O(\epsilon^{t}) \vert /\epsilon^{t}\leq C$, and $O_{1}(\epsilon^{t})$ $(\epsilon\rightarrow\epsilon_{0})$ means that there exist constants $C_{1}$, $C_{2}>0$ such that $C_{1}\epsilon^{t}\leq O_{1}(\epsilon^{t})\leq C_{2}\epsilon^{t}$ as $\epsilon\rightarrow\epsilon_{0}$. In a Banach space X, we denote by ‘→’ and ‘⇀’ strong and weak convergence, respectively. A functional $\mathcal{F}\in \mathscr{C}^{1}(X, \mathbb{R})$ is called to satisfy the $(PS)_{c}$ condition if each sequence $\{w_{n}\}$ in X satisfying $\mathcal{F}(w_{n})\rightarrow c$ in $\mathbb{R}$, $\mathcal{F}^{\prime}(w_{n})\rightarrow0$ in $X^{\ast}$ contains a strongly convergent subsequence.

The energy functional corresponding to problem $(\mathscr{P}^{Q}_{0})$ is defined on $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ by

It follows from (q.2) and the Rellich inequality (2.1) that $\mathcal{F}$ is a well-defined $\mathscr{C}^{1}$ functional on $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$. Then the critical points of $\mathcal{F}$ correspond to weak solutions of problem $(\mathscr{P}^{Q}_{0})$. According to the principle of symmetric criticality (see Lemma 3.1), any critical point of $\mathcal{F}$ in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ is also a solution of $(\mathscr{P}^{Q}_{0})$ in $(\mathscr {D}^{2, 2}(\mathbb{R}^{N}))^{2}$. This means that $(u, v)\in (\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ satisfies $(\mathscr{P}^{Q}_{0})$ if and only if, for any $(\varphi_{1}, \varphi_{2})\in(\mathscr{D}^{2, 2}(\mathbb{R}^{N}))^{2}$,

Lemma 3.1

If $Q(x)$ is a G-invariant function, then $\mathcal{F}^{\prime}(u, v)=0$ in $(\mathscr{D}_{G}^{-2, 2}(\mathbb{R}^{N}))^{2}$ implies $\mathcal{F}^{\prime}(u, v)=0$ in $(\mathscr{D}^{-2, 2}(\mathbb{R}^{N}))^{2}$.

Proof

The proof is similar to that of [9, Lemma 1] and is omitted here. □

For $\mu\in[0, \overline{\mu})$, $\varsigma_{i}\in(0, +\infty)$, $\alpha_{i}$, $\beta_{i}>1$, and $\alpha_{i}+\beta_{i}=2^{\ast\ast}\ (i=1, \ldots, m)$, we define

where $\tau_{\min}>0$ is a minimal point of $\mathscr{B}(\tau)$ and hence a root of the equation

Lemma 3.2

Let $y_{\epsilon}(x)$ be the minimizer of $\mathcal{A}_{\mu}$ defined in (2.4), $\mu\in[0, \overline{\mu})$, $\varsigma_{i}\in(0, +\infty)$, $\alpha_{i}$, $\beta_{i}>1$, and $\alpha_{i}+\beta_{i}=2^{\ast\ast}\ (i=1, \ldots, m)$. Then we have the following statements.

1. (i)

   $\mathcal{A}_{\mu, m}=\mathscr{B}(\tau_{\min})\mathcal{A}_{\mu}$;

2. (ii)

   $\mathcal{A}_{\mu, m}$ has the minimizer $(y_{\epsilon}(x), \tau_{\min}y_{\epsilon}(x))$ for all $\epsilon>0$.

Proof

The proof is a repeat of that in [19, Theorem 2.2] (see also [21, Theorem 5]) and hence is omitted here. □

To find conditions under which the Palais–Smale condition holds, we need the following concentration compactness principle due to Lions [34].

Lemma 3.3

Let $\{(u_{n}, v_{n})\}$ be a weakly convergent sequence to $(u, v)$ in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ such that $\vert \Delta u_{n} \vert ^{2}\rightharpoonup\eta^{(1)}$, $\vert \Delta v_{n} \vert ^{2}\rightharpoonup\eta^{(2)}$, $\vert u_{n} \vert ^{\alpha_{i}} \vert v_{n} \vert ^{\beta_{i}} \rightharpoonup\nu^{(i)}\ (i=1, \ldots, m)$, $\vert x \vert ^{-4} \vert u_{n} \vert ^{2}\rightharpoonup\gamma^{(1)}$, and $\vert x \vert ^{-4} \vert v_{n} \vert ^{2}\rightharpoonup\gamma^{(2)}$ in the sense of measures. Then there exists some at most countable set $\mathscr{J}$, $\{\eta_{j}^{(1)}\geq 0\}_{j\in\mathscr{J}\cup\{0\}}$, $\{\eta_{j}^{(2)}\geq 0\}_{j\in\mathscr{J}\cup\{0\}}$, $\{\nu_{j}^{(i)}\geq 0\}_{j\in\mathscr{J}\cup\{0\}}$, $\gamma_{0}^{(1)}\geq0$, $\gamma_{0}^{(2)}\geq0$, $\{x_{j}\}_{j\in\mathscr{J}}\subset\mathbb{R}^{N}\backslash\{0\}$ such that

1. (a)

   $\eta^{(1)}\geq \vert \Delta u \vert ^{2}+ \sum_{j\in\mathscr {J}}\eta_{j}^{(1)}\delta_{x_{j}}+\eta_{0}^{(1)}\delta_{0}$, $\eta^{(2)}\geq \vert \Delta v \vert ^{2}+ \sum_{j\in\mathscr {J}}\eta_{j}^{(2)}\delta_{x_{j}}+\eta_{0}^{(2)}\delta_{0}$,

2. (b)

   $\nu^{(i)}= \vert u \vert ^{\alpha_{i}} \vert v \vert ^{\beta_{i}}+ \sum_{j\in\mathscr {J}}\nu_{j}^{(i)}\delta_{x_{j}}+\nu_{0}^{(i)}\delta_{0}$, $i=1, \ldots, m$,

3. (c)

   $\gamma^{(1)}= \vert x \vert ^{-4} \vert u \vert ^{2}+\gamma_{0}^{(1)}\delta_{0}$, $\gamma^{(2)}= \vert x \vert ^{-4} \vert v \vert ^{2}+\gamma_{0}^{(2)}\delta_{0}$,

4. (d)

   $\mathcal{A}_{0, m} (\sum_{i=1}^{m}\varsigma_{i}\nu_{j}^{(i)} )^{\frac{2}{2^{\ast\ast}}}\leq\eta_{j}^{(1)}+\eta_{j}^{(2)}$,

5. (e)

   $\mathcal{A}_{\mu, m} (\sum_{i=1}^{m}\varsigma_{i}\nu_{0}^{(i)} )^{\frac{2}{2^{\ast\ast}}}\leq\eta_{0}^{(1)}+\eta_{0}^{(2)} -\mu(\gamma_{0}^{(1)}+\gamma_{0}^{(2)})$,

where $\delta_{x_{j}}$, $j\in\mathscr{J}\cup \{0 \}$, is a Dirac mass of 1 concentrated at $x_{j}\in\mathbb{R}^{N}$.

To establish the existence results for problem $(\mathscr{P}^{Q}_{0})$, we need the following local $(PS)_{c}$ condition, which is indispensable for the proof of Theorem 2.1.

Lemma 3.4

Assume that (q.1) and (q.2) hold. Then the $(PS)_{c}$ condition in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ holds for $\mathcal{F}$ if

Proof

We follow closely the arguments in [9, Proposition 2]. It is trivial to check that the $(PS)_{c}$ sequence $\{(u_{n}, v_{n})\}$ of $\mathcal{F}$ is bounded in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$. Then we may assume that $(u_{n}, v_{n})\rightharpoonup(u, v)$ in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$. In view of Lemma 3.3, there exist measures $\eta^{(1)}$, $\eta^{(2)}$, $\nu^{(i)}\ (i=1, \ldots, m)$, $\gamma^{(1)}$, and $\gamma^{(2)}$ such that relations (a)–(e) of this lemma hold. We begin by considering the concentration at the point $x_{j}\in\mathbb{R}^{N}\backslash\{0\}$, $j\in\mathscr{J}$. For $\epsilon>0$ small, we define the cut-off function $\psi_{x_{j}}^{\epsilon}(x)\in\mathscr {C}_{0}^{\infty}(\mathbb{R}^{N})$ such that $0\leq\psi_{x_{j}}^{\epsilon}(x)\leq1$, $\psi_{x_{j}}^{\epsilon}(x)=1$ in $B_{\epsilon}(x_{j})$, $\psi_{x_{j}}^{\epsilon}(x)=0$ on $\mathbb{R}^{N}\backslash B_{2\epsilon}(x_{j})$, $\vert \nabla\psi_{x_{j}}^{\epsilon} \vert \leq 2/\epsilon$, and $\vert \Delta\psi_{x_{j}}^{\epsilon} \vert \leq 2/\epsilon^{2}$ on $\mathbb{R}^{N}$. Then, by Lemma 3.1, $\lim_{n\rightarrow\infty}\langle\mathcal{F}^{\prime}(u_{n}, v_{n}), (u_{n}\psi_{x_{j}}^{\epsilon}, v_{n}\psi_{x_{j}}^{\epsilon})\rangle=0$; hence, combining (3.2), the Hölder inequality, and the Sobolev inequality, we derive

As $\epsilon\rightarrow0$, it follows from (3.8) and Lemma 3.3 that

This means that the concentration of the measures $\nu^{(i)}\ (i=1, \ldots, m)$ cannot occur at points where $Q(x_{j})\leq0$. By virtue of (3.9) and (d) of Lemma 3.3, we conclude that either (i) $\nu_{j}^{(i)}=0\ (i=1, \ldots, m)$ or (ii) $\sum_{i=1}^{m}\varsigma_{i}\nu_{j}^{(i)}\geq (\mathcal{A}_{0, m}/ \Vert Q_{+} \Vert _{\infty})^{N/4}$. Let us now study the possibility of concentration at $x=0$ and at ∞. By the argument similar to that of $x_{j}\in \mathbb{R}^{N}\backslash\{0\}$, we find $\eta_{0}^{(1)}+\eta_{0}^{(2)} -\mu(\gamma_{0}^{(1)}+\gamma_{0}^{(2)}) -Q(0)\sum_{i=1}^{m}\varsigma_{i}\nu_{0}^{(i)} \leq0$. Together with (e) of Lemma 3.3, it follows that either (iii) $\nu_{0}^{(i)} =0\ (i=1, \ldots, m)$ or (iv) $\sum_{i=1}^{m}\varsigma_{i}\nu_{0}^{(i)}\geq (\mathcal{A}_{\mu, m}/Q_{+}(0))^{N/4}$. To discuss the concentration at infinity of the sequence $\{(u_{n}, v_{n})\}$, we define the following quantities:

1. (1)

   $\eta_{\infty}^{(1)}=\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}}\int_{ \vert x \vert >R} \vert \Delta u_{n} \vert ^{2}\,dx$, $\eta_{\infty}^{(2)}=\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}}\int_{ \vert x \vert >R} \vert \Delta v_{n} \vert ^{2}\,dx$,

2. (2)

   $\nu_{\infty}^{(i)}= \lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}}\int_{ \vert x \vert >R} \vert u_{n} \vert ^{\alpha_{i}} \vert v_{n} \vert ^{\beta_{i}}\,dx$, $i=1, \ldots, m$,

3. (3)

   $\gamma_{\infty}^{(1)}=\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}}\int_{ \vert x \vert >R} \vert x \vert ^{-4} \vert u_{n} \vert ^{2}\,dx$, $\gamma_{\infty}^{(2)}=\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}}\int_{ \vert x \vert >R} \vert x \vert ^{-4} \vert v_{n} \vert ^{2}\,dx$.

It is obvious that $\eta_{\infty}^{(1)}$, $\eta_{\infty}^{(2)}$, $\nu_{\infty}^{(i)}\ (i=1, \ldots, m)$, $\gamma_{\infty}^{(1)}$, and $\gamma_{\infty}^{(2)}$ defined by (1)–(3) exist and are finite. For $R>1$, let $\psi_{R}(x)\in\mathscr {C}^{\infty}(\mathbb{R}^{N})$ be a function such that $0\leq\psi_{R}(x)\leq1$, $\psi_{R}(x)=1$ for $\vert x \vert >R+1$, $\psi_{R}(x)=0$ for $\vert x \vert < R$, $\vert \nabla\psi_{R} \vert \leq2/R$, and $\vert \Delta\psi_{R} \vert \leq2/R^{2}$. Because the sequence $\{(u_{n}\psi_{R}, v_{n}\psi_{R})\}$ is bounded in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$, we deduce from (3.2) and the fact that $\alpha_{i}+\beta_{i}=2^{\ast\ast}\ (i=1, \ldots, m)$ that

Furthermore, by utilizing the Hölder inequality and the Sobolev inequality, we obtain

Similarly, we have $\lim_{R\rightarrow \infty}\overline{\lim_{n\rightarrow\infty}}\int_{\mathbb{R}^{N}} (2 \vert \Delta v_{n}\langle\nabla v_{n}, \nabla \psi_{R}\rangle \vert + \vert v_{n}\Delta v_{n}\Delta\psi_{R} \vert )\,dx=0$. Consequently, it follows from (3.10) and definitions (1)–(3) of the quantities $\eta_{\infty}^{(1)}$, $\eta_{\infty}^{(2)}$, $\nu_{\infty}^{(i)}\ (i=1, \ldots, m)$, $\gamma_{\infty}^{(1)}$, and $\gamma_{\infty}^{(2)}$ that

Moreover, in view of (3.3), we find $\mathcal{A}_{\mu, m}(\sum_{i=1}^{m} \varsigma_{i}\nu_{\infty}^{(i)})^{\frac{2}{2^{\ast\ast}}} \leq\eta_{\infty}^{(1)}+\eta_{\infty}^{(2)} -\mu(\gamma_{\infty}^{(1)}+\gamma_{\infty}^{(2)})$. This, combined with (3.11), implies that either (v) $\nu_{\infty}^{(i)}=0\ (i=1, \ldots, m)$ or (vi) $\sum_{i=1}^{m}\varsigma_{i}\nu_{\infty}^{(i)} \geq(\mathcal{A}_{\mu, m}/Q_{+}(\infty))^{N/4}$. In the following, we claim that (ii), (iv), and (vi) cannot occur. For every continuous nonnegative function ψ such that $0\leq\psi(x)\leq 1$ on $\mathbb{R}^{N}$, we find

Note that the measures $\nu^{(i)}\ (i=1, \ldots, m)$ are bounded and G-invariant. This means that if (ii) holds, then the set $\mathscr{J}$ must be finite. Moreover, if $x_{j}\neq0$ is a singular point of $\nu^{(i)}\ (i=1, \ldots, m)$, so is $gx_{j}$ for each $g\in G$, and the mass of $\nu^{(i)}\ (i=1, \ldots, m)$ concentrated at $gx_{j}$ is the same for every $g\in G$. Assuming that (ii) occurs for some $j\in\mathscr{J}$ with $x_{j}\neq0$, we choose ψ with compact support so that $\psi(gx_{j})=1$ for every $g\in G$, and we derive

which is impossible. Similarly, assuming that (iv) holds for $x=0$, we take ψ with compact support so that $\psi(0)=1$, and we have

a contradiction to (3.7). Finally, if (vi) occurs, we choose $\psi=\psi_{R}$ to obtain

which contradicts (3.7). Hence, $\nu_{j}^{(i)}=0\ (i=1, \ldots, m)$ for all $j\in\mathscr{J}\cup\{0, \infty\}$, and this yields

Finally, taking into account $\lim_{n\rightarrow\infty}\langle \mathcal{F}^{\prime}(u_{n}, v_{n})-\mathcal{F}^{\prime}(u, v), (u_{n}-u, v_{n}-v)\rangle=0$, we naturally deduce $(u_{n}, v_{n})\rightarrow(u, v)$ as $n\rightarrow\infty$ in $(\mathscr {D}^{2, 2}(\mathbb{R}^{N}))^{2}$. □

Thanks to Lemma 3.4, we immediately obtain the following result.

Corollary 3.1

If $\vert G \vert =+\infty$ and $Q_{+}(0)=Q_{+}(\infty)=0$, then the functional $\mathcal{F}$ satisfies the $(PS)_{c}$ condition for every $c\in\mathbb{R}$.

Proof of Theorem 2.1

Let $y_{\epsilon}$ be the extremal function satisfying (2.4)–(2.10). We now choose $\epsilon>0$ such that (2.11) is fulfilled. It is clear from (q.2), (3.1), and (3.2) that there exist constants $\alpha_{0}>0$ and $\rho>0$ such that $\mathcal{F}(u, v)\geq\alpha_{0}$ for all $\Vert (u, v) \Vert _{\mu}=\rho$. Moreover, if we set $u=y_{\epsilon}$, $v=\tau_{\min}y_{\epsilon}$, and

with $t\geq0$, then $\max_{t\geq0}\Phi(t)$ is attained for some finite $\overline{t}>0$ with $\Phi^{\prime}(\overline{t})=0$. This yields

Besides, because $\mathcal{F}(ty_{\epsilon}, t\tau_{\min}y_{\epsilon})\rightarrow-\infty$ as $t\rightarrow+\infty$, there exists $t_{0}>0$ such that $\Vert (t_{0}y_{\epsilon}, t_{0}\tau_{\min}y_{\epsilon }) \Vert _{\mu}>\rho$ and $\mathcal{F}(t_{0}y_{\epsilon}, t_{0}\tau_{\min}y_{\epsilon})<0$. Now, we define

where $\Gamma=\{\gamma\in\mathscr{C}([0, 1], (\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}); \gamma(0)=(0, 0), \mathcal{F}(\gamma(1))<0, \Vert \gamma(1) \Vert _{\mu }>\rho\}$. It follows directly from (2.5), (2.11), (3.4), (3.5), (3.7), (3.12), (3.13), and Lemma 3.2 that

If $c_{0}< c_{0}^{\ast}$, then the $(PS)_{c}$ condition holds by Lemma 3.4. Thus we arrive at the conclusion by the mountain pass theorem in [35]. If $c_{0}=c_{0}^{\ast}$, then $\gamma(t)=(tt_{0}y_{\epsilon}, tt_{0}\tau_{\min}y_{\epsilon})$, with $0\leq t\leq1$, is a path in Γ such that $\max_{t\in [0, 1]}\mathcal{F}(\gamma(t))=c_{0}$. Hence, either $\Phi^{\prime}(\overline{t})=0$ and we are done, or γ can be deformed to a path $\widetilde{\gamma}\in\Gamma$ with $\max_{t\in[0, 1]}\mathcal{F}(\widetilde{\gamma}(t))< c_{0}$ and we have a contradiction. Thus we conclude from Lemma 3.1 that there exists a nontrivial G-invariant solution $(u_{0}, v_{0})\in(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N})\backslash\{0\})^{2}$ to problem $(\mathscr {P}_{0}^{Q})$ and the results follow. □

Proof of Corollary 2.1

In view of (2.6) and Theorem 2.1, it is sufficient to prove that

for some $\epsilon>0$, where $\widetilde{Q}=\max\{ \vert G \vert ^{\frac{2-2^{\ast\ast}}{2}} (\mathcal{A}_{0}/\mathcal{A}_{\mu})^{-\frac{2^{\ast\ast}}{2}} \Vert Q_{+} \Vert _{\infty}, Q_{+}(0), Q_{+}(\infty)\}$.

Part (1), case (i). By virtue of (3.14), we need to show that

for certain $\epsilon>0$. By the hypothesis, we choose $\varrho_{0}>0$ so that $Q(x)\geq Q(0)+\xi_{0} \vert x \vert ^{2^{\ast\ast}(l_{2}(\mu)-\Lambda_{0})}$ for $\vert x \vert \leq\varrho_{0}$. It follows from $2^{\ast\ast}\Lambda_{0}=N$ and (2.8) that

as $\epsilon\rightarrow0$. On the other hand, for any $\epsilon>0$, we deduce from (2.8), (2.9), and the fact that $2^{\ast\ast}l_{2}(\mu)>N$ that

for some constant $\overline{C}_{1}>0$ independent of ϵ. Combining (3.16) and (3.17), we obtain (3.15) for ϵ sufficiently small.

Part (1), case (ii). By the hypothesis, we choose $\varrho_{1}>0$ so that $\vert Q(x)-Q(0) \vert \leq\xi_{1} \vert x \vert ^{\varsigma}$ for $\vert x \vert \leq\varrho_{1}$. Taking into account $\varsigma>2^{\ast\ast}(l_{2}(\mu)-\Lambda_{0})>0$, $N-1+\varsigma-2^{\ast\ast}l_{2}(\mu)>-1$ and $N-1-2^{\ast\ast}l_{2}(\mu)<-1$, we derive

Thus, by (2.8), (2.12), and the Lebesgue dominated convergence theorem, we have

Hence (3.15) holds for ϵ small enough.

Part (2), case (i). According to (3.14), we need to prove that

for certain $\epsilon>0$. By the assumption, we take $\varrho_{2}>0$ such that $Q(x)\geq Q(\infty)+ \xi_{2} \vert x \vert ^{-2^{\ast\ast}(\Lambda_{0}-l_{1}(\mu))}$ for all $\vert x \vert \geq\varrho_{2}$. It follows from (2.7) that

as $\epsilon\rightarrow+\infty$. On the other hand, for any $\epsilon>0$, we conclude from (2.7), (q.2), and the fact that $N-1-2^{\ast\ast}l_{1}(\mu)>-1$ that

for some constant $\overline{C}_{2}>0$ independent of $\epsilon>0$. By putting these two estimates together, we obtain (3.18) for $\epsilon>0$ large enough.

Part (2), case (ii). By the assumption, we take $\varrho_{3}>0$ such that $\vert Q(x)-Q(\infty) \vert \leq \xi_{3} \vert x \vert ^{-\kappa}$ for all $\vert x \vert \geq\varrho_{3}$. Taking into account $\kappa>2^{\ast\ast}(\Lambda_{0}-l_{1}(\mu))>0$, $N-1-\kappa-2^{\ast\ast}l_{1}(\mu)<-1$ and $N-1-2^{\ast\ast}l_{1}(\mu)>-1$, we find

Therefore, by (2.7), (2.13), and the Lebesgue dominated convergence theorem, we obtain

Thus (3.18) holds for $\epsilon>0$ enough large. Similar to the above, we find that part (3) follows. □

To prove Theorem 2.2, we need the following symmetric mountain pass theorem (see [36] or [37, Theorem 9.12]).

Lemma 3.5

Let X be an infinite dimensional Banach space, and let $\mathcal{F}\in\mathscr{C}^{1}(X, \mathbb{R})$ be an even functional satisfying the $(PS)_{c}$ condition for each c and $\mathcal{F}(0)=0$. Furthermore, one supposes that:

1. (i)

   there exist constants $\widetilde{\alpha}>0$ and $\rho>0$ such that $\mathcal{F}(w)\geq\widetilde{\alpha}$ for all $\Vert w \Vert =\rho$;

2. (ii)

   there exists an increasing sequence of subspaces $\{X_{k}\}$ of X, with $\dim X_{k}=k$, such that for every k one can find a constant $R_{k}>0$ such that $\mathcal{F}(w)\leq0$ for all $w\in X_{k}$ with $\Vert w \Vert \geq R_{k}$.

Then $\mathcal{F}$ possesses a sequence of critical values $\{c_{k}\}$ tending to ∞ as $k\rightarrow\infty$.

Proof of Theorem 2.2

We follow closely the arguments in [9, Theorem 3] (see also [38, Theorem 3]). By virtue of Lemma 3.5 with $X=(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ and $w=(u, v)\in X$, we easily see from (q.2), (2.2), (3.1), and (3.3) that

Thanks to $2^{\ast\ast}>2$, there exist constants $\widetilde{\alpha}>0$ and $\rho>0$ such that $\mathcal{F}(u, v)\geq\widetilde{\alpha}$ for any $(u, v)$ with $\Vert (u, v) \Vert _{\mu}=\rho$. To find an appropriate sequence of finite dimensional subspaces of $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$, we set $\Omega=\{x\in\mathbb{R}^{N}; Q(x)>0\}$. The set Ω is G-invariant, and we can define $(\mathscr{D}_{G}^{2, 2}(\Omega))^{2}$, which is the subspace of G-invariant functions of $(\mathscr{D}^{2, 2}(\Omega))^{2}$. Extending functions in $(\mathscr{D}_{G}^{2, 2}(\Omega))^{2}$ by 0 outside Ω, we can presume that $(\mathscr{D}_{ G}^{2, 2}(\Omega))^{2}\subset(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$. Let $\{X_{k}\}$ be an increasing sequence of subspaces of $(\mathscr{D}_{G}^{2, 2}(\Omega))^{2}$ with $\dim X_{k}=k$ for every k. As in [38, Theorem 3], we define $\varphi_{1, k}$, …, $\varphi_{k, k}\in\mathscr {C}_{0}^{\infty}(\mathbb{R}^{N})$ such that $0\leq\varphi_{i, k}\leq1$, $\operatorname{supp}(\varphi_{i, k})\cap\operatorname{supp}(\varphi_{j, k})=\emptyset$, $i\neq j$, and

Taking $e_{i, k}=(a\varphi_{i, k}, b\varphi_{i, k})\in X_{k}$, $i=1, \ldots, k$, and $X_{k}=\operatorname{span}\{e_{1, k}, \ldots, e_{k, k}\}$, where a and b are two positive constants, we conclude from the construction of $X_{k}$ that $\dim X_{k}=k$ for every k. Therefore, there exists a constant $\epsilon(k)>0$ such that

for all $(\tilde{u}, \tilde{v})=\sum_{i=1}^{k}t_{i, k}e_{i, k}\in X_{k}$, with $\Vert (\tilde{u}, \tilde{v}) \Vert _{\mu}=1$. Hence, if $(u, v)\in X_{k}\backslash\{(0, 0)\}$, then we write $(u, v)=t(\tilde{u}, \tilde{v})$, with $t= \Vert (u, v) \Vert _{\mu}$ and $\Vert (\tilde{u}, \tilde{v}) \Vert _{\mu}=1$. Therefore, we derive

for $t>0$ sufficiently large. By Corollary 3.1 and Lemma 3.5, we conclude that there exists a sequence of critical values $c_{k}\rightarrow\infty$ as $k\rightarrow\infty$ and the results follow. □

Proof of Corollary 2.2

Because $Q(x)$ is radial, we know that the corresponding group $G=O(\mathbb{N})$ and $\vert G \vert =+\infty$. By Corollary 3.1, $\mathcal{F}$ satisfies the $(PS)_{c}$ condition for every $c\in\mathbb{R}$. Therefore, we deduce from Theorem 2.2 that the results follow. □

The purpose of this section is to investigate problem $(\mathscr {P}_{\sigma}^{\overline{Q}})$ and prove Theorem 2.3; here we always presume that $\sigma>0$ and $Q(x)\equiv\overline{Q}>0$ is a constant. The corresponding energy functional of problem $(\mathscr{P}_{\sigma}^{\overline{Q}})$ is defined on $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ by

where $1< q<2$. In view of (h.2), (2.3), and the Hölder inequality, we find

It follows from (4.1) and (4.2) that $\mathscr{E}_{\sigma}\in\mathscr{C}^{1}((\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}, \mathbb{R})$ and there exists a one-to-one correspondence between the weak solutions of $(\mathscr {P}_{\sigma}^{\overline{Q}})$ and the critical points of $\mathscr{E}_{\sigma}$. We now observe that an analogously symmetric criticality principle of Lemma 3.1 clearly holds. Consequently, the weak solutions of problem $(\mathscr {P}_{\sigma}^{\overline{Q}})$ are exactly the critical points of the functional $\mathscr{E}_{\sigma}$.

Lemma 4.1

Assume that (h.1) and (h.2) hold. Then there exists a positive constant M depending only on N, q, $\mathcal{A}_{\mu}$, and $\Vert h \Vert _{\theta}$, such that any bounded sequence $\{(u_{n}, v_{n})\}\subset(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ satisfying

contains a convergent subsequence.

Proof

By the hypothesis, $\{(u_{n}, v_{n})\}$ is bounded in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$. Hence we obtain a subsequence, still denoted by $\{(u_{n}, v_{n})\}$, satisfying $(u_{n}, v_{n})\rightharpoonup(u, v)$ in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$, $(u_{n}, v_{n})\rightarrow(u, v)$ a.e. in $\mathbb{R}^{N}$ and $(u_{n}, v_{n})\rightarrow(u, v)$ in $(L_{\operatorname{loc}}^{r}(\mathbb{R}^{N}))^{2}$ for all $r\in[1, 2^{\ast\ast})$. By virtue of (h.2), the Hölder inequality and the Lebesgue dominated theorem, we derive

Applying the standard argument, we easily check from (4.4) that $(u, v)$ is a critical point of $\mathscr{E}_{\sigma}$. Further, in view of (h.2), (4.1), (4.2), and the Hölder inequality, by direct calculation, we obtain

where $M=(2-q)(\frac{qN}{4})^{\frac{q}{2-q}} (\frac{2^{\ast\ast}-q}{2^{\ast\ast}q}\mathcal{A}_{\mu}^{-\frac{q}{2}} \Vert h \Vert _{\theta})^{\frac{2}{2-q}}$ is a positive constant. We now set $\widetilde{u}_{n}=u_{n}-u$ and $\widetilde{v}_{n}=v_{n}-v$. Then, by the Brezis–Lieb lemma [39] and arguing as in [40, Lemma 2.1], we have

Taking into account $\mathscr{E}_{\sigma}(u_{n}, v_{n})=c+o_{n}(1)$ and $\mathscr{E}_{\sigma}^{\prime}(u_{n}, v_{n})=o_{n}(1)$, we conclude from (4.1), (4.4), (4.6), and (4.7) that

and

As a result, for a subsequence $\{(\widetilde{u}_{n}, \widetilde{v}_{n})\}$, we find

as $n\rightarrow\infty$. It follows from (3.3) that $\mathcal{A}_{\mu, m}(\overline{\xi}/\overline{Q})^{\frac{2}{2^{\ast\ast}}}\leq \overline{\xi}$. This yields either $\overline{\xi}=0$ or $\overline{\xi}\geq\overline{Q}^{1-\frac{N}{4}}\mathcal{A}_{\mu, m}^{\frac{N}{4}}$. If $\overline{\xi}\geq \overline{Q}^{1-\frac{N}{4}}\mathcal{A}_{\mu, m}^{\frac{N}{4}}$, then we deduce from (4.5), (4.8), and (4.9) that

which contradicts (4.3). Therefore, we obtain $\Vert (\widetilde{u}_{n}, \widetilde{v}_{n}) \Vert _{\mu }^{2}\rightarrow 0$ as $n\rightarrow+\infty$, and hence, $(u_{n}, v_{n})\rightarrow (u, v)$ in $(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$. The conclusion of this lemma follows. □

Lemma 4.2

Assume that (h.1) and (h.2) hold. Then there exists $\sigma_{1}^{\ast}>0$ such that for any $\sigma\in(0, \sigma_{1}^{\ast})$ the following geometric conditions for $\mathscr{E}_{\sigma}(u, v)$ hold:

1. (i)

   $\mathscr{E}_{\sigma}(0, 0)=0$; there exist constants $\overline{\alpha}>0$ and $\rho>0$ such that $\mathscr{E}_{\sigma}(u, v)\geq\overline{\alpha}$ for all $\Vert (u, v) \Vert _{\mu}=\rho$;

2. (ii)

   there exists $(e_{u}, e_{v})\in (\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}$ such that $\Vert (e_{u}, e_{v}) \Vert _{\mu}>\rho$ and $\mathscr{E}_{\sigma}(e_{u}, e_{v})< 0$.

Proof

In view of (h.2), (3.3), (4.1), (4.2), and the Hölder inequality, by direct computation, we derive

for any $\varsigma_{0}\in(0, \frac{1}{2})$, where $C(\varsigma_{0})=(\frac{2}{q}-1)\varsigma_{0} [C \Vert h \Vert _{\theta}/(2\varsigma_{0})]^{2/(2-q)}$ is a positive constant. It follows from the last inequality in (4.10) that there exist constants $\overline{\alpha}>0$, $\rho>0$, and $\sigma_{1}^{\ast}>0$ such that $\mathscr{E}_{\sigma}(u, v)\geq\overline{\alpha}>0$ for all $\Vert (u, v) \Vert _{\mu}=\rho$, $\varsigma_{0}\in(0, \frac{1}{2})$ and $\sigma\in(0, \sigma_{1}^{\ast})$. This yields (i).

On the other hand, taking into account $\int_{\mathbb{R}^{N}}h(x)( \vert u \vert ^{q}+ \vert v \vert ^{q})\,dx\geq0$, we deduce from (4.1) that there exists $(\check{u}, \check{v})\in(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N})\backslash\{0\})^{2}$ such that $\mathscr {E}_{\sigma}(t\check{u}, t\check{v})\rightarrow-\infty$ as $t\rightarrow+\infty$. Therefore, we can choose $(e_{u}, e_{v})=(T\check{u}, T\check{v})$ ($T>0$ large enough) such that $\Vert (e_{u}, e_{v}) \Vert _{\mu}>\rho$ and $\mathscr {E}_{\sigma}(e_{u}, e_{v})< 0$. Thus (ii) follows. □

Lemma 4.3

Assume that (h.1) and (h.2) hold. Then there exists $\sigma_{2}^{\ast}>0$ such that

for any $\sigma\in(0, \sigma_{2}^{\ast})$ and small $\epsilon>0$, where $M>0$ is given in Lemma 4.1 and $\tau_{\min}>0$ satisfies (3.4)–(3.6).

Proof

Similar to the proof in Alves [41, Theorem 3], we define the functions

and

with $t\geq0$. Note that $\widetilde{\Psi}(0)=0$, $\widetilde{\Psi}(t)>0$ for $t\rightarrow0^{+}$, and $\lim_{t\rightarrow+\infty}\widetilde{\Psi}(t)=-\infty$. Hence, $\sup_{t\geq0}\widetilde{\Psi}(t)$ can be achieved at some finite $t_{\epsilon}^{0}>0$ at which $\widetilde{\Psi}^{\prime}(t)$ becomes zero. In view of (2.5), (2.6), (3.4)–(3.6), (4.13), and Lemma 3.2, by simple arithmetic, we derive

Let $\overline{\sigma}>0$ be such that

On the one hand, by virtue of (h.1), (h.2), (2.5), and (4.12), we conclude that

and there exists $T_{0}\in(0, 1)$ independent of ϵ such that

On the other hand, it follows from (4.12), (4.13), and (4.14) that

Now, taking $\sigma>0$ such that $-\frac{\sigma}{q}T_{0}^{q}(1+\tau_{\min}^{q}) \int_{\mathbb{R}^{N}}h(x)y_{\epsilon}^{q}\,dx <-M\sigma^{\frac{2}{2-q}}$, namely

we find from (4.16) that

Choosing $\sigma_{2}^{\ast}=\min\{\overline{\sigma},\widetilde{\sigma}\}$, we deduce from (4.15) and (4.17) that

which implies (4.11). Hence the results of this lemma follow. □

Proof of Theorem 2.3

Taking $\rho>0$ and $\sigma^{\ast}=\min\{\sigma_{1}^{\ast}, \sigma_{2}^{\ast}\}$, for $0<\sigma<\sigma^{\ast}$, given in the proofs of Lemmas 4.2 and 4.3, we define

where $\overline{\mathcal{B}}_{\rho}(0)=\{(u, v)\in(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}; \Vert (u, v) \Vert _{\mu}\leq\rho\}$. It is easy to see that the metric space $\overline{\mathcal{B}}_{\rho}(0)$ is complete. According to the Ekeland variational principle [42], we deduce that there exists a sequence $\{(u_{n}, v_{n})\}\subset\overline{\mathcal{B}}_{\rho}(0)$ such that $\mathscr{E}_{\sigma}(u_{n}, v_{n})\rightarrow c_{1}$ and $\mathscr{E}_{\sigma}^{\prime}(u_{n}, v_{n})\rightarrow0$ as $n\rightarrow\infty$.

Let $\varphi_{0}$, $\psi_{0}\in\mathscr {C}_{0}^{\infty}(\mathbb{R}^{N})$ be the G-invariant functions such that $\varphi_{0}$, $\psi_{0}> 0$. It follows from (h.1) and (h.2) that $\int_{\mathbb{R}^{N}}h(x)( \varphi_{0}^{q}+\psi_{0}^{q})\,dx>0$. In view of $1< q<2<2^{\ast\ast}$, we find that there exists $\tilde{t}_{0}=\tilde{t}_{0}(\varphi_{0}, \psi_{0})>0$ sufficiently small such that

This yields

By virtue of Lemma 4.1, $\mathscr{E}_{\sigma}$ admits a nontrivial critical point $(u_{1}, v_{1})$ with $\mathscr{E}_{\sigma}(u_{1}, v_{1})=c_{1}<0$. Applying the principle of symmetric criticality, we obtain that $(u_{1}, v_{1})$ is a nontrivial G-invariant solution of problem $(\mathscr{P}_{\sigma}^{\overline{Q}})$.

Furthermore, we now define

where $\Gamma=\{\gamma\in\mathscr{C}([0, 1], (\mathscr{D}_{ G}^{2, 2}(\mathbb{R}^{N}))^{2}); \gamma(0)=(0, 0), \gamma(1)=(e_{u}, e_{v})\}$. It follows from Lemmas 4.2 and 4.3 that

This, combined with the mountain pass theorem, implies that $c_{2}$ is another nonzero critical value of $\mathscr{E}_{\sigma}$. Similar to the above arguments, problem $(\mathscr{P}_{\sigma}^{\overline{Q}})$ possesses another nontrivial G-invariant solution $(u_{2}, v_{2})$ with $\mathscr{E}_{\sigma}(u_{2}, v_{2})=c_{2}>0$. □

Acknowledgements

ZD and DX are supported by the National Natural Science Foundation of China (Nos. 11471235; 11601052) and Chongqing Research Program of Basic Research and Frontier Technology (No. cstc2017jcyjBX0037). YH is supported by the National Natural Science Foundation of China (No. 11471235).

Availability of data and materials

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

Affiliations

1. Key Lab of Intelligent Analysis and Decision on Complex Systems, Chongqing University of Posts and Telecommunications, Chongqing, P.R. China

   • Zhiying Deng
   •  & Dong Xu

2. Department of Mathematics, Soochow University, Suzhou, P.R. China

   • Yisheng Huang

Contributions

The authors declare that this study was independently finished. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Zhiying Deng.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Reprints and Permissions