The energy functional corresponding to problem \((\mathscr{P}^{Q}_{0})\) is defined on \((\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}\) by
$$ \mathcal{F}(u, v)=\frac{1}{2} \bigl\Vert (u, v) \bigr\Vert _{\mu}^{2} -\frac{1}{2^{\ast\ast}} \int_{\mathbb{R}^{N}} Q(x)\sum_{i=1}^{m} \varsigma_{i} \vert u \vert ^{\alpha_{i}} \vert v \vert ^{\beta_{i}}\,dx. $$
(3.1)
It follows from (q.2) and the Rellich inequality (2.1) that \(\mathcal{F}\) is a well-defined \(\mathscr{C}^{1}\) functional on \((\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}\). Then the critical points of \(\mathcal{F}\) correspond to weak solutions of problem \((\mathscr{P}^{Q}_{0})\). According to the principle of symmetric criticality (see Lemma 3.1), any critical point of \(\mathcal{F}\) in \((\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}\) is also a solution of \((\mathscr{P}^{Q}_{0})\) in \((\mathscr {D}^{2, 2}(\mathbb{R}^{N}))^{2}\). This means that \((u, v)\in (\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}\) satisfies \((\mathscr{P}^{Q}_{0})\) if and only if, for any \((\varphi_{1}, \varphi_{2})\in(\mathscr{D}^{2, 2}(\mathbb{R}^{N}))^{2}\),
$$\begin{aligned} & \bigl\langle \mathcal{F}^{\prime}(u, v), ( \varphi_{1}, \varphi_{2}) \bigr\rangle \\ &\quad= \int_{\mathbb{R}^{N}} \biggl(\Delta u\Delta\varphi_{1}+\Delta v \Delta\varphi_{2}-\mu\frac{u\varphi_{1} +v\varphi_{2}}{ \vert x \vert ^{4}} \biggr)\,dx \\ &\qquad-\frac{1}{2^{\ast\ast}} \int_{\mathbb{R}^{N}}Q(x) \Biggl(\varphi_{1}\sum _{i=1}^{m} \varsigma_{i} \alpha_{i} \vert u \vert ^{\alpha_{i}-2}u \vert v \vert ^{\beta_{i}} +\varphi_{2}\sum_{i=1}^{m} \varsigma_{i}\beta_{i} \vert u \vert ^{\alpha_{i}} \vert v \vert ^{\beta_{i}-2}v \Biggr)\,dx=0. \end{aligned}$$
(3.2)
Lemma 3.1
If
\(Q(x)\)
is a
G-invariant function, then
\(\mathcal{F}^{\prime}(u, v)=0\)
in
\((\mathscr{D}_{G}^{-2, 2}(\mathbb{R}^{N}))^{2}\)
implies
\(\mathcal{F}^{\prime}(u, v)=0\)
in
\((\mathscr{D}^{-2, 2}(\mathbb{R}^{N}))^{2}\).
Proof
The proof is similar to that of [9, Lemma 1] and is omitted here. □
For \(\mu\in[0, \overline{\mu})\), \(\varsigma_{i}\in(0, +\infty)\), \(\alpha_{i}\), \(\beta_{i}>1\), and \(\alpha_{i}+\beta_{i}=2^{\ast\ast}\ (i=1, \ldots, m)\), we define
$$\begin{aligned} &\mathcal{A}_{\mu, m}\triangleq\inf_{(u, v)\in (\mathscr{D}^{2, 2}(\mathbb{R}^{N})\backslash\{0\})^{2}} \frac{\int_{\mathbb{R}^{N}} ( \vert \Delta u \vert ^{2}+ \vert \Delta v \vert ^{2}-\mu \frac{u^{2}+v^{2}}{ \vert x \vert ^{4}} )\,dx}{ (\int_{\mathbb{R}^{N}}\sum_{i=1}^{m}\varsigma_{i} \vert u \vert ^{\alpha_{i}} \vert v \vert ^{\beta_{i}}\,dx )^{\frac{2}{2^{\ast\ast}}}}, \end{aligned}$$
(3.3)
$$\begin{aligned} & \mathscr{B}(\tau)\triangleq\frac{1+\tau^{2}}{ (\sum_{i=1}^{m}\varsigma_{i}\tau^{\beta_{i}} )^{\frac{2}{2^{\ast\ast}}}},\quad \tau\geq0, \end{aligned}$$
(3.4)
$$\begin{aligned} &\mathscr{B}(\tau_{\min})\triangleq\min _{\tau\geq 0}\mathscr{B}(\tau)>0, \end{aligned}$$
(3.5)
where \(\tau_{\min}>0\) is a minimal point of \(\mathscr{B}(\tau)\) and hence a root of the equation
$$ \sum_{i=1}^{m} \varsigma_{i}\tau^{\beta_{i}-1} \bigl(\alpha_{i}\tau ^{2}-\beta_{i} \bigr) =0,\quad \tau\geq0. $$
(3.6)
Lemma 3.2
Let
\(y_{\epsilon}(x)\)
be the minimizer of
\(\mathcal{A}_{\mu}\)
defined in (2.4), \(\mu\in[0, \overline{\mu})\), \(\varsigma_{i}\in(0, +\infty)\), \(\alpha_{i}\), \(\beta_{i}>1\), and
\(\alpha_{i}+\beta_{i}=2^{\ast\ast}\ (i=1, \ldots, m)\). Then we have the following statements.
-
(i)
\(\mathcal{A}_{\mu, m}=\mathscr{B}(\tau_{\min})\mathcal{A}_{\mu}\);
-
(ii)
\(\mathcal{A}_{\mu, m}\)
has the minimizer
\((y_{\epsilon}(x), \tau_{\min}y_{\epsilon}(x))\)
for all
\(\epsilon>0\).
Proof
The proof is a repeat of that in [19, Theorem 2.2] (see also [21, Theorem 5]) and hence is omitted here. □
To find conditions under which the Palais–Smale condition holds, we need the following concentration compactness principle due to Lions [34].
Lemma 3.3
Let
\(\{(u_{n}, v_{n})\}\)
be a weakly convergent sequence to
\((u, v)\)
in
\((\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}\)
such that
\(\vert \Delta u_{n} \vert ^{2}\rightharpoonup\eta^{(1)}\), \(\vert \Delta v_{n} \vert ^{2}\rightharpoonup\eta^{(2)}\), \(\vert u_{n} \vert ^{\alpha_{i}} \vert v_{n} \vert ^{\beta_{i}} \rightharpoonup\nu^{(i)}\ (i=1, \ldots, m)\), \(\vert x \vert ^{-4} \vert u_{n} \vert ^{2}\rightharpoonup\gamma^{(1)}\), and
\(\vert x \vert ^{-4} \vert v_{n} \vert ^{2}\rightharpoonup\gamma^{(2)}\)
in the sense of measures. Then there exists some at most countable set
\(\mathscr{J}\), \(\{\eta_{j}^{(1)}\geq 0\}_{j\in\mathscr{J}\cup\{0\}}\), \(\{\eta_{j}^{(2)}\geq 0\}_{j\in\mathscr{J}\cup\{0\}}\), \(\{\nu_{j}^{(i)}\geq 0\}_{j\in\mathscr{J}\cup\{0\}}\), \(\gamma_{0}^{(1)}\geq0\), \(\gamma_{0}^{(2)}\geq0\), \(\{x_{j}\}_{j\in\mathscr{J}}\subset\mathbb{R}^{N}\backslash\{0\}\)
such that
-
(a)
\(\eta^{(1)}\geq \vert \Delta u \vert ^{2}+ \sum_{j\in\mathscr {J}}\eta_{j}^{(1)}\delta_{x_{j}}+\eta_{0}^{(1)}\delta_{0}\), \(\eta^{(2)}\geq \vert \Delta v \vert ^{2}+ \sum_{j\in\mathscr {J}}\eta_{j}^{(2)}\delta_{x_{j}}+\eta_{0}^{(2)}\delta_{0}\),
-
(b)
\(\nu^{(i)}= \vert u \vert ^{\alpha_{i}} \vert v \vert ^{\beta_{i}}+ \sum_{j\in\mathscr {J}}\nu_{j}^{(i)}\delta_{x_{j}}+\nu_{0}^{(i)}\delta_{0}\), \(i=1, \ldots, m\),
-
(c)
\(\gamma^{(1)}= \vert x \vert ^{-4} \vert u \vert ^{2}+\gamma_{0}^{(1)}\delta_{0}\), \(\gamma^{(2)}= \vert x \vert ^{-4} \vert v \vert ^{2}+\gamma_{0}^{(2)}\delta_{0}\),
-
(d)
\(\mathcal{A}_{0, m} (\sum_{i=1}^{m}\varsigma_{i}\nu_{j}^{(i)} )^{\frac{2}{2^{\ast\ast}}}\leq\eta_{j}^{(1)}+\eta_{j}^{(2)}\),
-
(e)
\(\mathcal{A}_{\mu, m} (\sum_{i=1}^{m}\varsigma_{i}\nu_{0}^{(i)} )^{\frac{2}{2^{\ast\ast}}}\leq\eta_{0}^{(1)}+\eta_{0}^{(2)} -\mu(\gamma_{0}^{(1)}+\gamma_{0}^{(2)})\),
where
\(\delta_{x_{j}}\), \(j\in\mathscr{J}\cup \{0 \}\), is a Dirac mass of 1 concentrated at
\(x_{j}\in\mathbb{R}^{N}\).
To establish the existence results for problem \((\mathscr{P}^{Q}_{0})\), we need the following local \((PS)_{c}\) condition, which is indispensable for the proof of Theorem 2.1.
Lemma 3.4
Assume that (q.1) and (q.2) hold. Then the
\((PS)_{c}\)
condition in
\((\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}\)
holds for
\(\mathcal{F}\)
if
$$ c< c_{0}^{\ast}\triangleq\frac{2}{N} \min \bigl\{ \vert G \vert \mathcal{A}_{0, m}^{\frac{N}{4}} \Vert Q_{+} \Vert _{\infty}^{1-\frac{N}{4}}, \mathcal{A}_{\mu, m}^{\frac{N}{4}}Q_{+}(0)^{1-\frac{N}{4}}, \mathcal{A}_{\mu, m}^{\frac{N}{4}}Q_{+}(\infty)^{1-\frac{N}{4}} \bigr\} . $$
(3.7)
Proof
We follow closely the arguments in [9, Proposition 2]. It is trivial to check that the \((PS)_{c}\) sequence \(\{(u_{n}, v_{n})\}\) of \(\mathcal{F}\) is bounded in \((\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}\). Then we may assume that \((u_{n}, v_{n})\rightharpoonup(u, v)\) in \((\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}\). In view of Lemma 3.3, there exist measures \(\eta^{(1)}\), \(\eta^{(2)}\), \(\nu^{(i)}\ (i=1, \ldots, m)\), \(\gamma^{(1)}\), and \(\gamma^{(2)}\) such that relations (a)–(e) of this lemma hold. We begin by considering the concentration at the point \(x_{j}\in\mathbb{R}^{N}\backslash\{0\}\), \(j\in\mathscr{J}\). For \(\epsilon>0\) small, we define the cut-off function \(\psi_{x_{j}}^{\epsilon}(x)\in\mathscr {C}_{0}^{\infty}(\mathbb{R}^{N})\) such that \(0\leq\psi_{x_{j}}^{\epsilon}(x)\leq1\), \(\psi_{x_{j}}^{\epsilon}(x)=1\) in \(B_{\epsilon}(x_{j})\), \(\psi_{x_{j}}^{\epsilon}(x)=0\) on \(\mathbb{R}^{N}\backslash B_{2\epsilon}(x_{j})\), \(\vert \nabla\psi_{x_{j}}^{\epsilon} \vert \leq 2/\epsilon\), and \(\vert \Delta\psi_{x_{j}}^{\epsilon} \vert \leq 2/\epsilon^{2}\) on \(\mathbb{R}^{N}\). Then, by Lemma 3.1, \(\lim_{n\rightarrow\infty}\langle\mathcal{F}^{\prime}(u_{n}, v_{n}), (u_{n}\psi_{x_{j}}^{\epsilon}, v_{n}\psi_{x_{j}}^{\epsilon})\rangle=0\); hence, combining (3.2), the Hölder inequality, and the Sobolev inequality, we derive
$$\begin{aligned} & \int_{\mathbb{R}^{N}} \psi_{x_{j}}^{\epsilon} \Biggl\{ d \eta^{(1)}+d\eta^{(2)} -\mu \bigl(d\gamma^{(1)}+d \gamma^{(2)} \bigr) -Q(x) \sum_{i=1}^{m} \frac{\varsigma_{i}}{2^{\ast\ast}} (\alpha_{i}+\beta_{i} )\,d \nu^{(i)} \Biggr\} \\ &\quad\leq\overline{\lim_{n\rightarrow\infty}} \int_{\mathbb{R}^{N}} \bigl\{ 2 \bigl\vert \Delta u_{n} \bigl\langle \nabla u_{n}, \nabla\psi_{x_{j}}^{\epsilon} \bigr\rangle +\Delta v_{n} \bigl\langle \nabla v_{n}, \nabla \psi_{x_{j}}^{\epsilon} \bigr\rangle \bigr\vert + \bigl\vert (u_{n}\Delta u_{n}+v_{n}\Delta v_{n} )\Delta\psi_{x_{j}}^{\epsilon} \bigr\vert \bigr\} \,dx \\ &\quad\leq\sup_{n\geq 1} \biggl( \int_{\mathbb{R}^{N}} \vert \Delta u_{n} \vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggl[2\overline{\lim _{n\rightarrow \infty}} \biggl( \int_{\mathbb{R}^{N}} \vert \nabla u_{n} \vert ^{2} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \\ &\qquad{} +\overline{\lim _{n\rightarrow\infty}} \biggl( \int_{\mathbb{R}^{N}} \vert u_{n} \vert ^{2} \bigl\vert \Delta\psi_{x_{j}} ^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggr] \\ &\qquad{} +\sup_{n\geq1} \biggl( \int_{\mathbb{R}^{N}} \vert \Delta v_{n} \vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggl[2\overline{\lim _{n\rightarrow \infty}} \biggl( \int_{\mathbb{R}^{N}} \vert \nabla v_{n} \vert ^{2} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \\ &\qquad{} +\overline{\lim _{n\rightarrow\infty}} \biggl( \int_{\mathbb{R}^{N}} \vert v_{n} \vert ^{2} \bigl\vert \Delta\psi_{x_{j}} ^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggr] \\ &\quad\leq C \biggl\{ \biggl( \int_{\mathbb{R}^{N}} \vert \nabla u \vert ^{2} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac {1}{2}}+ \biggl( \int_{\mathbb{R}^{N}} \vert u \vert ^{2} \bigl\vert \Delta \psi_{x_{j}}^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac {1}{2}}+ \biggl( \int_{\mathbb{R}^{N}} \vert v \vert ^{2} \bigl\vert \Delta \psi_{x_{j}}^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \\ &\qquad{} + \biggl( \int_{\mathbb{R}^{N}} \vert \nabla v \vert ^{2} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggr\} \leq C \biggl\{ \biggl( \int_{B_{2\epsilon}(x_{j})} \vert \nabla u \vert ^{\frac{2N}{N-2}}\,dx \biggr)^{\frac{N-2}{2N}} \biggl( \int_{\mathbb{R}^{N}} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{N}\,dx \biggr)^{\frac{1}{N}} \\ &\qquad{} + \biggl( \int_{B_{2\epsilon}(x_{j})} \vert u \vert ^{2^{\ast\ast}}\,dx \biggr)^{\frac{1}{2^{\ast\ast}}} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert \Delta\psi_{x_{j}}^{\epsilon} \bigr\vert ^{\frac{N}{2}} \biggr) ^{\frac{2}{N}} \\ &\qquad{}+ \biggl( \int_{B_{2\epsilon}(x_{j})} \vert v \vert ^{2^{\ast\ast}}\,dx \biggr)^{\frac{1}{2^{\ast\ast}}} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert \Delta\psi_{x_{j}}^{\epsilon} \bigr\vert ^{\frac{N}{2}} \biggr)^{\frac{2}{N}} \\ &\qquad{} + \biggl( \int_{B_{2\epsilon}(x_{j})} \vert \nabla v \vert ^{\frac{2N}{N-2}}\,dx \biggr)^{\frac{N-2}{2N}} \biggl( \int_{\mathbb{R}^{N}} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{N}\,dx \biggr)^{\frac{1}{N}} \biggr\} \leq C \biggl\{ \biggl( \int_{B_{2\epsilon}(x_{j})} \vert \nabla u \vert ^{\frac{2N}{N-2}}\,dx \biggr)^{\frac{N-2}{2N}} \\ &\qquad{} + \biggl( \int_{B_{2\epsilon}(x_{j})} \vert \Delta u \vert ^{2}\,dx \biggr)^{\frac{1}{2}}+ \biggl( \int_{B_{2\epsilon}(x_{j})} \vert \Delta v \vert ^{2}\,dx \biggr)^{\frac{1}{2}}+ \biggl( \int_{B_{2\epsilon}(x_{j})} \vert \nabla v \vert ^{\frac{2N}{N-2}}\,dx \biggr)^{\frac{N-2}{2N}} \biggr\} . \end{aligned}$$
(3.8)
As \(\epsilon\rightarrow0\), it follows from (3.8) and Lemma 3.3 that
$$ Q(x_{j})\sum_{i=1}^{m} \varsigma_{i}\nu_{j}^{(i)}\geq \eta_{j}^{(1)}+\eta_{j}^{(2)}. $$
(3.9)
This means that the concentration of the measures \(\nu^{(i)}\ (i=1, \ldots, m)\) cannot occur at points where \(Q(x_{j})\leq0\). By virtue of (3.9) and (d) of Lemma 3.3, we conclude that either (i) \(\nu_{j}^{(i)}=0\ (i=1, \ldots, m)\) or (ii) \(\sum_{i=1}^{m}\varsigma_{i}\nu_{j}^{(i)}\geq (\mathcal{A}_{0, m}/ \Vert Q_{+} \Vert _{\infty})^{N/4}\). Let us now study the possibility of concentration at \(x=0\) and at ∞. By the argument similar to that of \(x_{j}\in \mathbb{R}^{N}\backslash\{0\}\), we find \(\eta_{0}^{(1)}+\eta_{0}^{(2)} -\mu(\gamma_{0}^{(1)}+\gamma_{0}^{(2)}) -Q(0)\sum_{i=1}^{m}\varsigma_{i}\nu_{0}^{(i)} \leq0\). Together with (e) of Lemma 3.3, it follows that either (iii) \(\nu_{0}^{(i)} =0\ (i=1, \ldots, m)\) or (iv) \(\sum_{i=1}^{m}\varsigma_{i}\nu_{0}^{(i)}\geq (\mathcal{A}_{\mu, m}/Q_{+}(0))^{N/4}\). To discuss the concentration at infinity of the sequence \(\{(u_{n}, v_{n})\}\), we define the following quantities:
-
(1)
\(\eta_{\infty}^{(1)}=\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}}\int_{ \vert x \vert >R} \vert \Delta u_{n} \vert ^{2}\,dx\), \(\eta_{\infty}^{(2)}=\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}}\int_{ \vert x \vert >R} \vert \Delta v_{n} \vert ^{2}\,dx\),
-
(2)
\(\nu_{\infty}^{(i)}= \lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}}\int_{ \vert x \vert >R} \vert u_{n} \vert ^{\alpha_{i}} \vert v_{n} \vert ^{\beta_{i}}\,dx\), \(i=1, \ldots, m\),
-
(3)
\(\gamma_{\infty}^{(1)}=\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}}\int_{ \vert x \vert >R} \vert x \vert ^{-4} \vert u_{n} \vert ^{2}\,dx\), \(\gamma_{\infty}^{(2)}=\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}}\int_{ \vert x \vert >R} \vert x \vert ^{-4} \vert v_{n} \vert ^{2}\,dx\).
It is obvious that \(\eta_{\infty}^{(1)}\), \(\eta_{\infty}^{(2)}\), \(\nu_{\infty}^{(i)}\ (i=1, \ldots, m)\), \(\gamma_{\infty}^{(1)}\), and \(\gamma_{\infty}^{(2)}\) defined by (1)–(3) exist and are finite. For \(R>1\), let \(\psi_{R}(x)\in\mathscr {C}^{\infty}(\mathbb{R}^{N})\) be a function such that \(0\leq\psi_{R}(x)\leq1\), \(\psi_{R}(x)=1\) for \(\vert x \vert >R+1\), \(\psi_{R}(x)=0\) for \(\vert x \vert < R\), \(\vert \nabla\psi_{R} \vert \leq2/R\), and \(\vert \Delta\psi_{R} \vert \leq2/R^{2}\). Because the sequence \(\{(u_{n}\psi_{R}, v_{n}\psi_{R})\}\) is bounded in \((\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}\), we deduce from (3.2) and the fact that \(\alpha_{i}+\beta_{i}=2^{\ast\ast}\ (i=1, \ldots, m)\) that
$$\begin{aligned} 0={}&\lim_{n\rightarrow\infty} \bigl\langle \mathcal{F}^{\prime}(u_{n}, v_{n}), (u_{n}\psi_{R}, v_{n} \psi_{R}) \bigr\rangle \\ ={}&\lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \Biggl\{ \Biggl( \vert \Delta u_{n} \vert ^{2}+ \vert \Delta v_{n} \vert ^{2}- \mu\frac{ \vert u_{n} \vert ^{2} + \vert v_{n} \vert ^{2}}{ \vert x \vert ^{4}}-Q(x)\sum_{i=1}^{m} \varsigma_{i} \vert u_{n} \vert ^{\alpha_{i}} \vert v_{n} \vert ^{\beta_{i}} \Biggr)\psi_{R} \\ &{} + \bigl(2\Delta u_{n}\langle\nabla u_{n}, \nabla \psi_{R}\rangle+u_{n}\Delta u_{n}\Delta \psi_{R}+2\Delta v_{n}\langle\nabla v_{n}, \nabla\psi_{R}\rangle+v_{n}\Delta v_{n}\Delta \psi_{R} \bigr) \Biggr\} \,dx. \end{aligned}$$
(3.10)
Furthermore, by utilizing the Hölder inequality and the Sobolev inequality, we obtain
$$\begin{aligned} &\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}} \int_{\mathbb{R}^{N}} \bigl(2 \bigl\vert \Delta u_{n}\langle \nabla u_{n}, \nabla \psi_{R}\rangle \bigr\vert + \vert u_{n}\Delta u_{n}\Delta\psi_{R} \vert \bigr)\,dx \\ &\quad\leq\lim_{R\rightarrow\infty} \overline{\lim_{n\rightarrow\infty}} \biggl( \int_{\mathbb {R}^{N}} \vert \Delta u_{n} \vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggl[2 \biggl( \int_{\mathbb{R}^{N}} \vert \nabla u_{n} \vert ^{2} \vert \nabla\psi_{R} \vert ^{2}\,dx \biggr)^{\frac{1}{2}}\\ &\qquad{}+ \biggl( \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2} \vert \Delta\psi_{R} \vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggr] \\ &\quad\leq C\lim_{R\rightarrow \infty} \biggl\{ \biggl( \int_{R< \vert x \vert < R+1} \vert \nabla u \vert ^{2} \vert \nabla\psi_{R} \vert ^{2}\,dx \biggr)^{\frac{1}{2}}+ \biggl( \int_{R< \vert x \vert < R+1} \vert u \vert ^{2} \vert \Delta \psi_{R} \vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggr\} \\ &\quad\leq C\lim_{R\rightarrow \infty} \biggl\{ \biggl( \int_{R< \vert x \vert < R+1} \vert \nabla u \vert ^{\frac{2N}{N-2}}\,dx \biggr)^{\frac{N-2}{2N}}+ \biggl( \int_{R< \vert x \vert < R+1} \vert \Delta u \vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggr\} =0. \end{aligned}$$
Similarly, we have \(\lim_{R\rightarrow \infty}\overline{\lim_{n\rightarrow\infty}}\int_{\mathbb{R}^{N}} (2 \vert \Delta v_{n}\langle\nabla v_{n}, \nabla \psi_{R}\rangle \vert + \vert v_{n}\Delta v_{n}\Delta\psi_{R} \vert )\,dx=0\). Consequently, it follows from (3.10) and definitions (1)–(3) of the quantities \(\eta_{\infty}^{(1)}\), \(\eta_{\infty}^{(2)}\), \(\nu_{\infty}^{(i)}\ (i=1, \ldots, m)\), \(\gamma_{\infty}^{(1)}\), and \(\gamma_{\infty}^{(2)}\) that
$$ Q_{+}(\infty)\sum_{i=1}^{m} \varsigma_{i}\nu_{\infty}^{(i)} \geq \eta_{\infty}^{(1)}+\eta_{\infty}^{(2)} -\mu \bigl( \gamma_{\infty}^{(1)}+\gamma_{\infty}^{(2)} \bigr). $$
(3.11)
Moreover, in view of (3.3), we find \(\mathcal{A}_{\mu, m}(\sum_{i=1}^{m} \varsigma_{i}\nu_{\infty}^{(i)})^{\frac{2}{2^{\ast\ast}}} \leq\eta_{\infty}^{(1)}+\eta_{\infty}^{(2)} -\mu(\gamma_{\infty}^{(1)}+\gamma_{\infty}^{(2)})\). This, combined with (3.11), implies that either (v) \(\nu_{\infty}^{(i)}=0\ (i=1, \ldots, m)\) or (vi) \(\sum_{i=1}^{m}\varsigma_{i}\nu_{\infty}^{(i)} \geq(\mathcal{A}_{\mu, m}/Q_{+}(\infty))^{N/4}\). In the following, we claim that (ii), (iv), and (vi) cannot occur. For every continuous nonnegative function ψ such that \(0\leq\psi(x)\leq 1\) on \(\mathbb{R}^{N}\), we find
$$\begin{aligned} c&=\lim_{n\rightarrow\infty} \biggl(\mathcal{F}(u_{n}, v_{n}) -\frac{1}{2^{\ast\ast}} \bigl\langle \mathcal{F}^{\prime}(u_{n}, v_{n}), (u_{n}, v_{n}) \bigr\rangle \biggr) \\ &=\frac{2}{N} \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \biggl( \vert \Delta u_{n} \vert ^{2}+ \vert \Delta v_{n} \vert ^{2}-\mu \frac{ \vert u_{n} \vert ^{2}+ \vert v_{n} \vert ^{2}}{ \vert x \vert ^{4}} \biggr)\,dx \\ &\geq\frac{2}{N}\overline{\lim_{n\rightarrow\infty}} \int_{\mathbb{R}^{N}} \biggl( \vert \Delta u_{n} \vert ^{2}+ \vert \Delta v_{n} \vert ^{2}-\mu \frac{ \vert u_{n} \vert ^{2}+ \vert v_{n} \vert ^{2}}{ \vert x \vert ^{4}} \biggr)\psi(x)\,dx. \end{aligned}$$
Note that the measures \(\nu^{(i)}\ (i=1, \ldots, m)\) are bounded and G-invariant. This means that if (ii) holds, then the set \(\mathscr{J}\) must be finite. Moreover, if \(x_{j}\neq0\) is a singular point of \(\nu^{(i)}\ (i=1, \ldots, m)\), so is \(gx_{j}\) for each \(g\in G\), and the mass of \(\nu^{(i)}\ (i=1, \ldots, m)\) concentrated at \(gx_{j}\) is the same for every \(g\in G\). Assuming that (ii) occurs for some \(j\in\mathscr{J}\) with \(x_{j}\neq0\), we choose ψ with compact support so that \(\psi(gx_{j})=1\) for every \(g\in G\), and we derive
$$\begin{aligned} c&\geq\frac{2}{N} \vert G \vert \bigl(\eta_{j}^{(1)}+ \eta_{j}^{(2)} \bigr)\geq \frac{2}{N} \vert G \vert \mathcal{A}_{0, m} \Biggl(\sum_{i=1}^{m} \varsigma_{i}\nu_{j}^{(i)} \Biggr) ^{\frac{2}{2^{\ast\ast}}} \\ &\geq\frac{2}{N} \vert G \vert \mathcal{A}_{0, m} \bigl( \mathcal{A}_{0, m}/ \Vert Q_{+} \Vert _{\infty} \bigr)^{\frac{2}{2^{\ast \ast}-2}} =\frac{2}{N} \vert G \vert \mathcal{A}_{0, m}^{\frac{N}{4}} \Vert Q_{+} \Vert _{\infty}^{1-\frac{N}{4}}, \end{aligned}$$
which is impossible. Similarly, assuming that (iv) holds for \(x=0\), we take ψ with compact support so that \(\psi(0)=1\), and we have
$$\begin{aligned} c&\geq\frac{2}{N} \bigl(\eta_{0}^{(1)}+ \eta_{0}^{(2)} -\mu\gamma_{0}^{(1)}-\mu \gamma_{0}^{(2)} \bigr)\geq \frac{2}{N} \mathcal{A}_{\mu, m} \Biggl(\sum_{i=1}^{m} \varsigma_{i}\nu_{0}^{(i)} \Biggr) ^{\frac{2}{2^{\ast\ast}}} \\ &\geq\frac{2}{N}\mathcal{A}_{\mu, m} \bigl(\mathcal{A}_{\mu, m}/Q_{+}(0) \bigr)^{\frac{2}{2^{\ast\ast}-2}} =\frac{2}{N}\mathcal{A}_{\mu, m}^{\frac{N}{4}}Q_{+}(0)^{1-\frac{N}{4}}, \end{aligned}$$
a contradiction to (3.7). Finally, if (vi) occurs, we choose \(\psi=\psi_{R}\) to obtain
$$\begin{aligned} c&\geq\frac{2}{N} \bigl(\eta_{\infty}^{(1)}+ \eta_{\infty}^{(2)} -\mu\gamma_{\infty}^{(1)}-\mu \gamma_{\infty}^{(2)} \bigr)\geq \frac{2}{N} \mathcal{A}_{\mu, m} \Biggl(\sum_{i=1}^{m} \varsigma_{i}\nu_{\infty}^{(i)} \Biggr) ^{\frac{2}{2^{\ast\ast}}} \\ &\geq\frac{2}{N}\mathcal{A}_{\mu, m} \bigl(\mathcal{A}_{\mu, m}/Q_{+}( \infty) \bigr)^{\frac{2}{2^{\ast\ast}-2}} =\frac{2}{N}\mathcal{A}_{\mu, m}^{\frac{N}{4}}Q_{+}( \infty)^{1-\frac{N}{4}}, \end{aligned}$$
which contradicts (3.7). Hence, \(\nu_{j}^{(i)}=0\ (i=1, \ldots, m)\) for all \(j\in\mathscr{J}\cup\{0, \infty\}\), and this yields
$$\lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \sum_{i=1}^{m} \varsigma_{i} \vert u_{n} \vert ^{\alpha_{i}} \vert v_{n} \vert ^{\beta_{i}}\,dx = \int_{\mathbb{R}^{N}}\sum_{i=1}^{m} \varsigma_{i} \vert u \vert ^{\alpha _{i}} \vert v \vert ^{\beta_{i}} \,dx. $$
Finally, taking into account \(\lim_{n\rightarrow\infty}\langle \mathcal{F}^{\prime}(u_{n}, v_{n})-\mathcal{F}^{\prime}(u, v), (u_{n}-u, v_{n}-v)\rangle=0\), we naturally deduce \((u_{n}, v_{n})\rightarrow(u, v)\) as \(n\rightarrow\infty\) in \((\mathscr {D}^{2, 2}(\mathbb{R}^{N}))^{2}\). □
Thanks to Lemma 3.4, we immediately obtain the following result.
Corollary 3.1
If
\(\vert G \vert =+\infty\)
and
\(Q_{+}(0)=Q_{+}(\infty)=0\), then the functional
\(\mathcal{F}\)
satisfies the
\((PS)_{c}\)
condition for every
\(c\in\mathbb{R}\).
Proof of Theorem 2.1
Let \(y_{\epsilon}\) be the extremal function satisfying (2.4)–(2.10). We now choose \(\epsilon>0\) such that (2.11) is fulfilled. It is clear from (q.2), (3.1), and (3.2) that there exist constants \(\alpha_{0}>0\) and \(\rho>0\) such that \(\mathcal{F}(u, v)\geq\alpha_{0}\) for all \(\Vert (u, v) \Vert _{\mu}=\rho\). Moreover, if we set \(u=y_{\epsilon}\), \(v=\tau_{\min}y_{\epsilon}\), and
$$\begin{aligned} \Phi(t)={}&\mathcal{F}(ty_{\epsilon}, t\tau_{\min}y_{\epsilon})= \frac{t^{2}}{2} \bigl(1+\tau_{\min}^{2} \bigr) \int_{\mathbb{R}^{N}} \biggl( \vert \Delta y_{\epsilon} \vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{ \vert x \vert ^{4}} \biggr)\,dx\\ &{} -\frac{t^{2^{\ast\ast}}}{2^{\ast\ast}} \sum _{i=1}^{m}\varsigma_{i} \tau_{\min}^{\beta_{i}} \int_{\mathbb{R}^{N}} Q(x)y_{\epsilon}^{2^{\ast\ast}}\,dx \end{aligned}$$
with \(t\geq0\), then \(\max_{t\geq0}\Phi(t)\) is attained for some finite \(\overline{t}>0\) with \(\Phi^{\prime}(\overline{t})=0\). This yields
$$ \max_{t\geq0}\Phi(t)=\mathcal{F}( \overline{t}y_{\epsilon}, \overline{t}\tau_{\min}y_{\epsilon})= \frac{2}{N} \biggl\{ \frac{ (1+\tau_{\min}^{2} ) \int_{\mathbb{R}^{N}} ( \vert \Delta y_{\epsilon} \vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{ \vert x \vert ^{4}} )\,dx}{ (\sum_{i=1}^{m}\varsigma_{i}\tau_{\min}^{\beta_{i}} \int_{\mathbb{R}^{N}}Q(x)y_{\epsilon}^{2^{\ast\ast}}\,dx ) ^{\frac{2}{2^{\ast\ast}}}} \biggr\} ^{\frac{2^{\ast\ast}}{2^{\ast\ast}-2}}. $$
(3.12)
Besides, because \(\mathcal{F}(ty_{\epsilon}, t\tau_{\min}y_{\epsilon})\rightarrow-\infty\) as \(t\rightarrow+\infty\), there exists \(t_{0}>0\) such that \(\Vert (t_{0}y_{\epsilon}, t_{0}\tau_{\min}y_{\epsilon }) \Vert _{\mu}>\rho\) and \(\mathcal{F}(t_{0}y_{\epsilon}, t_{0}\tau_{\min}y_{\epsilon})<0\). Now, we define
$$ c_{0}=\inf_{\gamma\in\Gamma}\max _{t\in[0, 1]}\mathcal{F} \bigl(\gamma(t) \bigr), $$
(3.13)
where \(\Gamma=\{\gamma\in\mathscr{C}([0, 1], (\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}); \gamma(0)=(0, 0), \mathcal{F}(\gamma(1))<0, \Vert \gamma(1) \Vert _{\mu }>\rho\}\). It follows directly from (2.5), (2.11), (3.4), (3.5), (3.7), (3.12), (3.13), and Lemma 3.2 that
$$\begin{aligned} c_{0}&\leq\mathcal{F}(\overline{t}y_{\epsilon}, \overline{t} \tau_{\min}y_{\epsilon})=\frac{2}{N} \biggl\{ \frac{ (1+\tau_{\min}^{2} ) \int_{\mathbb{R}^{N}} ( \vert \Delta y_{\epsilon} \vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{ \vert x \vert ^{4}} )\,dx}{ (\sum_{i=1}^{m}\varsigma_{i}\tau_{\min}^{\beta_{i}} \int_{\mathbb{R}^{N}}Q(x)y_{\epsilon}^{2^{\ast\ast}}\,dx ) ^{\frac{2}{2^{\ast\ast}}}} \biggr\} ^{\frac{2^{\ast\ast}}{2^{\ast\ast}-2}} \\ &\leq\frac{2}{N} \biggl\{ \frac{\mathscr{B}(\tau_{\min})\int_{\mathbb{R}^{N}} ( \vert \Delta y_{\epsilon} \vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{ \vert x \vert ^{4}} )\,dx}{ (\max \{ \vert G \vert ^{\frac{2-2^{\ast\ast}}{2}} \mathcal{A}_{0}^{-\frac{2^{\ast\ast}}{2}} \Vert Q_{+} \Vert _{\infty}, \mathcal{A}_{\mu}^{-\frac{2^{\ast\ast}}{2}}Q_{+}(0), \mathcal{A}_{\mu}^{-\frac{2^{\ast\ast}}{2}}Q_{+}(\infty) \} ) ^{\frac{2}{2^{\ast\ast}}}} \biggr\} ^{\frac{2^{\ast\ast}}{2^{\ast\ast}-2}} \\ &=\frac{2}{N}\min \bigl\{ \vert G \vert \mathcal{A}_{0, m}^{\frac{N}{4}} \Vert Q_{+} \Vert _{\infty}^{1-\frac{N}{4}}, \mathcal{A}_{\mu, m}^{\frac{N}{4}}Q_{+}(0)^{1-\frac{N}{4}}, \mathcal{A}_{\mu, m}^{\frac{N}{4}}Q_{+}(\infty)^{1-\frac{N}{4}} \bigr\} =c_{0}^{\ast}. \end{aligned}$$
If \(c_{0}< c_{0}^{\ast}\), then the \((PS)_{c}\) condition holds by Lemma 3.4. Thus we arrive at the conclusion by the mountain pass theorem in [35]. If \(c_{0}=c_{0}^{\ast}\), then \(\gamma(t)=(tt_{0}y_{\epsilon}, tt_{0}\tau_{\min}y_{\epsilon})\), with \(0\leq t\leq1\), is a path in Γ such that \(\max_{t\in [0, 1]}\mathcal{F}(\gamma(t))=c_{0}\). Hence, either \(\Phi^{\prime}(\overline{t})=0\) and we are done, or γ can be deformed to a path \(\widetilde{\gamma}\in\Gamma\) with \(\max_{t\in[0, 1]}\mathcal{F}(\widetilde{\gamma}(t))< c_{0}\) and we have a contradiction. Thus we conclude from Lemma 3.1 that there exists a nontrivial G-invariant solution \((u_{0}, v_{0})\in(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N})\backslash\{0\})^{2}\) to problem \((\mathscr {P}_{0}^{Q})\) and the results follow. □
Proof of Corollary 2.1
In view of (2.6) and Theorem 2.1, it is sufficient to prove that
$$ \int_{\mathbb{R}^{N}} \bigl(Q(x)-\widetilde{Q} \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx\geq0 $$
(3.14)
for some \(\epsilon>0\), where \(\widetilde{Q}=\max\{ \vert G \vert ^{\frac{2-2^{\ast\ast}}{2}} (\mathcal{A}_{0}/\mathcal{A}_{\mu})^{-\frac{2^{\ast\ast}}{2}} \Vert Q_{+} \Vert _{\infty}, Q_{+}(0), Q_{+}(\infty)\}\).
Part (1), case (i). By virtue of (3.14), we need to show that
$$ \epsilon^{-2^{\ast\ast}l_{2}(\mu)} \int_{\mathbb{R}^{N}} \bigl(Q(x)-Q(0) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx\geq0 $$
(3.15)
for certain \(\epsilon>0\). By the hypothesis, we choose \(\varrho_{0}>0\) so that \(Q(x)\geq Q(0)+\xi_{0} \vert x \vert ^{2^{\ast\ast}(l_{2}(\mu)-\Lambda_{0})}\) for \(\vert x \vert \leq\varrho_{0}\). It follows from \(2^{\ast\ast}\Lambda_{0}=N\) and (2.8) that
$$\begin{aligned} &\epsilon^{-2^{\ast\ast}l_{2}(\mu)} \int_{ \vert x \vert \leq\varrho_{0}} \bigl(Q(x)-Q(0) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \\ &\quad\geq\xi_{0} \int_{ \vert x \vert \leq\varrho_{0}}\epsilon^{-2^{\ast\ast }l_{2}(\mu)} \vert x \vert ^{2^{\ast\ast}(l_{2}(\mu)-\Lambda_{0})} U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \\ &\quad=\xi_{0} \int_{ \vert x \vert \leq\varrho_{0}} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon } \biggr) ^{l_{2}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}} \vert x \vert ^{-N}\,dx\rightarrow+\infty \end{aligned}$$
(3.16)
as \(\epsilon\rightarrow0\). On the other hand, for any \(\epsilon>0\), we deduce from (2.8), (2.9), and the fact that \(2^{\ast\ast}l_{2}(\mu)>N\) that
$$\begin{aligned} &\biggl\vert \epsilon^{-2^{\ast\ast}l_{2}(\mu)} \int_{ \vert x \vert > \varrho_{0}} \bigl(Q(x)-Q(0) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \biggr\vert \\ &\quad\leq \int_{ \vert x \vert > \varrho_{0}}\frac{ \vert Q(x)-Q(0) \vert }{ \vert x \vert ^{2^{\ast\ast}l_{2}(\mu)}} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{2}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}}\,dx \\ &\quad \leq C \int_{ \vert x \vert > \varrho_{0}}\frac{1}{ \vert x \vert ^{2^{\ast\ast}l_{2}(\mu)}}\,dx\leq \overline{C}_{1} \end{aligned}$$
(3.17)
for some constant \(\overline{C}_{1}>0\) independent of ϵ. Combining (3.16) and (3.17), we obtain (3.15) for ϵ sufficiently small.
Part (1), case (ii). By the hypothesis, we choose \(\varrho_{1}>0\) so that \(\vert Q(x)-Q(0) \vert \leq\xi_{1} \vert x \vert ^{\varsigma}\) for \(\vert x \vert \leq\varrho_{1}\). Taking into account \(\varsigma>2^{\ast\ast}(l_{2}(\mu)-\Lambda_{0})>0\), \(N-1+\varsigma-2^{\ast\ast}l_{2}(\mu)>-1\) and \(N-1-2^{\ast\ast}l_{2}(\mu)<-1\), we derive
$$\begin{aligned} &\epsilon^{-2^{\ast\ast}l_{2}(\mu)} \int_{\mathbb{R}^{N}} \bigl\vert Q(x)-Q(0) \bigr\vert U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \\ &\quad= \int_{\mathbb{R}^{N}}\frac{ \vert Q(x)-Q(0) \vert }{ \vert x \vert ^{2^{\ast\ast }l_{2}(\mu)}} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{2}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}}\,dx \\ &\quad\leq C \int_{\mathbb{R}^{N}}\frac{ \vert Q(x)-Q(0) \vert }{ \vert x \vert ^{2^{\ast\ast }l_{2}(\mu)}}\,dx \\ &\quad\leq C \biggl(\xi_{1} \int_{ \vert x \vert \leq\varrho_{1}} \vert x \vert ^{\varsigma-2^{\ast\ast}l_{2}(\mu)}\,dx + \int_{ \vert x \vert >\varrho_{1}} \bigl\vert Q(x)-Q(0) \bigr\vert \vert x \vert ^{-2^{\ast\ast}l_{2}(\mu )}\,dx \biggr) \\ &\quad\leq C \biggl( \int_{0}^{\varrho_{1}} r^{N-1+\varsigma-2^{\ast\ast}l_{2}(\mu)}\,dr + \int_{\varrho_{1}}^{+\infty}r^{N-1-2^{\ast\ast}l_{2}(\mu)}\,dr \biggr) < +\infty. \end{aligned}$$
Thus, by (2.8), (2.12), and the Lebesgue dominated convergence theorem, we have
$$\begin{aligned} & \lim_{\epsilon\rightarrow0} \int_{\mathbb{R}^{N}}\epsilon^{-2^{\ast\ast}l_{2}(\mu)} \bigl(Q(x)-Q(0) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \\ &\quad =\lim_{\epsilon\rightarrow 0} \int_{\mathbb{R}^{N}} \bigl(Q(x)-Q(0) \bigr) \vert x \vert ^{-2^{\ast\ast }l_{2}(\mu)} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{2}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}}\,dx \\ &\quad =C \int_{\mathbb{R}^{N}} \bigl(Q(x)-Q(0) \bigr) \vert x \vert ^{-2^{\ast\ast }l_{2}(\mu)}\,dx>0. \end{aligned}$$
Hence (3.15) holds for ϵ small enough.
Part (2), case (i). According to (3.14), we need to prove that
$$ \epsilon^{-2^{\ast\ast}l_{1}(\mu)} \int_{\mathbb{R}^{N}} \bigl(Q(x)-Q(\infty) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx\geq0 $$
(3.18)
for certain \(\epsilon>0\). By the assumption, we take \(\varrho_{2}>0\) such that \(Q(x)\geq Q(\infty)+ \xi_{2} \vert x \vert ^{-2^{\ast\ast}(\Lambda_{0}-l_{1}(\mu))}\) for all \(\vert x \vert \geq\varrho_{2}\). It follows from (2.7) that
$$\begin{aligned} &\epsilon^{-2^{\ast\ast}l_{1}(\mu)} \int_{ \vert x \vert \geq \varrho_{2}} \bigl(Q(x)-Q(\infty) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \\ &\quad = \int_{ \vert x \vert \geq \varrho_{2}} \bigl(Q(x)-Q(\infty) \bigr) \vert x \vert ^{-2^{\ast\ast}l_{1}(\mu)} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{1}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}}\,dx \\ &\quad \geq\xi_{2} \int_{ \vert x \vert \geq\varrho_{2}} \vert x \vert ^{-N} \biggl[ \biggl( \frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{1}(\mu)}U_{\mu} \biggl( \frac{ \vert x \vert }{\epsilon} \biggr) \biggr] ^{2^{\ast\ast}}\,dx\rightarrow+\infty \end{aligned}$$
as \(\epsilon\rightarrow+\infty\). On the other hand, for any \(\epsilon>0\), we conclude from (2.7), (q.2), and the fact that \(N-1-2^{\ast\ast}l_{1}(\mu)>-1\) that
$$\begin{aligned} &\biggl\vert \int_{ \vert x \vert \leq \varrho_{2}}\epsilon^{-2^{\ast\ast}l_{1}(\mu)} \bigl(Q(x)-Q(\infty ) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \biggr\vert \\ &\quad \leq \int_{ \vert x \vert \leq \varrho_{2}}\frac{ \vert Q(x)-Q(\infty) \vert }{ \vert x \vert ^{2^{\ast\ast}l_{1}(\mu)}} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{1}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}}\,dx \\ &\quad\leq C \int_{ \vert x \vert \leq \varrho_{2}}\frac{ \vert Q(x)-Q(\infty) \vert }{ \vert x \vert ^{2^{\ast\ast}l_{1}(\mu )}}\,dx\leq C \int_{0}^{\varrho_{2}}r^{N-1-2^{\ast\ast}l_{1}(\mu)}\,dr\leq \overline{C}_{2} \end{aligned}$$
for some constant \(\overline{C}_{2}>0\) independent of \(\epsilon>0\). By putting these two estimates together, we obtain (3.18) for \(\epsilon>0\) large enough.
Part (2), case (ii). By the assumption, we take \(\varrho_{3}>0\) such that \(\vert Q(x)-Q(\infty) \vert \leq \xi_{3} \vert x \vert ^{-\kappa}\) for all \(\vert x \vert \geq\varrho_{3}\). Taking into account \(\kappa>2^{\ast\ast}(\Lambda_{0}-l_{1}(\mu))>0\), \(N-1-\kappa-2^{\ast\ast}l_{1}(\mu)<-1\) and \(N-1-2^{\ast\ast}l_{1}(\mu)>-1\), we find
$$\begin{aligned} &\epsilon^{-2^{\ast\ast}l_{1}(\mu)} \int_{\mathbb{R}^{N}} \bigl\vert Q(x)-Q(\infty) \bigr\vert U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \\ &\quad = \int_{\mathbb{R}^{N}}\frac{ \vert Q(x)-Q(\infty) \vert }{ \vert x \vert ^{2^{\ast\ast }l_{1}(\mu)}} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{1}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}}\,dx\\ &\quad \leq C \int_{\mathbb{R}^{N}}\frac{ \vert Q(x)-Q(\infty) \vert }{ \vert x \vert ^{2^{\ast\ast }l_{1}(\mu)}}\,dx \\ &\quad \leq C \biggl(\xi_{3} \int_{ \vert x \vert \geq \varrho_{3}} \vert x \vert ^{-\kappa-2^{\ast\ast}l_{1}(\mu)}\,dx + \int_{ \vert x \vert \leq \varrho_{3}} \bigl\vert Q(x)-Q(\infty) \bigr\vert \vert x \vert ^{-2^{\ast\ast}l_{1}(\mu)}\,dx \biggr) \\ &\quad \leq C \biggl( \int_{ \varrho_{3}}^{+\infty}r^{N-1-\kappa-2^{\ast\ast}l_{1}(\mu)}\,dr + \int_{0}^{\varrho_{3}}r^{N-1-2^{\ast\ast}l_{1}(\mu)}\,dr \biggr)< +\infty. \end{aligned}$$
Therefore, by (2.7), (2.13), and the Lebesgue dominated convergence theorem, we obtain
$$\begin{aligned} & \lim_{\epsilon\rightarrow+\infty} \int_{\mathbb{R}^{N}}\epsilon^{-2^{\ast\ast}l_{1}(\mu)} \bigl(Q(x)-Q(\infty) \bigr) U_{\mu}^{2^{\ast\ast}} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr)\,dx \\ &\quad =\lim_{\epsilon\rightarrow +\infty} \int_{\mathbb{R}^{N}}\frac{Q(x)-Q(\infty)}{ \vert x \vert ^{2^{\ast \ast}l_{1}(\mu)}} \biggl[ \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) ^{l_{1}(\mu)}U_{\mu} \biggl(\frac{ \vert x \vert }{\epsilon} \biggr) \biggr]^{2^{\ast \ast}}\,dx \\ &\quad =C \int_{\mathbb{R}^{N}} \bigl(Q(x)-Q(\infty) \bigr) \vert x \vert ^{-2^{\ast\ast }l_{1}(\mu)}\,dx>0. \end{aligned}$$
Thus (3.18) holds for \(\epsilon>0\) enough large. Similar to the above, we find that part (3) follows. □
To prove Theorem 2.2, we need the following symmetric mountain pass theorem (see [36] or [37, Theorem 9.12]).
Lemma 3.5
Let
X
be an infinite dimensional Banach space, and let
\(\mathcal{F}\in\mathscr{C}^{1}(X, \mathbb{R})\)
be an even functional satisfying the
\((PS)_{c}\)
condition for each
c
and
\(\mathcal{F}(0)=0\). Furthermore, one supposes that:
-
(i)
there exist constants
\(\widetilde{\alpha}>0\)
and
\(\rho>0\)
such that
\(\mathcal{F}(w)\geq\widetilde{\alpha}\)
for all
\(\Vert w \Vert =\rho\);
-
(ii)
there exists an increasing sequence of subspaces
\(\{X_{k}\}\)
of
X, with
\(\dim X_{k}=k\), such that for every
k
one can find a constant
\(R_{k}>0\)
such that
\(\mathcal{F}(w)\leq0\)
for all
\(w\in X_{k}\)
with
\(\Vert w \Vert \geq R_{k}\).
Then
\(\mathcal{F}\)
possesses a sequence of critical values
\(\{c_{k}\}\)
tending to ∞ as
\(k\rightarrow\infty\).
Proof of Theorem 2.2
We follow closely the arguments in [9, Theorem 3] (see also [38, Theorem 3]). By virtue of Lemma 3.5 with \(X=(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}\) and \(w=(u, v)\in X\), we easily see from (q.2), (2.2), (3.1), and (3.3) that
$$\mathcal{F}(u, v)\geq\frac{1}{2} \bigl\Vert (u, v) \bigr\Vert _{\mu}^{2}-\frac{1}{2^{\ast\ast}} \Vert Q \Vert _{\infty} \mathcal{A}_{\mu, m}^{-\frac{2^{\ast\ast}}{2}} \bigl\Vert (u, v) \bigr\Vert _{\mu}^{2^{\ast\ast}}. $$
Thanks to \(2^{\ast\ast}>2\), there exist constants \(\widetilde{\alpha}>0\) and \(\rho>0\) such that \(\mathcal{F}(u, v)\geq\widetilde{\alpha}\) for any \((u, v)\) with \(\Vert (u, v) \Vert _{\mu}=\rho\). To find an appropriate sequence of finite dimensional subspaces of \((\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}\), we set \(\Omega=\{x\in\mathbb{R}^{N}; Q(x)>0\}\). The set Ω is G-invariant, and we can define \((\mathscr{D}_{G}^{2, 2}(\Omega))^{2}\), which is the subspace of G-invariant functions of \((\mathscr{D}^{2, 2}(\Omega))^{2}\). Extending functions in \((\mathscr{D}_{G}^{2, 2}(\Omega))^{2}\) by 0 outside Ω, we can presume that \((\mathscr{D}_{ G}^{2, 2}(\Omega))^{2}\subset(\mathscr{D}_{G}^{2, 2}(\mathbb{R}^{N}))^{2}\). Let \(\{X_{k}\}\) be an increasing sequence of subspaces of \((\mathscr{D}_{G}^{2, 2}(\Omega))^{2}\) with \(\dim X_{k}=k\) for every k. As in [38, Theorem 3], we define \(\varphi_{1, k}\), …, \(\varphi_{k, k}\in\mathscr {C}_{0}^{\infty}(\mathbb{R}^{N})\) such that \(0\leq\varphi_{i, k}\leq1\), \(\operatorname{supp}(\varphi_{i, k})\cap\operatorname{supp}(\varphi_{j, k})=\emptyset\), \(i\neq j\), and
$$\bigl\vert \operatorname{supp}(\varphi_{i, k})\cap\Omega \bigr\vert >0,\qquad \bigl\vert \operatorname{supp}(\varphi_{j, k})\cap\Omega \bigr\vert >0, \quad\forall i, j\in\{1, \ldots, k\}. $$
Taking \(e_{i, k}=(a\varphi_{i, k}, b\varphi_{i, k})\in X_{k}\), \(i=1, \ldots, k\), and \(X_{k}=\operatorname{span}\{e_{1, k}, \ldots, e_{k, k}\}\), where a and b are two positive constants, we conclude from the construction of \(X_{k}\) that \(\dim X_{k}=k\) for every k. Therefore, there exists a constant \(\epsilon(k)>0\) such that
$$\begin{aligned} &\int_{\Omega}Q(x) \bigl(\varsigma_{1} \vert \tilde{u} \vert ^{\alpha_{1}} \vert \tilde{v} \vert ^{\beta_{1}} + \cdots+\varsigma_{m} \vert \tilde{u} \vert ^{\alpha_{m}} \vert \tilde{v} \vert ^{\beta _{m}} \bigr)\,dx \\ &\quad= \int_{\Omega}Q(x) \Biggl( \varsigma_{1} \Biggl\vert \sum_{i=1}^{k}at_{i, k} \varphi_{i, k} \Biggr\vert ^{\alpha_{1}} \Biggl\vert \sum_{i=1}^{k}bt_{i, k} \varphi_{i, k} \Biggr\vert ^{\beta_{1}}+\cdots \\ &\qquad{}+ \varsigma_{m} \Biggl\vert \sum_{i=1}^{k}at_{i, k} \varphi_{i, k} \Biggr\vert ^{\alpha_{m}} \Biggl\vert \sum _{i=1}^{k}bt_{i, k} \varphi_{i, k} \Biggr\vert ^{\beta_{m}} \Biggr)\,dx \geq\epsilon(k) \end{aligned}$$
for all \((\tilde{u}, \tilde{v})=\sum_{i=1}^{k}t_{i, k}e_{i, k}\in X_{k}\), with \(\Vert (\tilde{u}, \tilde{v}) \Vert _{\mu}=1\). Hence, if \((u, v)\in X_{k}\backslash\{(0, 0)\}\), then we write \((u, v)=t(\tilde{u}, \tilde{v})\), with \(t= \Vert (u, v) \Vert _{\mu}\) and \(\Vert (\tilde{u}, \tilde{v}) \Vert _{\mu}=1\). Therefore, we derive
$$\mathcal{F}(u, v)=\frac{1}{2}t^{2} -\frac{1}{2^{\ast\ast}}t^{2^{\ast\ast}} \int_{\Omega}Q(x) \sum_{i=1}^{m} \varsigma_{i} \vert \tilde{u} \vert ^{\alpha_{i}} \vert \tilde {v} \vert ^{\beta_{i}} \,dx\leq\frac{1}{2}t^{2} - \frac{\epsilon(k)}{2^{\ast\ast}}t^{2^{\ast\ast}}\leq0 $$
for \(t>0\) sufficiently large. By Corollary 3.1 and Lemma 3.5, we conclude that there exists a sequence of critical values \(c_{k}\rightarrow\infty\) as \(k\rightarrow\infty\) and the results follow. □
Proof of Corollary 2.2
Because \(Q(x)\) is radial, we know that the corresponding group \(G=O(\mathbb{N})\) and \(\vert G \vert =+\infty\). By Corollary 3.1, \(\mathcal{F}\) satisfies the \((PS)_{c}\) condition for every \(c\in\mathbb{R}\). Therefore, we deduce from Theorem 2.2 that the results follow. □