Positive symmetric results for a weighted quasilinear elliptic system with multiple critical exponents in $\mathbb{R}^{N}$

Abstract

This work focuses on the symmetric solutions for a weighted quasilinear elliptic system involving multiple critical exponents in $\mathbb{R}^{N}$. Based upon the Caffarelli-Kohn-Nirenberg inequality and the symmetric criticality principle due to Palais, we prove a variety of symmetric results under certain appropriate hypotheses on the singular potentials and the parameters.

1 Introduction

The present paper is dedicated to studying the following singular quasilinear elliptic system:

$$\begin{aligned} \textstyle\begin{cases}\mathscr{L}_{p}^{\beta} u=\frac{K(x)\vert x\vert ^{\alpha}}{p^{\ast}(\beta, \alpha)} (\mu_{1}\varsigma_{1}\vert u\vert ^{\varsigma_{1}-2}u \vert v\vert ^{\tau_{1}}+ \mu_{2}\varsigma_{2}\vert u\vert ^{\varsigma_{2}-2}u\vert v\vert ^{\tau_{2}} )+ h(x) \vert u \vert ^{q-2}u \quad\text{in } \mathbb{R}^{N}, \\ \mathscr{L}_{p}^{\beta} v=\frac{K(x)\vert x\vert ^{\alpha}}{p^{\ast}(\beta, \alpha)} ( \mu_{1}\tau_{1}\vert u\vert ^{\varsigma_{1}}\vert v \vert ^{\tau_{1}-2}v +\mu_{2}\tau_{2}\vert u\vert ^{\varsigma_{2}}\vert v\vert ^{\tau_{2}-2}v ) +h(x) \vert v\vert ^{q-2}v, \quad\text{in } \mathbb{R}^{N}, \\ u(x), v(x)\rightarrow0,\quad \text{as } \vert x\vert \rightarrow +\infty, \end{cases}\displaystyle \end{aligned}$$

(1.1)

where $\mathscr{L}_{p}^{\beta}\triangleq -\operatorname{div}(\vert x\vert ^{\beta} \vert \nabla\cdot \vert ^{p-2}\nabla\cdot)$ is a quasilinear elliptic operator, $1< p< N$, $\beta\leq0$, $N+\beta-p>0$, $N+\alpha>0$, $\alpha+p>\beta$, $\beta\geq\frac{\alpha p}{p^{\ast}(\beta, \alpha)}$, $0<\mu_{i}<+\infty$, and $\varsigma_{i}$, $\tau_{i}>1$ satisfy $\varsigma_{i}+\tau_{i}=p^{\ast}(\beta, \alpha)\ (i=1, 2)$, $p< q< p^{\ast}(\beta, \alpha)$, $p^{\ast}(\beta, \alpha)\triangleq\frac{(N+\alpha)p}{N+\beta-p}$ is the critical Hardy-Sobolev exponent and $p^{\ast}(0, 0)=p^{\ast}\triangleq\frac{Np}{N-p}$ is the critical Sobolev exponent, K and h are G-symmetric functions (G is a closed subgroup of $O(\mathbb{N})$; see Section 2 for details) satisfying certain appropriate hypotheses which will be elaborated later.

The critical elliptic problems like (1.1) have been widely investigated in recent years, starting with the pioneering work of Brezis and Nirenberg [1]. Since we are interested in problems with critical exponents and singular weighted functions, we refer the reader to [2–8] and the references therein. These scalar elliptic equations related to singular potentials, together with the corresponding elliptic systems, arise naturally in a wide range of physical fields and various economical prototypes [9]. Recently, Deng and Jin in [10] dealt with the following scalar semilinear elliptic equation of the type:

$$ -\Delta u=\mu\frac{u}{\vert x\vert ^{2}}+K(x)\vert x\vert ^{-s}u^{2^{\ast}(s)-1} \quad\text{and} \quad u>0 \quad\text{in } \mathbb{R}^{N}, $$

(1.2)

where $N> 2$, $s\in[0, 2)$, $\mu\in[0, (\frac{N-2}{2})^{2})$, $2^{\ast}(s)\triangleq\frac{2(N-s)}{N-2}$ and $2^{\ast}(0)=2^{\ast}\triangleq\frac{2N}{N-2}$, and $K(x)\in\mathscr{C}(\mathbb{R}^{N})\cap L^{\infty}(\mathbb{R}^{N})$ fulfills several assumptions with respect to a subgroup G of $O(\mathbb{N})$. By utilizing the standard variational approach, together with the symmetric criticality principle of Palais [11], the authors in [10] attained several valuable symmetric results to problem (1.2) under different conditions on the weighted function $K(x)$. Borrowing ideas from [10], Deng and Huang [12] recently established some symmetric results for the singular quasilinear elliptic equations in a G-symmetric bounded domain. Finally, it is worthwhile to point out that when the right-hand side critical term $\vert x\vert ^{-s}u^{2^{\ast}(s)-1}$ in (1.2) is substituted by a nonlinear term $f(u)$, such as $f(u)=u^{q-1}$ with $1< q<2^{\ast}$ or $q=2^{\ast}$, there have been a variety of elegant results on G-symmetric solutions in [13–15]. These results provide us a new insight into the corresponding problems.

For the systems of singular elliptic equations with critical nonlinearities, various studies concerning the existence and multiplicity of nontrivial solutions have also been presented in the last decades (see [16–20] for example). Among these, Cai and Kang [17] investigated the following singular critical elliptic system:

$$\begin{aligned} \textstyle\begin{cases}Lu=\frac{\mu_{1}\varsigma_{1}}{2^{\ast}} \vert u\vert ^{\varsigma _{1}-2}u\vert v \vert ^{\tau_{1}}+ \frac{\mu_{2}\varsigma_{2}}{2^{\ast}} \vert u\vert ^{\varsigma _{2}-2}u\vert v\vert ^{\tau_{2}} +a_{1}\vert u\vert ^{q_{1}-2}u+a_{2}v, \quad\text{in } \Omega, \\ Lv=\frac{\mu_{1}\tau_{1}}{2^{\ast}} \vert u\vert ^{\varsigma_{1}}\vert v\vert ^{\tau_{1}-2}v +\frac{\mu_{2}\tau_{2}}{2^{\ast}} \vert u\vert ^{\varsigma_{2}}\vert v \vert ^{\tau_{2}-2}v +a_{2}u+a_{3}\vert v\vert ^{q_{2}-2}v, \quad\text{in } \Omega, \\ u=v=0, \quad\text{on } \partial\Omega, \end{cases}\displaystyle \end{aligned}$$

(1.3)

where $\Omega\subset\mathbb{R}^{N} (N\geq3)$ is a smooth bounded domain, $0\in\Omega$, $L=-\Delta-\mu \vert x\vert ^{-2}$, $\mu<(\frac{N-2}{2})^{2}$, $a_{j}\in\mathbb{R}\ (j=1, 2, 3)$, $0<\mu_{i}<+\infty$, $2\leq q_{i}<2^{\ast}$, and $\varsigma_{i}$, $\tau_{i}>1$ satisfy $\varsigma_{i}+\tau_{i}=2^{\ast}\ (i=1, 2)$. Note that $\vert u\vert ^{\varsigma_{i}-2}u\vert v\vert ^{\tau_{i}}$ and $\vert u\vert ^{\varsigma_{i}}\vert v\vert ^{\tau_{i}-2}v\ (i=1, 2)$ in (1.3) are called strongly coupled critical terms. By means of variational arguments and analytic techniques, the authors established the existence of positive solutions to (1.3) under certain suitable conditions on the parameters $\mu_{i}$, $q_{i}\ (i=1, 2)$ and $a_{j}\in\mathbb{R}\ (j=1, 2, 3)$. Very recently, considerable attention has been devoted to the singular critical elliptic systems like (1.3). Many existence and multiplicity results of positive solutions have been obtained with various assumptions; we would like to mention the papers by Kang [21], Nyamoradi and Hsu [22], Chen and Zou [23] and the references therein contained.

However, with respect to symmetric solutions for nonlinear elliptic systems, we remark that several symmetric results for singular problems were established in [24–26] and when $G=O(\mathbb{N})$, a handful of radial and nonradial results for nonsingular problems were obtained in [27]. Stimulated by [10, 13, 17], in this work we are devoted to seeking the symmetric solutions for the singular quasilinear elliptic system (1.1). Nevertheless, due to the nonlinear perturbations $h(x)\vert u\vert ^{q-2}u$ and $h(x)\vert v\vert ^{q-2}v$, and the singularities caused not only by the operator $\mathscr{L}_{p}^{\beta}$ but also by the strongly coupled critical terms $\vert x\vert ^{\alpha} \vert u\vert ^{\varsigma_{i}-2}u\vert v\vert ^{\tau_{i}}$ and $\vert x\vert ^{\alpha} \vert u\vert ^{\varsigma_{i}}\vert v\vert ^{\tau_{i}-2}v\ (i=1, 2)$, the quasilinear elliptic system (1.1) gets more sophisticated to deal with than (1.2) and (1.3), and hence we have no choice but to confront more difficulties. To the best of our knowledge, even in the particular cases $\beta=\alpha=0$ and $p=2$, it seems like little work on the symmetric solutions for the problem (1.1). Let $K_{0}>0$ be a constant. In this work, applying the symmetric criticality principle of Palais and variational methods, we will treat both the cases of $h=0$, $K(x)\not\equiv K_{0}$ and $h\neq0$, $K(x)\equiv K_{0}$.

The remainder of this article is schemed as follows. The variational framework and some preliminaries are presented, and the main results of this article are stated in Section 2. We detail the proofs of the symmetric results for the cases $h=0$ and $K(x)\not\equiv K_{0}$ in Section 3, while the existence results for the cases $h\neq0$ and $K(x)\equiv K_{0}$ are proved in Section 4.

2 Preliminaries and main results

Let $O(\mathbb{N})$ be the group of orthogonal linear transformations of $\mathbb{R}^{N}$ with natural action and let $G\subset O(\mathbb{N})$ be a closed subgroup. For any point $x\in \mathbb{R}^{N}$, the set $G_{x}=\{\widetilde{x}\in\mathbb{R}^{N}; \widetilde{x}=gx, g\in G\}$ is called an orbit of x. The cardinality of the orbit $G_{x}$ will be denoted $\vert G_{x}\vert$. Denote $\vert G\vert =\inf_{0\neq x\in\mathbb{R}^{N}}\vert G_{x}\vert$. In particular, $\vert G\vert$ may be +∞. A measurable function f is called G-symmetric if for all $x\in \mathbb{R}^{N}\backslash\{0\}$, $f(gx)=f(x)$ holds. For example, considering the natural action of $O(\mathbb{N})$ on $\mathbb{R}^{N}\backslash\{0\}$, We easily find that in this case, $\vert G\vert =+\infty$, the orbits are the sphere $\partial B_{R}(0)\ (R>0)$, and G-symmetric functions are the radial functions. Further examples can be found in [10].

Let $\mathscr {D}_{\beta}^{1,p}(\mathbb{R}^{N})$ denote the closure of $\mathscr{C}_{0}^{\infty}(\mathbb{R}^{N})$ functions with respect to the norm

$$\Vert u\Vert = \biggl( \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert \nabla u \vert ^{p}\,dx \biggr)^{1/p}. $$

It is well known that the Caffarelli-Kohn-Nirenberg inequality [2] asserts that there exists a constant $C=C(N, p, \beta, \alpha)>0$ such that

$$ \biggl( \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert u\vert ^{p^{\ast}(\beta, \alpha)}\,dx \biggr) ^{\frac{p}{p^{\ast}(\beta, \alpha)}}\leq C \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert \nabla u \vert ^{p}\,dx, \quad \forall u\in\mathscr{D}_{\beta}^{1,p} \bigl(\mathbb{R}^{N} \bigr), $$

(2.1)

where $1< p< N$, $\beta\leq0$, $N+\beta-p>0$, $N+\alpha>0$, $\alpha+p>\beta$, $\beta\geq\frac{\alpha p}{p^{\ast}(\beta, \alpha)}$ and $p^{\ast}(\beta, \alpha)=\frac{(N+\alpha)p}{N+\beta-p}$. If $\alpha=\beta-p$ and $p^{\ast}(\beta, \beta-p)=p$, then we obtain the following weighted Hardy inequality (see [2, 5]):

$$ \biggl(\frac{N+\beta-p}{p} \biggr)^{p} \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta -p}\vert u\vert ^{p}\,dx \leq \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert \nabla u \vert ^{p}\,dx, \quad \forall u\in\mathscr{D}_{\beta}^{1,p} \bigl(\mathbb{R}^{N} \bigr). $$

(2.2)

Now, we define the product space $(\mathscr{D}_{\beta}^{1, p}(\mathbb{R}^{N}))^{2}$ endowed with the norm

$$\bigl\Vert (u, v) \bigr\Vert = \bigl(\Vert u\Vert ^{p}+ \Vert v\Vert ^{p} \bigr)^{1/p},\quad \forall(u, v)\in \bigl( \mathscr{D}_{\beta}^{1, p} \bigl(\mathbb{R}^{N} \bigr) \bigr)^{2}. $$

The natural functional space to investigate (1.1) is the Banach space $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$, which is the subspace of $(\mathscr {D}_{\beta}^{1, p}(\mathbb{R}^{N}))^{2}$ consisting of all G-symmetric functions. This work is devoted to the study of the following systems:

$$\begin{aligned} \bigl(\mathscr{P}^{K}_{h} \bigr) \textstyle\begin{cases} \mathscr{L}_{p}^{\beta} u=\frac{K(x)\vert x\vert ^{\alpha}}{p^{\ast}(\beta, \alpha)} (\mu_{1}\varsigma_{1}\vert u \vert ^{\varsigma_{1}-2}u\vert v\vert ^{\tau_{1}}+ \mu_{2} \varsigma_{2}\vert u\vert ^{\varsigma_{2}-2}u\vert v\vert ^{\tau_{2}} )+ h(x) \vert u\vert ^{q-2}u &\text{in } \mathbb{R}^{N}, \\ \mathscr{L}_{p}^{\beta} v=\frac{K(x)\vert x\vert ^{\alpha}}{p^{\ast}(\beta, \alpha)} ( \mu_{1}\tau_{1}\vert u\vert ^{\varsigma_{1}}\vert v \vert ^{\tau_{1}-2}v +\mu_{2}\tau_{2}\vert u\vert ^{\varsigma_{2}}\vert v\vert ^{\tau_{2}-2}v ) +h(x) \vert v\vert ^{q-2}v &\text{in } \mathbb{R}^{N}, \\ (u, v)\in (\mathscr{D}_{\beta, G}^{1, p} ( \mathbb{R}^{N} ) )^{2}, \text{ and } u>0, v>0 & \text{in } \mathbb{R}^{N}. \end{cases}\displaystyle \end{aligned}$$

To clearly describe the results of this work, several notations should be presented:

$$\begin{aligned} & S\triangleq\inf_{u\in\mathscr{D}_{\beta}^{1, p}(\mathbb{R}^{N})\backslash\{0\}} \frac{\int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert \nabla u\vert ^{p}\,dx}{ (\int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert u\vert ^{p^{\ast}(\beta, \alpha)}\,dx ) ^{\frac{p}{p^{\ast}(\beta, \alpha)}}}, \end{aligned}$$

(2.3)

$$\begin{aligned} & y_{\epsilon}(x)\triangleq C\epsilon^{\frac{N+\beta-p}{(\alpha-\beta+p)p}} \bigl( \epsilon +\vert x\vert ^{\frac{\alpha-\beta+p}{p-1}} \bigr)^{-\frac{N+\beta-p}{\alpha -\beta+p}}, \end{aligned}$$

(2.4)

where $\epsilon>0$, and the constant $C=C(N, p, \alpha, \beta)>0$, depending only on N, p, α and β. According to [5], we remark that $y_{\epsilon}(x)$ fulfills the following equations:

$$ \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert \nabla y_{\epsilon} \vert ^{p}\,dx=1 $$

(2.5)

and

$$\int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha}y_{\epsilon}^{p^{\ast}(\beta, \alpha)-1} \varphi \,dx=S^{-\frac{p^{\ast}(\beta, \alpha)}{p}} \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert \nabla y_{\epsilon} \vert ^{p-2}\nabla y_{\epsilon}\nabla\varphi \,dx $$

for all $\varphi\in\mathscr{D}_{\beta}^{1, p}(\mathbb{R}^{N})$. Furthermore, we obtain (let $\varphi=y_{\epsilon}$)

$$ \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha}y_{\epsilon}^{p^{\ast}(\beta, \alpha)} \,dx=S^{-\frac{p^{\ast}(\beta, \alpha)}{p}}=S^{-\frac{N+\alpha}{N+\beta-p}}. $$

(2.6)

We presume that the functions $K(x)$ and $h(x)$ verify the following hypotheses.

• (k.1) $K(x)$ is G-symmetric on $\mathbb{R}^{N}$.

• (k.2) $K(x)\in\mathscr{C}(\mathbb{R}^{N})\cap L^{\infty}(\mathbb{R}^{N})$, and $K_{+}(x)\not\equiv0$, where $K_{+}(x)=\max\{0, K(x)\}$.

• (h.1) $h(x)$ is G-symmetric on $\mathbb{R}^{N}$.

• (h.2) $h(x)$ is nonnegative and locally bounded in $\mathbb{R}^{N}\backslash\{0\}$, $h(x)=O(\vert x\vert ^{\alpha})$ in the bounded neighborhood $\mathscr{O}$ of the origin, $h(x)=O(\vert x\vert ^{\vartheta})$ as $\vert x\vert \rightarrow\infty$, $-N<\beta-p<\vartheta<\alpha$, $p^{\ast}(\beta, \vartheta)< q< p^{\ast}(\beta, \alpha)$, where $p^{\ast}(\beta, \vartheta)=\frac{(N+\vartheta)p}{N+\beta-p}$.

The main results of this work are summarized in the following.

Theorem 2.1

Suppose that (k.1) and (k.2) hold. If

$$ \int_{\mathbb{R}^{N}}K(x)\vert x\vert ^{\alpha}y_{\epsilon}^{p^{\ast}(\beta, \alpha)} \,dx \geq S^{-\frac{N+\alpha}{N+\beta-p}}\max \bigl\{ \vert G\vert ^{-\frac{\alpha-\beta+p}{N+\beta-p}}\Vert K_{+}\Vert _{\infty}, K_{+}(0), K_{+}(\infty) \bigr\} >0 $$

(2.7)

for some $\epsilon>0$, where $K_{+}(\infty)=\limsup_{\vert x\vert \rightarrow\infty}K_{+}(x)$, then problem $(\mathscr{P}^{K}_{0})$ has at least one positive solution in $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$.

Corollary 2.1

Suppose that (k.1) and (k.2) hold. Then we have the following statements.

• (1) Problem $(\mathscr{P}^{K}_{0})$ possesses at least one positive solution if

  $$K(0)>0,\quad K(0)\geq\max \bigl\{ \vert G\vert ^{-\frac{\alpha-\beta+p}{N+\beta-p}}\Vert K_{+}\Vert _{\infty}, K_{+}(\infty) \bigr\} $$

  and either (i) $K(x)\geq K(0)+\Lambda_{0} \vert x\vert ^{\frac{N+\alpha}{p-1}}$ for certain $\Lambda_{0}>0$ and $\vert x\vert$ small or (ii) $\vert K(x)-K(0)\vert \leq\Lambda_{1}\vert x\vert ^{\theta}$ for certain constants $\Lambda_{1}>0$, $\theta>\frac{N+\alpha}{p-1}$ and $\vert x\vert$ small and

  $$ \int_{\mathbb{R}^{N}} \bigl(K(x)-K(0) \bigr) \vert x\vert ^{-\frac{Np+\alpha}{p-1}}\,dx>0. $$

  (2.8)

• (2) Problem $(\mathscr{P}^{K}_{0})$ has at least one positive solution if $\lim_{\vert x\vert \rightarrow\infty}K(x)=K(\infty)$ exists and is positive,

  $$K(\infty)\geq\max \bigl\{ \vert G\vert ^{-\frac{\alpha-\beta+p}{N+\beta-p}}\Vert K_{+}\Vert _{\infty}, K_{+}(0) \bigr\} , $$

  and either (i) $K(x)\geq K(\infty)+\Lambda_{2} \vert x\vert ^{-(N+\alpha)}$ for certain $\Lambda_{2}>0$ and large $\vert x\vert$ or (ii) $\vert K(x)-K(\infty)\vert \leq\Lambda_{3}\vert x\vert ^{-\iota}$ for certain constants $\Lambda_{3}>0$, $\iota>N+\alpha$ and large $\vert x\vert$ and

  $$ \int_{\mathbb{R}^{N}} \bigl(K(x)-K(\infty) \bigr) \vert x\vert ^{\alpha}\,dx>0. $$

  (2.9)

• (3) If $K(x)\geq K(\infty)=K(0)>0$ on $\mathbb{R}^{N}$ and $K(\infty)=K(0)\geq \vert G\vert ^{-\frac{\alpha-\beta+p}{N+\beta-p}}\Vert K_{+}\Vert _{\infty}$, then problem $(\mathscr{P}^{K}_{0})$ admits at least one positive solution.

Theorem 2.2

Suppose that $\vert G\vert =+\infty$ and $K_{+}(0)=K_{+}(\infty)=0$. Then problem $(\mathscr{P}^{K}_{0})$ possesses infinitely many G-symmetric solutions.

Corollary 2.2

If K is a radially symmetric function such that $K_{+}(0)=K_{+}(\infty)=0$, then problem $(\mathscr{P}^{K}_{0})$ possesses infinitely many solutions which are radially symmetric.

Theorem 2.3

Let $K_{0}>0$ be a constant. Suppose that $K(x)\equiv K_{0}$ and (h.1) and (h.2) hold. If

$$ \max \biggl\{ p^{\ast}(\beta, \vartheta), \frac{(N+\alpha)(p-1)}{N+\beta-p}, p^{\ast}(\beta, \alpha)-\frac{p}{p-1} \biggr\} < q< p^{\ast}(\beta, \alpha), $$

(2.10)

where $-N<\beta-p<\vartheta<\alpha$, $p^{\ast}(\beta, \vartheta)=\frac{(N+\vartheta)p}{N+\beta-p}$ and $p^{\ast}(\beta, \alpha)=\frac{(N+\alpha)p}{N+\beta-p}$, then problem $(\mathscr{P}^{K_{0}}_{h})$ possesses at least one positive solution in $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$.

Remark 2.1

The main results of this work extend and complement that of [10, 12, 13, 24]. Even in the particular cases $\beta=\alpha=0$ and $p=2$, the above results to problem $(\mathscr{P}^{K}_{h})$ are new on $\mathbb{R}^{N}$.

In the sequel, we denote by $B_{r}(x)$ a ball centered at x with radius r. For simplicity, we use the same C or $C_{i}\ (i=1, 2, \ldots)$ to denote various generic positive constants. $o_{n}(1)$ denotes a datum which tends to 0 as $n\rightarrow\infty$. The dual space of $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$ ($(\mathscr{D}_{\beta}^{1, p}(\mathbb{R}^{N}))^{2}$, respectively) is denoted by $(\mathscr {D}_{\beta, G}^{-1, p^{\prime}}(\mathbb{R}^{N}))^{2}$ ($(\mathscr {D}_{\beta}^{-1, p^{\prime}}(\mathbb{R}^{N}))^{2}$, respectively), where $\frac{1}{p}+\frac{1}{p^{\prime}}=1$. In a given Banach space X, we denote by ‘→’ and ‘⇀’ strong and weak convergence, respectively. Hereafter, $L^{q}(\mathbb{R}^{N}, h(x))$ denotes the weighted $L^{q}(\mathbb{R}^{N})$ space with the norm $(\int_{\mathbb{R}^{N}}h(x)\vert u\vert ^{q}\,dx)^{1/q}$. A functional $\mathscr{F}\in\mathscr{C}^{1}(X, \mathbb{R})$ is said to satisfy the $(PS)_{c}$ condition if each sequence $\{w_{n}\}$ in X satisfying $\mathscr{F}(w_{n})\rightarrow c$, $\mathscr{F}^{\prime}(w_{n})\rightarrow0$ in $X^{\ast}$ has a subsequence, which strongly converges to certain element in X.

3 Existence and multiplicity results for problem $(\mathscr {P}^{K}_{0})$

The energy functional corresponding to problem $(\mathscr{P}^{K}_{0})$ is defined on $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$ by

$$ \mathscr{F}(u, v)=\frac{1}{p} \bigl\Vert (u, v) \bigr\Vert ^{p}-\frac{1}{p^{\ast}(\beta, \alpha)} \int_{\mathbb{R}^{N}} K(x)\vert x\vert ^{\alpha} \bigl( \mu_{1}\vert u\vert ^{\varsigma_{1}}\vert v\vert ^{\tau_{1}} +\mu_{2}\vert u\vert ^{\varsigma_{2}}\vert v\vert ^{\tau_{2}} \bigr)\,dx. $$

(3.1)

Then $\mathscr{F}\in\mathscr{C}^{1}((\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}, \mathbb{R})$ and it is well known that the weak solutions of problem $(\mathscr{P}_{0}^{K})$ correspond to the critical points of $\mathscr{F}(u, v)$ on $(\mathscr {D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$. More precisely, according to the symmetric criticality principle due to Palais (see Lemma 3.1), we say that $(u, v)\in(\mathscr {D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$ is a weak solution of $(\mathscr{P}^{K}_{0})$, if for any $(\varphi_{1}, \varphi_{2})\in (\mathscr{D}_{\beta}^{1, p}(\mathbb{R}^{N}))^{2}$,

$$\begin{aligned} & \int_{\mathbb{R}^{N}} \biggl\{ \vert x\vert ^{\beta} \bigl( \vert \nabla u\vert ^{p-2}\nabla u\nabla\varphi_{1}+ \vert \nabla v\vert ^{p-2}\nabla v\nabla\varphi_{2} \bigr)-\frac{K(x)\vert x\vert ^{\alpha}}{p^{\ast}(\beta, \alpha)} \bigl[ \bigl(\mu_{1}\varsigma_{1} \vert u\vert ^{\varsigma_{1}-2}u\vert v\vert ^{\tau_{1}} \\ &\quad{}+\mu_{2}\varsigma_{2}\vert u\vert ^{\varsigma_{2}-2}u \vert v\vert ^{\tau_{2}} \bigr)\varphi_{1} + \bigl( \mu_{1}\tau_{1}\vert u\vert ^{\varsigma_{1}} \vert v \vert ^{\tau_{1}-2}v +\mu_{2}\tau_{2}\vert u \vert ^{\varsigma_{2}}\vert v\vert ^{\tau_{2}-2}v \bigr)\varphi _{2} \bigr] \biggr\} \,dx=0. \end{aligned}$$

(3.2)

The proof of the following lemma is straightforward, we can find a similar proof in [13], Lemma 1, (see also [27], Proposition 2.8).

Lemma 3.1

If $K(x)$ is a G-symmetric function, then $\mathscr{F}^{\prime}(u, v)=0$ in $(\mathscr{D}_{\beta, G}^{-1, p^{\prime}}(\mathbb{R}^{N}))^{2}$ implies $\mathscr{F}^{\prime}(u, v)=0$ in $(\mathscr{D}_{\beta}^{-1, p^{\prime}}(\mathbb{R}^{N}))^{2}$.

For $0<\mu_{i}<+\infty$, $\varsigma_{i}$, $\tau_{i}>1$ and $\varsigma_{i}+\tau_{i}=p^{\ast}(\beta, \alpha)\ ( i=1, 2)$, we define

$$\begin{aligned} & S_{\mu_{1}, \mu_{2}}\triangleq\inf_{(u, v)\in (\mathscr{D}_{\beta}^{1, p}(\mathbb{R}^{N})\backslash\{0\})^{2}} \frac{\int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} (\vert \nabla u\vert ^{p}+\vert \nabla v\vert ^{p} )\,dx}{ [\int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} (\mu_{1}\vert u\vert ^{\varsigma_{1}} \vert v\vert ^{\tau_{1}}+\mu_{2}\vert u\vert ^{\varsigma_{2}} \vert v\vert ^{\tau_{2}} )\,dx ]^{\frac{p}{p^{\ast}(\beta, \alpha)}}}, \end{aligned}$$

(3.3)

$$\begin{aligned} & \mathscr{A}(\xi)\triangleq\frac{1+\xi^{p}}{ (\mu_{1}\xi ^{\tau_{1}} +\mu_{2}\xi^{\tau_{2}} )^{\frac{p}{p^{\ast}(\beta, \alpha)}}},\quad \xi\geq0, \end{aligned}$$

(3.4)

$$\begin{aligned} & \mathscr{A}(\xi_{\min})\triangleq\min _{\xi\geq 0}\mathscr{A}(\xi)>0, \end{aligned}$$

(3.5)

where $\xi_{\min}>0$ is a minimal point of $\mathscr{A}(\xi)$ and therefore a root of the equation

$$ \mu_{2}\varsigma_{2}\xi^{\tau_{2}-\tau_{1}+p}- \mu_{2}\tau_{2}\xi^{\tau_{2}-\tau_{1}} +\mu_{1} \varsigma_{1}\xi^{p}-\mu_{1}\tau_{1}=0, \quad \xi\geq0. $$

(3.6)

Lemma 3.2

Let $y_{\epsilon}(x)$ be the minimizer of S defined in (2.3) and (2.4), $0<\mu_{i}<+\infty$, $\varsigma_{i}$, $\tau_{i}>1$ and $\varsigma_{i}+\tau_{i}=p^{\ast}(\beta, \alpha)\ (i=1, 2)$. Then we have the following statements.

• (i) $S_{\mu_{1}, \mu_{2}}=\mathscr{A}(\xi_{\min})S$;

• (ii) $S_{\mu_{1}, \mu_{2}}$ has the minimizer $(y_{\epsilon}(x), \xi_{\min}y_{\epsilon}(x))$ for all $\epsilon>0$.

Proof

The proof is a repeat of that in [17], Theorem 1.1, and therefore is omitted here. □

In order to establish our conditions under which the Palais-Smale condition holds, we need the following concentration compactness principle in [28] (see also [4], Lemma 2.2).

Lemma 3.3

Let $\{(u_{n}, v_{n})\}$ be a weakly convergent sequence to $(u, v)$ in $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$ such that $\vert x\vert ^{\beta} \vert \nabla u_{n}\vert ^{p}\rightharpoonup\eta^{(1)}$, $\vert x\vert ^{\beta} \vert \nabla v_{n}\vert ^{p}\rightharpoonup\eta^{(2)}$, $\vert x\vert ^{\alpha} \vert u_{n}\vert ^{\varsigma_{1}}\vert v_{n}\vert ^{\tau_{1}} \rightharpoonup\nu^{(1)}$, and $\vert x\vert ^{\alpha} \vert u_{n}\vert ^{\varsigma_{2}}\vert v_{n}\vert ^{\tau_{2}} \rightharpoonup\nu^{(2)}$ in the sense of measures. Then there exists some at most countable set $\mathscr{J}$, $\{\eta_{j}^{(1)}\geq0\}_{j\in\mathscr{J}\cup\{0\}}$, $\{\eta_{j}^{(2)}\geq0\}_{j\in\mathscr{J}\cup\{0\}}$, $\{\nu_{j}^{(1)}\geq0\}_{j\in\mathscr{J}\cup\{0\}}$, $\{\nu_{j}^{(2)}\geq0\}_{j\in\mathscr{J}\cup\{0\}}$, $\{x_{j}\}_{j\in\mathscr{J}}\subset\mathbb{R}^{N}\backslash\{0\}$ such that

• (a) $\eta^{(1)}\geq \vert x\vert ^{\beta} \vert \nabla u\vert ^{p}+ \sum_{j\in\mathscr {J}}\eta_{j}^{(1)}\delta_{x_{j}}+\eta_{0}^{(1)}\delta_{0}$, $\eta^{(2)}\geq \vert x\vert ^{\beta} \vert \nabla v\vert ^{p}+ \sum_{j\in\mathscr {J}}\eta_{j}^{(2)}\delta_{x_{j}}+\eta_{0}^{(2)}\delta_{0}$,

• (b) $\nu^{(1)}=\vert x\vert ^{\alpha} \vert u\vert ^{\varsigma_{1}}\vert v\vert ^{\tau_{1}}+ \sum_{j\in\mathscr {J}}\nu_{j}^{(1)}\delta_{x_{j}}+\nu_{0}^{(1)}\delta_{0}$, $\nu^{(2)}=\vert x\vert ^{\alpha} \vert u\vert ^{\varsigma_{2}}\vert v\vert ^{\tau_{2}}+ \sum_{j\in\mathscr {J}}\nu_{j}^{(2)}\delta_{x_{j}}+\nu_{0}^{(2)}\delta_{0}$,

• (c) $S_{\mu_{1}, \mu_{2}} (\mu_{1}\nu_{j}^{(1)}+\mu_{2}\nu_{j}^{(2)} )^{\frac{p}{p^{\ast}(\beta, \alpha)}}\leq\eta_{j}^{(1)}+\eta_{j}^{(2)}$, $j\in\mathscr{J}\cup \{0\}$,

where $\delta_{x_{j}}$, $j\in\mathscr{J}\cup \{0 \}$, is a Dirac mass of 1 concentrated at $x_{j}\in\mathbb{R}^{N}$.

To obtain symmetric solutions for system $(\mathscr{P}_{0}^{K})$, we prove the following local $(PS)_{c}$ condition, which is indispensable for the proof of Theorem 2.1.

Lemma 3.4

Suppose that (k.1) and (k.2) hold. Then the $(PS)_{c}$ condition in $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$ holds for $\mathscr{F}$ if

$$ c< c_{0}^{\ast}\triangleq\frac{\alpha-\beta+p}{ (N+\alpha)p}\min \biggl\{ \frac{\vert G\vert S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}}{ \Vert K_{+}\Vert _{\infty}^{\frac{N+\beta-p}{\alpha-\beta+p}}}, \frac{S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}}{ K_{+}(0)^{\frac{N+\beta-p}{\alpha-\beta+p}}}, \frac{S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}}{ K_{+}(\infty)^{\frac{N+\beta-p}{\alpha-\beta+p}}} \biggr\} . $$

(3.7)

Proof

The proof is analogous to that of [13], Proposition 2, but we exhibit it here for completeness. Let $\{(u_{n}, v_{n})\}\subset(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$ be a $(PS)_{c}$ sequence for $\mathscr {F}$ with $c< c_{0}^{\ast}$. Then, we easily see from (k.2), (2.1), (2.2) and (3.1) that $\{(u_{n}, v_{n})\}$ is bounded in $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$, and we may presume that $(u_{n}, v_{n})\rightharpoonup(u, v)$ in $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$. Thanks to Lemma 3.3, there exist measures $\eta^{(1)}$, $\eta^{(2)}$, $\nu^{(1)}$ and $\nu^{(2)}$ such that relations (a)-(c) of this lemma hold. Let $x_{j}\neq0$ be a singular point of measures $\eta^{(1)}$, $\eta^{(2)}$, $\nu^{(1)}$ and $\nu^{(2)}$. We define a function $\psi_{x_{j}}^{\epsilon}\in\mathscr {C}_{0}^{\infty}(\mathbb{R}^{N})$ such that $0\leq\psi_{x_{j}}^{\epsilon}\leq1$, $\psi_{x_{j}}^{\epsilon}=1$ in $B_{\epsilon}(x_{j})$, $\psi_{x_{j}}^{\epsilon}=0$ on $\mathbb{R}^{N}\backslash B_{2\epsilon}(x_{j})$ and $\vert \nabla\psi_{x_{j}}^{\epsilon} \vert \leq2/\epsilon$ on $\mathbb{R}^{N}$. According to Lemma 3.1, $\lim_{n\rightarrow\infty}\langle\mathscr{F}^{\prime}(u_{n}, v_{n}), (u_{n}\psi_{x_{j}}^{\epsilon}, v_{n}\psi_{x_{j}}^{\epsilon})\rangle=0$; thus, combining (2.1), (3.2), the Hölder inequality, and the fact that $p^{\ast}(\beta, p^{\ast}\beta/p)=p^{\ast}$, we derive

$$\begin{aligned} & \int_{\mathbb{R}^{N}} \psi_{x_{j}}^{\epsilon} \bigl(d \eta^{(1)}+d\eta^{(2)} \bigr) - \int_{\mathbb{R}^{N}}K(x)\psi_{x_{j}}^{\epsilon} \bigl( \mu_{1}\,d\nu^{(1)} +\mu_{2}\,d\nu^{(2)} \bigr) \\ &\quad\leq\limsup_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \bigl(\vert x\vert ^{\beta} \vert u_{n}\vert \vert \nabla u_{n}\vert ^{p-1} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert +\vert x \vert ^{\beta }\vert v_{n}\vert \vert \nabla v_{n}\vert ^{p-1} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert \bigr)\,dx \\ &\quad\leq\sup_{n\geq1} \biggl( \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert \nabla u_{n}\vert ^{p}\,dx \biggr)^{\frac{p-1}{p}}\limsup _{n\rightarrow\infty} \biggl( \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert u_{n}\vert ^{p} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{p}\,dx \biggr)^{\frac{1}{p}} \\ &\qquad{} +\sup_{n\geq1} \biggl( \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert \nabla v_{n}\vert ^{p}\,dx \biggr)^{\frac{p-1}{p}}\limsup _{n\rightarrow\infty} \biggl( \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert v_{n}\vert ^{p} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{p}\,dx \biggr)^{\frac{1}{p}} \\ &\quad\leq C \biggl\{ \biggl( \int_{\mathbb{R}^{N}} \vert x\vert ^{\beta} \vert u\vert ^{p} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{p}\,dx \biggr)^{\frac{1}{p}} + \biggl( \int_{\mathbb{R}^{N}} \vert x\vert ^{\beta} \vert v\vert ^{p} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{p}\,dx \biggr)^{\frac{1}{p}} \biggr\} \\ &\quad\leq C \biggl\{ \biggl( \int_{B_{2\epsilon}(x_{j})} \vert x\vert ^{\frac{p^{\ast}\beta}{p}} \vert u\vert ^{p^{\ast}}\,dx \biggr)^{\frac {1}{p^{\ast}}} + \biggl( \int_{B_{2\epsilon}(x_{j})} \vert x\vert ^{\frac{p^{\ast}\beta}{p}} \vert v\vert ^{p^{\ast}}\,dx \biggr)^{\frac {1}{p^{\ast}}} \biggr\} \biggl( \int_{\mathbb{R}^{N}} \bigl\vert \nabla\psi_{x_{j}}^{\epsilon} \bigr\vert ^{N} \biggr)^{\frac{1}{N}} \\ &\quad\leq C \biggl\{ \biggl( \int_{B_{2\epsilon}(x_{j})}\vert x\vert ^{\beta} \vert \nabla u \vert ^{p}\,dx \biggr)^{\frac{1}{p}}+ \biggl( \int_{B_{2\epsilon}(x_{j})} \vert x\vert ^{\beta} \vert \nabla v \vert ^{p}\,dx \biggr)^{\frac{1}{p}} \biggr\} . \end{aligned}$$

(3.8)

Passing to the limit as $\epsilon\rightarrow0$, we deduce from Lemma 3.3 and (3.8) that

$$ K(x_{j}) \bigl(\mu_{1} \nu_{j}^{(1)}+ \mu_{2}\nu_{j}^{(2)} \bigr)\geq \eta_{j}^{(1)}+\eta_{j}^{(2)}. $$

(3.9)

In view of (3.9), we find that the concentration of $\nu^{(1)}$ and $\nu^{(2)}$ cannot occur at points where $K(x_{j})\leq0$, namely, if $K(x_{j})\leq0$ then $\eta_{j}^{(1)}=\eta_{j}^{(2)}=\nu_{j}^{(1)}=\nu_{j}^{(2)}=0$. Applying (3.9) and (c) of Lemma 3.3, we deduce that either (i) $\nu_{j}^{(1)}=\nu_{j}^{(2)}=0$ or (ii) $\mu_{1}\nu_{j}^{(1)}+\mu_{2}\nu_{j}^{(2)}\geq(S_{\mu_{1}, \mu_{2}}/\Vert K_{+}\Vert _{\infty})^{\frac{N+\alpha}{\alpha-\beta+p}}$. For the point $x=0$, as in the case $x_{j}\neq0$, we get

$$\eta_{0}^{(1)}+\eta_{0}^{(2)} -K(0) \bigl(\mu_{1}\nu_{0}^{(1)}+\mu_{2} \nu_{0}^{(2)} \bigr)\leq0. $$

This, combined with (c) of Lemma 3.3, implies that either (iii) $\nu_{0}^{(1)}=\nu_{0}^{(2)}=0$ or (iv) $\mu_{1}\nu_{0}^{(1)}+\mu_{2}\nu_{0}^{(2)}\geq(S_{\mu_{1}, \mu_{2}}/K_{+}(0))^{\frac{N+\alpha}{\alpha-\beta+p}}$. To discuss the possibility of concentration of the sequence $\{(u_{n}, v_{n})\}$ at infinity, we define the following quantities:

• (1) $\eta_{\infty}^{(1)}=\lim_{R\rightarrow\infty}\limsup_{n\rightarrow\infty}\int_{\vert x\vert >R}\vert x\vert ^{\beta} \vert \nabla u_{n}\vert ^{p}\,dx$, $\eta_{\infty}^{(2)}=\lim_{R\rightarrow\infty} \limsup_{n\rightarrow\infty}\int_{\vert x\vert >R}\vert x\vert ^{\beta} \vert \nabla v_{n}\vert ^{p}\,dx$,

• (2) $\nu_{\infty}^{(1)}=\lim_{R\rightarrow\infty}\limsup_{n\rightarrow\infty}\int_{\vert x\vert >R}\vert x\vert ^{\alpha} \vert u_{n}\vert ^{\varsigma_{1}}\vert v_{n}\vert ^{\tau_{1}}\,dx$,

• (3) $\nu_{\infty}^{(2)}=\lim_{R\rightarrow\infty}\limsup_{n\rightarrow\infty}\int_{\vert x\vert >R}\vert x\vert ^{\alpha} \vert u_{n}\vert ^{\varsigma_{2}}\vert v_{n}\vert ^{\tau_{2}}\,dx$.

It is obvious that $\eta_{\infty}^{(1)}$, $\eta_{\infty}^{(2)}$, $\nu_{\infty}^{(1)}$ and $\nu_{\infty}^{(2)}$ defined by (1)-(3) exist and are finite. For $R>1$, let $\psi_{R}(x)$ be a function in $\mathscr{C}^{1}(\mathbb{R}^{N})$ such that $0\leq\psi_{R}(x)\leq 1$ on $\mathbb{R}^{N}$, $\psi_{R}(x)=1$ for $\vert x\vert >R+1$, $\psi_{R}(x)=0$ for $\vert x\vert < R$ and $\vert \nabla\psi_{R}\vert \leq2/R$. Because the sequence $\{(u_{n}\psi_{R}, v_{n}\psi_{R})\}$ is bounded in $(\mathscr{D}_{\beta}^{1, p}(\mathbb{R}^{N}))^{2}$, we deduce from (3.1) and the fact that $\varsigma_{i}+\tau_{i}=p^{\ast}(\beta, \alpha)\ (i=1, 2)$ that

$$\begin{aligned} 0={}&\lim_{n\rightarrow\infty} \bigl\langle \mathscr{F}^{\prime}(u_{n}, v_{n}), (u_{n}\psi_{R}, v_{n} \psi_{R}) \bigr\rangle \\ ={}&\lim_{n\rightarrow\infty} \biggl\{ \int_{\mathbb{R}^{N}} \bigl(\vert x\vert ^{\beta} \vert \nabla u_{n}\vert ^{p-2}\nabla u_{n} \nabla(u_{n}\psi_{R})+\vert x\vert ^{\beta} \vert \nabla v_{n}\vert ^{p-2}\nabla v_{n} \nabla(v_{n}\psi_{R}) \bigr)\,dx \\ &{} - \int_{\mathbb{R}^{N}}\frac{K(x)\vert x\vert ^{\alpha}}{p^{\ast}(\beta, \alpha)} \bigl(\mu_{1}( \varsigma_{1}+\tau_{1})\vert u_{n}\vert ^{\varsigma _{1}}\vert v_{n}\vert ^{\tau_{1}} + \mu_{2}(\varsigma_{2}+\tau_{2})\vert u_{n}\vert ^{\varsigma _{2}}\vert v_{n}\vert ^{\tau_{2}} \bigr) \psi_{R}\,dx \biggr\} \\ =&\lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \bigl\{ \vert x\vert ^{\beta} \bigl(\vert \nabla u_{n}\vert ^{p}\psi_{R}+\vert \nabla v_{n}\vert ^{p}\psi_{R}+u_{n} \vert \nabla u_{n}\vert ^{p-2}\nabla u_{n} \nabla\psi_{R} \\ &{} +v_{n}\vert \nabla v_{n}\vert ^{p-2} \nabla v_{n}\nabla\psi_{R} \bigr)-K(x)\vert x\vert ^{\alpha} \bigl(\mu_{1}\vert u_{n}\vert ^{\varsigma_{1}}\vert v_{n}\vert ^{\tau_{1}} + \mu_{2}\vert u_{n}\vert ^{\varsigma_{2}}\vert v_{n}\vert ^{\tau_{2}} \bigr)\psi_{R} \bigr\} \,dx. \end{aligned}$$

(3.10)

Furthermore, combining (2.1) and the Hölder inequality, we derive

$$\begin{aligned} &\lim_{R\rightarrow\infty} \limsup_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \vert x\vert ^{\beta} \vert u_{n}\vert \nabla u_{n} \vert ^{p-2} \nabla u_{n}\nabla\psi_{R}+v_{n} \vert \nabla v_{n} \vert ^{p-2}\nabla v_{n} \nabla \psi_{R} \vert \,dx \\ &\quad\leq\lim_{R\rightarrow\infty} \limsup_{n\rightarrow\infty} \biggl\{ \biggl( \int_{R< \vert x\vert < R+1} \vert x\vert ^{\beta} \vert u_{n}\vert ^{p}\vert \nabla\psi_{R}\vert ^{p}\,dx \biggr)^{\frac{1}{p}} \biggl( \int_{\mathbb{R}^{N}} \vert x\vert ^{\beta} \vert \nabla u_{n}\vert ^{p}\,dx \biggr)^{\frac{p-1}{p}} \\ &\qquad{} + \biggl( \int_{R< \vert x\vert < R+1} \vert x\vert ^{\beta} \vert v_{n}\vert ^{p}\vert \nabla\psi_{R}\vert ^{p}\,dx \biggr)^{\frac{1}{p}} \biggl( \int_{\mathbb{R}^{N}} \vert x\vert ^{\beta} \vert \nabla v_{n}\vert ^{p}\,dx \biggr)^{\frac{p-1}{p}} \biggr\} \\ &\quad\leq C\lim_{R\rightarrow \infty} \biggl\{ \biggl( \int_{R< \vert x\vert < R+1} \vert x\vert ^{\beta} \vert u\vert ^{p}\vert \nabla\psi_{R}\vert ^{p}\,dx \biggr)^{\frac{1}{p}}+ \biggl( \int_{R< \vert x\vert < R+1} \vert x\vert ^{\beta} \vert v\vert ^{p}\vert \nabla\psi_{R}\vert ^{p}\,dx \biggr)^{\frac{1}{p}} \biggr\} \\ &\quad\leq C\lim_{R\rightarrow\infty} \biggl\{ \biggl( \int_{R< \vert x\vert < R+1} \vert x\vert ^{\frac{p^{\ast}\beta}{p}} \vert u\vert ^{p^{\ast}} \biggr)^{\frac{1}{p^{\ast}}} + \biggl( \int_{R< \vert x\vert < R+1} \vert x\vert ^{\frac{p^{\ast}\beta}{p}} \vert v\vert ^{p^{\ast}} \biggr)^{\frac{1}{p^{\ast}}} \biggr\} \biggl( \int_{\mathbb{R}^{N}} \vert \nabla\psi_{R}\vert ^{N} \biggr)^{\frac{1}{N}} \\ &\quad\leq C\lim_{R\rightarrow \infty} \biggl\{ \biggl( \int_{R< \vert x\vert < R+1} \vert x\vert ^{\beta} \vert \nabla u \vert ^{p}\,dx \biggr)^{\frac{1}{p}}+ \biggl( \int_{R< \vert x\vert < R+1} \vert x\vert ^{\beta} \vert \nabla v \vert ^{p}\,dx \biggr)^{\frac{1}{p}} \biggr\} =0. \end{aligned}$$

Consequently, we obtain from the definitions (1)-(3) and (3.10)

$$ K_{+}(\infty) \bigl(\mu_{1} \nu_{\infty}^{(1)} +\mu_{2}\nu_{\infty}^{(2)} \bigr) \geq\eta_{\infty}^{(1)}+\eta_{\infty}^{(2)}. $$

(3.11)

Moreover, by means of (3.3), we have $S_{\mu_{1}, \mu_{2}}(\mu_{1}\nu_{\infty}^{(1)} +\mu_{2}\nu_{\infty}^{(2)})^{p/p^{\ast}(\beta, \alpha)} \leq\eta_{\infty}^{(1)}+\eta_{\infty}^{(2)}$. This, combined with (3.11), implies that either (v) $\nu_{\infty}^{(1)}=\nu_{\infty}^{(2)}=0$ or (vi) $\mu_{1}\nu_{\infty}^{(1)}+\mu_{2}\nu_{\infty}^{(2)}\geq(S_{\mu_{1}, \mu_{2}}/K_{+}(\infty))^{\frac{N+\alpha}{\alpha-\beta+p}}$. In the following, we show that (ii), (iv) and (vi) cannot occur. For any nonnegative continuous function ψ such that $0\leq\psi(x)\leq1$ on $\mathbb{R}^{N}$, we have

$$\begin{aligned} c&=\lim_{n\rightarrow\infty} \biggl(\mathscr{F}(u_{n}, v_{n}) -\frac{1}{p^{\ast}(\beta, \alpha)} \bigl\langle \mathscr{F}^{\prime}(u_{n}, v_{n}), (u_{n}, v_{n}) \bigr\rangle \biggr) \\ &=\frac{\alpha-\beta+p}{(N+\alpha)p} \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \vert x\vert ^{\beta} \bigl(\vert \nabla u_{n}\vert ^{p}+\vert \nabla v_{n}\vert ^{p} \bigr)\,dx \\ &\geq\frac{\alpha-\beta+p}{(N+\alpha)p}\limsup_{n\rightarrow \infty} \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \bigl(\vert \nabla u_{n}\vert ^{p}+\vert \nabla v_{n}\vert ^{p} \bigr)\psi(x)\,dx. \end{aligned}$$

The alternative (i) or (ii) implies that $\mathscr {J}$ must be finite because the measures $\nu^{(1)}$ and $\nu^{(2)}$ are bounded. Moreover, $\nu^{(1)}$ and $\nu^{(2)}$ must be G-invariant because the functions $(u_{n}, v_{n})$ are G-symmetric. This implies that if $x_{j}\neq0$ is a singular point of $\nu^{(1)}$ and $\nu^{(2)}$, so is $gx_{j}$ for every $g\in G$, and the mass of $\nu^{(1)}$ and $\nu^{(2)}$ concentrated at $gx_{j}$ is the same for every $g\in G$. If (ii) holds for $x_{j}\neq0$, then we choose ψ with compact support so that $\psi(gx_{j})=1$ for every $g\in G$ and we derive

$$\begin{aligned} c&\geq\frac{\alpha-\beta+p}{(N+\alpha)p} \vert G\vert \bigl(\eta_{j}^{(1)}+ \eta_{j}^{(2)} \bigr)\geq \frac{\alpha-\beta+p}{(N+\alpha)p}\vert G\vert S_{\mu_{1}, \mu_{2}} \bigl(\mu_{1}\nu_{j}^{(1)} + \mu_{2}\nu_{j}^{(2)} \bigr) ^{\frac{p}{p^{\ast}(\beta, \alpha)}} \\ &\geq\frac{\alpha-\beta+p}{(N+\alpha)p}\vert G\vert S_{\mu_{1}, \mu_{2}} \bigl(S_{\mu_{1}, \mu_{2}}/ \Vert K_{+}\Vert _{\infty} \bigr)^{\frac{p}{p^{\ast}(\beta, \alpha)-p}} = \frac{(\alpha-\beta+p)\vert G\vert S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}}{ (N+\alpha)p\Vert K_{+}\Vert _{\infty}^{\frac{N+\beta-p}{\alpha-\beta+p}}}, \end{aligned}$$

which contradicts (3.7). Similarly, if (iv) occurs at $x=0$, we take ψ with compact support, so that $\psi(0)=1$ and we have

$$\begin{aligned} c&\geq\frac{\alpha-\beta+p}{(N+\alpha)p} \bigl(\eta_{0}^{(1)}+ \eta_{0}^{(2)} \bigr)\geq \frac{\alpha-\beta+p}{(N+\alpha)p}S_{\mu_{1}, \mu_{2}} \bigl(\mu_{1}\nu_{0}^{(1)} +\mu_{2} \nu_{0}^{(2)} \bigr) ^{\frac{p}{p^{\ast}(\beta, \alpha)}} \\ &\geq\frac{\alpha-\beta+p}{(N+\alpha)p}S_{\mu_{1}, \mu_{2}} \bigl(S_{\mu_{1}, \mu_{2}}/K_{+}(0) \bigr)^{\frac{p}{p^{\ast}(\beta, \alpha)-p}} =\frac{(\alpha-\beta+p)S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}}{ (N+\alpha)pK_{+}(0)^{\frac{N+\beta-p}{\alpha-\beta+p}}}, \end{aligned}$$

which is impossible. Finally, if (vi) holds we choose $\psi=\psi_{R}$ to obtain

$$\begin{aligned} c&\geq\frac{\alpha-\beta+p}{(N+\alpha)p} \bigl(\eta_{\infty}^{(1)}+ \eta_{\infty}^{(2)} \bigr)\geq \frac{\alpha-\beta+p}{(N+\alpha)p}S_{\mu_{1}, \mu_{2}} \bigl(\mu_{1}\nu_{\infty}^{(1)} +\mu_{2} \nu_{\infty}^{(2)} \bigr) ^{\frac{p}{p^{\ast}(\beta, \alpha)}} \\ &\geq\frac{\alpha-\beta+p}{(N+\alpha)p}S_{\mu_{1}, \mu_{2}} \bigl(S_{\mu_{1}, \mu_{2}}/K_{+}( \infty) \bigr)^{\frac{p}{p^{\ast}(\beta, \alpha)-p}} =\frac{(\alpha-\beta+p)S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}}{ (N+\alpha)pK_{+}(\infty)^{\frac{N+\beta-p}{\alpha-\beta+p}}}, \end{aligned}$$

a contradiction with (3.7). As a result, we find that $\nu_{j}^{(1)}=\nu_{j}^{(2)}=0$ for all $j\in\mathscr{J}\cup\{0, \infty\}$. This implies

$$\begin{aligned} &\lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \vert x\vert ^{\alpha} \bigl( \mu_{1}\vert u_{n}\vert ^{\varsigma_{1}}\vert v_{n}\vert ^{\tau_{1}} +\mu_{2}\vert u_{n}\vert ^{\varsigma_{2}}\vert v_{n}\vert ^{\tau_{2}} \bigr)\,dx \\ &\quad= \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \bigl( \mu_{1}\vert u\vert ^{\varsigma_{1}}\vert v\vert ^{\tau_{1}} +\mu_{2}\vert u\vert ^{\varsigma_{2}}\vert v\vert ^{\tau_{2}} \bigr)\,dx. \end{aligned}$$

Finally, by virtue of the fact that $\mathscr{F}^{\prime}(u, v)=0$ and $\lim_{n\rightarrow\infty}\langle \mathscr{F}^{\prime}(u_{n}, v_{n})-\mathscr{F}^{\prime}(u, v), (u_{n}-u, v_{n}-v)\rangle=0$, we obtain $(u_{n}, v_{n})\rightarrow (u, v)$ in $(\mathscr{D}_{\beta}^{1, p}(\mathbb{R}^{N}))^{2}$. □

According to Lemma 3.4, we immediately obtain the following result.

Corollary 3.1

If $\vert G\vert =+\infty$ and $K_{+}(0)=K_{+}(\infty)=0$, then the functional $\mathscr{F}$ fulfills the $(PS)_{c}$ condition for each $c\in\mathbb{R}$.

Proof of Theorem 2.1

Our argument is based upon the mountain pass theorem in [29] (see also [1]). Firstly, we choose $\epsilon>0$ such that (2.7) is satisfied. Thanks to (k.2), (3.1) and (3.3), we derive

$$\mathscr{F}(u, v)\geq\frac{1}{p} \bigl\Vert (u, v) \bigr\Vert ^{p}-\frac{1}{p^{\ast}(\beta, \alpha)}\Vert K\Vert _{\infty}S_{\mu_{1}, \mu_{2}}^{-\frac{p^{\ast}(\beta, \alpha)}{p}} \bigl\Vert (u, v) \bigr\Vert ^{p^{\ast}(\beta, \alpha)}. $$

Consequently, we deduce from $p^{\ast}(\beta, \alpha)>p$ that there exist constants $\alpha_{0}>0$ and $\rho>0$ such that $\mathscr{F}(u, v)\geq\alpha_{0}$ for all $\Vert (u, v)\Vert =\rho$. Let $y_{\epsilon}$ be the extremal function satisfying (2.4), (2.5) and (2.7). Now we set $u=y_{\epsilon}$, $v=\xi_{\min}y_{\epsilon}$ and

$$\begin{aligned} \Phi(t)={}&\mathscr{F}(tu, tv)=\mathscr{F}(ty_{\epsilon}, t \xi_{\min}y_{\epsilon})=\frac{t^{p}}{p} \bigl(1+ \xi_{\min}^{p} \bigr) \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta }\vert \nabla y_{\epsilon} \vert ^{p}\,dx \\ &{}-\frac{t^{p^{\ast}(\beta, \alpha)}}{p^{\ast}(\beta, \alpha)} \bigl(\mu_{1}\xi_{\min}^{\tau_{1}} + \mu_{2}\xi_{\min}^{\tau_{2}} \bigr) \int_{\mathbb{R}^{N}} K(x)\vert x\vert ^{\alpha}y_{\epsilon}^{p^{\ast}(\beta, \alpha)} \,dx,\quad t\geq0. \end{aligned}$$

It is trivial to verify that $\Phi(t)$ has a unique maximum at some $\overline{t}>0$. An easy computation gives us this value

$$\overline{t}= \biggl\{ \frac{ (1+\xi_{\min}^{p} ) \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert \nabla y_{\epsilon} \vert ^{p}\,dx}{ (\mu_{1}\xi_{\min}^{\tau_{1}}+\mu_{2}\xi_{\min}^{\tau _{2}} ) \int_{\mathbb{R}^{N}}K(x)\vert x\vert ^{\alpha}y_{\epsilon}^{p^{\ast}(\beta, \alpha)}\,dx} \biggr\} ^{\frac{1}{p^{\ast}(\beta, \alpha)-p}}. $$

Thus, we derive

$$\begin{aligned} \max_{t\geq 0}\Phi(t)&=\mathscr{F}( \overline{t}y_{\epsilon}, \overline{t}\xi_{\min}y_{\epsilon}) \\ &=\frac{\alpha-\beta+p}{(N+\alpha)p} \biggl\{ \frac{ (1+\xi_{\min}^{p} ) \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert \nabla y_{\epsilon} \vert ^{p}\,dx}{ [ (\mu_{1}\xi_{\min}^{\tau_{1}}+\mu_{2}\xi_{\min }^{\tau_{2}} ) \int_{\mathbb{R}^{N}}K(x)\vert x\vert ^{\alpha}y_{\epsilon}^{p^{\ast}(\beta, \alpha)}\,dx ] ^{\frac{p}{p^{\ast}(\beta, \alpha)}}} \biggr\} ^{\frac{N+\alpha}{\alpha-\beta+p}}. \end{aligned}$$

(3.12)

Moreover, because $\mathscr{F}(ty_{\epsilon}, t\xi_{\min}y_{\epsilon})\rightarrow-\infty$ as $t\rightarrow\infty$, we choose $t_{0}>0$ such that $\Vert (t_{0}y_{\epsilon}, t_{0}\xi_{\min}y_{\epsilon})\Vert >\rho$ and $\mathscr{F}(t_{0}y_{\epsilon}, t_{0}\xi_{\min}y_{\epsilon})<0$, and set

$$ c_{0}=\inf_{\gamma\in\Gamma}\max _{t\in[0, 1]}\mathscr{F} \bigl(\gamma(t) \bigr), $$

(3.13)

where $\Gamma=\{\gamma\in\mathscr{C}([0, 1], (\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}); \gamma(0)=(0, 0), \mathscr{F}(\gamma(1))<0, \Vert \gamma(1)\Vert >\rho\}$. Combining (2.5), (2.7), (3.5), (3.7), (3.12), (3.13), and Lemma 3.2, we have

$$\begin{aligned} &c_{0}\leq\mathscr{F}(\overline{t}y_{\epsilon}, \overline{t} \xi_{\min}y_{\epsilon}) \\ &\quad=\frac{\alpha-\beta+p}{(N+\alpha)p} \biggl\{ \frac{ (1+\xi_{\min}^{p} ) \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert \nabla y_{\epsilon} \vert ^{p}\,dx}{ [ (\mu_{1}\xi_{\min}^{\tau_{1}}+\mu_{2}\xi_{\min }^{\tau_{2}} ) \int_{\mathbb{R}^{N}}K(x)\vert x\vert ^{\alpha}y_{\epsilon}^{p^{\ast}(\beta, \alpha)}\,dx ] ^{\frac{p}{p^{\ast}(\beta, \alpha)}}} \biggr\} ^{\frac{N+\alpha}{\alpha-\beta+p}} \\ &\quad\leq\frac{\alpha-\beta+p}{(N+\alpha)p} \biggl\{ \frac{\mathscr{A}(\xi_{\min})\int_{\mathbb {R}^{N}}\vert x\vert ^{\beta} \vert \nabla y_{\epsilon} \vert ^{p}\,dx}{ [S^{\frac{-(N+\alpha)}{N+\beta-p}}\max \{\vert G\vert ^{-\frac{\alpha-\beta+p}{N+\beta-p}}\Vert K_{+}\Vert _{\infty}, K_{+}(0), K_{+}(\infty) \} ] ^{\frac{N+\beta-p}{N+\alpha}}} \biggr\} ^{\frac{N+\alpha}{\alpha-\beta+p}} \\ &\quad=\frac{\alpha-\beta+p}{ (N+\alpha)p}\min \biggl\{ \frac{\vert G\vert S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}}{ \Vert K_{+}\Vert _{\infty}^{\frac{N+\beta-p}{\alpha-\beta+p}}}, \frac{S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}}{ K_{+}(0)^{\frac{N+\beta-p}{\alpha-\beta+p}}}, \frac{S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}}{ K_{+}(\infty)^{\frac{N+\beta-p}{\alpha-\beta+p}}} \biggr\} =c_{0}^{\ast}. \end{aligned}$$

If $c_{0}< c_{0}^{\ast}$, then the $(PS)_{c}$ condition holds by Lemma 3.4 and the conclusion follows by the mountain pass theorem. On the other hand, if $c_{0}=c_{0}^{\ast}$, then $\gamma(t)=(tt_{0}y_{\epsilon}, tt_{0}\xi_{\min}y_{\epsilon})$, with $0\leq t\leq1$, is a path in Γ such that $\max_{t\in [0, 1]}\mathscr{F}(\gamma(t))=c_{0}$. As a result, either $\Phi^{\prime}(\overline{t})=0$ and we are done, or γ can be deformed to a path $\widetilde{\gamma}\in\Gamma$ satisfying $\max_{t\in[0, 1]}\mathscr{F}(\widetilde{\gamma}(t))< c_{0}$, which is a contradiction with (3.13). This part says that a nontrivial solution $(u_{0}, v_{0})\in(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$ of $(\mathscr{P}_{0}^{K})$ exists. Now we have just to show that the solution $(u_{0}, v_{0})$ can be chosen to be positive on $\mathbb{R}^{N}$. Due to the fact that $\mathscr{F}(u_{0}, v_{0})=\mathscr{F}(\vert u_{0}\vert , \vert v_{0}\vert )$, we obtain

$$\begin{aligned} 0&= \bigl\langle \mathscr{F}^{\prime}(u_{0}, v_{0}), (u_{0}, v_{0}) \bigr\rangle \\ &= \bigl\Vert (u_{0}, v_{0}) \bigr\Vert ^{p} - \int_{\mathbb{R}^{N}}K(x)\vert x\vert ^{\alpha} \bigl( \mu_{1}\vert u_{0}\vert ^{\varsigma_{1}}\vert v_{0}\vert ^{\tau_{1}} +\mu_{2}\vert u_{0}\vert ^{\varsigma_{2}}\vert v_{0}\vert ^{\tau_{2}} \bigr)\,dx. \end{aligned}$$

This implies $\int_{\mathbb{R}^{N}}K(x)\vert x\vert ^{\alpha} (\mu_{1}\vert u_{0}\vert ^{\varsigma_{1}}\vert v_{0}\vert ^{\tau_{1}} +\mu_{2}\vert u_{0}\vert ^{\varsigma_{2}}\vert v_{0}\vert ^{\tau_{2}})\,dx>0$, and hence, $c_{0}=\mathscr{F}(\vert u_{0}\vert , \vert v_{0}\vert )=\max_{t\geq 0}\mathscr{F}(t\vert u_{0}\vert , t\vert v_{0}\vert )$. As a result, either $(\vert u_{0}\vert , \vert v_{0}\vert )$ is a critical point of $\mathscr{F}$ or $\gamma(t)=(tt_{0}\vert u_{0}\vert , tt_{0}\vert v_{0}\vert )$, with $\mathscr{F}(t_{0}\vert u_{0}\vert , t_{0}\vert v_{0}\vert )<0$, can be deformed to a path $\widetilde{\gamma}(t)$ with $\max_{t\in[0, 1]}\mathscr{F}(\widetilde{\gamma}(t))< c_{0}$, which contradicts (3.13). Consequently, we may presume that $u_{0}\geq0$, $v_{0}\geq0$ on $\mathbb{R}^{N}$ and the fact that $u_{0}>0$, $v_{0}>0$ on $\mathbb{R}^{N}$ follows by the strong maximum principle. □

Proof of Corollary 2.1

Firstly, we remark that due to the identity (2.6), inequality (2.7) is equivalent to $\int_{\mathbb{R}^{N}}(K(x)-\overline{K})\vert x\vert ^{\alpha} y_{\epsilon}^{p^{\ast}(\beta, \alpha)}\,dx\geq 0$ for certain $\epsilon>0$, or equivalently

$$ \int_{\mathbb{R}^{N}}\frac{ (K(x)-\overline{K} )\vert x\vert ^{\alpha}}{ (\epsilon+\vert x\vert ^{\frac{\alpha-\beta+p}{p-1}} )^{\frac {(N+\alpha)p}{\alpha-\beta+p}}}\,dx\geq0 $$

(3.14)

for certain $\epsilon>0$, where $\overline{K}=\max\{\vert G\vert ^{-\frac{\alpha-\beta+p}{N+\beta-p}}\Vert K_{+}\Vert _{\infty}, K_{+}(0), K_{+}(\infty)\}$.

Part (1), case (i). Taking into account (3.14), we need to show that

$$ \int_{\mathbb{R}^{N}}\frac{ (K(x)-K(0) )\vert x\vert ^{\alpha}}{ (\epsilon+\vert x\vert ^{\frac{\alpha-\beta+p}{p-1}} )^{\frac {(N+\alpha)p}{\alpha-\beta+p}}}\,dx\geq0 $$

(3.15)

for certain $\epsilon>0$. We choose $\varrho_{0}>0$ so that $K(x)\geq K(0)+\Lambda_{0} \vert x\vert ^{\frac{N+\alpha}{p-1}}$ for $\vert x\vert \leq\varrho_{0}$. Thanks to $\frac{N+\alpha}{p-1}+\alpha -\frac{\alpha-\beta+p}{p-1}\cdot\frac{(N+\alpha)p}{\alpha-\beta+p}=-N$, we derive

$$\begin{aligned} & \int_{\vert x\vert \leq\varrho_{0}}\frac{ (K(x)-K(0) )\vert x\vert ^{\alpha}}{ (\epsilon+\vert x\vert ^{\frac{\alpha-\beta+p}{p-1}} )^{\frac {(N+\alpha)p}{\alpha-\beta+p}}}\,dx \\ &\quad\geq\Lambda_{0} \int_{\vert x\vert \leq\varrho_{0}}\frac{\vert x\vert ^{\frac{N+\alpha}{p-1}+\alpha}}{ (\epsilon+\vert x\vert ^{\frac{\alpha-\beta+p}{p-1}} )^{\frac {(N+\alpha)p}{\alpha-\beta+p}}}\,dx\rightarrow+\infty \end{aligned}$$

(3.16)

as $\epsilon\rightarrow0$. Moreover, for any $\epsilon>0$, we obtain

$$\begin{aligned} \int_{\vert x\vert > \varrho_{0}}\frac{\vert K(x)-K(0)\vert \vert x\vert ^{\alpha}}{ (\epsilon+\vert x\vert ^{\frac{\alpha-\beta+p}{p-1}} )^{\frac {(N+\alpha)p}{\alpha-\beta+p}}}\,dx&\leq \int_{\vert x\vert > \varrho_{0}} \bigl\vert K(x)-K(0) \bigr\vert \vert x\vert ^{\alpha-\frac{(N+\alpha)p}{p-1}}\,dx \\ &\leq C \int_{\vert x\vert > \varrho_{0}}\vert x\vert ^{-\frac{Np+\alpha}{p-1}}\,dx \leq \overline{C}_{1} \end{aligned}$$

(3.17)

for certain constant $\overline{C}_{1}>0$ independent of ϵ. By virtue of (3.16) and (3.17), we obtain (3.15) for ϵ small enough.

Part (1), case (ii). We choose $\varrho_{1}>0$ so that $\vert K(x)-K(0)\vert \leq\Lambda_{1}\vert x\vert ^{\theta}$ for $\vert x\vert \leq\varrho_{1}$. Because $\theta>\frac{N+\alpha}{p-1}$, we conclude from the fact $N-1+\alpha+\theta-\frac{(N+\alpha)p}{p-1}>-1$ and $N-1+\alpha-\frac{(N+\alpha)p}{p-1}<-1$ that

$$\begin{aligned} \int_{\mathbb{R}^{N}}\frac{\vert K(x)-K(0)\vert \vert x\vert ^{\alpha}}{ (\epsilon+\vert x\vert ^{\frac{\alpha-\beta+p}{p-1}} )^{\frac {(N+\alpha)p}{\alpha-\beta+p}}}\,dx&\leq C \biggl( \int_{\vert x\vert \leq\varrho_{1}} \vert x\vert ^{\alpha+\theta-\frac{(N+\alpha)p}{p-1}}\,dx + \int_{\vert x\vert >\varrho_{1}}\vert x\vert ^{\alpha-\frac{(N+\alpha)p}{p-1}}\,dx \biggr) \\ &\leq C \biggl( \int_{0}^{\varrho_{1}} r^{N-1+\alpha+\theta-\frac{(N+\alpha)p}{p-1}}\,dr + \int_{\varrho_{1}}^{+\infty}r^{N-1+\alpha-\frac{(N+\alpha )p}{p-1}}\,dr \biggr) \\ &\leq C. \end{aligned}$$

Then by (2.8) and the Lebesgue dominated convergence theorem, we derive

$$\lim_{\epsilon\rightarrow0} \int_{\mathbb{R}^{N}}\frac{ (K(x)-K(0) )\vert x\vert ^{\alpha}}{ (\epsilon+\vert x\vert ^{\frac{\alpha-\beta+p}{p-1}} )^{\frac {(N+\alpha)p}{\alpha-\beta+p}}}\,dx= \int_{\mathbb{R}^{N}} \bigl(K(x)-K(0) \bigr) \vert x\vert ^{-\frac{Np+\alpha}{p-1}}\,dx>0. $$

Hence (3.15) holds for ϵ small enough.

Part (2), case (i). In view of (3.14), it is enough to prove that

$$ \int_{\mathbb{R}^{N}}\frac{ (K(x)-K(\infty) ) \vert x\vert ^{\alpha}\epsilon^{\frac{(N+\alpha)p}{\alpha-\beta+p}}}{ (\epsilon+\vert x\vert ^{\frac{\alpha-\beta +p}{p-1}} )^{\frac{(N+\alpha)p}{\alpha-\beta+p}}}\,dx\geq0 $$

(3.18)

for certain $\epsilon>0$. We choose $\varrho_{2}>0$ such that $K(x)\geq K(\infty)+\Lambda_{2}\vert x\vert ^{-(N+\alpha)}$ for all $\vert x\vert \geq \varrho_{2}$. Then

$$\int_{\vert x\vert \geq\varrho_{2}}\frac{ (K(x)-K(\infty) ) \vert x\vert ^{\alpha}\epsilon^{\frac{(N+\alpha)p}{\alpha-\beta+p}}}{ (\epsilon+\vert x\vert ^{\frac{\alpha-\beta +p}{p-1}} )^{\frac{(N+\alpha)p}{\alpha-\beta+p}}}\,dx\geq\Lambda_{2} \int_{\vert x\vert \geq \varrho_{2}}\frac{ \vert x\vert ^{-N}\epsilon^{\frac{(N+\alpha)p}{\alpha-\beta+p}}}{ (\epsilon+\vert x\vert ^{\frac{\alpha-\beta +p}{p-1}} )^{\frac{(N+\alpha)p}{\alpha-\beta+p}}}\,dx\rightarrow+\infty $$

as $\epsilon\rightarrow+\infty$. Furthermore, for any $\epsilon>0$, we deduce from (k.2) and the fact that $\alpha>-N$ that

$$\int_{\vert x\vert \leq\varrho_{2}}\frac{\vert K(x)-K(\infty)\vert \vert x\vert ^{\alpha}\epsilon^{\frac{(N+\alpha)p}{\alpha-\beta+p}}}{ (\epsilon+\vert x\vert ^{\frac{\alpha-\beta +p}{p-1}} )^{\frac{(N+\alpha)p}{\alpha-\beta+p}}}\,dx\leq \int_{\vert x\vert \leq \varrho_{2}} \bigl\vert K(x)-K(\infty) \bigr\vert \vert x \vert ^{\alpha}\,dx\leq\overline{C}_{2} $$

for certain constant $\overline{C}_{2}>0$ independent of $\epsilon>0$. These two estimates combined together imply (3.18) for $\epsilon>0$ sufficiently large.

Part (2), case (ii). We choose $\varrho_{3}>0$ such that $\vert K(x)-K(\infty)\vert \leq\Lambda_{3}\vert x\vert ^{-\iota}$ for all $\vert x\vert \geq \varrho_{3}$. Because $\iota>N+\alpha>0$, we have

$$\begin{aligned} \int_{\mathbb{R}^{N}}\frac{\vert K(x)-K(\infty)\vert \vert x\vert ^{\alpha}\epsilon^{\frac{(N+\alpha)p}{\alpha-\beta+p}}}{ (\epsilon+\vert x\vert ^{\frac{\alpha-\beta +p}{p-1}} )^{\frac{(N+\alpha)p}{\alpha-\beta+p}}}\,dx&\leq \int_{\mathbb{R}^{N}} \bigl\vert K(x)-K(\infty) \bigr\vert \vert x \vert ^{\alpha}\,dx \\ &\leq\Lambda_{3} \int_{\vert x\vert \geq\varrho_{3}}\vert x\vert ^{\alpha-\iota}\,dx + \int_{\vert x\vert \leq\varrho_{3}} \bigl\vert K(x)-K(\infty) \bigr\vert \vert x \vert ^{\alpha}\,dx \\ &< +\infty. \end{aligned}$$

Thus, by (2.9) and the Lebesgue dominated convergence theorem, we find

$$\lim_{\epsilon\rightarrow+\infty} \int_{\mathbb{R}^{N}}\frac{ (K(x)-K(\infty) ) \vert x\vert ^{\alpha}\epsilon^{\frac{(N+\alpha)p}{\alpha-\beta+p}}}{ (\epsilon +\vert x\vert ^{\frac{\alpha-\beta+p}{p-1}} )^{\frac{(N+\alpha)p}{\alpha-\beta+p}}}\,dx= \int_{\mathbb{R}^{N}} \bigl(K(x)-K(\infty) \bigr) \vert x\vert ^{\alpha}\,dx>0 $$

and (3.18) holds for $\epsilon>0$ sufficiently large. Similarly to the above, we conclude that part (3) holds. □

To establish Theorem 2.2, we employ the following version of the symmetric mountain pass theorem (see [30], Theorem 9.12).

Lemma 3.5

Let X be an infinite dimensional Banach space and let $\mathscr{F}\in\mathscr{C}^{1}(X, \mathbb{R})$ be an even functional satisfying $(PS)_{c}$ condition for each c and $\mathscr{F}(0)=0$. Furthermore, one supposes that:

• (i) there exist constants $\overline{\alpha}>0$ and $\rho>0$ such that $\mathscr{F}(w)\geq\overline{\alpha}$ for all $\Vert w\Vert =\rho$;

• (ii) there exists an increasing sequence of subspaces $\{X_{m}\}$ of X, with $\dim X_{m}=m$, such that for every m one can find a constant $R_{m}>0$ such that $\mathscr {F}(w)\leq0$ for all $w\in X_{m}$ with $\Vert w\Vert \geq R_{m}$.

Then $\mathscr{F}$ possesses a sequence of critical values $\{c_{m}\}$ tending to ∞ as $m\rightarrow\infty$.

Proof of Theorem 2.2

We follow closely the arguments in [13], Theorem 3, (see also [18], Theorem 3). Applying Lemma 3.5 with $X=(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$ and $w=(u, v)\in X$, we deduce from (k.2), (3.1), and (3.3) that

$$\mathscr{F}(u, v)\geq\frac{1}{p} \bigl\Vert (u, v) \bigr\Vert ^{p}-\frac{1}{p^{\ast}(\beta, \alpha)}\Vert K\Vert _{\infty} S_{\mu_{1}, \mu_{2}}^{-\frac{p^{\ast}(\beta, \alpha)}{p}} \bigl\Vert (u, v) \bigr\Vert ^{p^{\ast}(\beta, \alpha)}. $$

Because $p^{\ast}(\beta, \alpha)>p>1$, there exist constants $\overline{\alpha}>0$ and $\rho>0$ such that $\mathscr{F}(u, v)\geq\overline{\alpha}$ for any $(u, v)$ with $\Vert (u, v)\Vert =\rho$. In order to seek an appropriate sequence of finite dimensional subspaces of $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$, we set $\Omega=\{x\in\mathbb{R}^{N}; K(x)>0\}$. Since Ω is G-symmetric, we can define by $(\mathscr{D}_{\beta, G}^{1, p}(\Omega))^{2}$ the subspace of $(\mathscr{D}_{\beta}^{1, p}(\Omega))^{2}$ consisting of al G-symmetric functions (see Section 2). Extending the functions in $(\mathscr{D}_{\beta, G}^{1, p}(\Omega))^{2}$ outside Ω by 0, we can presume that $(\mathscr{D}_{\beta, G}^{1, p}(\Omega))^{2}\subset(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$. Let $\{X_{m}\}$ be an increasing sequence of subspaces of $(\mathscr{D}_{\beta, G}^{1, p}(\Omega))^{2}$ with $\dim X_{m}=m$ for every m. As in [18], we define $\varphi_{1, m}, \ldots,\varphi_{m, m}\in\mathscr{C}_{0}^{\infty}(\mathbb{R}^{N})$ such that $0\leq \varphi_{i, m}\leq1$, $\operatorname{supp}(\varphi_{i, m})\cap\operatorname{supp}(\varphi_{j, m})=\emptyset$, $i\neq j$, and

$$\bigl\vert \operatorname{supp}(\varphi_{i, m})\cap\Omega \bigr\vert >0,\qquad \bigl\vert \operatorname{supp}(\varphi_{j, m})\cap\Omega \bigr\vert >0,\quad \forall i, j\in\{1, \ldots, m\}. $$

Taking $e_{i, m}=(a\varphi_{i, m}, b\varphi_{i, m})\in X_{m}$, $i=1, \ldots, m$, and $X_{m}=\operatorname{span}\{e_{1, m}, \ldots, e_{m, m}\}$, where a and b are two positive constants, we deduce from the construction of $X_{m}$ that $\dim X_{m}=m$ for each m. Then there exists a constant $\epsilon_{m}>0$ such that

$$\begin{aligned} & \int_{\Omega}K(x)\vert x\vert ^{\alpha} \bigl( \mu_{1}\vert \tilde{u}\vert ^{\varsigma_{1}}\vert \tilde{v}\vert ^{\tau_{1}} +\mu_{2}\vert \tilde{u}\vert ^{\varsigma_{2}}\vert \tilde{v}\vert ^{\tau_{2}} \bigr)\,dx \\ &\quad= \int_{\Omega}K(x)\vert x\vert ^{\alpha} \Biggl( \mu_{1} \Biggl\vert \sum_{i=1}^{m}at_{i, m} \varphi_{i, m} \Biggr\vert ^{\varsigma_{1}} \Biggl\vert \sum _{i=1}^{m}bt_{i, m} \varphi_{i, m} \Biggr\vert ^{\tau_{1}} \\ &\qquad{} +\mu_{2} \Biggl\vert \sum_{i=1}^{m}at_{i, m} \varphi_{i, m} \Biggr\vert ^{\varsigma_{2}} \Biggl\vert \sum _{i=1}^{m}bt_{i, m}\varphi_{i, m} \Biggr\vert ^{\tau_{2}} \Biggr)\,dx \geq\epsilon_{m} \end{aligned}$$

for all $(\tilde{u}, \tilde{v})=\sum_{i=1}^{m}t_{i, m}e_{i, m}\in X_{m}$, with $\Vert (\tilde{u}, \tilde{v})\Vert =1$. Hence, if $(u, v)\in X_{m}\backslash\{(0, 0)\}$, then we write $(u, v)=t(\tilde{u}, \tilde{v})$, with $t=\Vert (u, v)\Vert$ and $\Vert (\tilde{u}, \tilde{v})\Vert =1$. As a result, we derive

$$\begin{aligned} \mathscr{F}(u, v)&=\frac{1}{p}t^{p} -\frac{1}{p^{\ast}(\beta, \alpha)}t^{p^{\ast}(\beta, \alpha)} \int_{\Omega}K(x)\vert x\vert ^{\alpha} \bigl( \mu_{1}\vert \tilde{u}\vert ^{\varsigma_{1}}\vert \tilde{v}\vert ^{\tau_{1}} +\mu_{2}\vert \tilde{u}\vert ^{\varsigma_{2}}\vert \tilde{v}\vert ^{\tau_{2}} \bigr)\,dx \\ &\leq\frac{1}{p}t^{p} -\frac{\epsilon_{m}}{p^{\ast}(\beta, \alpha)}t^{p^{\ast}(\beta, \alpha)}\leq0 \end{aligned}$$

for t sufficiently large. Combining Lemma 3.5 and Corollary 3.1, we conclude that there exists a sequence of critical values $c_{m}\rightarrow\infty$ as $m\rightarrow\infty$ and the results follow. □

Proof of Corollary 2.2

Because $K(x)$ is radially symmetric, we easily see that the corresponding group $G=O(\mathbb{N})$ and $\vert G\vert =+\infty$. By virtue of Corollary 3.1, $\mathscr{F}$ fulfills the $(PS)_{c}$ condition for each $c\in\mathbb{R}$. Consequently, we conclude from Theorem 2.2 that the results follow. □

4 Existence results for problem $(\mathscr{P}_{h}^{K_{0}})$

The purpose of this section is to study problem $(\mathscr {P}_{h}^{K_{0}})$ and detail the proof of Theorem 2.3; here we always presume that $K(x)\equiv K_{0}>0$ is a constant on $\mathbb{R}^{N}$. First of all, we present the following compact embedding result, which is crucial for the proof of Theorem 2.3.

Lemma 4.1

Suppose that (h.2) is satisfied. Then the inclusion of $(\mathscr{D}_{\beta}^{1, p}(\mathbb{R}^{N}))^{2}$ in $(L^{q}(\mathbb{R}^{N}, h(x)))^{2}$ is compact. Moreover, if h fulfills (h.1) and (h.2) and $G\subset O(\mathbb{N})$ is closed, then $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$ is compactly embedded in $(L^{q}(\mathbb{R}^{N}, h(x)))^{2}$.

Proof

Similar to the argument of [6], Lemma 4.1, we choose $R_{1}>0$ and $R_{2}>0$ such that $0< R_{1}< R_{2}$. In view of (h.2), we employ the following integrals:

$$\begin{aligned} &I(u, v)= \int_{\mathbb{R}^{N}}h(x) \bigl(\vert u\vert ^{q}+\vert v \vert ^{q} \bigr)\,dx,\qquad I_{1}(u, v)= \int_{\vert x\vert < R_{1}}h(x) \bigl(\vert u\vert ^{q}+\vert v \vert ^{q} \bigr)\,dx, \\ &I_{2}(u, v)= \int_{\vert x\vert >R_{2}}h(x) \bigl(\vert u\vert ^{q}+\vert v \vert ^{q} \bigr)\,dx, \\ & I_{3}(u, v)= \int_{R_{1}\leq \vert x\vert \leq R_{2}}h(x) \bigl(\vert u\vert ^{q}+\vert v \vert ^{q} \bigr)\,dx. \end{aligned}$$

For $R_{1}>0$ small enough, we obtain from (h.2), (2.1), the Hölder inequality, and the fact that $N-1+\alpha>-1$, $(N+\alpha)(1-\frac{q}{p^{\ast}(\beta, \alpha)})>0$

$$\begin{aligned} I_{1}(u, v)={}& \int_{\vert x\vert < R_{1}}h(x) \bigl(\vert u\vert ^{q}+\vert v \vert ^{q} \bigr)\,dx\leq C \int_{\vert x\vert < R_{1}}\vert x\vert ^{\alpha} \bigl(\vert u \vert ^{q}+\vert v\vert ^{q} \bigr)\,dx \\ \leq{}& C \biggl\{ \biggl( \int_{\vert x\vert < R_{1}}\vert x\vert ^{\alpha} \vert u\vert ^{p^{\ast}(\beta, \alpha)}\,dx \biggr)^{\frac{q}{p^{\ast}(\beta, \alpha)}} \biggl( \int_{\vert x\vert < R_{1}}\vert x\vert ^{\alpha} \,dx \biggr)^{\frac{p^{\ast}(\beta, \alpha)-q}{p^{\ast}(\beta, \alpha)}} \\ &{} + \biggl( \int_{\vert x\vert < R_{1}}\vert x\vert ^{\alpha} \vert v\vert ^{p^{\ast}(\beta, \alpha)}\,dx \biggr)^{\frac{q}{p^{\ast}(\beta, \alpha)}} \biggl( \int_{\vert x\vert < R_{1}}\vert x\vert ^{\alpha} \,dx \biggr)^{\frac{p^{\ast}(\beta, \alpha)-q}{p^{\ast}(\beta, \alpha)}} \biggr\} \\ \leq{}& C \biggl\{ \biggl( \int_{\vert x\vert < R_{1}}\vert x\vert ^{\beta} \vert \nabla u \vert ^{p}\,dx \biggr)^{\frac{q}{p}} \biggl( \int_{\vert x\vert < R_{1}}\vert x\vert ^{\alpha} \,dx \biggr)^{\frac{p^{\ast}(\beta, \alpha)-q}{p^{\ast}(\beta, \alpha)}} \\ & {}+ \biggl( \int_{\vert x\vert < R_{1}}\vert x\vert ^{\beta} \vert \nabla v \vert ^{p}\,dx \biggr)^{\frac{q}{p}} \biggl( \int_{\vert x\vert < R_{1}}\vert x\vert ^{\alpha} \,dx \biggr)^{\frac{p^{\ast}(\beta, \alpha)-q}{p^{\ast}(\beta, \alpha)}} \biggr\} \\ \leq{}& C \biggl[ \int_{\vert x\vert < R_{1}}\vert x\vert ^{\beta} \bigl(\vert \nabla u\vert ^{p}+\vert \nabla v\vert ^{p} \bigr)\,dx \biggr]^{\frac{q}{p}} \biggl( \int_{0}^{R_{1}}r^{N-1+\alpha} \,dr \biggr)^{\frac{p^{\ast}(\beta, \alpha)-q}{p^{\ast}(\beta, \alpha)}} \\ \leq{}& C \bigl\Vert (u, v) \bigr\Vert ^{q}R_{1}^{(N+\alpha)(1-\frac{q}{p^{\ast}(\beta, \alpha)})} \rightarrow0 \end{aligned}$$

(4.1)

as $R_{1}\rightarrow0$. Moreover, for $R_{2}>0$ large enough, we conclude from (h.2), the Hölder inequality, and the fact that $N-1+\frac{\vartheta p^{\ast}(\beta, \alpha)-\alpha q}{p^{\ast}(\beta, \alpha)-q}<-1$, $N+\vartheta-\frac{(N+\alpha)q}{p^{\ast}(\beta, \alpha)}<0$ that

$$\begin{aligned} I_{2}(u, v)={}& \int_{\vert x\vert >R_{2}}h(x) \bigl(\vert u\vert ^{q}+\vert v \vert ^{q} \bigr)\,dx\leq C \int_{\vert x\vert >R_{2}}\vert x\vert ^{\vartheta} \bigl(\vert u \vert ^{q}+\vert v\vert ^{q} \bigr)\,dx \\ \leq{}& C \biggl\{ \biggl( \int_{\vert x\vert >R_{2}}\vert x\vert ^{\alpha} \vert u\vert ^{p^{\ast}(\beta, \alpha)}\,dx \biggr)^{\frac{q}{p^{\ast}(\beta, \alpha)}} \biggl( \int_{\vert x\vert >R_{2}}\vert x\vert ^{\frac{\vartheta p^{\ast}(\beta, \alpha)-\alpha q}{p^{\ast}(\beta, \alpha)-q}} \,dx \biggr)^{\frac{p^{\ast}(\beta, \alpha)-q}{p^{\ast}(\beta, \alpha)}} \\ &{} + \biggl( \int_{\vert x\vert >R_{2}}\vert x\vert ^{\alpha} \vert v\vert ^{p^{\ast}(\beta, \alpha)}\,dx \biggr)^{\frac{q}{p^{\ast}(\beta, \alpha)}} \biggl( \int_{\vert x\vert >R_{2}}\vert x\vert ^{\frac{\vartheta p^{\ast}(\beta, \alpha)-\alpha q}{p^{\ast}(\beta, \alpha)-q}} \,dx \biggr)^{\frac{p^{\ast}(\beta, \alpha)-q}{p^{\ast}(\beta, \alpha)}} \biggr\} \\ \leq {}&C \biggl\{ \biggl( \int_{\vert x\vert >R_{2}}\vert x\vert ^{\beta} \vert \nabla u \vert ^{p}\,dx \biggr)^{\frac{q}{p}} \biggl( \int_{\vert x\vert >R_{2}}\vert x\vert ^{\frac {\vartheta p^{\ast}(\beta, \alpha)-\alpha q}{p^{\ast}(\beta, \alpha)-q}} \,dx \biggr)^{\frac{p^{\ast}(\beta, \alpha)-q}{p^{\ast}(\beta, \alpha)}} \\ &{} + \biggl( \int_{\vert x\vert >R_{2}}\vert x\vert ^{\beta} \vert \nabla v \vert ^{p}\,dx \biggr)^{\frac{q}{p}} \biggl( \int_{\vert x\vert >R_{2}}\vert x\vert ^{\frac {\vartheta p^{\ast}(\beta, \alpha)-\alpha q}{p^{\ast}(\beta, \alpha)-q}} \,dx \biggr)^{\frac{p^{\ast}(\beta, \alpha)-q}{p^{\ast}(\beta, \alpha)}} \biggr\} \\ \leq{}& C \biggl[ \int_{\vert x\vert >R_{2}}\vert x\vert ^{\beta} \bigl(\vert \nabla u\vert ^{p}+\vert \nabla v\vert ^{p} \bigr)\,dx \biggr]^{\frac{q}{p}} \biggl( \int_{R_{2}}^{+\infty }r^{N-1+\frac{\vartheta p^{\ast}(\beta, \alpha)-\alpha q}{p^{\ast}(\beta, \alpha)-q}} \,dr \biggr)^{\frac{p^{\ast}(\beta, \alpha)-q}{p^{\ast}(\beta, \alpha)}} \\ \leq{}& C \bigl\Vert (u, v) \bigr\Vert ^{q}R_{2}^{N+\vartheta-\frac{(N+\alpha)q}{p^{\ast}(\beta, \alpha)}} \rightarrow0 \end{aligned}$$

(4.2)

as $R_{2}\rightarrow\infty$. Let $\{(u_{n}, v_{n})\}$ be bounded in $(\mathscr{D}_{\beta}^{1, p}(\mathbb{R}^{N}))^{2}$ and set $\omega=\{x\in\mathbb{R}^{N}; R_{1}<\vert x\vert <R_{2}\}$. Obviously, we can presume $(u_{n}, v_{n})\rightharpoonup(u, v)$ in $(\mathscr{D}_{\beta}^{1, p}(\mathbb{R}^{N}))^{2}$. Now it is trivial to verify that $\lim_{n\rightarrow\infty}I_{3}(u_{n}-u, v_{n}-v)=0$ by utilizing the local boundedness of $h(x)$ and the compactness of the inclusion of $(\mathscr{D}_{\beta}^{1, p}(\omega))^{2}$ in $(L^{q}(\omega))^{2}$. Consequently, we deduce from (4.1) and (4.2) that $\lim_{n\rightarrow\infty}I(u_{n}-u, v_{n}-v)=0$ by taking $R_{1}\rightarrow0$ and $R_{2}\rightarrow+\infty$. This implies that the inclusion of $(\mathscr{D}_{\beta}^{1, p}(\mathbb{R}^{N}))^{2}$ in $(L^{q}(\mathbb{R}^{N}, h(x)))^{2}$ is compact. Furthermore, because $O(\mathbb{N})$ is a compact Lie group and $G\subset O(\mathbb{N})$ is closed, G is compact. Therefore, combining the arguments in Schneider [31], Corollaries 3.4 and 3.2, and the first part of the proof, we conclude that $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$ is compactly embedded in $(L^{q}(\mathbb{R}^{N}, h(x)))^{2}$ and the results follow. □

To establish the existence of positive G-symmetric solutions of problem $(\mathscr{P}_{h}^{K_{0}})$, we consider the energy functional $J: (\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}\rightarrow\mathbb{R}$ defined by

$$\begin{aligned} J(u, v)={}&\frac{1}{p} \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \bigl(\vert \nabla u\vert ^{p}+\vert \nabla v\vert ^{p} \bigr)\,dx \\ &{}- \frac{K_{0}}{p^{\ast}(\beta, \alpha)} \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \bigl(\mu _{1} \bigl\vert u^{+} \bigr\vert ^{\varsigma_{1}} \bigl\vert v^{+} \bigr\vert ^{\tau_{1}} +\mu_{2} \bigl\vert u^{+} \bigr\vert ^{\varsigma_{2}} \bigl\vert v^{+} \bigr\vert ^{\tau_{2}} \bigr)\,dx \\ &{}- \frac{1}{q} \int_{\mathbb{R}^{N}}h(x) \bigl( \bigl\vert u^{+} \bigr\vert ^{q}+ \bigl\vert v^{+} \bigr\vert ^{q} \bigr)\,dx, \end{aligned}$$

(4.3)

where $u^{+}=\max\{0,u\}$ and $v^{+}=\max\{0,v\}$. It is easy to check from (h.1), (h.2), (2.1), (4.3), and Lemma 4.1 that J is well defined and of $\mathscr {C}^{1}$. By an analogous symmetric criticality principle of Lemma 3.1, we mention that the weak solutions of problem $(\mathscr{P}_{h}^{K_{0}})$ are exactly the critical points of J.

Let $y_{\epsilon}(x)$ be the extremal function satisfying (2.4)-(2.6). In view of (h.2), we choose $\varrho>0$ such that $B_{2\varrho}(0)\subset\mathscr{O}$ and define a function $\phi\in\mathscr{C}^{1}(\mathbb{R}^{N})$ such that $\phi(x)=1$ for $\vert x\vert \leq\varrho$, $\phi(x)=0$ for $\vert x\vert \geq2\varrho$, $0\leq\phi(x)\leq1$ and $\vert \nabla\phi \vert \leq4/\varrho$ on $\mathbb{R}^{N}$. Following the arguments in [1, 3], we obtain from (2.4)-(2.6)

$$\begin{aligned} &\Vert \phi y_{\epsilon} \Vert ^{p}= \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \bigl\vert \nabla( \phi y_{\epsilon}) \bigr\vert ^{p}\,dx=1+O \bigl( \epsilon^{\frac{N+\beta-p}{\alpha -\beta+p}} \bigr), \end{aligned}$$

(4.4)

$$\begin{aligned} & \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert \phi y_{\epsilon} \vert ^{p^{\ast}(\beta, \alpha)}\,dx= S^{-\frac{N+\alpha}{N+\beta-p}}+O \bigl( \epsilon^{\frac{N+\alpha }{\alpha-\beta+p}} \bigr), \end{aligned}$$

(4.5)

$$\begin{aligned} & \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert \phi y_{\epsilon} \vert ^{q}\,dx= \textstyle\begin{cases} O (\epsilon^{\frac{q(N+\beta-p)}{p(\alpha-\beta+p)}} ),& 1\leq q< \frac{(p-1)(N+\alpha)}{N+\beta-p}, \\ O (\epsilon^{\frac{q(N+\beta-p)}{p(\alpha-\beta+p)}}\vert \ln \epsilon \vert ),& q= \frac{(p-1)(N+\alpha)}{N+\beta-p}, \\ O (\epsilon^{\frac{(p-1)[N+\alpha-\frac{q}{p}(N+\beta-p)]}{\alpha-\beta+p}} ), &\frac{(p-1)(N+\alpha)}{N+\beta-p}< q< p^{\ast}( \beta, \alpha). \end{cases}\displaystyle \end{aligned}$$

(4.6)

Set $V_{\epsilon}=\phi y_{\epsilon}/\Vert \phi y_{\epsilon} \Vert$; then by (4.4), (4.5), and (4.6) we obtain $\Vert V_{\epsilon} \Vert =1$ and

$$\begin{aligned} & \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert V_{\epsilon} \vert ^{p^{\ast}(\beta, \alpha)}\,dx = \int_{\mathbb{R}^{N}}\frac{\vert x\vert ^{\alpha} \vert \phi y_{\epsilon} \vert ^{p^{\ast}(\beta, \alpha)}}{\Vert \phi y_{\epsilon} \Vert ^{p^{\ast}(\beta, \alpha)}}\,dx=S^{-\frac{N+\alpha}{N+\beta-p}} +O \bigl( \epsilon^{\frac{N+\beta-p}{\alpha-\beta+p}} \bigr), \end{aligned}$$

(4.7)

$$\begin{aligned} &\textstyle\begin{cases} C_{1}\epsilon^{\frac{q(N+\beta-p)}{p(\alpha-\beta+p)}}\leq \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert V_{\epsilon} \vert ^{q}\leq C_{2}\epsilon^{\frac{q(N+\beta-p)}{p(\alpha-\beta+p)}},\quad 1\leq q< \frac{(p-1)(N+\alpha)}{N+\beta-p}, \\ C_{3}\epsilon^{\frac{q(N+\beta-p)}{p(\alpha-\beta+p)}}\vert \ln \epsilon \vert \leq \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert V_{\epsilon} \vert ^{q}\leq C_{4} \epsilon^{\frac{q(N+\beta-p)}{p(\alpha-\beta+p)}}\vert \ln \epsilon \vert ,\quad q=\frac{(p-1)(N+\alpha)}{N+\beta-p}, \\ C_{5}\epsilon^{\frac{N+\alpha-\frac{q}{p}(N+\beta-p)}{\frac{\alpha-\beta+p}{p-1}}}\leq \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert V_{\epsilon} \vert ^{q}\leq C_{6} \epsilon^{\frac{N+\alpha-\frac{q}{p}(N+\beta-p)}{\frac{\alpha-\beta+p}{p-1}}},\quad \frac{N+\alpha}{\frac{N+\beta-p}{p-1}}< q< p^{\ast}(\beta, \alpha). \end{cases}\displaystyle \end{aligned}$$

(4.8)

Lemma 4.2

Suppose that (2.10), (h.1), and (h.2) hold. Then there exists a pair of functions $(\overline{u}, \overline{v})\in (\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N})\backslash \{0\})^{2}$ such that

$$ \sup_{t\geq0}J (t\overline{u}, t\overline{v} ) < \frac{\alpha-\beta+p}{ (N+\alpha)p}K_{0}^{-\frac{N+\beta-p}{\alpha-\beta+p}}S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}. $$

(4.9)

Proof

Recall that $\xi_{\min}>0$ satisfies (3.4)-(3.6), and $V_{\epsilon}=\phi y_{\epsilon}/\Vert \phi y_{\epsilon} \Vert$ fulfills (4.7) and (4.8). Now, we claim that $(V_{\epsilon}, \xi_{\min}V_{\epsilon})$ fulfills (4.9) for $\epsilon>0$ small enough. Indeed, if we set

$$\begin{aligned} \Psi(t)={}&J (tV_{\epsilon}, t\xi_{\min}V_{\epsilon} )= \frac{1+\xi_{\min}^{p}}{p} t^{p} \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert \nabla V_{\epsilon} \vert ^{p}\,dx \\ &{} -\frac{\mu_{1}\xi_{\min}^{\tau_{1}} +\mu_{2}\xi_{\min}^{\tau_{2}}}{p^{\ast}(\beta, \alpha)} t^{p^{\ast}(\beta, \alpha)} K_{0} \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert V_{\epsilon} \vert ^{p^{\ast }(\beta, \alpha)}\,dx -\frac{1+\xi_{\min}^{q} }{q}t^{q} \int_{\mathbb{R}^{N}}h(x) \vert V_{\epsilon} \vert ^{q} \,dx \end{aligned}$$

and

$$\widetilde{\Psi}(t)=\frac{1+\xi_{\min}^{p}}{p} t^{p} \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert \nabla V_{\epsilon} \vert ^{p}\,dx-\frac{\mu_{1}\xi_{\min}^{\tau_{1}} +\mu_{2}\xi_{\min}^{\tau_{2}}}{p^{\ast}(\beta, \alpha)} t^{p^{\ast}(\beta, \alpha)} K_{0} \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert V_{\epsilon} \vert ^{p^{\ast }(\beta, \alpha)} $$

with $t\geq0$, then we conclude from (h.2) and the fact that $p^{\ast}(\beta, \alpha)>q>p>1$ that $\Psi(0)=0$, $\Psi(t)>0$ for $t\rightarrow0^{+}$, and $\lim_{t\rightarrow +\infty}\Psi(t)=-\infty$. Consequently, $\sup_{t\geq0}\Psi(t)$ can be achieved at certain $t_{\epsilon}>0$ for which we derive

$$\begin{aligned} & \bigl(1+\xi_{\min}^{p} \bigr)t_{\epsilon}^{p-1} \int_{\mathbb {R}^{N}}\vert x\vert ^{\beta} \vert \nabla V_{\epsilon} \vert ^{p}\,dx-K_{0} \bigl( \mu_{1}\xi_{\min}^{\tau_{1}} +\mu_{2} \xi_{\min}^{\tau_{2}} \bigr)t_{\epsilon}^{p^{\ast}(\beta, \alpha)-1} \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert V_{\epsilon} \vert ^{p^{\ast }(\beta, \alpha)} \\ &\quad {}- \bigl(1+\xi_{\min}^{q} \bigr)t_{\epsilon}^{q-1} \int_{\mathbb{R}^{N}}h(x)\vert V_{\epsilon} \vert ^{q} \,dx=0. \end{aligned}$$

(4.10)

Therefore, by (h.2), (4.7), and (4.10) we obtain

$$ 0< \overline{C}_{3}\leq t_{\epsilon}\leq \biggl\{ \frac{ (1+\xi_{\min}^{p} ) \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert \nabla V_{\epsilon} \vert ^{p}\,dx}{K_{0} (\mu_{1}\xi_{\min}^{\tau_{1}} +\mu_{2}\xi_{\min}^{\tau_{2}} )\int_{\mathbb{R}^{N}} \vert x\vert ^{\alpha} \vert V_{\epsilon} \vert ^{p^{\ast}(\beta, \alpha)}\,dx} \biggr\} ^{\frac{1}{p^{\ast}(\beta, \alpha)-p}}\triangleq t_{\epsilon}^{0} \leq\overline{C}_{4}, $$

(4.11)

where $\overline{C}_{3}>0$, $\overline{C}_{4}>0$ are constants independent of ϵ. Moreover, $\widetilde{\Psi}(t)$ is a monotonically increasing function on the interval $[0, t_{\epsilon}^{0}]$ and attains its maximum at $t_{\epsilon}^{0}$, together with Lemma 3.2, (4.3), (4.7), (4.11), and $h(x)\geq C\vert x\vert ^{\alpha}$, which is directly derived from (h.2), we have

$$\begin{aligned} \Psi(t_{\epsilon})={}&\widetilde{\Psi}(t_{\epsilon}) - \frac{t_{\epsilon}^{q}}{q} \bigl(1+\xi_{\min}^{q} \bigr) \int_{\mathbb{R}^{N}}h(x)\vert V_{\epsilon} \vert ^{q} \,dx \leq\widetilde{\Psi} \bigl(t_{\epsilon}^{0} \bigr) -C \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert V_{\epsilon} \vert ^{q}\,dx \\ =&{}\frac{\alpha-\beta+p}{(N+\alpha)p} \biggl\{ \frac{\mathscr{A}(\xi_{\min}) \int_{\mathbb{R}^{N}}\vert x\vert ^{\beta} \vert \nabla V_{\epsilon} \vert ^{p}\,dx}{ (K_{0}\int_{\mathbb{R}^{N}} \vert x\vert ^{\alpha} \vert V_{\epsilon} \vert ^{p^{\ast}(\beta, \alpha)}\,dx )^{\frac{N+\beta-p}{N+\alpha}}} \biggr\} ^{\frac{N+\alpha }{\alpha-\beta+p}} -C \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert V_{\epsilon} \vert ^{q}\,dx \\ ={}&\frac{\alpha-\beta+p}{ (N+\alpha)p}K_{0}^{-\frac{N+\beta-p}{\alpha-\beta+p}} \biggl\{ \frac{\mathscr{A}(\xi_{\min})}{ [S^{\frac{-(N+\alpha)}{N+\beta-p}} +O(\epsilon^{\frac{N+\beta-p}{\alpha-\beta+p}})]^{\frac{N+\beta -p}{N+\alpha}}} \biggr\} ^{\frac{N+\alpha}{\alpha-\beta+p}}-C \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert V_{\epsilon} \vert ^{q}\,dx \\ ={}&\frac{\alpha-\beta+p}{ (N+\alpha)p}K_{0}^{-\frac{N+\beta-p}{\alpha-\beta+p}}S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}} +O \bigl(\epsilon^{\frac{N+\beta-p}{\alpha-\beta+p}} \bigr)-C \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert V_{\epsilon} \vert ^{q}\,dx. \end{aligned}$$

(4.12)

Furthermore, it is easy to check from (2.10) that

$$ \frac{N+\beta-p}{\alpha-\beta+p}> \frac{(p-1)[N+\alpha-\frac{q}{p}(N+\beta-p)]}{\alpha-\beta+p}. $$

(4.13)

Therefore, by choosing $\epsilon>0$ sufficiently small, we obtain from (4.8), (4.12), and (4.13)

$$\sup_{t\geq0}J (tV_{\epsilon}, t\xi_{\min}V_{\epsilon} )=\Psi(t_{\epsilon}) < \frac{\alpha-\beta+p}{ (N+\alpha)p}K_{0}^{-\frac{N+\beta-p}{\alpha-\beta+p}}S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}. $$

Hence, we find that $(V_{\epsilon}, \xi_{\min}V_{\epsilon})$ fulfills (4.9) for $\epsilon>0$ small enough and the conclusion follows. □

Lemma 4.3

Suppose that (h.1) and (h.2) hold. Then the $(PS)_{c}$ condition in $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$ holds for J if

$$ c< \frac{\alpha-\beta+p}{ (N+\alpha)p}K_{0}^{-\frac{N+\beta-p}{\alpha-\beta+p}}S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}. $$

(4.14)

Proof

Let $\{(u_{n}, v_{n})\}$ be a sequence in $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$ such that $J(u_{n}, v_{n})\rightarrow c$ and $J^{\prime}(u_{n}, v_{n})\rightarrow0$ in $(\mathscr{D}_{\beta, G}^{-1, p^{\prime}}(\mathbb{R}^{N}))^{2}$ with c satisfying (4.14). Then, for n large enough, we deduce from $1< p< p^{\ast}(\beta, \vartheta)< q< p^{\ast}(\beta, \alpha)$ that

$$\begin{aligned} c+1\geq{}& J(u_{n}, v_{n})-\frac{1}{q} \bigl\langle J^{\prime}(u_{n}, v_{n}), (u_{n}, v_{n}) \bigr\rangle +\frac{1}{q} \bigl\langle J^{\prime}(u_{n}, v_{n}), (u_{n}, v_{n}) \bigr\rangle \\ ={}& \biggl(\frac{1}{p}-\frac{1}{q} \biggr) \bigl\Vert (u_{n}, v_{n}) \bigr\Vert ^{p}+ \frac{1}{q} \bigl\langle J^{\prime}(u_{n}, v_{n}), (u_{n}, v_{n}) \bigr\rangle \\ &{} + \biggl(\frac{1}{q} -\frac{1}{p^{\ast}(\beta, \alpha)} \biggr)K_{0} \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \bigl( \mu_{1} \bigl\vert u_{n}^{+} \bigr\vert ^{\varsigma_{1}} \bigl\vert v_{n}^{+} \bigr\vert ^{\tau_{1}}+\mu_{2} \bigl\vert u_{n}^{+} \bigr\vert ^{\varsigma_{2}} \bigl\vert v_{n}^{+} \bigr\vert ^{\tau_{2}} \bigr)\,dx \\ \geq{}& \biggl(\frac{1}{p}-\frac{1}{q} \biggr) \bigl\Vert (u_{n}, v_{n}) \bigr\Vert ^{p}+o_{n}(1) \bigl\Vert (u_{n}, v_{n}) \bigr\Vert . \end{aligned}$$

It follows that $\{(u_{n}, v_{n})\}$ is bounded. We can assume, going if necessary to a subsequence, that $(u_{n}, v_{n})\rightharpoonup(u, v)$ in $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$ and in $(L^{p^{\ast}(\beta, \alpha)}(\mathbb{R}^{N}, \vert x\vert ^{\alpha}))^{2}$; moreover, according to Lemma 4.1, $(u_{n}, v_{n})\rightarrow(u, v)$ in $(L^{q}(\mathbb{R}^{N}, h(x)))^{2}$ and a.e. on $\mathbb{R}^{N}$. Applying a standard variational method, we find that $(u, v)$ is a critical point of J, and hence

$$\begin{aligned} J(u, v)={}& \biggl(\frac{1}{p}-\frac{1}{p^{\ast}(\beta, \alpha)} \biggr) K_{0} \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \bigl( \mu_{1} \bigl\vert u^{+} \bigr\vert ^{\varsigma_{1}} \bigl\vert v^{+} \bigr\vert ^{\tau_{1}}+\mu_{2} \bigl\vert u^{+} \bigr\vert ^{\varsigma_{2}} \bigl\vert v^{+} \bigr\vert ^{\tau_{2}} \bigr)\,dx \\ &{}+ \biggl(\frac{1}{p}-\frac{1}{q} \biggr) \int_{\mathbb{R}^{N}}h(x) \bigl( \bigl\vert u^{+} \bigr\vert ^{q}+ \bigl\vert v^{+} \bigr\vert ^{q} \bigr)\,dx\geq0. \end{aligned}$$

(4.15)

Let $\widetilde{u}_{n}=u_{n}-u$ and $\widetilde{v}_{n}=v_{n}-v$. Then by the Brezis-Lieb lemma [32] and arguing as in [33], Lemma 2.1, we have

$$\begin{aligned} & \bigl\Vert (\widetilde{u}_{n}, \widetilde{v}_{n}) \bigr\Vert ^{p}= \bigl\Vert (u_{n}, v_{n}) \bigr\Vert ^{p}- \bigl\Vert (u, v) \bigr\Vert ^{p}+o_{n}(1), \end{aligned}$$

(4.16)

$$\begin{aligned} & \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \bigl\vert \widetilde{u}_{n}^{+} \bigr\vert ^{\varsigma_{i}} \bigl\vert \widetilde{v}_{n}^{+} \bigr\vert ^{\tau_{i}} \,dx = \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \bigl\vert u_{n}^{+} \bigr\vert ^{\varsigma_{i}} \bigl\vert v_{n}^{+} \bigr\vert ^{\tau_{i}}\,dx \\ &\phantom{\int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \vert \widetilde{u}_{n}^{+} \vert ^{\varsigma_{i}} \vert \widetilde{v}_{n}^{+} \vert ^{\tau_{i}}\,dx =}{}- \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \bigl\vert u^{+} \bigr\vert ^{\varsigma_{i}} \bigl\vert v^{+} \bigr\vert ^{\tau_{i}}\,dx+o_{n}(1),\quad i=1, 2. \end{aligned}$$

(4.17)

Taking account into $J(u_{n}, v_{n})=c+o_{n}(1)$ and $J^{\prime}(u_{n}, v_{n})=o_{n}(1)$, we deduce from (4.3), (4.16), and (4.17) that

$$\begin{aligned} c+o_{n}(1)={}&J(u_{n}, v_{n})=J(u, v)+\frac{1}{p} \bigl\Vert (\widetilde{u}_{n}, \widetilde{v}_{n}) \bigr\Vert ^{p} \\ &{}-\frac{K_{0}}{p^{\ast}(\beta, \alpha)} \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \bigl( \mu_{1} \bigl\vert \widetilde{u}_{n}^{+} \bigr\vert ^{\varsigma_{1}} \bigl\vert \widetilde{v}_{n}^{+} \bigr\vert ^{\tau _{1}}+\mu_{2} \bigl\vert \widetilde{u}_{n}^{+} \bigr\vert ^{\varsigma_{2}} \bigl\vert \widetilde{v}_{n}^{+} \bigr\vert ^{\tau _{2}} \bigr)\,dx+o_{n}(1) \end{aligned}$$

(4.18)

and

$$ \bigl\Vert (\widetilde{u}_{n}, \widetilde{v}_{n}) \bigr\Vert ^{p} -K_{0} \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \bigl( \mu_{1} \bigl\vert \widetilde{u}_{n}^{+} \bigr\vert ^{\varsigma_{1}} \bigl\vert \widetilde{v}_{n}^{+} \bigr\vert ^{\tau_{1}}+\mu_{2} \bigl\vert \widetilde{u}_{n}^{+} \bigr\vert ^{\varsigma_{2}} \bigl\vert \widetilde{v}_{n}^{+} \bigr\vert ^{\tau_{2}} \bigr)\,dx=o_{n}(1). $$

(4.19)

Therefore, for a subsequence $\{(\widetilde{u}_{n}, \widetilde{v}_{n})\}$, we get

$$\bigl\Vert (\widetilde{u}_{n}, \widetilde{v}_{n}) \bigr\Vert ^{p} \rightarrow l\geq0 \quad\text{and}\quad K_{0} \int_{\mathbb{R}^{N}}\vert x\vert ^{\alpha} \bigl( \mu_{1} \bigl\vert \widetilde{u}_{n}^{+} \bigr\vert ^{\varsigma_{1}} \bigl\vert \widetilde{v}_{n}^{+} \bigr\vert ^{\tau_{1}}+\mu_{2} \bigl\vert \widetilde{u}_{n}^{+} \bigr\vert ^{\varsigma_{2}} \bigl\vert \widetilde{v}_{n}^{+} \bigr\vert ^{\tau_{2}} \bigr)\,dx\rightarrow l $$

as $n\rightarrow\infty$. By virtue of (3.3), we derive $S_{\mu_{1}, \mu_{2}}(l/K_{0})^{\frac{p}{p^{\ast}(\beta, \alpha)}}\leq l$, which implies either $l=0$ or $l\geq K_{0}^{-\frac{N+\beta-p}{\alpha-\beta+p}}S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}$. If $l\geq K_{0}^{-\frac{N+\beta-p}{\alpha-\beta+p}}S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}$, then we obtain from (4.15), (4.18), and (4.19)

$$c=J(u, v)+ \biggl(\frac{1}{p}-\frac{1}{p^{\ast}(\beta, \alpha)} \biggr)l \geq \frac{\alpha-\beta+p}{(N+\alpha)p} K_{0}^{-\frac{N+\beta-p}{\alpha-\beta+p}}S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}, $$

which is a contradiction with (4.14). As a result, we have $\Vert (\widetilde{u}_{n}, \widetilde{v}_{n})\Vert ^{p}\rightarrow0$ as $n\rightarrow\infty$, and thus $(u_{n}, v_{n})\rightarrow(u, v)$ in $(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$. □

Proof of Theorem 2.3

For any $(u, v)\in(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N})\backslash\{0\})^{2}$, we find from Lemma 4.1, (2.1), (4.3), and the Hölder inequality that

$$J(u, v)\geq\frac{1}{p} \bigl\Vert (u, v) \bigr\Vert ^{p}- \frac{K_{0}}{p^{\ast}(\beta, \alpha)} S_{\mu_{1}, \mu_{2}}^{-\frac{p^{\ast}(\beta, \alpha)}{p}} \bigl\Vert (u, v) \bigr\Vert ^{p^{\ast}(\beta, \alpha)}-C \bigl\Vert (u, v) \bigr\Vert ^{q}. $$

In view of $p< q< p^{\ast}(\beta, \alpha)$, there exist constants $\widetilde{\alpha}>0$ and $\rho>0$ such that $J(u, v)\geq\widetilde{\alpha}$ for all $\Vert (u, v)\Vert =\rho$. On the other hand, taking account into $\lim_{t\rightarrow\infty}J(tu, tv)\rightarrow-\infty$, we see that there exists $\widetilde{t}>0$ such that $\Vert (\widetilde{t}u, \widetilde{t}v)\Vert >\rho$ and $J(\widetilde{t}u, \widetilde{t}v)<0$. Now, we set

$$c_{1}=\inf_{\gamma\in\Gamma}\max_{t\in[0, 1]}J \bigl(\gamma(t) \bigr), $$

where $\Gamma=\{\gamma\in\mathscr{C}([0, 1], (\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}); \gamma(0)=(0, 0), J(\gamma(1))<0, \Vert \gamma(1)\Vert >\rho\}$. According to the mountain pass theorem in [29], we deduce that there exists a sequence $\{(u_{n}, v_{n})\}\subset(\mathscr{D}_{\beta, G}^{1, p}(\mathbb{R}^{N}))^{2}$ such that $J(u_{n}, v_{n})\rightarrow c_{1}\geq\widetilde{\alpha}$, $J^{\prime}(u_{n}, v_{n})\rightarrow 0$ as $n\rightarrow\infty$. Let $(\overline{u}, \overline{v})$ be the functions attained in Lemma 4.2. Then we have

$$0< \widetilde{\alpha}\leq c_{1}\leq\sup_{t\in[0, 1]}J(t \widetilde{t}\overline{u}, t\widetilde{t}\overline{v})< \frac{\alpha-\beta+p}{(N+\alpha)p} K_{0}^{-\frac{N+\beta-p}{\alpha-\beta+p}}S_{\mu_{1}, \mu_{2}}^{\frac{N+\alpha}{\alpha-\beta+p}}. $$

By Lemma 4.3 and the above inequality, we obtain a critical point $(u_{1}, v_{1})$ of J satisfying $(\mathscr {P}_{h}^{K_{0}})$. Taking $u_{1}^{-}=\min\{0, u_{1}\}$ and $v_{1}^{-}=\min\{0, v_{1}\}$ as the test functions, we derive $0=\langle J^{\prime}(u_{1}, v_{1}), (u_{1}^{-}, v_{1}^{-})\rangle=\Vert (u_{1}^{-}, v_{1}^{-})\Vert ^{p}$, which implies $u_{1}\geq0$ and $v_{1}\geq0$ on $\mathbb{R}^{N}$. Finally, according to the strong maximum principle, we have $u_{1}>0$ and $v_{1}>0$ on $\mathbb{R}^{N}$. Therefore, we conclude from the symmetric criticality principle that $(u_{1}, v_{1})$ is a positive G-symmetric solution of $(\mathscr{P}_{h}^{K_{0}})$. □