The existence of positive solutions for the Neumann problem of p-Laplacian elliptic systems with Sobolev critical exponent

Abstract

The paper aims to consider a class of p-Laplacian elliptic systems with a double Sobolev critical exponent. We obtain the existence result of the above problem under the Neumann boundary for some suitable range of the parameters in the systems.

1 Introduction and main results

We consider the following elliptic system with Neumann boundary:

$$\begin{aligned} \textstyle\begin{cases} -\Delta _{p} u+\lambda _{1} u^{p-1}= \vert u \vert ^{p^{*}-2}u+ \frac{\alpha }{p^{*}} \vert u \vert ^{\alpha -2} \vert v \vert ^{\beta } u, \quad x\in \Omega, \\ -\Delta _{p} v+\lambda _{2} v^{p-1}= \vert v \vert ^{p^{*}-2}v+ \frac{\beta }{p^{*}} \vert u \vert ^{\alpha } \vert v \vert ^{\beta -2} v, \quad x\in \Omega, \\ u>0,\qquad v>0, \qquad x\in \Omega, \\ \frac{\partial u}{\partial n}=\frac{\partial v}{\partial n}=0,\quad x\in \partial \Omega, \end{cases}\displaystyle \end{aligned}$$

(1.1)

where $\alpha, \beta >1$ satisfy $\alpha +\beta =p^{*}=\frac{Np}{N-p}$. Meanwhile, Ω is a bounded domain in $\mathbb{R}^{N}\ (N\geq 3)$ which has a smooth $C^{2}$ boundary.

When $p = 2$, the Dirichlet boundary problems for the system (1.1) are generally studied. Due to the “loss of compactness,” a direct application of the classical method is invalid for such problems. In [1], Brezis and Nirenberg first studied the Dirichlet problem with a critical Sobolev exponent for the following nonlinear elliptic equation:

$$\begin{aligned} \textstyle\begin{cases} { - \Delta u = {{ \vert u \vert }^{{2^{*}} - 2}}u + f(x,u),}\quad{x \in \Omega, } \\ {u > 0,}\quad{x \in \Omega, } \\ {u = 0,}\quad {x \in \partial \Omega. } \end{cases}\displaystyle \end{aligned}$$

(1.2)

In order to prove the existence of a sequence satisfying the $(PS)_{(C)}$ condition, the authors used a function which can achieve the best Sobolev constant as the test function to estimate the energy. This overcame the difficulties which were caused by the embedding with the critical exponent that made the minimization sequences not compact. In recent years, the problem of elliptic equations (systems) with critical exponents has attracted the attention of many researchers, and related theory has also made great progress, e.g., results on the existence and nonexistence of nontrivial and multiple solutions were proved, and also some properties of solutions (see [2–8]). At the same time, in recent years, the problem with critical exponents in fractional elliptic equations, such as fractional Kirchhoff problems and Schrödinger–Kirchhoff-type problems involving the fractional p-Laplacian, have also attracted the attention of researchers (see [9–11]).

The nonlinear elliptic system (1.1) arises from mathematical physics, when studying Bose–Einstein condensation, some reaction–diffusion shadow systems, or static problems of chemotaxis models, and any more. It also has a wide range of applications in mathematical biology, financial mathematics, and so on; more applications can be found in [12–16]. When $p=2,\alpha = \beta = 0$, equation (1.1) has the following form:

$$\begin{aligned} \textstyle\begin{cases} { - \Delta u + \lambda u = {u^{{2^{*}} - 1}}},\quad{x \in \Omega, } \\ {u > 0,}\quad{x \in \Omega, } \\ {\frac{{\partial u}}{{\partial n}} = 0,}\quad{x \in \partial \Omega. } \end{cases}\displaystyle \end{aligned}$$

(1.3)

This problem has attracted the attention of many researchers, and many interesting and important results have been proved [17–19]. In [18], by using the minimax theorem and mountain pass lemma, Wang proved that (1.3) has a nontrivial solution when the minimax is lower than the threshold value $\frac{1}{2N}S^{\frac{N}{2}}$ and $\lambda >0$ is appropriately large. In [17], the existence of the least energy solutions $u_{\lambda }$ is proved, under some conditions, i.e., $\exists \lambda _{0} >0$ such that, when $\lambda > \lambda _{0}$, $Q(u_{\lambda })=S_{\lambda }$ and

$$Q({u_{\lambda }}) = {S_{\lambda }}: = \mathop{\inf } _{u \in {H^{1}}(\Omega )\backslash \{ 0\} } Q(u) = \mathop{\inf } _{u \in {H^{1}}(\Omega )\backslash \{ 0\} } \frac{{\int _{\Omega }{({{ \vert {\nabla u} \vert }^{2}} + \lambda {u^{2}})\,dx} }}{{{{(\int _{\Omega }{{u^{{2^{*}}}}\,dx} )}^{\frac{2}{{{2^{*}}}}}}}}. $$

When $p=2,\alpha, \beta >0$, Yang (see [20]) used the method of the blow-up to solve the single critical growth problem as follows:

$$\begin{aligned} \textstyle\begin{cases} { - \Delta u + \lambda u = \frac{\alpha }{{{2^{{*}}}}}{{ \vert u \vert }^{\alpha - 2}}u{{ \vert v \vert }^{\beta }}},\quad{x \in \Omega }, \\ { - \Delta v + \lambda v = \frac{\beta }{{{2^{{*}}}}}{{ \vert u \vert }^{\alpha }}{{ \vert v \vert }^{\beta - 2}}v},\quad{x \in \Omega }, \\ {u,v > 0,}\quad{x \in \Omega, } \\ {\frac{{\partial u}}{{\partial n}} = \frac{{\partial v}}{{\partial n}} = 0,}\quad{x \in \partial \Omega, } \end{cases}\displaystyle \end{aligned}$$

(1.4)

and the author also discussed the asymptotic behavior of the least energy solution when $\mu \rightarrow \infty$ and $\lambda \rightarrow \infty$. The concentration–compactness principles are very different when comparing Dirichlet and Neumann boundary problems. In [21], Chabrowski and Yang used the concentration–compactness principle under the Neumann boundary to study the existence of the least energy solution of the above equations with a potential form, and also discussed the concentration phenomenon.

With the in-depth study of problem (1.3), researchers have extended this kind of problem to more general p-Laplacian equations or systems. Because of the wide and practical application background, researchers have used the equations in pharmacology, biology, non-Newtonian fluids, etc., and got crucial results [22–25]. In [26], Wang discussed the following equation with Neumann boundary and critical exponent and proved that it has positive solutions:

$$\begin{aligned} \textstyle\begin{cases} { - {\Delta _{p}}u = {u^{{p^{*}} - 1}} + f(x,u), u > 0,}\quad{x \in \Omega, } \\ {\frac{{\partial u}}{{\partial n}} = 0,}\quad{x \in \partial \Omega. } \end{cases}\displaystyle \end{aligned}$$

(1.5)

For the p-Laplacian equation with Neumann boundary and subcritical growth exponents, we can use a standard variational principle to deduce the existence results for the solutions. However, for the problem (1.5), the corresponding energy functional loses compactness, so the standard variational principles are invalid. The Dirichlet boundary problems corresponding to such equations have also been widely studied (see [1, 27, 28]). However, the method of dealing with the Dirichlet boundary problem is no longer applicable to Neumann boundary problem owning to the best Sobolev constant of the embedding $W_{0} ^{1,p}(\Omega ) \hookrightarrow L^{P^{*}}(\Omega )$, which is only depends on $N,p$, and not on Ω. Under the following conditions, Wang proved that (1.5) has a positive solution by virtue of the local convexity at a point of ∂Ω:

$$\begin{aligned} &\mathop{\lim } _{u \to 0} \frac{{f(x,u)}}{{{u^{p - 1}}}} = a(x) \le 0,\quad a(x) \ne 0, \\ &f(x,u) \ge - A{u^{p - 1}} - B{u^{t - 1}},\quad A > 0, B > 0, t \in \biggl(p - 1,\frac{{n(p - 1)}}{{n - p}} \biggr). \end{aligned}$$

However, the problem with a doubly critical growth involved in this paper is rarely studied. When $p=2$, in [29], Peng, Peng, and Wang considered the doubly critical Dirichlet boundary problem:

$$\begin{aligned} \textstyle\begin{cases} { - \Delta u = {{ \vert u \vert }^{{2^{*}} - 2}} + \frac{\alpha }{{{2^{{*}}}}}{{ \vert v \vert }^{\beta }}}{{ \vert u \vert }^{\alpha - 2}}u,\quad{x \in \Omega, } \\ { - \Delta v = {{ \vert v \vert }^{{2^{*}} - 2}} + \frac{\beta }{{{2^{{*}}}}}{{ \vert u \vert }^{\alpha }}{{ \vert v \vert }^{\beta - 2}}v,}\quad{x \in \Omega, } \\ {u,v > 0,}\quad{x \in \Omega, } \\ {u = v = 0,}\quad{x \in \partial \Omega. } \end{cases}\displaystyle \end{aligned}$$

(1.6)

In [30], the authors considered the single equation case of the above problem, for the embedding $D^{1,2}(R^{N}) \hookrightarrow L^{2^{*}}(R^{N})$, and they deduced that the following radial function $U(x)$ achieves the best Sobolev constant:

$$\begin{aligned} U(x) = \frac{(N(N - 2))^{\frac{{N - 2}}{4}}}{(1 + { \vert x \vert ^{2}})^{ \frac{{N - 2}}{2}}}. \end{aligned}$$

With similar methods, the authors of [29] proved that the following form of the best Sobolev constant can be achieved:

$$\begin{aligned} {S_{\alpha,\beta }}: = \mathop{\inf } _{(u,v) \in D_{0}^{1,2} \Omega )\backslash \{ (0,0)\} } \frac{{\int _{{\mathbb{R}^{N}}} {({{ \vert {\nabla u} \vert }^{2}} + {{ \vert {\nabla v} \vert }^{2}})\,dx} }}{{{{(\int _{{\mathbb{R}^{N}}} {({{ \vert u \vert }^{{2^{*}}}} + {{ \vert v \vert }^{{2^{*}}}} + {{ \vert u \vert }^{\alpha }}{{ \vert v \vert }^{\beta }})\,dx} )}^{\frac{2}{{{2^{*}}}}}}}} = f({ \tau _{\min }})S \end{aligned}$$

and

$$\begin{aligned} f({\tau _{\min }}): = \mathop{\min } _{\tau \ge 0} f(\tau ) = \mathop{\min } _{\tau \ge 0} \frac{{1 + {\tau ^{2}}}}{{{{(1 + {\tau ^{\beta }} + {\tau ^{{2^{*}}}})}^{\frac{2}{{{2^{*}}}}}}}} < 1. \end{aligned}$$

The authors derived the uniqueness result for the least energy solutions when $\alpha, \beta, N$ satisfied some conditions, and the form of the least energy solution is $(sU_{x_{0},\varepsilon }, bU_{x_{0},\varepsilon })$. We are interested in whether we can use this form of extremal function to research the Neumann boundary problems for the p-Laplacian system whose positive solutions exist in more general case. Therefore, we select the extremal function of the above-mentioned form and use a similar method to that in [1], choose an extremal function as the test function for the energy estimation as in [26], and also need to make the energy corresponding to the Palais–Smale sequence lower than the threshold. Subsequently, we overcome the difficulties caused by the appearance of doubly critical terms $|u|^{p^{*}}, |v|^{p^{*}},|u|^{\alpha }|v|^{\beta }$ in (1.1), which leads to the emergence of noncompactness, and, naturally, the existence of a positive solution for the equations (1.1) can be obtained.

Now, the main theorem of this paper can be presented as follows:

Theorem 1.1

If the following conditions for the parameters $\alpha, \beta, N$ are satisfied:

$$\begin{aligned} \alpha,\beta > 1,\quad \alpha + \beta = {p^{*}} = \frac{{Np}}{{N - p}},\quad N \ge {p^{2}}, \end{aligned}$$

then there is at least one positive solution for (1.1), when $\lambda _{1}, \lambda _{2}$ are sufficiently large.

2 Preliminary results

In this section, it is necessary to present some definitions and preliminary lemmas, which are going to be used to prove our basic estimates and main results.

First, denote $X=W^{1,p}(\Omega ) \times W^{1,p}(\Omega )$, where its norm is as follows:

$$\begin{aligned} \bigl\Vert (u,v) \bigr\Vert :=\biggl( \int _{\Omega }\bigl( \vert \nabla u \vert ^{p}+u^{p} \bigr)\,dx+ \int _{\Omega }\bigl( \vert \nabla v \vert ^{p}+v^{p} \bigr)\,dx\biggr)^{\frac{1}{p}}. \end{aligned}$$

Here $W^{1,p}(\Omega )$ is a Sobolev space, and it has the following norm:

$$\begin{aligned} \Vert u \Vert =( \int _{\Omega }\bigl( \vert \nabla u \vert ^{p}+u^{p} \,dx\bigr)^{\frac{1}{p}}. \end{aligned}$$

We define its corresponding energy functional $J: X\rightarrow \mathbb{R}$ as follows:

$$\begin{aligned} \begin{aligned} J(u,v)={}&\frac{1}{p} \int _{\Omega } \bigl( \vert \nabla u \vert ^{p}+ \lambda _{1} u^{p}+ \vert \nabla v \vert ^{p}+\lambda _{2}v^{p}\bigr)\,dx \\ &{}-\frac{1}{p^{*}} \int _{\Omega } \bigl(\bigl(u^{+}\bigr)^{p^{*}}+ \bigl(v^{+}\bigr)^{p^{*}}+\bigl(u^{+} \bigr)^{ \alpha }\bigl(v^{+}\bigr)^{\beta }\bigr)\,dx. \end{aligned} \end{aligned}$$

Now, we define the weak solution:

Definition 2.1

If $\forall (\varphi, \psi )\in X$, when $u>0$, $v>0$, and $(u,v)\in X$, we have

$$\begin{aligned} & \int _{\Omega }\bigl( \vert \nabla u \vert ^{p-2} \nabla u \nabla \varphi +\lambda _{1}u^{p-1}\varphi + \vert \nabla v \vert ^{p-2}\nabla v \nabla \psi +\lambda _{2}v^{p-1} \psi \bigr)\,dx- \int _{\Omega } \vert u \vert ^{p^{*}-2}u \varphi \,dx \\ &\quad{}- \int _{\Omega } \vert v \vert ^{p^{*}-2}v\psi \,dx- \frac{\alpha }{p^{*}} \int _{ \Omega } \vert u \vert ^{\alpha -1} \vert v \vert ^{\beta }\varphi \,dx- \frac{\beta }{p^{*}} \int _{\Omega } \vert u \vert ^{\alpha } \vert v \vert ^{\beta -1}\psi \,dx =0, \end{aligned}$$

then $(u,v)$ is called a weak solution of problem (1.1).

In order to prove that the least energy solution of (1.1) exists, and derive the solution’s form, we consider the following equation:

$$\begin{aligned} \textstyle\begin{cases} { - {\Delta _{p}}u = {{ \vert u \vert }^{{p^{*}} - 2}}u,}&{x \in \Omega, } \\ {u \in W_{0}^{1,p}(\Omega ).} \end{cases}\displaystyle \end{aligned}$$

(2.1)

From [30], we know that, when $\Omega =\mathbb{R}^{N}$, the function

$$\begin{aligned} U(x) = {\bigl(N(N - p)\bigr)^{\frac{{N - p}}{p^{2}}}} {\bigl(1 + { \vert x \vert ^{ \frac{p}{p-1}}}\bigr)^{ - \frac{{N - p}}{p}}} \end{aligned}$$

(2.2)

is a radial function which solves (2.1). Meanwhile, the best Sobolev constant S can be achieved by $U(x)$ for the embedding $W^{1, p}(\mathbb{R}^{N})\hookrightarrow L^{p^{*}}(\mathbb{R}^{N})$, where

$$\begin{aligned} S: = \mathop{\inf } _{u \in {W^{1,p}}(\mathbb{R}^{N}) \backslash \{ 0\} } \frac{{{{ \Vert u \Vert }^{p}}}}{{ \vert u \vert _{{p^{*}}}^{p}}} = \frac{{{{ \Vert U \Vert }^{p}}}}{{ \vert U \vert _{{p^{*}}}^{p}}}. \end{aligned}$$

(2.3)

For $x_{0}\in \mathbb{R}^{N}, \varepsilon >0$, we denote

$$\begin{aligned} U_{x_{0}, \varepsilon }(x):=\varepsilon ^{\frac{p-N}{p(p-1)}}U \biggl(\frac{x-x_{0}}{\varepsilon }\biggr). \end{aligned}$$

(2.4)

From [30], we know that the $(N+2)$-dimensional manifold of the following form consists of almost all functions which can achieve the best Sobolev constant S:

$$\begin{aligned} \overline{\mathcal{M}}:=\bigl\{ cU_{x_{0}, \varepsilon }, c\in \mathbb{R} \setminus \{{0}\}, x_{0}\in \mathbb{R}^{N}, \varepsilon >0\bigr\} . \end{aligned}$$

Set

$$\begin{aligned} {S_{\alpha,\beta }}: = \mathop{\inf } _{(u,v) \in X \backslash \{ (0,0)\} } \frac{{\int _{\mathbb{R}^{N}} {({{ \vert {\nabla u} \vert }^{p}} + {{ \vert {\nabla v} \vert }^{p}})\,dx} }}{{{{(\int _{\mathbb{R}^{N}} {({{ \vert u \vert }^{{p^{*}}}} + {{ \vert v \vert }^{{p^{*}}}} + {{ \vert u \vert }^{\alpha }}{{ \vert v \vert }^{\beta }})\,dx} )}^{\frac{p}{{{p^{*}}}}}}}}. \end{aligned}$$

(2.5)

Supposing $(sU_{x_{0}, \varepsilon }, bU_{x_{0}, \varepsilon })$ is a positive solution corresponding to the problem (1.1), we have

$$\begin{aligned} \bigl({p^{*}} + \alpha {\tau ^{\beta }}\bigr){s^{{p^{*}} - 2}} = {p^{*}} = \bigl({p^{*}} { \tau ^{{p^{*}} - 2}} + \beta {\tau ^{\beta - 2}}\bigr){s^{{p^{*}} - 2}},\qquad b = \tau s. \end{aligned}$$

Therefore,

$$\begin{aligned} {p^{*}} + \alpha {\tau ^{\beta }} - \beta {\tau ^{\beta - 2}} - {p^{*}} { \tau ^{{p^{*}} - 2}} = 0 \end{aligned}$$

(2.6)

and

$$\begin{aligned} {s^{{p^{*}} - 2}} = \frac{{{p^{*}}}}{{{p^{*}} + \alpha {\tau ^{\beta }}}}. \end{aligned}$$

(2.7)

Subsequently, we find that if all of the least energy solutions of (1.1) have the form $(sU_{x_{0}, \varepsilon }, bU_{x_{0}, \varepsilon })$, where $s,b$ are constants, then we know

$$\begin{aligned} {S_{\alpha,\beta }} = \frac{{1 + {\tau ^{p}}}}{{{{{{(1 + }}{\tau ^{\beta }} + {\tau ^{{p^{*}}}})}^{\frac{p}{{{p^{*}}}}}}}}S. \end{aligned}$$

Setting $f(\tau ):= \frac{1+\tau ^{p}}{(1+\tau ^{\beta }+\tau ^{p^{*}})^{\frac{p}{p^{*}}}}$, it is easy to see that

$$\begin{aligned} f({\tau _{\min }}): = \mathop{\min } _{{\tau _{\min }} \ge 0} f( \tau ) \le 1. \end{aligned}$$

By a similar method as that in [29], under the assumptions of Theorem 1.1, we derive that the following form is suitable for the least energy solutions of (1.1), which can be found in [29]:

Lemma 2.1

(The special case $p=2$; see [29])

Assume $\Omega =\mathbb{R}^{N}, N\geq p^{2}$, and parameters $\alpha >1, \beta >1$, satisfy $\alpha + \beta =p^{*}$. If $(u_{0}, v_{0})$ is the least energy solution of (1.6), then $\exists ! \tau _{\min }>0$, satisfying $f({\tau _{\min }}): = \mathop{\min } _{\tau \ge 0} f(\tau ) = \mathop{\min } _{\tau \ge 0} \frac{{1 + {\tau ^{2}}}}{{{{(1 + {\tau ^{\beta }} + {\tau ^{{2^{*}}}})}^{\frac{2}{{{2^{*}}}}}}}} < 1$, and

$$\begin{aligned} (u_{0}, v_{0})=(sU_{x_{0}, \varepsilon }, bU_{x_{0}, \varepsilon }), \end{aligned}$$

where $b=s\tau _{\min }, x_{0}\in \mathbb{R}^{N}, \varepsilon >0$, and s satisfies (2.7).

Lemma 2.2

For $\forall x_{0}\in \mathbb{R}^{N}$, denote $\widetilde{D}=B_{1}(x_{0})\cap \{x_{N}>h(x')\}$, and let $B_{1}(x_{0})$ be the unit ball centered at $x_{0}$, with $h(x')$ being a $C^{1}$ function defined in the set $\{x'\in \mathbb{R}^{N-1}: |x'-x_{0}'|<1\}$ where $(x_{0})_{N}=h((x_{0})_{1},\dots,(x_{0})_{N-1})$, and ∇h vanishes at $x_{0}'=((x_{0})_{1},\dots,(x_{0})_{N-1})$ (that is, $\nabla h=0$ at this point). If $u,v\in W^{1,p}(B_{1}(x_{0})), \operatorname{supp} u\subset B_{1}(x_{0}), \operatorname{supp} v\subset B_{1}(x_{0})$, then there is a constant $C(\delta )$, which depends on δ, such that the following conclusions hold:

(1) When $h\equiv 0$,

$$\begin{aligned} \int _{\widetilde{D}}\bigl( \vert \nabla u \vert ^{p}+ \vert \nabla v \vert ^{p}\bigr)\,dx\geq 2^{- \frac{p}{N}}S_{\alpha, \beta } \biggl( \int _{\widetilde{D}}\bigl( \vert u \vert ^{p^{*}}+ \vert v \vert ^{p^{*}}+ \vert u \vert ^{ \alpha } \vert v \vert ^{\beta }\bigr)\,dx\biggr)^{\frac{p}{p^{*}}}. \end{aligned}$$

(2.8)

(2) For $\forall \varepsilon >0, \exists \delta (\varepsilon )$, $\nabla h\leq \delta$, such that

$$\begin{aligned} \int _{\widetilde{D}}\bigl( \vert \nabla u \vert ^{p}+ \vert \nabla v \vert ^{p}\bigr)\,dx\geq \bigl(2^{- \frac{p}{N}}S_{\alpha, \beta }- \varepsilon \bigr) \biggl( \int _{\widetilde{D}}\bigl( \vert u \vert ^{p^{*}}+ \vert v \vert ^{p^{*}}+ \vert u \vert ^{ \alpha } \vert v \vert ^{\beta }\bigr)\,dx\biggr)^{\frac{p}{p^{*}}}. \end{aligned}$$

(2.9)

Proof

(1) We estimate

$$\begin{aligned} \int _{\widetilde{D}}\bigl( \vert \nabla u \vert ^{p}+ \vert \nabla v \vert ^{p}\bigr)\,dx &= \frac{1}{2} \int _{B_{1}(x_{0})}\bigl( \vert \nabla u \vert ^{p}+ \vert \nabla u \vert ^{p}\bigr)\,dx \\ &\geq \frac{1}{2}S_{\alpha, \beta }\biggl( \int _{B_{1}(x_{0})}\bigl( \vert u \vert ^{p^{*}}+ \vert v \vert ^{p^{*}}+ \vert u \vert ^{ \alpha } \vert v \vert ^{\beta }\bigr)\,dx\biggr)^{\frac{p}{p^{*}}} \\ &=2^{-\frac{p}{N}}S_{\alpha, \beta }\biggl( \int _{\widetilde{D}}\bigl( \vert u \vert ^{p^{*}}+ \vert v \vert ^{p^{*}}+ \vert u \vert ^{ \alpha } \vert v \vert ^{\beta }\bigr)\,dx\biggr)^{\frac{p}{p^{*}}}. \end{aligned}$$

(2) By a translation transformation, letting $y'=x', y_{n}=x_{n}-h(x')>0$, we straighten the boundary of D̃, and complete the proof. □

First of all, we have the following result which is necessary to verify that the equations (1.1) satisfy the mountain pass lemma:

Lemma 2.3

When the assumptions of Theorem 1.1are satisfied, the following conclusions hold:

(1) If $r>0, \delta >0$, $\|(u, v)\|=r$, then $J(u,v)\geq \delta >0$;

(2) $\exists (u_{0}, v_{0})\in X$, such that $\|(u_{0}, v_{0})\|>r, J(u_{0},v_{0})<0$ hold.

Proof

(1) Because $W^{1,p}(\Omega )\hookrightarrow L^{p^{*}}(\Omega )$ is continuous, by Hölder inequality, we have:

$$\begin{aligned} \begin{aligned}& \biggl( \int _{\Omega } u^{p^{*}}\,dx \biggr)^{\frac{1}{p^{*}}} \leq C_{1} \Vert u \Vert _{W^{1,p}}, \\ &\biggl( \int _{\Omega } v^{p^{*}}\,dx \biggr)^{\frac{1}{p^{*}}}\leq C_{2} \Vert v \Vert _{W^{1,p}}, \\ &\int _{\Omega } \vert u \vert ^{\alpha } \vert v \vert ^{\beta }\,dx \leq \biggl( \int _{\Omega } u^{p^{*}}\,dx \biggr)^{\frac{\alpha }{p^{*}}} \biggl( \int _{\Omega } v^{p^{*}}\,dx \biggr)^{ \frac{\beta }{p^{*}}} \\ &\phantom{\int _{\Omega } \vert u \vert ^{\alpha } \vert v \vert ^{\beta }\,dx }\leq C_{3} \biggl( \int _{\Omega } \vert \nabla u \vert ^{p}+ \vert u \vert ^{p}\,dx \biggr)^{ \frac{\alpha }{2}} \biggl( \int _{\Omega } \vert \nabla v \vert ^{p}+ \vert v \vert ^{p}\,dx \biggr)^{ \frac{\beta }{2}} \\ &\phantom{\int _{\Omega } \vert u \vert ^{\alpha } \vert v \vert ^{\beta }\,dx }=C_{3} \Vert u \Vert _{W^{1,p}}^{\alpha } \Vert v \Vert _{W^{1,p}}^{\beta }. \end{aligned} \end{aligned}$$

(2.10)

Hence,

$$\begin{aligned} \begin{aligned} J(u,v)={}&\frac{1}{p} \int _{\Omega } \bigl( \vert \nabla u \vert ^{p}+ \lambda _{1} u^{p}+ \vert \nabla v \vert ^{p}+\lambda _{2}v^{p}\bigr)\,dx \\ &{} -\frac{1}{p^{*}} \int _{\Omega } \bigl(\bigl(u^{+}\bigr)^{p^{*}}+ \bigl(v^{+}\bigr)^{p^{*}}+\bigl(u^{+} \bigr)^{ \alpha }\bigl(v^{+}\bigr)^{\beta }\bigr)\,dx \\ \geq{}& \frac{1}{p}C_{4} \bigl\Vert (u,v) \bigr\Vert ^{p}-\frac{1}{p^{*}}C_{1} \Vert u \Vert _{W^{1,p}( \Omega )}^{p^{*}} -\frac{1}{p^{*}}C_{2} \Vert v \Vert _{W^{1,p}(\Omega )}^{p^{*}} \\ &{} -\frac{1}{p^{*}}C_{3} \Vert u \Vert _{W^{1,p}(\Omega )}^{\alpha } \Vert v \Vert _{W^{1,p}( \Omega )}^{\beta } \\ \geq{}& \frac{1}{p}C_{4} \bigl\Vert (u,v) \bigr\Vert ^{p}-\frac{1}{p^{*}}C_{5} \bigl\Vert (u, v) \bigr\Vert ^{p^{*}} \end{aligned} \end{aligned}$$

As a consequence of $\frac{1}{p}-\frac{1}{p^{*}}=\frac{1}{N}>0$, choosing $\|(u, v)\|=r$ small enough, we obtain the desired result $J(u, v)\geq \delta >0$.

(2) Now we estimate

$$\begin{aligned} \begin{aligned} J(tu,tv)={}&\frac{t^{p}}{p} \int _{\Omega } \bigl( \vert \nabla u \vert ^{p}+ \lambda _{1} u^{p}+ \vert \nabla v \vert ^{p}+\lambda _{2}v^{p}\bigr)\,dx \\ &{}-\frac{t^{p^{*}}}{p^{*}} \int _{\Omega } \bigl(\bigl(u^{+}\bigr)^{p^{*}}+ \bigl(v^{+}\bigr)^{p^{*}}+\bigl(u^{+} \bigr)^{ \alpha }\bigl(v^{+}\bigr)^{\beta }\bigr)\,dx \\ \geq{}& \frac{t^{p}}{p}C_{6} \bigl\Vert (u,v) \bigr\Vert ^{p}-\frac{t^{p^{*}}}{p^{*}}C_{7} \bigl\Vert (u, v) \bigr\Vert ^{p^{*}}. \end{aligned} \end{aligned}$$

Hence, letting t tend to infinity,

$$\begin{aligned} \mathop{\lim } _{t \to \infty } J(tu,tv) = - \infty, \end{aligned}$$

so we can choose $t_{0}\in \mathbb{R}$, such that $\|(t_{0}u, t_{0}v)\|>r, J(t_{0}u, t_{0}v)<0$, and then, setting $(u_{0}, v_{0})=(t_{0}u, t_{0}v)$, the proof is completed. □

Next, we introduce the following lemmas (see [31]), which are useful in verifying that the energy functional is lower than the threshold, and whose functional corresponds to the Palais–Smale sequence.

Lemma 2.4

If {$u_{n}$} is a bounded sequence in $L^{p}(\Omega )$ such that $u_{n} \rightarrow u$ a.e., then

$$\begin{aligned} \mathop{\lim } _{n \to \infty } \biggl({ \int _{\Omega }{ \vert {{u_{n}}} \vert } ^{p}}\,dx - { \int _{\Omega }{ \vert {{u_{n}} - u} \vert } ^{p}}\,dx \biggr) = { \int _{\Omega }{ \vert u \vert } ^{p}}\,dx. \end{aligned}$$

(2.11)

Lemma 2.5

If

$$\begin{aligned} \begin{aligned} &u_{n}\rightharpoonup u \quad\textit{in } W^{1,p}( \Omega ), \\ &v_{n}\rightharpoonup v \quad\textit{in } W^{1,p}(\Omega ) \end{aligned} \end{aligned}$$

then

$$\begin{aligned} \mathop{\lim } _{n \to \infty } \biggl({ \int _{\Omega }{ \vert {{u_{n}}} \vert } ^{\alpha }} { \vert {{v_{n}}} \vert ^{\beta }}\,dx - { \int _{\Omega }{ \vert {{u_{n}} - u} \vert } ^{\alpha }} { \vert {{v_{n}} - v} \vert ^{\beta }}\,dx \biggr) = { \int _{\Omega }{ \vert u \vert } ^{\alpha }} { \vert v \vert ^{\beta }}\,dx. \end{aligned}$$

(2.12)

Lemma 2.6

(see [32])

Assume {${u_{n}}$} is the $(PS)$ sequence corresponding to $J_{\lambda, p}$, and $u_{n}\rightharpoonup u$, then it has finitely many points in Ω, we denote them by $x_{1},x_{2}, \dots, x_{k} \in \Omega$, which make $u_{n}\rightarrow u$ hold in $W_{\mathrm{loc}}^{1,p}(\Omega \setminus \{x_{1},x_{2}, \dots, x_{k} \})$.

Denote

$$\begin{aligned} {S_{{\lambda _{1}}, \lambda _{2}}}: = \mathop{\inf } _{u,v \in {W^{1,p}}(\Omega )\backslash \{ 0\} } \frac{{\int _{\Omega }{({{ \vert {\nabla u} \vert }^{p}} + {\lambda _{1}}{u^{p}} + {{ \vert {\nabla v} \vert }^{p}} + {\lambda _{2}}{v^{p}})\,dx} }}{{{{[\int _{\Omega }{({u^{{p^{*}}}} + {v^{{p^{*}}}} + {u^{\alpha }}{v^{\beta }})\,dx} ]}^{\frac{p}{{{p^{*}}}}}}}}. \end{aligned}$$

Lemma 2.7

Suppose {${(u_{n}, v_{n})}$} is a sequence in X, satisfying

$$\begin{aligned} J(u_{n}, v_{n})\rightarrow c,\qquad J(u_{n}, v_{n})\rightarrow 0, \end{aligned}$$

(2.13)

and

$$\begin{aligned} c < \min \biggl\{ \frac{1}{N}S_{{\lambda _{1}},{\lambda _{2}}}^{ \frac{N}{p}}, \frac{1}{{2N}}S_{\alpha,\beta }^{\frac{N}{p}} \biggr\} . \end{aligned}$$

(2.14)

Then the system (1.1) has a solution $(u, v)\in X$ and $J(u, v)\leq c$.

Proof

By Lemma 2.3, $\exists {(u_{n}, v_{n})}$ which satisfies (2.13). Let

$$\begin{aligned} &c = \mathop{\inf } _{\gamma \in \Gamma } \mathop{\sup } _{t \in [0,1]} J\bigl(\gamma (t)\bigr), \\ &\Gamma = \bigl\{ \gamma \in C\bigl([0,1],X\bigr):\gamma (0) = (0,0),J\bigl( \gamma (1)\bigr) < 0 \bigr\} . \end{aligned}$$

As a consequence, we can see that

$$\begin{aligned} \begin{aligned}& c + o(1) = J({u_{n}},{v_{n}}) \\ &\phantom{c + o(1) }=\frac{1}{p} \int _{\Omega }{{{ \vert {\nabla {u_{n}}} \vert }^{p}} + {\lambda _{1}}u_{n}^{p} + {{ \vert {\nabla {v_{n}}} \vert }^{p}} + { \lambda _{2}}v_{n}^{p}} \,dx \\ &\phantom{c + o(1) =}{} -\frac{1}{{{p^{*}}}} \int _{\Omega }{{{\bigl(u_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(v_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(u_{n}^{+} \bigr)}^{\alpha }} {{\bigl(v_{n}^{+} \bigr)}^{\beta }}} \,dx, \end{aligned} \end{aligned}$$

(2.15)

$$\begin{aligned} \begin{aligned} &o(1) \bigl\Vert \bigl(\|\varphi, \psi ) \bigr\Vert \bigr) \\ &\quad= \bigl\langle J'({u_{n}},{v_{n}}), ( \varphi, \psi ) \bigr\rangle \\ &\quad= \int _{\Omega }{{{ \vert {\nabla {u_{n}}} \vert }^{p - 2}}\nabla {u_{n}} \nabla \varphi + {\lambda _{1}}u_{n}^{p - 1}\varphi + {{ \vert { \nabla {v_{n}}} \vert }^{p - 2}}\nabla {v_{n}}\nabla \psi + { \lambda _{2}}v_{n}^{p - 1}\psi } \,dx \\ &\qquad{} - \int _{\Omega }{{{\bigl(u_{n}^{+} \bigr)}^{{p^{*}} - 1}}\varphi + {{\bigl(v_{n}^{+} \bigr)}^{{p^{*}} - 1}}\psi } \,dx - \frac{\alpha }{{{p^{*}}}} \int _{\Omega }{{{\bigl(u_{n}^{+} \bigr)}^{\alpha - 1}} {{\bigl(v_{n}^{+} \bigr)}^{\beta }} \varphi } \,dx \\ &\qquad{} - \frac{\beta }{{{p^{*}}}} \int _{\Omega }{{{\bigl(u_{n}^{+} \bigr)}^{\alpha }} {{\bigl(v_{n}^{+} \bigr)}^{\beta - 1}}\psi } \,dx, \quad\forall ( \varphi, \psi ) \in X. \end{aligned} \end{aligned}$$

(2.16)

Setting $(\varphi, \psi )=(u_{n}, v_{n})$ and substituting it into equation (2.16), we then have

$$\begin{aligned} \begin{aligned} o(1) \bigl\Vert \bigl(\|\varphi, \psi ) \bigr\Vert \bigr) ={}& \bigl\langle J'({u_{n}},{v_{n}}), (\varphi, \psi ) \bigr\rangle \\ ={}& \int _{\Omega }{{{ \vert {\nabla {u_{n}}} \vert }^{p}} + {\lambda _{1}}u_{n}^{p} + {{ \vert {\nabla {v_{n}}} \vert }^{p}} + {\lambda _{2}}v_{n}^{p}} \,dx \\ &{} - \int _{\Omega }{{{\bigl(u_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(v_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(u_{n}^{+} \bigr)}^{\alpha }} {{\bigl(v_{n}^{+} \bigr)}^{\beta }}} \,dx. \end{aligned} \end{aligned}$$

(2.17)

Combining (2.17) with (2.15), we obtain

$$\begin{aligned} \begin{aligned} c + o(1) & \geq J({u_{n}},{v_{n}}) - \frac{1}{{{p^{*}}}} \bigl\langle J'({u_{n}},{v_{n}}),({u_{n}},{v_{n}}) \bigr\rangle \\ &=\biggl(\frac{1}{p}-\frac{1}{p^{*}}\biggr) \int _{\Omega }{{{ \vert {\nabla {u_{n}}} \vert }^{p}} + {\lambda _{1}}u_{n}^{p} + {{ \vert {\nabla {v_{n}}} \vert }^{p}} + {\lambda _{2}}v_{n}^{p}} \,dx \\ &\geq \frac{1}{N}C_{4} \bigl\Vert (u_{n}, v_{n}) \bigr\Vert . \end{aligned} \end{aligned}$$

This implies that $(u_{n}, v_{n})$ is bounded, hence, ∃C such that $\|(u_{n}, v_{n})\|\leq C$, $\|u_{n}\|_{W^{1,p}(\Omega )}\leq C$, and $\|v_{n}\|_{W^{1,p}(\Omega )}\leq C$. Moreover, there exist $u\in W^{1,p}(\Omega ), v\in W^{1,p}(\Omega )$ such that $(u_{n}, v_{n}) \rightharpoonup (u,v)$ in X. By Lemma 2.6, we have the following results:

$$\begin{aligned} &u_{n}\rightharpoonup u,\qquad v_{n} \rightharpoonup v \quad\text{in } W^{1,p}(\Omega ), \\ &u_{n}\rightarrow u,\qquad v_{n}\rightarrow v \quad\text{in } W_{ \mathrm{loc}}^{1,p}\bigl(\Omega \setminus \{x_{1},x_{2}, \dots, x_{k}\}\bigr), \\ &u_{n}\rightarrow u,\qquad v_{n}\rightarrow v \quad\text{in } L^{p}( \Omega ), \\ &u_{n}\rightharpoonup u,\qquad v_{n}\rightharpoonup v\quad \text{in } L^{p^{*}}(\Omega ), \\ &\nabla u_{n}\rightarrow \nabla u,\qquad \nabla v_{n} \rightarrow \nabla v \quad\text{in } L_{\mathrm{loc}}^{p}\bigl(\Omega \setminus \{x_{1},x_{2}, \dots, x_{k}\}\bigr), \\ &u_{n}\rightharpoonup u,\qquad v_{n}\rightharpoonup v\quad \text{a.e. in }\Omega. \end{aligned}$$

By calculating the limit of both sides of (2.16), we obtain

$$\begin{aligned} \begin{aligned} 0={}&\mathop{\lim } _{n \to \infty } \bigl\langle J'({u_{n}},{v_{n}}),( \varphi,\psi ) \bigr\rangle \\ ={}& \int _{\Omega }{{{ \vert {\nabla u} \vert }^{p - 2}} \nabla u \nabla \varphi + {\lambda _{1}} {u^{p - 1}}\varphi + {{ \vert { \nabla v} \vert }^{p - 2}}\nabla v\nabla \psi + {\lambda _{2}} {v^{p - 1}}\psi } \,dx \\ &{} - \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{{p^{*}} - 1}} \varphi + {{\bigl({v^{+} }\bigr)}^{{p^{*}} - 1}}\psi } \,dx - \frac{\alpha }{{{p^{*}}}} \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{\alpha - 1}} {{\bigl({v^{+} }\bigr)}^{\beta }}\varphi } \,dx \\ & {}- \frac{\beta }{{{p^{*}}}} \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{\alpha }} {{\bigl({v^{+} }\bigr)}^{\beta - 1}}\psi } \,dx \quad\forall (\varphi, \psi )\in X. \end{aligned} \end{aligned}$$

(2.18)

By Definition 2.1, we know that $(u,v)$ is a weak solution of (1.1). We need to verify that $(u,v)$ is a nontrivial solution in the following. Let $\widetilde{\omega _{n}}=u_{n}-u, \widetilde{\sigma _{n}}=v_{n}-v$, then by Lemmas 2.4, 2.5, and 2.6, we see that

$$\begin{aligned} &\int _{\Omega }{{{\bigl(u_{n}^{+} \bigr)}^{{p^{*}}}}} \,dx = \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{{p^{*}}}}} \,dx + \int _{\Omega }{{{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}}} \,dx + o(1), \\ &\int _{\Omega }{{{\bigl(v_{n}^{+} \bigr)}^{{p^{*}}}}} \,dx = \int _{\Omega }{{{\bigl({v^{+} }\bigr)}^{{p^{*}}}}} \,dx + \int _{\Omega }{{{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}}} \,dx + o(1), \\ &\int _{\Omega }{{{\bigl(u_{n}^{+} \bigr)}^{\alpha }} {{\bigl(v_{n}^{+} \bigr)}^{\beta }}} \,dx = \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{\alpha }} {{\bigl({v^{+} }\bigr)}^{\beta }}} \,dx + \int _{\Omega }{{{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{\alpha }} {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{\beta }}} \,dx + o(1), \\ &\int _{\Omega }{{ \vert {\nabla {u_{n}}} \vert }^{p}}\,dx = \int _{\Omega }{{{ \vert \nabla u \vert }^{p}}} \,dx + \int _{\Omega }{{{ \vert \nabla {{ \tilde{\omega }}_{n}} \vert }^{p}}} \,dx + o(1), \\ &\int _{\Omega }{{ \vert {\nabla {v_{n}}} \vert }^{p}}\,dx = \int _{\Omega }{{{ \vert \nabla v \vert }^{p}}} \,dx + \int _{\Omega }{{{ \vert \nabla {{ \tilde{\sigma }}_{n}} \vert }^{p}}} \,dx + o(1), \\ &\int _{\Omega }{{{({{\tilde{\omega }}_{n}})}^{p}}} = o(1),\qquad \int _{\Omega }{{{({{\tilde{\sigma }}_{n}})}^{p}}} = o(1). \end{aligned}$$

By (2.15) and the fact $J(u_{n}, v_{n})\rightarrow c, J'(u_{n}, v_{n})\rightarrow 0$, where ${(u_{n}, v_{n})}\subset X$, we have

$$\begin{aligned} \begin{aligned} &c + o(1)=J(u,v) + \frac{1}{p} \int _{\Omega }{{{ \vert {\nabla {{ \tilde{\omega }}_{n}}} \vert }^{p}}\,dx + {{ \vert {\nabla {{ \tilde{\sigma }}_{n}}} \vert }^{p}}} \,dx \\ &\phantom{c + o(1)=} {}-\frac{1}{{{p^{*}}}} \int _{\Omega }{{{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{\alpha }} {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{\beta }}} \,dx, \end{aligned} \end{aligned}$$

(2.19)

$$\begin{aligned} &\int _{\Omega } \vert \nabla \tilde{\omega }_{n} \vert ^{p}\,dx + \vert \nabla \tilde{\sigma }_{n} \vert ^{p} \,dx - \int _{\Omega }{{{\bigl( \tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{\alpha }} {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{\beta }}} \,dx = 0. \end{aligned}$$

(2.20)

Hence,

$$\begin{aligned} \begin{aligned} \mathop{\lim } _{n \to \infty } \int _{\Omega }{{{ \vert {\nabla {{\tilde{\omega }}_{n}}} \vert }^{p}}\,dx + {{ \vert { \nabla {{ \tilde{\sigma }}_{n}}} \vert }^{p}}} \,dx &= \mathop{\lim } _{n \to \infty } \int _{\Omega }{{{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{\alpha }} {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{\beta }}} \,dx \\ &= l\geq 0. \end{aligned} \end{aligned}$$

For $\forall \varepsilon >0$, where ε is a small suitable positive constant, we denote by $(\phi _{i})_{i=1}^{m}$ a partition of unity on Ω̄, satisfying ∀i, $\operatorname{diam}(\operatorname{supp} \phi _{i})\leq \rho$, where $\operatorname{diam}(D)$ means the diameter of the domain D. Therefore, by Lemma 2.2, provided that ρ is small enough, we have

$$\begin{aligned} \begin{aligned} & \int _{\Omega }{{{ \bigl\vert {\nabla (u{\phi _{i}})} \bigr\vert }^{p}} + {{ \bigl\vert {\nabla (v{\phi _{i}})} \bigr\vert }^{p}}\,dx} \\ &\quad \ge \bigl({2^{ - \frac{p}{N}}} {S_{\alpha,\beta }} - \varepsilon \bigr){\biggl( \int _{\Omega }{{{ \bigl\vert {(u{\phi _{i}})} \bigr\vert }^{{p^{*}}}} + {{ \bigl\vert {(v{\phi _{i}})} \bigr\vert }^{{p^{*}}}} + {{ \bigl\vert {(u{\phi _{i}})} \bigr\vert }^{\alpha }} {{ \bigl\vert {(v{\phi _{i}})} \bigr\vert }^{\beta }}\,dx} \biggr)^{ \frac{p}{{{p^{*}}}}}}. \end{aligned} \end{aligned}$$

Employing Young inequality with ε, for $\forall 1\leq i\leq m, u,v \in W^{1,p}(\Omega )$, we have

$$\begin{aligned} \begin{aligned} & \biggl( \int _{\Omega }{{{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{\alpha }} {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{\beta }}} \,dx \biggr)^{ \frac{p}{{{p^{*}}}}} \\ &\quad\leq \Biggl( \int _{\Omega }{\sum_{i = 1}^{m} {{\phi _{i}}^{ \frac{{{p^{*}}}}{p}}\bigl[{{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl( \tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{\alpha }} {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{\beta }}\bigr]} } \,dx \Biggr)^{ \frac{p}{{{p^{*}}}}} \\ &\quad\leq \sum_{i = 1}^{m} {{{ \biggl( \int _{\Omega }{{{\bigl({\phi _{i}}^{ \frac{1}{p}} \tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl({ \phi _{i}}^{ \frac{1}{p}}\tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl({\phi _{i}}^{ \frac{1}{p}}\tilde{ \omega }_{n}^{+} \bigr)}^{\alpha }} {{\bigl({\phi _{i}}^{ \frac{1}{p}}\tilde{\sigma }_{n}^{+} \bigr)}^{\beta }}} \,dx \biggr)}^{ \frac{p}{{{p^{*}}}}}}} \\ &\quad\leq {\bigl({2^{ - \frac{p}{N}}} {S_{\alpha,\beta }} - \varepsilon \bigr)^{ - 1}} \sum_{i = 1}^{m} { \int _{\Omega }{{{ \bigl\vert {\nabla \bigl({{ \tilde{\omega }}_{n}} {\phi _{i}}^{\frac{1}{p}}\bigr)} \bigr\vert }^{p}} + {{ \bigl\vert {\nabla \bigl({{\tilde{\sigma }}_{n}} {\phi _{i}}^{\frac{1}{p}}\bigr)} \bigr\vert }^{p}}\,dx} } \\ &\quad\leq {\bigl({2^{ - \frac{p}{N}}} {S_{\alpha,\beta }} - \varepsilon \bigr)^{ - 1}}\\ &\qquad{}\times\biggl[(1 + \varepsilon ) \int _{\Omega }{{{ \vert {\nabla {{\tilde{\omega }}_{n}}} \vert }^{p}} + {{ \vert {\nabla {{\tilde{ \sigma }}_{n}}} \vert }^{p}}} \,dx + C(\varepsilon ) \int _{\Omega }{{{({{\tilde{\omega }}_{n}})}^{p}} + {{({{ \tilde{\sigma }}_{n}})}^{p}}\,dx\biggr]}. \end{aligned} \end{aligned}$$

(2.21)

Hence, $l\geq ( \frac{2^{-\frac{p}{N}}S_{\alpha, \beta }-\varepsilon }{1+\varepsilon } )l^{\frac{p}{p^{*}}}$, and we discuss the following two cases:

If $l=0$, then it is easy to see that $(u_{n}, v_{n})\rightarrow (u, v)$ in X, so $(u,v)\not \equiv (0,0)$. If $u\not \equiv 0, v\equiv 0$, then

$$\begin{aligned} c = \frac{1}{p} \int _{\Omega }{{{ \vert {\nabla u} \vert }^{p}} + { \lambda _{1}} {u^{{p}}}} \,dx - \frac{1}{{{p^{*}}}} \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{{p^{*}}}}} \,dx \ge \frac{1}{N}S_{{\lambda _{1}}, 0}^{\frac{N}{p}}. \end{aligned}$$

Similarly, if $u\equiv 0, v\not \equiv 0$, then

$$\begin{aligned} c = \frac{1}{p} \int _{\Omega }{{{ \vert {\nabla v} \vert }^{p}} + { \lambda _{2}} {v^{{p}}}} \,dx - \frac{1}{{{p^{*}}}} \int _{\Omega }{{{\bigl({v^{+} }\bigr)}^{{p^{*}}}}} \,dx \ge \frac{1}{N}S_{0, {\lambda _{2}}}^{\frac{N}{p}}. \end{aligned}$$

This contradicts $c < \min \{ \frac{1}{N}S_{{\lambda _{1}},{\lambda _{2}}}^{ \frac{N}{p}},\frac{1}{{2N}}S_{\alpha,\beta }^{\frac{N}{p}} \}$. Thus, the solutions of system (1.1) are not semitrivial solutions.

If $l\neq 0$, that is, $l\geq \frac{1}{2}S_{\alpha, \beta }^{\frac{N}{p}}$, then we only need to verify $u\not \equiv 0, v\not \equiv 0$.

(i) Assume one of $u,v$ equals zero. It is natural to suppose $u\not \equiv 0, v\equiv 0$.

From $\langle J'({u_{n}},{v_{n}}),({u_{n}},{v_{n}}) \rangle \rightarrow 0$, it is easy to obtain

$$\begin{aligned} \int _{\Omega }{{{ \vert {\nabla u} \vert }^{p}} + { \lambda _{1}} {u^{{p}}}} \,dx - \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{{p^{*}}}}} \,dx = 0. \end{aligned}$$

Due to (2.19) and $l\geq \frac{1}{2}S_{\alpha, \beta }^{\frac{N}{p}}$, we see

$$\begin{aligned} \begin{aligned} c&= \biggl(\frac{1}{p} - \frac{1}{{{p^{*}}}} \biggr)l + \frac{1}{p} \int _{\Omega }{{{ \vert {\nabla u} \vert }^{p}} + { \lambda _{1}} {u^{p}}} \,dx - \frac{1}{{{p^{*}}}} \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{{p^{*}}}}} \,dx \\ &= \biggl(\frac{1}{p} - \frac{1}{{{p^{*}}}} \biggr)l + \biggl( \frac{1}{p} - \frac{1}{{{p^{*}}}} \biggr) \int _{\Omega }{{{ \vert {\nabla u} \vert }^{p}} + { \lambda _{1}} {u^{p}}} \,dx \\ &\geq \biggl(\frac{1}{p} - \frac{1}{{{p^{*}}}} \biggr)l\geq \frac{1}{2N}S_{ \alpha, \beta }^{\frac{N}{p}}. \end{aligned} \end{aligned}$$

Similarly, this is a contradiction to $c < \min \{ \frac{1}{N}S_{{\lambda _{1}},{\lambda _{2}}}^{ \frac{N}{p}},\frac{1}{{2N}}S_{\alpha,\beta }^{\frac{N}{p}} \}$.

(ii) $u\equiv 0, v\equiv 0$.

Now we can find $c = (\frac{1}{p} - \frac{1}{{{p^{*}}}})c\geq \frac{1}{2N}S_{\alpha, \beta }^{\frac{N}{p}}$, which is also a contradiction. In summary, $(u, v)$ is a nontrivial solution of system (1.1).

Combining (2.19) and (2.20), the following result can be obtained:

$$\begin{aligned} J(u,v) = c - \frac{1}{N} \int _{\Omega }{{{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}}} \,dx + o(1), \end{aligned}$$

implying $J(u, v)\leq c$. □

3 Basic estimates

For each $\varepsilon >0$, the extremal function of the best Sobolev constant has the following form:

$$\begin{aligned} u_{\varepsilon }(x)= \frac{\varepsilon ^{\frac{N-p}{p^{2}}}}{(\varepsilon + \vert x \vert ^{\frac{p}{p-1}})^{\frac{N-p}{p}}}. \end{aligned}$$

(3.1)

Suppose the ball centered at $\widetilde{x} \in \Omega$ is such that $B(\widetilde{x}, R)\subset \Omega$ and $\partial B(\widetilde{x}, R)\cap \overline{\Omega } \neq \emptyset$. Choosing $x_{0}\in \partial B(\widetilde{x}, R)\cap \overline{\Omega }$, the principal curvatures of ∂Ω at $x_{0}$ can be presented by $\gamma _{1}, \gamma _{2},\dots, \gamma _{N-1}$. Obviously, $\Omega \subset \{x_{n}>0 \}$, that is, $\Omega \subset \mathbb{R}_{+}^{N}$, where $\mathbb{R}_{+}^{N}=\{x=(x_{1}, x_{2},\dots, x_{N},)\in \mathbb{R}^{N}|x_{N}>0 \}$. Hence, the mean curvature of ∂Ω at $x_{0}$ is $\sum_{i = 1}^{N - 1} {{\gamma _{i}}}$, and we can represent the boundary near the origin by the following:

$$\begin{aligned} {x_{N}} = h\bigl(x'\bigr) = \frac{1}{2}\sum _{i = 1}^{N - 1} {{\gamma _{i}}x_{i}^{2}} + o\bigl({ \bigl\vert {x'} \bigr\vert ^{2}}\bigr), \end{aligned}$$

where $h(x')$ is as defined in Lemma 2.2, $x'=(x_{1}, x_{2},\dots, x_{N-1})\in D_{\delta }(0)$, and $(0)=B_{\delta }(0)\cap \{x_{N}=0\}$.

Set

$$\begin{aligned} \begin{aligned} &K_{1}(\varepsilon )= \int _{\Omega } \vert \nabla u_{ \varepsilon } \vert ^{p}\,dx,\qquad K_{2}(\varepsilon )= \int _{\Omega } \vert u_{ \varepsilon } \vert ^{p^{*}} \,dx, \\ &K_{3}(\varepsilon )= \int _{\Omega } \vert u_{\varepsilon } \vert ^{p} \,dx,\qquad g\bigl(x'\bigr)= \frac{1}{2}\sum _{i = 1}^{N - 1} {{\gamma _{i}}x_{i}^{2}}. \end{aligned} \end{aligned}$$

Next, $K_{1}(\varepsilon ), K_{2}(\varepsilon )$, and $K_{3}(\varepsilon )$ will be estimated.

Lemma 3.1

$$\begin{aligned} K_{1}(\varepsilon )&= \int _{\Omega } \vert \nabla u_{\varepsilon } \vert ^{p}\,dx= \int _{\mathbb{R}_{+}^{N}} \vert \nabla u_{\varepsilon } \vert ^{p}\,dx-I_{ \varepsilon }+o\bigl(\varepsilon ^{\frac{p-1}{p}}\bigr), \end{aligned}$$

(3.2)

$$\begin{aligned} K_{2}(\varepsilon )&= \int _{\Omega } \vert u_{\varepsilon } \vert ^{p^{*}} \,dx= \int _{ \mathbb{R}_{+}^{N}} \vert u_{\varepsilon } \vert ^{p^{*}} \,dx -\mathit{II}_{\varepsilon }+o\bigl( \varepsilon ^{\frac{p-1}{p}}\bigr), \end{aligned}$$

(3.3)

$$\begin{aligned} {K_{3}}(\varepsilon ) &= \textstyle\begin{cases} {O({\varepsilon ^{\frac{{N - p}}{p}}}\ln \varepsilon ) = O({ \varepsilon ^{p - 1}}\ln \varepsilon ),}\quad{N = {p^{2}}}, \\ {O({\varepsilon ^{\frac{{N - p}}{p}}}) = O({\varepsilon ^{ \frac{{{{(p - 1)}^{2}}}}{p}}}),}\quad{N = {p^{2}} - p + 1,} \\ {O({\varepsilon ^{p - 1}}),}\quad{N > {p^{2}}}, \end{cases}\displaystyle \end{aligned}$$

(3.4)

where

$$\begin{aligned} &I_{\varepsilon }= \int _{\mathbb{R}^{N-1}}\,dx' \int _{0}^{g(x')} {{{ \vert {\nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}}, \end{aligned}$$

(3.5)

$$\begin{aligned} &\mathit{II}_{\varepsilon }= \int _{\mathbb{R}^{N-1}}\,dx' \int _{0}^{g(x')} {u_{\varepsilon }^{{p^{*}}} \,d{x_{N}}}, \end{aligned}$$

(3.6)

and

$$\begin{aligned} &\mathop{\lim } _{\varepsilon \to 0} {\varepsilon ^{ - \frac{{p - 1}}{p}}} {{I}_{\varepsilon }}= { \biggl( \frac{{N - p}}{{p - 1}} \biggr)^{p}} \int _{{\mathbb{R}^{N - 1}}} { \frac{{{{ \vert {y'} \vert }^{\frac{p}{{p - 1}}}}g(y')}}{{{{ (1 + {{ \vert {y'} \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dy'}, \end{aligned}$$

(3.7)

$$\begin{aligned} &\mathop{\lim } _{\varepsilon \to 0} {\varepsilon ^{ - \frac{{p - 1}}{p}}} {{\mathit{II}}_{\varepsilon }}= \int _{{\mathbb{R}^{N - 1}}} { \frac{{g(y')}}{{{{ (1 + {{ \vert {y'} \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dy'}. \end{aligned}$$

(3.8)

Proof

By (3.1) and using $g(x')=\frac{1}{2}\sum_{i = 1}^{N - 1} {{\gamma _{i}}x_{i}^{2}}$, we see that $x_{N}=h(x')=g(x')+o(|x'|^{2})$, and

$$\begin{aligned} \vert \nabla u_{\varepsilon } \vert = \frac{(\frac{N-p}{p-1}) \vert x \vert ^{\frac{1}{p-1}}\varepsilon ^{\frac{N-p}{p^{2}}}}{(\varepsilon + \vert x \vert ^{\frac{p}{p-1}})^{\frac{N}{p}}}. \end{aligned}$$

(3.9)

(1) The estimate for $K_{1}(\varepsilon )$.

First, based on spherical coordinate transformation, we have

$$\begin{aligned} & \biggl\vert \int _{\Omega } \vert \nabla u_{\varepsilon } \vert ^{p}\,dx- \int _{ \mathbb{R}_{+}^{N}} \vert \nabla u_{\varepsilon } \vert ^{p}\,dx+ \int _{{D_{\delta }}(0)} {\,dx'} \int _{0}^{h(x')} {{{ \vert {\nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}} \biggr\vert \\ &\quad= \biggl\vert - \int _{\mathbb{R}_{+}^{N}\setminus \Omega } \vert \nabla u_{ \varepsilon } \vert ^{p}\,dx+ \int _{{D_{\delta }}(0)} {\,dx'} \int _{0}^{h(x')} {{{ \vert {\nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}} \biggr\vert \\ &\quad\leq \int _{\mathbb{R}_{+}^{N}\setminus B_{R}^{+}(0)} \vert \nabla u_{ \varepsilon } \vert ^{p}\,dx \\ &\quad= \int _{\mathbb{R}_{+}^{N}\setminus B_{R}^{+}(0)} \frac{(\frac{N-p}{p-1})^{p} \vert x \vert ^{\frac{p}{p-1}}\varepsilon ^{\frac{N-p}{p}}}{(\varepsilon + \vert x \vert ^{\frac{p}{p-1}})^{N}}\,dx \\ &\quad=\frac{1}{2} \int _{R}^{ + \infty } { \frac{{{{(\frac{{N - p}}{{p - 1}})}^{p}}{\varepsilon ^{\frac{{N - p}}{p}}}{r^{\frac{p}{{p - 1}}}}{\omega _{N}}{r^{N - 1}}}}{{(\varepsilon + {r^{\frac{p}{{p - 1}}}})^{N}}}\,dr} \\ &\quad\leq C{\varepsilon ^{\frac{{N - p}}{p}}} \int _{R}^{ + \infty } { \frac{1}{{{r^{\frac{{N - 1}}{{p - 1}}}}}}\,dr} \\ &\quad=O\bigl(\varepsilon ^{\frac{N-p}{p}}\bigr). \end{aligned}$$

(3.10)

Secondly,

$$\begin{aligned} \begin{aligned} & \int _{\mathbb{R}^{N-1}}\,dx'{ \int _{0}^{g(x')} {{{ \vert {\nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}} }-{ \int _{{D_{\delta }}(0)} {\,dx'} \int _{0}^{g(x')} {{{ \vert {\nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}} } \\ &\quad= \int _{\mathbb{R}^{N-1}\setminus D_{\delta }(0)}\,dx'{ \int _{0}^{g(x')} {{{ \vert {\nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}} } \\ &\quad= \int _{\mathbb{R}^{N-1}\setminus D_{\delta }(0)}\,dx' \int _{0}^{g(x')} { \frac{{{{(\frac{{N - p}}{{p - 1}})}^{p}}{\varepsilon ^{\frac{{N - p}}{p}}}{{ \vert x \vert }^{\frac{p}{{p - 1}}}}}}{{{{(\varepsilon + {{ \vert x \vert }^{\frac{p}{{p - 1}}}})}^{N}}}} \,d{x_{N}}}. \end{aligned} \end{aligned}$$

(3.11)

Continuing the above calculation, we get

$$\begin{aligned} \begin{aligned} \text{(3.11)}&\leq \int _{\mathbb{R}^{N-1}\setminus D_{ \delta }(0)}\,dx' \int _{0}^{g(x')} { \frac{{{{(\frac{{N - p}}{{p - 1}})}^{p}}{\varepsilon ^{\frac{{N - p}}{p}}}{{ \vert x' \vert }^{\frac{p}{{p - 1}}}}}}{{{{(\varepsilon + {{ \vert x' \vert }^{\frac{p}{{p - 1}}}})}^{N}}}} \,d{x_{N}}} \\ &= \int _{\mathbb{R}^{N-1}\setminus D_{\delta }(0)} \frac{{{{(\frac{{N - p}}{{p - 1}})}^{p}}{\varepsilon ^{\frac{{N - p}}{p}}}{{ \vert {x'} \vert }^{\frac{p}{{p - 1}}}}g(x')}}{{{{(\varepsilon + {{ \vert {x'} \vert }^{\frac{p}{{p - 1}}}})}^{N}}}}\,dx' \\ &= \int _{\mathbb{R}^{N-1}\setminus D_{\delta }(0)} \frac{{{{(\frac{{N - p}}{{p - 1}})}^{p}}{\varepsilon ^{\frac{{N - p}}{p}}}{{ \vert {x'} \vert }^{\frac{p}{{p - 1}}}}\frac{1}{2}\sum_{i = 1}^{N - 1} {{\gamma _{i}}x_{i}^{2}} }}{{{{(\varepsilon + {{ \vert {x'} \vert }^{\frac{p}{{p - 1}}}})}^{N}}}}\,dx'. \end{aligned} \end{aligned}$$

Performing the spherical coordinate transformation yields

$$\begin{aligned} \begin{aligned} \text{(3.11)}&\leq \frac{1}{2}\sum _{i = 1}^{N - 1} {{\gamma _{i}} \frac{1}{{N - 1}}} \int _{\mathbb{R}^{N-1}\setminus D_{ \delta }(0)} \frac{{{{(\frac{{N - p}}{{p - 1}})}^{p}}{\varepsilon ^{\frac{{N - p}}{p}}}{{ \vert {x'} \vert }^{\frac{p}{{p - 1}} + 2}}}}{{{{(\varepsilon + {{ \vert {x'} \vert }^{\frac{p}{{p - 1}}}})}^{N}}}}\,dx' \\ &=\frac{1}{2}\sum_{i = 1}^{N - 1} {{ \gamma _{i}} \frac{{{{(\frac{{N - p}}{{p - 1}})}^{p}}{\varepsilon ^{\frac{{N - p}}{p}}}{\omega _{N - 1}}}}{{N - 1}}} \int _{\delta }^{ + \infty } { \frac{{{r^{N - 2}}{r^{\frac{p}{{p - 1}} + 2}}}}{{{{(\varepsilon + {{ \vert r \vert }^{\frac{p}{{p - 1}}}})}^{N}}}}} \,dr \\ &=C\varepsilon ^{\frac{N-p}{p}} \int _{\delta }^{ + \infty } { \frac{1}{{{{ \vert r \vert }^{\frac{{N - p}}{{p - 1}}}}}}} \,dr \\ &=O\bigl(\varepsilon ^{\frac{N-p}{p}}\bigr). \end{aligned} \end{aligned}$$

Additionally,

$$\begin{aligned} & \int _{\mathbb{R}_{+}^{N}} \vert \nabla u_{\varepsilon } \vert ^{p}\,dx- \int _{{D_{\delta }}(0)} {\,dx'} \int _{0}^{h(x')} {{{ \vert {\nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}}+O\bigl( \varepsilon ^{\frac{N-p}{p}}\bigr) \\ &\quad= \int _{\mathbb{R}_{+}^{N}} \vert \nabla u_{\varepsilon } \vert ^{p}\,dx- \int _{{D_{\delta }}(0)} {\,dx'} \int _{0}^{g(x')} {{{ \vert {\nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}} \\ \begin{aligned} & \qquad{}- \int _{{D_{\delta }}(0)} {\,dx'} \int _{g(x')}^{h(x')} {{{ \vert { \nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}}+O\bigl( \varepsilon ^{ \frac{N-p}{p}}\bigr) \\ &\quad= \int _{\mathbb{R}_{+}^{N}} \vert \nabla u_{\varepsilon } \vert ^{p}\,dx- \int _{ \mathbb{R}^{N-1}}\,dx'{ \int _{0}^{g(x')} {{{ \vert {\nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}} } \end{aligned} \\ &\qquad{} - \int _{{D_{\delta }}(0)} {\,dx'} \int _{g(x')}^{h(x')} {{{ \vert { \nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}}+ \int _{\mathbb{R}^{N-1}}\,dx'{ \int _{0}^{g(x')} {{{ \vert {\nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}} } \\ &\qquad{} - \int _{{D_{\delta }}(0)} {\,dx'} \int _{0}^{g(x')} {{{ \vert { \nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}}+O\bigl( \varepsilon ^{ \frac{N-p}{p}}\bigr). \end{aligned}$$

(3.12)

By (3.5) and (3.11), and denoting $I_{\varepsilon }^{1}=\int _{{D_{\delta }}(0)} {\,dx'} \int _{g(x')}^{h(x')} {{{ \vert {\nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}}$, we see that

$$\begin{aligned} K_{1}(\varepsilon )= \int _{\Omega } \vert \nabla u_{\varepsilon } \vert ^{p}\,dx= \int _{\mathbb{R}_{+}^{N}} \vert \nabla u_{\varepsilon } \vert ^{p}\,dx-I_{ \varepsilon } -I_{\varepsilon }^{1}+O\bigl( \varepsilon ^{\frac{N-p}{p}}\bigr). \end{aligned}$$

In order to prove Lemma 3.1, we now estimate $I_{\varepsilon }^{1}$ as follows:

$$\begin{aligned} \begin{aligned} \bigl\vert I_{\varepsilon }^{1} \bigr\vert &= \biggl\vert \int _{{D_{\delta }}(0)} {\,dx'} \int _{g(x')}^{h(x')} {{{ \vert {\nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}} \biggr\vert \\ &= \biggl\vert { \biggl(\frac{{N - p}}{{p - 1}} \biggr)^{p}} { \varepsilon ^{ \frac{{N - p}}{p}}} \int _{{D_{\delta }}(0)} {\,dx'} \int _{g(x')}^{h(x')} { \frac{{{{ \vert x \vert }^{\frac{p}{{p - 1}}}}}}{{{{(\varepsilon + {{ \vert x \vert }^{\frac{p}{{p - 1}}}})}^{N}}}} \,d{x_{N}}} \biggr\vert \\ &\leq \biggl\vert { \biggl(\frac{{N - p}}{{p - 1}} \biggr)^{p}} { \varepsilon ^{ \frac{{N - p}}{p}}} \int _{{D_{\delta }}(0)} {\,dx'} \int _{g(x')}^{h(x')} { \frac{{\varepsilon + {{ \vert x \vert }^{\frac{p}{{p - 1}}}}}}{{{{(\varepsilon + {{ \vert x \vert }^{\frac{p}{{p - 1}}}})}^{N}}}} \,d{x_{N}}} \biggr\vert \\ &\leq { \biggl(\frac{{N - p}}{{p - 1}} \biggr)^{p}} {\varepsilon ^{ \frac{{N - p}}{p}}} \int _{{D_{\delta }}(0)} { \frac{{ \vert {h(x') - g(x')} \vert }}{{{{(\varepsilon + {{ \vert x' \vert }^{\frac{p}{{p - 1}}}})}^{N - 1}}}}\,dx'.} \end{aligned} \end{aligned}$$

(3.13)

For $\forall \theta >0$, using $h(x')=g(x')+o(|x'|^{2})$, we know that $\exists C(\theta )>0$ such that

$$\begin{aligned} \bigl\vert h\bigl(x'\bigr)-g\bigl(x'\bigr) \bigr\vert \leq \theta \bigl\vert x' \bigr\vert ^{2}+C( \theta ) \bigl\vert x' \bigr\vert ^{\frac{5}{2}}. \end{aligned}$$

By the above inequalities, we can estimate (3.13) as follows:

$$\begin{aligned} \begin{aligned} \bigl\vert I_{\varepsilon }^{1} \bigr\vert &\leq { \biggl(\frac{{N - p}}{{p - 1}} \biggr)^{p}} {\varepsilon ^{\frac{{N - p}}{p}}} \int _{{D_{\delta }}(0)} { \frac{{\theta {{ \vert {x'} \vert }^{2}} + C(\theta ){{ \vert {x'} \vert }^{\frac{5}{2}}}}}{{{{(\varepsilon + {{ \vert x \vert }^{\frac{p}{{p - 1}}}})}^{N - 1}}}}\,dx'} \\ &\leq C \bigl({\varepsilon ^{\frac{{p - 1}}{p}}}\theta + {\varepsilon ^{ \frac{{p - 1}}{p}}} {\varepsilon ^{\frac{{p - 1}}{{2p}}}}C(\theta ) \bigr) \\ &=C{\varepsilon ^{\frac{{p - 1}}{p}}} \bigl(\theta + {\varepsilon ^{ \frac{{p - 1}}{{2p}}}}C( \theta ) \bigr), \end{aligned} \end{aligned}$$

therefore $I_{\varepsilon }^{1}=o(\varepsilon ^{\frac{p-1}{p}})$.

(2) The estimate of $K_{2}(\varepsilon )$.

By similar calculations as for $K_{1}(\varepsilon )$, we obtain

$$\begin{aligned} \begin{aligned} & \biggl\vert \int _{\Omega } u_{\varepsilon }^{p^{*}}\,dx- \int _{ \mathbb{R}_{+}^{N}}u_{\varepsilon }^{p^{*}}\,dx+ \int _{{D_{\delta }}(0)} {\,dx'} \int _{0}^{h(x')} {{u_{\varepsilon }}^{{p^{{*}}}} \,d{x_{N}}} \biggr\vert \\ &\quad= \biggl\vert - \int _{\mathbb{R}_{+}^{N}\setminus \Omega } \vert u_{\varepsilon } \vert ^{p^{*}} \,dx+ \int _{{D_{\delta }}(0)} {\,dx'} \int _{0}^{h(x')} {{{ \vert {\nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}} \biggr\vert \\ &\quad\leq \int _{\mathbb{R}_{+}^{N}\setminus B_{R}^{+}(0) } \vert u_{ \varepsilon } \vert ^{p^{*}} \,dx \\ &\quad= \int _{\mathbb{R}_{+}^{N}\setminus B_{R}^{+}(0)} \frac{{{\varepsilon ^{\frac{N}{p}}}}}{{{{ (\varepsilon + {{ \vert x \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}} \\ &\quad\leq C{\varepsilon ^{\frac{{N}}{p}}} \int _{R}^{ + \infty } { \frac{1}{{{r^{\frac{{N+p - 1}}{{p - 1}}}}}}\,dr} \\ &\quad=O\bigl(\varepsilon ^{\frac{N}{p}}\bigr). \end{aligned} \end{aligned}$$

(3.14)

Similarly as for (3.11),

$$\begin{aligned} \begin{aligned} & \int _{\mathbb{R}^{N-1}}\,dx' \int _{0}^{g(x')} {u_{\varepsilon }^{{p^{*}}} \,d{x_{N}}} - \int _{{D_{\delta }}(0)} {\,dx'} \int _{0}^{g(x')} {{u_{\varepsilon }}^{{p^{{*}}}} \,d{x_{N}}} \\ &\quad= \int _{\mathbb{R}^{N-1}\setminus D_{\delta }(0)}\,dx' \int _{0}^{g(x')} {u_{\varepsilon }^{{p^{*}}} \,d{x_{N}}} \\ &\quad\leq \int _{\mathbb{R}^{N-1}\setminus D_{\delta }(0)}\,dx' \int _{0}^{g(x')} { \frac{{{\varepsilon ^{\frac{N}{p}}}}}{{{{ (\varepsilon + {{ \vert x' \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}} \,d{x_{N}}} \\ &\quad= \int _{\mathbb{R}^{N-1}\setminus D_{\delta }(0)} \frac{{{\varepsilon ^{\frac{N}{p}}}g(x')}}{{{{ (\varepsilon + {{ \vert {x'} \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dx' \\ &\quad= \int _{\mathbb{R}^{N-1}\setminus D_{\delta }(0)} \frac{{{\varepsilon ^{\frac{N}{p}}}\frac{1}{2}\sum_{i = 1}^{N - 1} {{\gamma _{i}}x_{i}^{2}} }}{{{{ (\varepsilon + {{ \vert {x'} \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dx'. \end{aligned} \end{aligned}$$

(3.15)

As before, performing a spherical coordinate transformation, we get

$$\begin{aligned} \begin{aligned} \text{(3.15)}&\leq \frac{1}{2}\sum _{i = 1}^{N - 1} {{\gamma _{i}} \frac{1}{{N - 1}}} \int _{\mathbb{R}^{N-1}\setminus D_{ \delta }(0)} \frac{{{\varepsilon ^{\frac{N}{p}}}{{ \vert {x'} \vert }^{2}}}}{{{{(\varepsilon + {{ \vert {x'} \vert }^{\frac{p}{{p - 1}}}})}^{N}}}}\,dx' \\ &=\frac{1}{2}\sum_{i = 1}^{N - 1} {{ \gamma _{i}} \frac{1}{{N - 1}}}\varepsilon ^{\frac{N}{p}}\omega _{N-1} \int _{\delta }^{ + \infty } { \frac{{{r^{2}}{r^{N - 2}}}}{{{{(\varepsilon + {r^{\frac{p}{{p - 1}}}})}^{N}}}}\,dr} \\ &\leq C\varepsilon ^{\frac{N}{p}} \end{aligned} \end{aligned}$$

and

$$\begin{aligned} \int _{\Omega }u_{\varepsilon \varrho }^{p^{*}}\,dx={}& \int _{ \mathbb{R}_{+}^{N}} u_{\varepsilon }^{p^{*}}\,dx- \int _{{D_{\delta }}(0)} {\,dx'} \int _{0}^{h(x')} {{u_{\varepsilon }}^{{p^{{*}}}} \,d{x_{N}}} +O\bigl( \varepsilon ^{\frac{N}{p}}\bigr) \\ ={}& \int _{\mathbb{R}_{+}^{N}} u_{\varepsilon }^{p^{*}}\,dx- \int _{{D_{\delta }}(0)} {\,dx'} \int _{0}^{g(x')} {{u_{\varepsilon }}^{{p^{{*}}}} \,d{x_{N}}} \\ \begin{aligned} &{} - \int _{{D_{\delta }}(0)} {\,dx'} \int _{g(x')}^{h(x')} {{u_{\varepsilon }}^{{p^{{*}}}} \,d{x_{N}}} +O\bigl(\varepsilon ^{\frac{N}{p}}\bigr) \\ ={}& \int _{\mathbb{R}_{+}^{N}} u_{\varepsilon }^{p^{*}}\,dx- \int _{ \mathbb{R}^{N-1}}\,dx' \int _{0}^{g(x')} {{u_{\varepsilon }}^{{p^{{*}}}} \,d{x_{N}}} \end{aligned} \\ & {}- \int _{{D_{\delta }}(0)} {\,dx'} \int _{g(x')}^{h(x')} {{u_{\varepsilon }}^{{p^{{*}}}} \,d{x_{N}}}+ \int _{\mathbb{R}^{N-1}}\,dx' \int _{0}^{g(x')} {{u_{\varepsilon }}^{{p^{{*}}}} \,d{x_{N}}} \\ & {}- \int _{{D_{\delta }}(0)} {\,dx'} \int _{0}^{g(x')} {{u_{\varepsilon }}^{{p^{{*}}}} \,d{x_{N}}} +O\bigl(\varepsilon ^{\frac{N}{p}}\bigr). \end{aligned}$$

(3.16)

Denoting $\mathit{II}_{\varepsilon }^{2}=\int _{{D_{\delta }}(0)} {\,dx'} \int _{g(x')}^{h(x')} {{u_{\varepsilon }}^{{p^{{*}}}}\,d{x_{N}}}$, we have

$$\begin{aligned} K_{2}(\varepsilon )= \int _{\Omega } u_{\varepsilon }^{p^{*}}\,dx= \int _{ \mathbb{R}_{+}^{N}} u_{\varepsilon }^{p^{*}} \,dx-\mathit{II}_{\varepsilon } -\mathit{II}_{ \varepsilon }^{2}+O\bigl(\varepsilon ^{\frac{N}{p}}\bigr). \end{aligned}$$

Estimating $\mathit{II}_{\varepsilon }^{2}$ gives

$$\begin{aligned} \begin{aligned} \bigl\vert \mathit{II}_{\varepsilon }^{2} \bigr\vert &= \biggl\vert \int _{{D_{\delta }}(0)} {\,dx'} \int _{0}^{h(x')} {{u_{\varepsilon }}^{{p^{{*}}}} \,d{x_{N}}} \biggr\vert \\ &\leq \biggl\vert \int _{{D_{\delta }}(0)} {\,dx'} \int _{g(x')}^{h(x')} { \frac{{{\varepsilon ^{\frac{N}{p}}}}}{{{{ (\varepsilon + {{ \vert x' \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}} \,d{x_{N}}} \biggr\vert \\ &\leq \varepsilon ^{\frac{N}{p}} \int _{D_{\delta }(0)} \frac{{\theta {{ \vert {x'} \vert }^{2}} + C(\theta ){{ \vert {x'} \vert }^{\frac{5}{2}}}}}{{{{ (\varepsilon + {{ \vert x' \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dx' \\ &\leq \varepsilon ^{\frac{N}{p}}\omega _{N-1} \int _{0}^{\delta }{ \frac{{\theta {r^{N}} + C(\theta ){r^{N + \frac{1}{2}}}}}{{{{ (\varepsilon + {{ \vert x \vert }^{\frac{p}{{p - 1}}}} )}^{N }}}}\,dr} \\ &\leq C \bigl(\varepsilon ^{\frac{p-1}{p}}\theta +\varepsilon ^{ \frac{p-1}{p}} \varepsilon ^{\frac{p-1}{2p}}C(\theta ) \bigr) \\ &\leq C\varepsilon ^{\frac{p-1}{p}} \bigl(\theta +\varepsilon ^{ \frac{p-1}{2p}}C( \theta ) \bigr) \\ &=o \bigl(\varepsilon ^{\frac{p-1}{p}} \bigr). \end{aligned} \end{aligned}$$

(3.17)

(3) The estimate of $K_{3}(\varepsilon )$.

Due to Ω being bounded, there must exist $R>0$ such that $\Omega \subset B_{R}(0)$, hence we can write

$$\begin{aligned} \int _{\Omega }u_{\varepsilon }^{p}\,dx&= \int _{\Omega } \frac{{{\varepsilon ^{\frac{{N - p}}{p}}}}}{{{{ (\varepsilon + {{ \vert x \vert }^{\frac{p}{{p - 1}}}} )}^{N - p}}}}\,dx \\ \begin{aligned} &= \int _{B_{R}(0)} \frac{{{\varepsilon ^{\frac{{N - p}}{p}}}}}{{{{ (\varepsilon + {{ \vert x \vert }^{\frac{p}{{p - 1}}}} )}^{N - p}}}}\,dx \\ &\leq \varepsilon ^{\frac{N-p}{p}} \int _{B_{R}(0)} \frac{{{{ ({r^{\frac{p}{{p - 1}}}} )}^{\frac{{p - 1}}{p}(N - 1)}}}}{{{{ (\varepsilon + {r^{\frac{p}{{p - 1}}}} )}^{N - p}}}}{ \omega _{N}}\,dr \end{aligned} \\ &= \textstyle\begin{cases} {O ({\varepsilon ^{\frac{{N - p}}{p}}} ) = O ({\varepsilon ^{ \frac{{{{(p - 1)}^{2}}}}{p}}} ),}&{N = {p^{2}} - p + 1}, \\ {O ({\varepsilon ^{\frac{{N - p}}{p}}}\ln \varepsilon ) = O({ \varepsilon ^{p - 1}}\ln \varepsilon ),}&{N = {p^{2},}} \\ {O({\varepsilon ^{p - 1}}),}&{N \ge {p^{2}} + 1.} \end{cases}\displaystyle \end{aligned}$$

(3.18)

(4) The estimates of $I_{\varepsilon }$ and $\mathit{II}_{\varepsilon }$.

Setting $x=\varepsilon ^{\frac{p-1}{p}}y$, by direct calculation, we derive

$$\begin{aligned} \begin{aligned} I_{\varepsilon }&= \int _{\mathbb{R}^{N-1}}\,dx' \int _{0}^{g(x')} {{{ \vert {\nabla {u_{\varepsilon }}} \vert }^{p}}\,d{x_{N}}} \\ &= \biggl(\frac{N-p}{p-1} \biggr)^{p}\varepsilon ^{\frac{N-p}{p}} \int _{ \mathbb{R}^{N-1}}\,dx' \int _{0}^{g(x')} { \frac{{{{ \vert x \vert }^{\frac{p}{{p - 1}}}}}}{{{{ (\varepsilon + {{ \vert x \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}} \,d{x_{N}}} \\ &= \biggl(\frac{N-p}{p-1} \biggr)^{p}\varepsilon ^{\frac{N-p}{p}} \int _{ \mathbb{R}^{N-1}} \bigl(\varepsilon ^{\frac{p-1}{p}} \bigr)^{N-1}\,dy' \int _{0}^{{\varepsilon ^{\frac{{p - 1}}{p}}}g(y')} { \frac{{{{ \vert y \vert }^{\frac{p}{{p - 1}}}}}}{{{{ (1 + {{ \vert y \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}{ \varepsilon ^{1 + \frac{{p - 1}}{p} - N}}\,d{y_{N}}} \\ &= \biggl(\frac{N-p}{p-1} \biggr)^{p} \int _{\mathbb{R}^{N-1}}\,dy' \int _{0}^{{ \varepsilon ^{\frac{{p - 1}}{p}}}g(y')} { \frac{{{{ \vert y \vert }^{\frac{p}{{p - 1}}}}}}{{{{ (1 + {{ \vert y \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}} \,d{y_{N}}} \\ &= \biggl(\frac{N-p}{p-1} \biggr)^{p} \int _{\mathbb{R}^{N-1}} \frac{{{\varepsilon ^{\frac{{p - 1}}{p}}}g(y') ({{ \vert {y'} \vert }^{\frac{p}{{p - 1}}}} + \varepsilon {{ \vert {g(y')} \vert }^{\frac{p}{{p - 1}}}} )}}{{{{ (1 + {{ \vert {y'} \vert }^{\frac{p}{{p - 1}}}} + \varepsilon {{ \vert {g(y')} \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dy'. \end{aligned} \end{aligned}$$

(3.19)

Hence,

$$\begin{aligned} \mathop{\lim } _{\varepsilon \to 0} {\varepsilon ^{ - \frac{{p - 1}}{p}}} {{I}_{\varepsilon }} = { \biggl(\frac{{N - p}}{{p - 1}} \biggr)^{p}} \int _{{\mathbb{R}^{N - 1}}} { \frac{{{{ \vert {y'} \vert }^{\frac{p}{{p - 1}}}}g(y')}}{{{{ (1 + {{ \vert {y'} \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dy'.} \end{aligned}$$

Similarly,

$$\begin{aligned} \begin{aligned} \mathit{II}_{\varepsilon }&= \int _{\mathbb{R}^{N-1}}\,dx' \int _{0}^{g(x')} {u_{\varepsilon }^{{p^{*}}} \,d{x_{N}}} \\ &= \int _{\mathbb{R}^{N-1}}\,dx' \int _{0}^{g(x')} { \frac{{{\varepsilon ^{\frac{N}{p}}}}}{{{{ (\varepsilon + {{ \vert x \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}} \,d{x_{N}}} \\ &= \int _{\mathbb{R}^{N-1}} \bigl(\varepsilon ^{\frac{p-1}{p}} \bigr)^{N-1}\,dy' \int _{0}^{{\varepsilon ^{\frac{{p - 1}}{p}}}g(y')} { \frac{{{\varepsilon ^{\frac{N}{p}}}}}{{{{ (\varepsilon + \varepsilon {{ \vert y \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}{ \varepsilon ^{\frac{{p - 1}}{p}}}\,d{y_{N}}} \\ &= \int _{\mathbb{R}^{N-1}}\,dy' \int _{0}^{{\varepsilon ^{ \frac{{p - 1}}{p}}}g(y')} { \frac{1}{{{{ (1 + {{ \vert y \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}} \,d{y_{N}}} \\ &= \int _{\mathbb{R}^{N-1}} \frac{{{\varepsilon ^{\frac{{p - 1}}{p}}}g(y')}}{{{{ (1 + {{ \vert {y'} \vert }^{\frac{p}{{p - 1}}}} + \varepsilon {{ \vert {g(y')} \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dy', \end{aligned} \end{aligned}$$

(3.20)

and so we have proved (3.8). □

Making the same variable substitution and setting $K_{1}=\int _{\mathbb{R}^{N}}|\nabla u_{\varepsilon }|^{p}$, we see that

$$\begin{aligned} \begin{aligned} K_{1}&= \int _{\mathbb{R}^{N}} \vert \nabla u_{\varepsilon } \vert ^{p} \\ &= \biggl(\frac{N-p}{p-1} \biggr)^{p}\varepsilon ^{\frac{N-p}{p}} \int _{ \mathbb{R}^{N}} \frac{{{{ \vert x \vert }^{\frac{p}{{p - 1}}}}}}{{{{ (\varepsilon + {{ \vert x \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dx \\ &= \biggl(\frac{N-p}{p-1} \biggr)^{p}\varepsilon ^{\frac{N-p}{p}} \int _{ \mathbb{R}^{N}} \frac{{\varepsilon {{ \vert y \vert }^{\frac{p}{{p - 1}}}}}}{{{{ (\varepsilon + \varepsilon {{ \vert y \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}{ \bigl({\varepsilon ^{\frac{{p - 1}}{p}}} \bigr)^{N}}\,dy \\ &= \biggl(\frac{N-p}{p-1} \biggr)^{p} \int _{\mathbb{R}^{N}} \frac{{{{ \vert y \vert }^{\frac{p}{{p - 1}}}}}}{{{{ (1 + {{ \vert y \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dy. \end{aligned} \end{aligned}$$

(3.21)

Setting $K_{2}=\int _{\mathbb{R}^{N}} u_{\varepsilon }^{p^{*}}\,dx$, by the same calculation,

$$\begin{aligned} K_{2}= \int _{\mathbb{R}^{N}} u_{\varepsilon }^{p^{*}}= \int _{ \mathbb{R}^{N}} \frac{1}{{{{ (1 + {{ \vert y \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dy. \end{aligned}$$

(3.22)

Therefore, we can immediately deduce the following lemma from the above calculation.

Lemma 3.2

$$\begin{aligned} \frac{K_{1}}{K_{2}^{\frac{p}{p^{*}}}}= \frac{K_{1}}{K_{2}^{\frac{N-p}{N}}}=S. \end{aligned}$$

(3.23)

4 Proof of the main results

In this section, we will give the proof of Theorem 1.1, but we first give a lemma.

Lemma 4.1

Under the conditions of Theorem 1.1, there exists at least one nonnegative function $(u, v)\in X\setminus {(0, 0)}$ such that

$$\begin{aligned} \mathop{\sup } _{t \ge 0} J\bigl(t(u,v)\bigr) < \frac{1}{{2N}}S_{ \alpha,\beta }^{\frac{N}{p}}. \end{aligned}$$

(4.1)

Proof

By Lemma 2.1, we only need to verify that $\forall \varepsilon >0$ small enough,

$$\begin{aligned} \mathop{\sup } _{t \ge 0} J\bigl(t(s{u_{\varepsilon }},k{u_{\varepsilon }}) \bigr) < \frac{1}{{2N}}S_{\alpha,\beta }^{\frac{N}{p}}. \end{aligned}$$

(4.2)

Because

$$\begin{aligned} &\mathop{\sup } _{t \ge 0} J\bigl(t(s{u_{\varepsilon }},k{u_{\varepsilon }}) \bigr) \\ &\quad=\mathop{\sup } _{t \ge 0} \biggl[\frac{{{t^{p}}}}{p} \int _{\Omega }{{{ \bigl\vert {\nabla (s{u_{\varepsilon }})} \bigr\vert }^{p}} + { \lambda _{1}} {{(s{u_{\varepsilon }})}^{p}} + {{ \bigl\vert {\nabla (k{u_{\varepsilon }})} \bigr\vert }^{p}} + {\lambda _{2}} {{(k{u_{\varepsilon }})}^{p}}\,dx} \\ &\qquad{} - \frac{{{t^{{p^{*}}}}}}{{{p^{*}}}} \int _{\Omega }{{{(s{u_{\varepsilon }})}^{{p^{*}}}} + {{(k{u_{\varepsilon }})}^{{p^{*}}}} + {{(s{u_{\varepsilon }})}^{\alpha }} {{(k{u_{\varepsilon }})}^{\beta }}\,dx} \biggr] \\ &\quad\leq \mathop{\sup } _{t \ge 0} \biggl[\frac{{{t^{p}}}}{p} \int _{\Omega }{{{ \bigl\vert {\nabla (s{u_{\varepsilon }})} \bigr\vert }^{p}} + {{ \bigl\vert {\nabla (k{u_{\varepsilon }})} \bigr\vert }^{p}}\,dx} \\ &\qquad{} - \frac{{{t^{{p^{*}}}}}}{{{p^{*}}}} \int _{\Omega }{{{(s{u_{\varepsilon }})}^{{p^{*}}}} + {{(k{u_{\varepsilon }})}^{{p^{*}}}} + {{(s{u_{\varepsilon }})}^{\alpha }} {{(k{u_{\varepsilon }})}^{\beta }}\,dx} \biggr] + o\bigl({ \varepsilon ^{\frac{{p - 1}}{p}}}\bigr) \\ &\quad=\mathop{\sup } _{t \ge 0} \biggl[\frac{{{t^{p}}}}{p} \int _{\Omega }{\bigl({s^{p}} + {k^{p}} \bigr){{ \bigl\vert {\nabla ({u_{\varepsilon }})} \bigr\vert }^{p}} \,dx} \\ & \qquad{}- \frac{{{t^{{p^{*}}}}}}{{{p^{*}}}} \int _{\Omega }{\bigl({s^{{p^{*}}}} + {k^{{p^{*}}}} + {s^{\alpha }} {k^{\beta }}\bigr){{({u_{\varepsilon }})}^{{p^{*}}}} \,dx} \biggr] + o\bigl({\varepsilon ^{\frac{{p - 1}}{p}}}\bigr) \\ &\quad= \mathop{\sup } _{t \ge 0} \biggl[\frac{{{t^{p}}}}{p} \bigl({s^{p}} + {k^{p}}\bigr){K_{1}}(\varepsilon ) - \frac{{{t^{{p^{*}}}}}}{{{p^{*}}}}\bigl({s^{{p^{*}}}} + {k^{{p^{*}}}} + {s^{\alpha }} {k^{\beta }}\bigr){K_{2}}(\varepsilon ) \biggr] + o\bigl({ \varepsilon ^{\frac{{p - 1}}{p}}}\bigr) \\ &\quad=\frac{1}{N}{ \biggl[ \frac{{{s^{p}} + {k^{p}}}}{{{{({s^{{p^{*}}}} + {k^{{p^{*}}}} + {s^{\alpha }}{k^{\beta }})}^{\frac{p}{{{p^{*}}}}}}}} \cdot \frac{{{K_{1}}(\varepsilon )}}{{{{({K_{2}}(\varepsilon ))}^{\frac{p}{{{p^{*}}}}}}}} \biggr]^{\frac{N}{p}}} + o\bigl({\varepsilon ^{\frac{{p - 1}}{p}}}\bigr) \\ &\quad=\frac{1}{N}{ \biggl[f({\tau _{\min }}) \cdot \frac{{{K_{1}}(\varepsilon )}}{{{{({K_{2}}(\varepsilon ))}^{\frac{p}{{{p^{*}}}}}}}} \biggr]^{\frac{N}{p}}} + o\bigl({\varepsilon ^{\frac{{p - 1}}{p}}}\bigr). \end{aligned}$$

Therefore, we need to prove

$$\begin{aligned} \frac{K_{1}(\varepsilon )}{[K_{2}(\varepsilon )]^{\frac{p}{p^{*}}}}< \biggl(\frac{1}{2} \biggr)^{\frac{p}{N}}S +o\bigl(\varepsilon ^{\frac{p-1}{p}}\bigr)= \frac{\frac{1}{2}K_{1}}{(\frac{1}{2}K_{2})^{\frac{p}{p^{*}}}} +o\bigl( \varepsilon ^{\frac{p-1}{p}}\bigr), \end{aligned}$$

(4.3)

which is equivalent to

$$\begin{aligned} \biggl[\frac{1}{2}K_{1}-I_{\varepsilon }+o \bigl(\varepsilon ^{\frac{p-1}{p}}\bigr) \biggr] \biggl(\frac{1}{2}K_{2} \biggr)^{\frac{p}{p^{*}}}< \frac{1}{2}K_{1} \biggl[ \frac{1}{2}K_{2} -\mathit{II}_{\varepsilon }+o\bigl(\varepsilon ^{\frac{p-1}{p}}\bigr) \biggr]^{\frac{p}{p^{*}}}+o\bigl(\varepsilon ^{\frac{p-1}{p}} \bigr), \end{aligned}$$

(4.4)

where

$$\begin{aligned} \begin{aligned} & \biggl[\frac{1}{2}K_{2} -\mathit{II}_{\varepsilon }+o\bigl(\varepsilon ^{ \frac{p-1}{p}}\bigr) \biggr]^{\frac{p}{p^{*}}} \\ &\quad\leq \biggl(\frac{1}{2}K_{2}-\mathit{II}_{\varepsilon } \biggr)^{\frac{p}{p^{*}}} +o\bigl( \varepsilon ^{\frac{p-1}{p}}\bigr) \\ &\quad\leq \biggl(\frac{1}{2}K_{2} \biggr)^{\frac{p}{p^{*}}} - \frac{p}{p^{*}} \biggl(\frac{1}{2}K_{2} \biggr)^{\frac{p}{p^{*}}-1}\mathit{II}_{\varepsilon } +o\bigl( \varepsilon ^{\frac{p-1}{p}}\bigr) \\ &\quad= \biggl(\frac{1}{2}K_{2} \biggr)^{\frac{p}{p^{*}}} - \frac{p}{p^{*}} \biggl( \frac{1}{2}K_{2} \biggr)^{-\frac{p}{N}}\mathit{II}_{\varepsilon } +o\bigl(\varepsilon ^{ \frac{p-1}{p}} \bigr). \end{aligned} \end{aligned}$$

(4.5)

Substituting into (4.4) yields

$$\begin{aligned} \frac{I_{\varepsilon }}{\mathit{II}_{\varepsilon }} >\frac{K_{1}}{K_{2}}\cdot \frac{N-p}{N}+o(1). \end{aligned}$$

(4.6)

This implies that it is necessary to prove (4.6) before we verify (4.1). In fact,

$$\begin{aligned} \mathop{\lim } _{\varepsilon \to 0} \frac{I_{\varepsilon }}{\mathit{II}_{\varepsilon }} &= \mathop{\lim } _{ \varepsilon \to 0} \frac{\varepsilon ^{-\frac{p-1}{p}} I_{\varepsilon }}{\varepsilon ^{-\frac{p-1}{p}} \mathit{II}_{\varepsilon }} \\ &= \biggl(\frac{N-p}{p-1} \biggr)^{p} \frac{\int _{\mathbb{R}^{N-1}}\frac{{g(y'){{ \vert {y'} \vert }^{\frac{p}{{p - 1}}}}}}{{{{ (1 + {{ \vert {y'} \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dy'}{\int _{\mathbb{R}^{N-1}}\frac{{g(y')}}{{{{ (1 + {{ \vert {y'} \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dy'} \\ &= \biggl(\frac{N-p}{p-1} \biggr)^{p} \frac{\frac{1}{{2(N - 1)}}\sum_{i = 1}^{N - 1} {{\gamma _{i}}} \int _{\mathbb{R}^{N-1}}\frac{{{{ \vert {y'} \vert }^{\frac{p}{{p - 1}} + 2}}}}{{{{ (1 + {{ \vert {y'} \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dy'}{\frac{1}{{2(N - 1)}}\sum_{i = 1}^{N - 1} {{\gamma _{i}}} \int _{\mathbb{R}^{N-1}}\frac{{{{ \vert {y'} \vert }^{2}}}}{{{{ (1 + {{ \vert {y'} \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dy'} \end{aligned}$$

Performing a spherical coordinate transformation, we obtain

$$\begin{aligned} \begin{aligned} \mathop{\lim } _{\varepsilon \to 0} \frac{I_{\varepsilon }}{\mathit{II}_{\varepsilon }} &= \biggl(\frac{N-p}{p-1} \biggr)^{p} \frac{\int _{0}^{ + \infty } {\frac{{{r^{\frac{p}{{p - 1}} + 2}}{\omega _{N - 1}}{r^{N - 2}}}}{{{{ (1 + {r^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dr} }{\int _{0}^{ + \infty } {\frac{{{r^{2}}{\omega _{N - 1}}{r^{N - 2}}}}{{{{ (1 + {r^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dr} } \\ &= \biggl(\frac{N-p}{p-1} \biggr)^{p} \frac{\int _{0}^{ + \infty } {\frac{{{r^{\frac{p}{{p - 1}} + N}}}}{{{{ (1 + {r^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dr} }{\int _{0}^{ + \infty } {\frac{{{r^{N}}}}{{{{ (1 + {r^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dr} }. \end{aligned} \end{aligned}$$

For all $k, \frac{p}{p-1}\leq k \leq \frac{p}{p-1} N$, integrating by parts, we have

$$\begin{aligned} \int _{0}^{ + \infty } { \frac{{{r^{k - \frac{p}{{p - 1}}}}}}{{{{ (1 + {r^{\frac{p}{{p - 1}}}} )}^{N - 1}}}}\,dr} = \frac{{p(N - 1)}}{{(p - 1)k - 1}} \int _{0}^{ + \infty } { \frac{{{r^{k}}}}{{{{ (1 + {r^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dr} \end{aligned}$$

and

$$\begin{aligned} \int _{0}^{ + \infty } { \frac{{{r^{k}}}}{{{{ (1 + {r^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dr} = \int _{0}^{ + \infty } { \frac{{{r^{k - \frac{p}{{p - 1}}}}}}{{{{ (1 + {r^{\frac{p}{{p - 1}}}} )}^{N - 1}}}}\,dr} - \int _{0}^{ + \infty } { \frac{{{r^{k - \frac{p}{{p - 1}}}}}}{{{{ (1 + {r^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dr}. \end{aligned}$$

Hence,

$$\begin{aligned} \int _{0}^{ + \infty } { \frac{{{r^{k}}}}{{{{ (1 + {r^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dr} = \frac{{(p - 1)k - 1}}{{pN - (p - 1) - (p - 1)k}} \int _{0}^{ + \infty } { \frac{{{r^{k - \frac{p}{{p - 1}}}}}}{{{{ (1 + {r^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dr}. \end{aligned}$$

Choosing $k=N+\frac{p}{p-1}$ gives

$$\begin{aligned} \frac{\int _{0}^{ + \infty } {\frac{{{r^{\frac{p}{{p - 1}} + N}}}}{{{{ (1 + {r^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dr} }{\int _{0}^{ + \infty } {\frac{{{r^{N}}}}{{{{ (1 + {r^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dr} } =\frac{(p-1)(N+1)}{N-(2p-1)}, \end{aligned}$$

thus we obtain

$$\begin{aligned} \mathop{\lim } _{\varepsilon \to 0} \frac{I_{\varepsilon }}{\mathit{II}_{\varepsilon }} = \frac{(N-p)^{p}(N+1)}{(p-1)^{p-1}(N-(2p-1))}. \end{aligned}$$

(4.7)

On the other hand,

$$\begin{aligned} \begin{aligned} \frac{N-p}{N}\frac{K_{1}}{K_{2}}&= \frac{N-p}{N} \biggl( \frac{N-p}{p-1} \biggr)^{p} \frac{\int _{\mathbb{R}^{N}}\frac{{{{ \vert {y'} \vert }^{\frac{p}{{p - 1}}}}}}{{{{ (1 + {{ \vert {y} \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dy}{\int _{\mathbb{R}^{N}}\frac{{1}}{{{{ (1 + {{ \vert {y'} \vert }^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dy'} \\ &=\frac{N-p}{N} \biggl(\frac{N-p}{p-1} \biggr)^{p} \frac{\int _{0}^{ + \infty } {\frac{{{r^{\frac{p}{{p - 1}} + N - 1}}}}{{{{ (1 + {r^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dr} }{\int _{0}^{ + \infty } {\frac{{{r^{N - 1}}}}{{{{ (1 + {r^{\frac{p}{{p - 1}}}} )}^{N}}}}\,dr} }. \end{aligned} \end{aligned}$$

Choosing $k=N+\frac{p}{p-1}-1$, we then have

$$\begin{aligned} \frac{N-p}{N}\frac{K_{1}}{K_{2}}=\frac{(N-p)^{p}}{(p-1)^{p-1}}, \end{aligned}$$

(4.8)

thus, under the condition $N\geq p^{2}$ in Theorem 1.1, there must exist $N>2p-1$, and then the proofs of (4.4) and Theorem 4.1 are completed. □

Finally, we will complete the proof of Theorem 1.1.

Proof of Theorem 1.1

Set

$$\begin{aligned} {c^{*}} = \mathop{\inf } _{(u,v) \in X} \Bigl\{ \mathop{\sup _{t \ge 0} J\bigl(t(u,v)\bigr)| {u,v \ge 0,(u,v) \ne (0,0)} } \Bigr\} , \end{aligned}$$

where $c^{*}>c$, and c represents the mountain pass level. Hence, we only need to verify

$$\begin{aligned} c< \min \biggl\{ \frac{1}{N}S_{{\lambda _{1}},{\lambda _{2}}}^{ \frac{N}{p}}, \frac{1}{{2N}}S_{\alpha,\beta }^{\frac{N}{p}} \biggr\} . \end{aligned}$$

By Lemmas 4.1 and 2.7, we know that system (1.1) has a nontrivial solution $(u,v)\in X, J(u,v)< c$.

Assume $(u,v)=(u_{1}, v_{1})$ is the constant solution of equation (1.1), then

$$\begin{aligned} \textstyle\begin{cases} {{\lambda _{1}}{u_{1}} = u_{1}^{{p^{*}} - 1} + \frac{\alpha }{{{p^{*}}}}u_{1}^{\alpha - 1}v_{1}^{\beta }},\quad{x \in \Omega, } \\ {{\lambda _{2}}{v_{1}} = v_{1}^{{p^{*}} - 1} + \frac{\beta }{{{p^{*}}}}u_{1}^{\alpha }v_{1}^{\beta - 1}},\quad{x \in \Omega, } \\ {{u_{1}},{v_{1}} > 0,}\quad{x \in \Omega. } \end{cases}\displaystyle \end{aligned}$$

Therefore,

$$\begin{aligned} &\textstyle\begin{cases} {{\lambda _{1}} = u_{1}^{{p^{*}} - 2} + \frac{\alpha }{{{p^{*}}}}u_{1}^{ \alpha - 2}v_{1}^{\beta }, }\quad{x \in \Omega }, \\ {{\lambda _{2}} = v_{1}^{{p^{*}} - 2} + \frac{\beta }{{{p^{*}}}}u_{1}^{\alpha }v_{1}^{\beta - 2}},\quad{x \in \Omega, } \end{cases}\displaystyle \end{aligned}$$

(4.9)

$$\begin{aligned} & v_{1}^{1-\frac{(2-\alpha )(2-\beta )}{\alpha \beta }}\geq 2^{ \frac{2}{\alpha \beta }},\qquad u_{1}^{1- \frac{(2-\alpha )(2-\beta )}{\alpha \beta }}\geq 2^{ \frac{2}{\alpha \beta }}. \end{aligned}$$

(4.10)

However, $1-\frac{(2-\alpha )(2-\beta )}{\alpha \beta }>0$, that is, (4.10) cannot be satisfied, which is a contradiction.

The functional which corresponds to system (1.1) is

$$\begin{aligned} \begin{aligned} J(u_{1}, v_{1})&= \frac{1}{p} \int _{\Omega }\bigl(\lambda _{1}u_{1}^{p}+ \lambda _{2}v_{1}^{p}\bigr)\,dx - \frac{1}{p^{*}} \int _{\Omega }\bigl(u_{1}^{p^{*}}+v_{1}^{p^{*}}+u_{1}^{ \alpha }v_{1}^{\beta } \bigr)\,dx \\ &=\frac{1}{N} \int _{\Omega }\bigl(\lambda _{1}u_{1}^{p}+ \lambda _{2}v_{1}^{p}\bigr)\,dx \\ &=C(N, \lambda _{1}, \lambda _{2})\operatorname{meas}(\Omega ). \end{aligned} \end{aligned}$$

When $\lambda _{1}, \lambda _{2}$ are large enough, we can obtain $J(u_{1}, v_{1})>c$, which creates a contradiction. And also because

$$\begin{aligned} \bigl\langle J'(u,v),\bigl(u^{-},v^{-} \bigr)\bigr\rangle =0, \end{aligned}$$

we get $\|(u^{-},v^{-})\|^{2}=0$ and $(u^{-},v^{-})=(0,0)$. This implies that $(u,v)$ is a nonnegative solution of system (1.1). By the strong maximum principle, we have $u>0,v>0$, and then the proof is finished. □