In this section, it is necessary to present some definitions and preliminary lemmas, which are going to be used to prove our basic estimates and main results.
First, denote \(X=W^{1,p}(\Omega ) \times W^{1,p}(\Omega )\), where its norm is as follows:
$$\begin{aligned} \bigl\Vert (u,v) \bigr\Vert :=\biggl( \int _{\Omega }\bigl( \vert \nabla u \vert ^{p}+u^{p} \bigr)\,dx+ \int _{\Omega }\bigl( \vert \nabla v \vert ^{p}+v^{p} \bigr)\,dx\biggr)^{\frac{1}{p}}. \end{aligned}$$
Here \(W^{1,p}(\Omega )\) is a Sobolev space, and it has the following norm:
$$\begin{aligned} \Vert u \Vert =( \int _{\Omega }\bigl( \vert \nabla u \vert ^{p}+u^{p} \,dx\bigr)^{\frac{1}{p}}. \end{aligned}$$
We define its corresponding energy functional \(J: X\rightarrow \mathbb{R}\) as follows:
$$\begin{aligned} \begin{aligned} J(u,v)={}&\frac{1}{p} \int _{\Omega } \bigl( \vert \nabla u \vert ^{p}+ \lambda _{1} u^{p}+ \vert \nabla v \vert ^{p}+\lambda _{2}v^{p}\bigr)\,dx \\ &{}-\frac{1}{p^{*}} \int _{\Omega } \bigl(\bigl(u^{+}\bigr)^{p^{*}}+ \bigl(v^{+}\bigr)^{p^{*}}+\bigl(u^{+} \bigr)^{ \alpha }\bigl(v^{+}\bigr)^{\beta }\bigr)\,dx. \end{aligned} \end{aligned}$$
Now, we define the weak solution:
Definition 2.1
If \(\forall (\varphi, \psi )\in X\), when \(u>0\), \(v>0\), and \((u,v)\in X\), we have
$$\begin{aligned} & \int _{\Omega }\bigl( \vert \nabla u \vert ^{p-2} \nabla u \nabla \varphi +\lambda _{1}u^{p-1}\varphi + \vert \nabla v \vert ^{p-2}\nabla v \nabla \psi +\lambda _{2}v^{p-1} \psi \bigr)\,dx- \int _{\Omega } \vert u \vert ^{p^{*}-2}u \varphi \,dx \\ &\quad{}- \int _{\Omega } \vert v \vert ^{p^{*}-2}v\psi \,dx- \frac{\alpha }{p^{*}} \int _{ \Omega } \vert u \vert ^{\alpha -1} \vert v \vert ^{\beta }\varphi \,dx- \frac{\beta }{p^{*}} \int _{\Omega } \vert u \vert ^{\alpha } \vert v \vert ^{\beta -1}\psi \,dx =0, \end{aligned}$$
then \((u,v)\) is called a weak solution of problem (1.1).
In order to prove that the least energy solution of (1.1) exists, and derive the solution’s form, we consider the following equation:
$$\begin{aligned} \textstyle\begin{cases} { - {\Delta _{p}}u = {{ \vert u \vert }^{{p^{*}} - 2}}u,}&{x \in \Omega, } \\ {u \in W_{0}^{1,p}(\Omega ).} \end{cases}\displaystyle \end{aligned}$$
(2.1)
From [30], we know that, when \(\Omega =\mathbb{R}^{N}\), the function
$$\begin{aligned} U(x) = {\bigl(N(N - p)\bigr)^{\frac{{N - p}}{p^{2}}}} {\bigl(1 + { \vert x \vert ^{ \frac{p}{p-1}}}\bigr)^{ - \frac{{N - p}}{p}}} \end{aligned}$$
(2.2)
is a radial function which solves (2.1). Meanwhile, the best Sobolev constant S can be achieved by \(U(x)\) for the embedding \(W^{1, p}(\mathbb{R}^{N})\hookrightarrow L^{p^{*}}(\mathbb{R}^{N})\), where
$$\begin{aligned} S: = \mathop{\inf } _{u \in {W^{1,p}}(\mathbb{R}^{N}) \backslash \{ 0\} } \frac{{{{ \Vert u \Vert }^{p}}}}{{ \vert u \vert _{{p^{*}}}^{p}}} = \frac{{{{ \Vert U \Vert }^{p}}}}{{ \vert U \vert _{{p^{*}}}^{p}}}. \end{aligned}$$
(2.3)
For \(x_{0}\in \mathbb{R}^{N}, \varepsilon >0\), we denote
$$\begin{aligned} U_{x_{0}, \varepsilon }(x):=\varepsilon ^{\frac{p-N}{p(p-1)}}U \biggl(\frac{x-x_{0}}{\varepsilon }\biggr). \end{aligned}$$
(2.4)
From [30], we know that the \((N+2)\)-dimensional manifold of the following form consists of almost all functions which can achieve the best Sobolev constant S:
$$\begin{aligned} \overline{\mathcal{M}}:=\bigl\{ cU_{x_{0}, \varepsilon }, c\in \mathbb{R} \setminus \{{0}\}, x_{0}\in \mathbb{R}^{N}, \varepsilon >0\bigr\} . \end{aligned}$$
Set
$$\begin{aligned} {S_{\alpha,\beta }}: = \mathop{\inf } _{(u,v) \in X \backslash \{ (0,0)\} } \frac{{\int _{\mathbb{R}^{N}} {({{ \vert {\nabla u} \vert }^{p}} + {{ \vert {\nabla v} \vert }^{p}})\,dx} }}{{{{(\int _{\mathbb{R}^{N}} {({{ \vert u \vert }^{{p^{*}}}} + {{ \vert v \vert }^{{p^{*}}}} + {{ \vert u \vert }^{\alpha }}{{ \vert v \vert }^{\beta }})\,dx} )}^{\frac{p}{{{p^{*}}}}}}}}. \end{aligned}$$
(2.5)
Supposing \((sU_{x_{0}, \varepsilon }, bU_{x_{0}, \varepsilon })\) is a positive solution corresponding to the problem (1.1), we have
$$\begin{aligned} \bigl({p^{*}} + \alpha {\tau ^{\beta }}\bigr){s^{{p^{*}} - 2}} = {p^{*}} = \bigl({p^{*}} { \tau ^{{p^{*}} - 2}} + \beta {\tau ^{\beta - 2}}\bigr){s^{{p^{*}} - 2}},\qquad b = \tau s. \end{aligned}$$
Therefore,
$$\begin{aligned} {p^{*}} + \alpha {\tau ^{\beta }} - \beta {\tau ^{\beta - 2}} - {p^{*}} { \tau ^{{p^{*}} - 2}} = 0 \end{aligned}$$
(2.6)
and
$$\begin{aligned} {s^{{p^{*}} - 2}} = \frac{{{p^{*}}}}{{{p^{*}} + \alpha {\tau ^{\beta }}}}. \end{aligned}$$
(2.7)
Subsequently, we find that if all of the least energy solutions of (1.1) have the form \((sU_{x_{0}, \varepsilon }, bU_{x_{0}, \varepsilon })\), where \(s,b\) are constants, then we know
$$\begin{aligned} {S_{\alpha,\beta }} = \frac{{1 + {\tau ^{p}}}}{{{{{{(1 + }}{\tau ^{\beta }} + {\tau ^{{p^{*}}}})}^{\frac{p}{{{p^{*}}}}}}}}S. \end{aligned}$$
Setting \(f(\tau ):= \frac{1+\tau ^{p}}{(1+\tau ^{\beta }+\tau ^{p^{*}})^{\frac{p}{p^{*}}}}\), it is easy to see that
$$\begin{aligned} f({\tau _{\min }}): = \mathop{\min } _{{\tau _{\min }} \ge 0} f( \tau ) \le 1. \end{aligned}$$
By a similar method as that in [29], under the assumptions of Theorem 1.1, we derive that the following form is suitable for the least energy solutions of (1.1), which can be found in [29]:
Lemma 2.1
(The special case \(p=2\); see [29])
Assume \(\Omega =\mathbb{R}^{N}, N\geq p^{2}\), and parameters \(\alpha >1, \beta >1\), satisfy \(\alpha + \beta =p^{*}\). If \((u_{0}, v_{0})\) is the least energy solution of (1.6), then \(\exists ! \tau _{\min }>0\), satisfying \(f({\tau _{\min }}): = \mathop{\min } _{\tau \ge 0} f(\tau ) = \mathop{\min } _{\tau \ge 0} \frac{{1 + {\tau ^{2}}}}{{{{(1 + {\tau ^{\beta }} + {\tau ^{{2^{*}}}})}^{\frac{2}{{{2^{*}}}}}}}} < 1\), and
$$\begin{aligned} (u_{0}, v_{0})=(sU_{x_{0}, \varepsilon }, bU_{x_{0}, \varepsilon }), \end{aligned}$$
where \(b=s\tau _{\min }, x_{0}\in \mathbb{R}^{N}, \varepsilon >0\), and s satisfies (2.7).
Lemma 2.2
For \(\forall x_{0}\in \mathbb{R}^{N}\), denote \(\widetilde{D}=B_{1}(x_{0})\cap \{x_{N}>h(x')\}\), and let \(B_{1}(x_{0})\) be the unit ball centered at \(x_{0}\), with \(h(x')\) being a \(C^{1}\) function defined in the set \(\{x'\in \mathbb{R}^{N-1}: |x'-x_{0}'|<1\}\) where \((x_{0})_{N}=h((x_{0})_{1},\dots,(x_{0})_{N-1})\), and ∇h vanishes at \(x_{0}'=((x_{0})_{1},\dots,(x_{0})_{N-1})\) (that is, \(\nabla h=0\) at this point). If \(u,v\in W^{1,p}(B_{1}(x_{0})), \operatorname{supp} u\subset B_{1}(x_{0}), \operatorname{supp} v\subset B_{1}(x_{0})\), then there is a constant \(C(\delta )\), which depends on δ, such that the following conclusions hold:
(1) When \(h\equiv 0\),
$$\begin{aligned} \int _{\widetilde{D}}\bigl( \vert \nabla u \vert ^{p}+ \vert \nabla v \vert ^{p}\bigr)\,dx\geq 2^{- \frac{p}{N}}S_{\alpha, \beta } \biggl( \int _{\widetilde{D}}\bigl( \vert u \vert ^{p^{*}}+ \vert v \vert ^{p^{*}}+ \vert u \vert ^{ \alpha } \vert v \vert ^{\beta }\bigr)\,dx\biggr)^{\frac{p}{p^{*}}}. \end{aligned}$$
(2.8)
(2) For \(\forall \varepsilon >0, \exists \delta (\varepsilon )\), \(\nabla h\leq \delta \), such that
$$\begin{aligned} \int _{\widetilde{D}}\bigl( \vert \nabla u \vert ^{p}+ \vert \nabla v \vert ^{p}\bigr)\,dx\geq \bigl(2^{- \frac{p}{N}}S_{\alpha, \beta }- \varepsilon \bigr) \biggl( \int _{\widetilde{D}}\bigl( \vert u \vert ^{p^{*}}+ \vert v \vert ^{p^{*}}+ \vert u \vert ^{ \alpha } \vert v \vert ^{\beta }\bigr)\,dx\biggr)^{\frac{p}{p^{*}}}. \end{aligned}$$
(2.9)
Proof
(1) We estimate
$$\begin{aligned} \int _{\widetilde{D}}\bigl( \vert \nabla u \vert ^{p}+ \vert \nabla v \vert ^{p}\bigr)\,dx &= \frac{1}{2} \int _{B_{1}(x_{0})}\bigl( \vert \nabla u \vert ^{p}+ \vert \nabla u \vert ^{p}\bigr)\,dx \\ &\geq \frac{1}{2}S_{\alpha, \beta }\biggl( \int _{B_{1}(x_{0})}\bigl( \vert u \vert ^{p^{*}}+ \vert v \vert ^{p^{*}}+ \vert u \vert ^{ \alpha } \vert v \vert ^{\beta }\bigr)\,dx\biggr)^{\frac{p}{p^{*}}} \\ &=2^{-\frac{p}{N}}S_{\alpha, \beta }\biggl( \int _{\widetilde{D}}\bigl( \vert u \vert ^{p^{*}}+ \vert v \vert ^{p^{*}}+ \vert u \vert ^{ \alpha } \vert v \vert ^{\beta }\bigr)\,dx\biggr)^{\frac{p}{p^{*}}}. \end{aligned}$$
(2) By a translation transformation, letting \(y'=x', y_{n}=x_{n}-h(x')>0\), we straighten the boundary of D̃, and complete the proof. □
First of all, we have the following result which is necessary to verify that the equations (1.1) satisfy the mountain pass lemma:
Lemma 2.3
When the assumptions of Theorem 1.1are satisfied, the following conclusions hold:
(1) If \(r>0, \delta >0\), \(\|(u, v)\|=r\), then \(J(u,v)\geq \delta >0\);
(2) \(\exists (u_{0}, v_{0})\in X\), such that \(\|(u_{0}, v_{0})\|>r, J(u_{0},v_{0})<0 \) hold.
Proof
(1) Because \(W^{1,p}(\Omega )\hookrightarrow L^{p^{*}}(\Omega )\) is continuous, by Hölder inequality, we have:
$$\begin{aligned} \begin{aligned}& \biggl( \int _{\Omega } u^{p^{*}}\,dx \biggr)^{\frac{1}{p^{*}}} \leq C_{1} \Vert u \Vert _{W^{1,p}}, \\ &\biggl( \int _{\Omega } v^{p^{*}}\,dx \biggr)^{\frac{1}{p^{*}}}\leq C_{2} \Vert v \Vert _{W^{1,p}}, \\ &\int _{\Omega } \vert u \vert ^{\alpha } \vert v \vert ^{\beta }\,dx \leq \biggl( \int _{\Omega } u^{p^{*}}\,dx \biggr)^{\frac{\alpha }{p^{*}}} \biggl( \int _{\Omega } v^{p^{*}}\,dx \biggr)^{ \frac{\beta }{p^{*}}} \\ &\phantom{\int _{\Omega } \vert u \vert ^{\alpha } \vert v \vert ^{\beta }\,dx }\leq C_{3} \biggl( \int _{\Omega } \vert \nabla u \vert ^{p}+ \vert u \vert ^{p}\,dx \biggr)^{ \frac{\alpha }{2}} \biggl( \int _{\Omega } \vert \nabla v \vert ^{p}+ \vert v \vert ^{p}\,dx \biggr)^{ \frac{\beta }{2}} \\ &\phantom{\int _{\Omega } \vert u \vert ^{\alpha } \vert v \vert ^{\beta }\,dx }=C_{3} \Vert u \Vert _{W^{1,p}}^{\alpha } \Vert v \Vert _{W^{1,p}}^{\beta }. \end{aligned} \end{aligned}$$
(2.10)
Hence,
$$\begin{aligned} \begin{aligned} J(u,v)={}&\frac{1}{p} \int _{\Omega } \bigl( \vert \nabla u \vert ^{p}+ \lambda _{1} u^{p}+ \vert \nabla v \vert ^{p}+\lambda _{2}v^{p}\bigr)\,dx \\ &{} -\frac{1}{p^{*}} \int _{\Omega } \bigl(\bigl(u^{+}\bigr)^{p^{*}}+ \bigl(v^{+}\bigr)^{p^{*}}+\bigl(u^{+} \bigr)^{ \alpha }\bigl(v^{+}\bigr)^{\beta }\bigr)\,dx \\ \geq{}& \frac{1}{p}C_{4} \bigl\Vert (u,v) \bigr\Vert ^{p}-\frac{1}{p^{*}}C_{1} \Vert u \Vert _{W^{1,p}( \Omega )}^{p^{*}} -\frac{1}{p^{*}}C_{2} \Vert v \Vert _{W^{1,p}(\Omega )}^{p^{*}} \\ &{} -\frac{1}{p^{*}}C_{3} \Vert u \Vert _{W^{1,p}(\Omega )}^{\alpha } \Vert v \Vert _{W^{1,p}( \Omega )}^{\beta } \\ \geq{}& \frac{1}{p}C_{4} \bigl\Vert (u,v) \bigr\Vert ^{p}-\frac{1}{p^{*}}C_{5} \bigl\Vert (u, v) \bigr\Vert ^{p^{*}} \end{aligned} \end{aligned}$$
As a consequence of \(\frac{1}{p}-\frac{1}{p^{*}}=\frac{1}{N}>0\), choosing \(\|(u, v)\|=r\) small enough, we obtain the desired result \(J(u, v)\geq \delta >0\).
(2) Now we estimate
$$\begin{aligned} \begin{aligned} J(tu,tv)={}&\frac{t^{p}}{p} \int _{\Omega } \bigl( \vert \nabla u \vert ^{p}+ \lambda _{1} u^{p}+ \vert \nabla v \vert ^{p}+\lambda _{2}v^{p}\bigr)\,dx \\ &{}-\frac{t^{p^{*}}}{p^{*}} \int _{\Omega } \bigl(\bigl(u^{+}\bigr)^{p^{*}}+ \bigl(v^{+}\bigr)^{p^{*}}+\bigl(u^{+} \bigr)^{ \alpha }\bigl(v^{+}\bigr)^{\beta }\bigr)\,dx \\ \geq{}& \frac{t^{p}}{p}C_{6} \bigl\Vert (u,v) \bigr\Vert ^{p}-\frac{t^{p^{*}}}{p^{*}}C_{7} \bigl\Vert (u, v) \bigr\Vert ^{p^{*}}. \end{aligned} \end{aligned}$$
Hence, letting t tend to infinity,
$$\begin{aligned} \mathop{\lim } _{t \to \infty } J(tu,tv) = - \infty, \end{aligned}$$
so we can choose \(t_{0}\in \mathbb{R}\), such that \(\|(t_{0}u, t_{0}v)\|>r, J(t_{0}u, t_{0}v)<0\), and then, setting \((u_{0}, v_{0})=(t_{0}u, t_{0}v)\), the proof is completed. □
Next, we introduce the following lemmas (see [31]), which are useful in verifying that the energy functional is lower than the threshold, and whose functional corresponds to the Palais–Smale sequence.
Lemma 2.4
If {\(u_{n}\)} is a bounded sequence in \(L^{p}(\Omega )\) such that \(u_{n} \rightarrow u\) a.e., then
$$\begin{aligned} \mathop{\lim } _{n \to \infty } \biggl({ \int _{\Omega }{ \vert {{u_{n}}} \vert } ^{p}}\,dx - { \int _{\Omega }{ \vert {{u_{n}} - u} \vert } ^{p}}\,dx \biggr) = { \int _{\Omega }{ \vert u \vert } ^{p}}\,dx. \end{aligned}$$
(2.11)
Lemma 2.5
If
$$\begin{aligned} \begin{aligned} &u_{n}\rightharpoonup u \quad\textit{in } W^{1,p}( \Omega ), \\ &v_{n}\rightharpoonup v \quad\textit{in } W^{1,p}(\Omega ) \end{aligned} \end{aligned}$$
then
$$\begin{aligned} \mathop{\lim } _{n \to \infty } \biggl({ \int _{\Omega }{ \vert {{u_{n}}} \vert } ^{\alpha }} { \vert {{v_{n}}} \vert ^{\beta }}\,dx - { \int _{\Omega }{ \vert {{u_{n}} - u} \vert } ^{\alpha }} { \vert {{v_{n}} - v} \vert ^{\beta }}\,dx \biggr) = { \int _{\Omega }{ \vert u \vert } ^{\alpha }} { \vert v \vert ^{\beta }}\,dx. \end{aligned}$$
(2.12)
Lemma 2.6
(see [32])
Assume {\({u_{n}}\)} is the \((PS)\) sequence corresponding to \(J_{\lambda, p}\), and \(u_{n}\rightharpoonup u\), then it has finitely many points in Ω, we denote them by \(x_{1},x_{2}, \dots, x_{k} \in \Omega \), which make \(u_{n}\rightarrow u\) hold in \(W_{\mathrm{loc}}^{1,p}(\Omega \setminus \{x_{1},x_{2}, \dots, x_{k} \})\).
Denote
$$\begin{aligned} {S_{{\lambda _{1}}, \lambda _{2}}}: = \mathop{\inf } _{u,v \in {W^{1,p}}(\Omega )\backslash \{ 0\} } \frac{{\int _{\Omega }{({{ \vert {\nabla u} \vert }^{p}} + {\lambda _{1}}{u^{p}} + {{ \vert {\nabla v} \vert }^{p}} + {\lambda _{2}}{v^{p}})\,dx} }}{{{{[\int _{\Omega }{({u^{{p^{*}}}} + {v^{{p^{*}}}} + {u^{\alpha }}{v^{\beta }})\,dx} ]}^{\frac{p}{{{p^{*}}}}}}}}. \end{aligned}$$
Lemma 2.7
Suppose {\({(u_{n}, v_{n})}\)} is a sequence in X, satisfying
$$\begin{aligned} J(u_{n}, v_{n})\rightarrow c,\qquad J(u_{n}, v_{n})\rightarrow 0, \end{aligned}$$
(2.13)
and
$$\begin{aligned} c < \min \biggl\{ \frac{1}{N}S_{{\lambda _{1}},{\lambda _{2}}}^{ \frac{N}{p}}, \frac{1}{{2N}}S_{\alpha,\beta }^{\frac{N}{p}} \biggr\} . \end{aligned}$$
(2.14)
Then the system (1.1) has a solution \((u, v)\in X\) and \(J(u, v)\leq c\).
Proof
By Lemma 2.3, \(\exists {(u_{n}, v_{n})}\) which satisfies (2.13). Let
$$\begin{aligned} &c = \mathop{\inf } _{\gamma \in \Gamma } \mathop{\sup } _{t \in [0,1]} J\bigl(\gamma (t)\bigr), \\ &\Gamma = \bigl\{ \gamma \in C\bigl([0,1],X\bigr):\gamma (0) = (0,0),J\bigl( \gamma (1)\bigr) < 0 \bigr\} . \end{aligned}$$
As a consequence, we can see that
$$\begin{aligned} \begin{aligned}& c + o(1) = J({u_{n}},{v_{n}}) \\ &\phantom{c + o(1) }=\frac{1}{p} \int _{\Omega }{{{ \vert {\nabla {u_{n}}} \vert }^{p}} + {\lambda _{1}}u_{n}^{p} + {{ \vert {\nabla {v_{n}}} \vert }^{p}} + { \lambda _{2}}v_{n}^{p}} \,dx \\ &\phantom{c + o(1) =}{} -\frac{1}{{{p^{*}}}} \int _{\Omega }{{{\bigl(u_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(v_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(u_{n}^{+} \bigr)}^{\alpha }} {{\bigl(v_{n}^{+} \bigr)}^{\beta }}} \,dx, \end{aligned} \end{aligned}$$
(2.15)
$$\begin{aligned} \begin{aligned} &o(1) \bigl\Vert \bigl(\|\varphi, \psi ) \bigr\Vert \bigr) \\ &\quad= \bigl\langle J'({u_{n}},{v_{n}}), ( \varphi, \psi ) \bigr\rangle \\ &\quad= \int _{\Omega }{{{ \vert {\nabla {u_{n}}} \vert }^{p - 2}}\nabla {u_{n}} \nabla \varphi + {\lambda _{1}}u_{n}^{p - 1}\varphi + {{ \vert { \nabla {v_{n}}} \vert }^{p - 2}}\nabla {v_{n}}\nabla \psi + { \lambda _{2}}v_{n}^{p - 1}\psi } \,dx \\ &\qquad{} - \int _{\Omega }{{{\bigl(u_{n}^{+} \bigr)}^{{p^{*}} - 1}}\varphi + {{\bigl(v_{n}^{+} \bigr)}^{{p^{*}} - 1}}\psi } \,dx - \frac{\alpha }{{{p^{*}}}} \int _{\Omega }{{{\bigl(u_{n}^{+} \bigr)}^{\alpha - 1}} {{\bigl(v_{n}^{+} \bigr)}^{\beta }} \varphi } \,dx \\ &\qquad{} - \frac{\beta }{{{p^{*}}}} \int _{\Omega }{{{\bigl(u_{n}^{+} \bigr)}^{\alpha }} {{\bigl(v_{n}^{+} \bigr)}^{\beta - 1}}\psi } \,dx, \quad\forall ( \varphi, \psi ) \in X. \end{aligned} \end{aligned}$$
(2.16)
Setting \((\varphi, \psi )=(u_{n}, v_{n})\) and substituting it into equation (2.16), we then have
$$\begin{aligned} \begin{aligned} o(1) \bigl\Vert \bigl(\|\varphi, \psi ) \bigr\Vert \bigr) ={}& \bigl\langle J'({u_{n}},{v_{n}}), (\varphi, \psi ) \bigr\rangle \\ ={}& \int _{\Omega }{{{ \vert {\nabla {u_{n}}} \vert }^{p}} + {\lambda _{1}}u_{n}^{p} + {{ \vert {\nabla {v_{n}}} \vert }^{p}} + {\lambda _{2}}v_{n}^{p}} \,dx \\ &{} - \int _{\Omega }{{{\bigl(u_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(v_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(u_{n}^{+} \bigr)}^{\alpha }} {{\bigl(v_{n}^{+} \bigr)}^{\beta }}} \,dx. \end{aligned} \end{aligned}$$
(2.17)
Combining (2.17) with (2.15), we obtain
$$\begin{aligned} \begin{aligned} c + o(1) & \geq J({u_{n}},{v_{n}}) - \frac{1}{{{p^{*}}}} \bigl\langle J'({u_{n}},{v_{n}}),({u_{n}},{v_{n}}) \bigr\rangle \\ &=\biggl(\frac{1}{p}-\frac{1}{p^{*}}\biggr) \int _{\Omega }{{{ \vert {\nabla {u_{n}}} \vert }^{p}} + {\lambda _{1}}u_{n}^{p} + {{ \vert {\nabla {v_{n}}} \vert }^{p}} + {\lambda _{2}}v_{n}^{p}} \,dx \\ &\geq \frac{1}{N}C_{4} \bigl\Vert (u_{n}, v_{n}) \bigr\Vert . \end{aligned} \end{aligned}$$
This implies that \((u_{n}, v_{n})\) is bounded, hence, ∃C such that \(\|(u_{n}, v_{n})\|\leq C\), \(\|u_{n}\|_{W^{1,p}(\Omega )}\leq C\), and \(\|v_{n}\|_{W^{1,p}(\Omega )}\leq C\). Moreover, there exist \(u\in W^{1,p}(\Omega ), v\in W^{1,p}(\Omega )\) such that \((u_{n}, v_{n}) \rightharpoonup (u,v) \) in X. By Lemma 2.6, we have the following results:
$$\begin{aligned} &u_{n}\rightharpoonup u,\qquad v_{n} \rightharpoonup v \quad\text{in } W^{1,p}(\Omega ), \\ &u_{n}\rightarrow u,\qquad v_{n}\rightarrow v \quad\text{in } W_{ \mathrm{loc}}^{1,p}\bigl(\Omega \setminus \{x_{1},x_{2}, \dots, x_{k}\}\bigr), \\ &u_{n}\rightarrow u,\qquad v_{n}\rightarrow v \quad\text{in } L^{p}( \Omega ), \\ &u_{n}\rightharpoonup u,\qquad v_{n}\rightharpoonup v\quad \text{in } L^{p^{*}}(\Omega ), \\ &\nabla u_{n}\rightarrow \nabla u,\qquad \nabla v_{n} \rightarrow \nabla v \quad\text{in } L_{\mathrm{loc}}^{p}\bigl(\Omega \setminus \{x_{1},x_{2}, \dots, x_{k}\}\bigr), \\ &u_{n}\rightharpoonup u,\qquad v_{n}\rightharpoonup v\quad \text{a.e. in }\Omega. \end{aligned}$$
By calculating the limit of both sides of (2.16), we obtain
$$\begin{aligned} \begin{aligned} 0={}&\mathop{\lim } _{n \to \infty } \bigl\langle J'({u_{n}},{v_{n}}),( \varphi,\psi ) \bigr\rangle \\ ={}& \int _{\Omega }{{{ \vert {\nabla u} \vert }^{p - 2}} \nabla u \nabla \varphi + {\lambda _{1}} {u^{p - 1}}\varphi + {{ \vert { \nabla v} \vert }^{p - 2}}\nabla v\nabla \psi + {\lambda _{2}} {v^{p - 1}}\psi } \,dx \\ &{} - \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{{p^{*}} - 1}} \varphi + {{\bigl({v^{+} }\bigr)}^{{p^{*}} - 1}}\psi } \,dx - \frac{\alpha }{{{p^{*}}}} \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{\alpha - 1}} {{\bigl({v^{+} }\bigr)}^{\beta }}\varphi } \,dx \\ & {}- \frac{\beta }{{{p^{*}}}} \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{\alpha }} {{\bigl({v^{+} }\bigr)}^{\beta - 1}}\psi } \,dx \quad\forall (\varphi, \psi )\in X. \end{aligned} \end{aligned}$$
(2.18)
By Definition 2.1, we know that \((u,v)\) is a weak solution of (1.1). We need to verify that \((u,v)\) is a nontrivial solution in the following. Let \(\widetilde{\omega _{n}}=u_{n}-u, \widetilde{\sigma _{n}}=v_{n}-v\), then by Lemmas 2.4, 2.5, and 2.6, we see that
$$\begin{aligned} &\int _{\Omega }{{{\bigl(u_{n}^{+} \bigr)}^{{p^{*}}}}} \,dx = \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{{p^{*}}}}} \,dx + \int _{\Omega }{{{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}}} \,dx + o(1), \\ &\int _{\Omega }{{{\bigl(v_{n}^{+} \bigr)}^{{p^{*}}}}} \,dx = \int _{\Omega }{{{\bigl({v^{+} }\bigr)}^{{p^{*}}}}} \,dx + \int _{\Omega }{{{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}}} \,dx + o(1), \\ &\int _{\Omega }{{{\bigl(u_{n}^{+} \bigr)}^{\alpha }} {{\bigl(v_{n}^{+} \bigr)}^{\beta }}} \,dx = \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{\alpha }} {{\bigl({v^{+} }\bigr)}^{\beta }}} \,dx + \int _{\Omega }{{{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{\alpha }} {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{\beta }}} \,dx + o(1), \\ &\int _{\Omega }{{ \vert {\nabla {u_{n}}} \vert }^{p}}\,dx = \int _{\Omega }{{{ \vert \nabla u \vert }^{p}}} \,dx + \int _{\Omega }{{{ \vert \nabla {{ \tilde{\omega }}_{n}} \vert }^{p}}} \,dx + o(1), \\ &\int _{\Omega }{{ \vert {\nabla {v_{n}}} \vert }^{p}}\,dx = \int _{\Omega }{{{ \vert \nabla v \vert }^{p}}} \,dx + \int _{\Omega }{{{ \vert \nabla {{ \tilde{\sigma }}_{n}} \vert }^{p}}} \,dx + o(1), \\ &\int _{\Omega }{{{({{\tilde{\omega }}_{n}})}^{p}}} = o(1),\qquad \int _{\Omega }{{{({{\tilde{\sigma }}_{n}})}^{p}}} = o(1). \end{aligned}$$
By (2.15) and the fact \(J(u_{n}, v_{n})\rightarrow c, J'(u_{n}, v_{n})\rightarrow 0\), where \({(u_{n}, v_{n})}\subset X\), we have
$$\begin{aligned} \begin{aligned} &c + o(1)=J(u,v) + \frac{1}{p} \int _{\Omega }{{{ \vert {\nabla {{ \tilde{\omega }}_{n}}} \vert }^{p}}\,dx + {{ \vert {\nabla {{ \tilde{\sigma }}_{n}}} \vert }^{p}}} \,dx \\ &\phantom{c + o(1)=} {}-\frac{1}{{{p^{*}}}} \int _{\Omega }{{{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{\alpha }} {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{\beta }}} \,dx, \end{aligned} \end{aligned}$$
(2.19)
$$\begin{aligned} &\int _{\Omega } \vert \nabla \tilde{\omega }_{n} \vert ^{p}\,dx + \vert \nabla \tilde{\sigma }_{n} \vert ^{p} \,dx - \int _{\Omega }{{{\bigl( \tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{\alpha }} {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{\beta }}} \,dx = 0. \end{aligned}$$
(2.20)
Hence,
$$\begin{aligned} \begin{aligned} \mathop{\lim } _{n \to \infty } \int _{\Omega }{{{ \vert {\nabla {{\tilde{\omega }}_{n}}} \vert }^{p}}\,dx + {{ \vert { \nabla {{ \tilde{\sigma }}_{n}}} \vert }^{p}}} \,dx &= \mathop{\lim } _{n \to \infty } \int _{\Omega }{{{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{\alpha }} {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{\beta }}} \,dx \\ &= l\geq 0. \end{aligned} \end{aligned}$$
For \(\forall \varepsilon >0\), where ε is a small suitable positive constant, we denote by \((\phi _{i})_{i=1}^{m}\) a partition of unity on Ω̄, satisfying ∀i, \(\operatorname{diam}(\operatorname{supp} \phi _{i})\leq \rho \), where \(\operatorname{diam}(D)\) means the diameter of the domain D. Therefore, by Lemma 2.2, provided that ρ is small enough, we have
$$\begin{aligned} \begin{aligned} & \int _{\Omega }{{{ \bigl\vert {\nabla (u{\phi _{i}})} \bigr\vert }^{p}} + {{ \bigl\vert {\nabla (v{\phi _{i}})} \bigr\vert }^{p}}\,dx} \\ &\quad \ge \bigl({2^{ - \frac{p}{N}}} {S_{\alpha,\beta }} - \varepsilon \bigr){\biggl( \int _{\Omega }{{{ \bigl\vert {(u{\phi _{i}})} \bigr\vert }^{{p^{*}}}} + {{ \bigl\vert {(v{\phi _{i}})} \bigr\vert }^{{p^{*}}}} + {{ \bigl\vert {(u{\phi _{i}})} \bigr\vert }^{\alpha }} {{ \bigl\vert {(v{\phi _{i}})} \bigr\vert }^{\beta }}\,dx} \biggr)^{ \frac{p}{{{p^{*}}}}}}. \end{aligned} \end{aligned}$$
Employing Young inequality with ε, for \(\forall 1\leq i\leq m, u,v \in W^{1,p}(\Omega )\), we have
$$\begin{aligned} \begin{aligned} & \biggl( \int _{\Omega }{{{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{\alpha }} {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{\beta }}} \,dx \biggr)^{ \frac{p}{{{p^{*}}}}} \\ &\quad\leq \Biggl( \int _{\Omega }{\sum_{i = 1}^{m} {{\phi _{i}}^{ \frac{{{p^{*}}}}{p}}\bigl[{{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl( \tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{\alpha }} {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{\beta }}\bigr]} } \,dx \Biggr)^{ \frac{p}{{{p^{*}}}}} \\ &\quad\leq \sum_{i = 1}^{m} {{{ \biggl( \int _{\Omega }{{{\bigl({\phi _{i}}^{ \frac{1}{p}} \tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl({ \phi _{i}}^{ \frac{1}{p}}\tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl({\phi _{i}}^{ \frac{1}{p}}\tilde{ \omega }_{n}^{+} \bigr)}^{\alpha }} {{\bigl({\phi _{i}}^{ \frac{1}{p}}\tilde{\sigma }_{n}^{+} \bigr)}^{\beta }}} \,dx \biggr)}^{ \frac{p}{{{p^{*}}}}}}} \\ &\quad\leq {\bigl({2^{ - \frac{p}{N}}} {S_{\alpha,\beta }} - \varepsilon \bigr)^{ - 1}} \sum_{i = 1}^{m} { \int _{\Omega }{{{ \bigl\vert {\nabla \bigl({{ \tilde{\omega }}_{n}} {\phi _{i}}^{\frac{1}{p}}\bigr)} \bigr\vert }^{p}} + {{ \bigl\vert {\nabla \bigl({{\tilde{\sigma }}_{n}} {\phi _{i}}^{\frac{1}{p}}\bigr)} \bigr\vert }^{p}}\,dx} } \\ &\quad\leq {\bigl({2^{ - \frac{p}{N}}} {S_{\alpha,\beta }} - \varepsilon \bigr)^{ - 1}}\\ &\qquad{}\times\biggl[(1 + \varepsilon ) \int _{\Omega }{{{ \vert {\nabla {{\tilde{\omega }}_{n}}} \vert }^{p}} + {{ \vert {\nabla {{\tilde{ \sigma }}_{n}}} \vert }^{p}}} \,dx + C(\varepsilon ) \int _{\Omega }{{{({{\tilde{\omega }}_{n}})}^{p}} + {{({{ \tilde{\sigma }}_{n}})}^{p}}\,dx\biggr]}. \end{aligned} \end{aligned}$$
(2.21)
Hence, \(l\geq ( \frac{2^{-\frac{p}{N}}S_{\alpha, \beta }-\varepsilon }{1+\varepsilon } )l^{\frac{p}{p^{*}}}\), and we discuss the following two cases:
If \(l=0\), then it is easy to see that \((u_{n}, v_{n})\rightarrow (u, v)\) in X, so \((u,v)\not \equiv (0,0)\). If \(u\not \equiv 0, v\equiv 0\), then
$$\begin{aligned} c = \frac{1}{p} \int _{\Omega }{{{ \vert {\nabla u} \vert }^{p}} + { \lambda _{1}} {u^{{p}}}} \,dx - \frac{1}{{{p^{*}}}} \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{{p^{*}}}}} \,dx \ge \frac{1}{N}S_{{\lambda _{1}}, 0}^{\frac{N}{p}}. \end{aligned}$$
Similarly, if \(u\equiv 0, v\not \equiv 0\), then
$$\begin{aligned} c = \frac{1}{p} \int _{\Omega }{{{ \vert {\nabla v} \vert }^{p}} + { \lambda _{2}} {v^{{p}}}} \,dx - \frac{1}{{{p^{*}}}} \int _{\Omega }{{{\bigl({v^{+} }\bigr)}^{{p^{*}}}}} \,dx \ge \frac{1}{N}S_{0, {\lambda _{2}}}^{\frac{N}{p}}. \end{aligned}$$
This contradicts \(c < \min \{ \frac{1}{N}S_{{\lambda _{1}},{\lambda _{2}}}^{ \frac{N}{p}},\frac{1}{{2N}}S_{\alpha,\beta }^{\frac{N}{p}} \} \). Thus, the solutions of system (1.1) are not semitrivial solutions.
If \(l\neq 0\), that is, \(l\geq \frac{1}{2}S_{\alpha, \beta }^{\frac{N}{p}}\), then we only need to verify \(u\not \equiv 0, v\not \equiv 0\).
(i) Assume one of \(u,v\) equals zero. It is natural to suppose \(u\not \equiv 0, v\equiv 0\).
From \(\langle J'({u_{n}},{v_{n}}),({u_{n}},{v_{n}}) \rangle \rightarrow 0\), it is easy to obtain
$$\begin{aligned} \int _{\Omega }{{{ \vert {\nabla u} \vert }^{p}} + { \lambda _{1}} {u^{{p}}}} \,dx - \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{{p^{*}}}}} \,dx = 0. \end{aligned}$$
Due to (2.19) and \(l\geq \frac{1}{2}S_{\alpha, \beta }^{\frac{N}{p}}\), we see
$$\begin{aligned} \begin{aligned} c&= \biggl(\frac{1}{p} - \frac{1}{{{p^{*}}}} \biggr)l + \frac{1}{p} \int _{\Omega }{{{ \vert {\nabla u} \vert }^{p}} + { \lambda _{1}} {u^{p}}} \,dx - \frac{1}{{{p^{*}}}} \int _{\Omega }{{{\bigl({u^{+} }\bigr)}^{{p^{*}}}}} \,dx \\ &= \biggl(\frac{1}{p} - \frac{1}{{{p^{*}}}} \biggr)l + \biggl( \frac{1}{p} - \frac{1}{{{p^{*}}}} \biggr) \int _{\Omega }{{{ \vert {\nabla u} \vert }^{p}} + { \lambda _{1}} {u^{p}}} \,dx \\ &\geq \biggl(\frac{1}{p} - \frac{1}{{{p^{*}}}} \biggr)l\geq \frac{1}{2N}S_{ \alpha, \beta }^{\frac{N}{p}}. \end{aligned} \end{aligned}$$
Similarly, this is a contradiction to \(c < \min \{ \frac{1}{N}S_{{\lambda _{1}},{\lambda _{2}}}^{ \frac{N}{p}},\frac{1}{{2N}}S_{\alpha,\beta }^{\frac{N}{p}} \} \).
(ii) \(u\equiv 0, v\equiv 0\).
Now we can find \(c = (\frac{1}{p} - \frac{1}{{{p^{*}}}})c\geq \frac{1}{2N}S_{\alpha, \beta }^{\frac{N}{p}}\), which is also a contradiction. In summary, \((u, v)\) is a nontrivial solution of system (1.1).
Combining (2.19) and (2.20), the following result can be obtained:
$$\begin{aligned} J(u,v) = c - \frac{1}{N} \int _{\Omega }{{{\bigl(\tilde{\omega }_{n}^{+} \bigr)}^{{p^{*}}}} + {{\bigl(\tilde{\sigma }_{n}^{+} \bigr)}^{{p^{*}}}}} \,dx + o(1), \end{aligned}$$
implying \(J(u, v)\leq c\). □