Existence of solutions for a general quasilinear elliptic system via perturbation method

Abstract

In this paper, we consider the following quasilinear elliptic system:

$$\{\begin{array}{cc}-{\sum }_{i,j=1}^N{D}_j(a_{ij}(x,u)D_{i}u)+\frac{1}{2}{\sum }_{i,j=1}^N{D}_s{a}_{ij}(x,u)D_{i}u{D}_{j}u=\frac{2\alpha }{\alpha +\beta }{|u|}^{\alpha -2}{|v|}^{\beta }u,\hfill & x\in \mathrm{\Omega },\hfill \\ -{\sum }_{i,j=1}^N{D}_j(b_{ij}(x,v)D_{i}v)+\frac{1}{2}{\sum }_{i,j=1}^N{D}_s{b}_{ij}(x,v)D_{i}v{D}_{j}v=\frac{2\beta }{\alpha +\beta }{|u|}^{\alpha }{|v|}^{\beta -2}v,\hfill & x\in \mathrm{\Omega },\hfill \\ u=0,v=0,\hfill & x\in \partial \mathrm{\Omega },\hfill \end{array}$$

where $D_{i}u=\frac{\partial u}{\partial x_i}$, $D_s{a}_{ij}(x,u)=\frac{\partial }{\partial u}{a}_{ij}(x,u)$, $D_s{b}_{ij}(x,v)=\frac{\partial }{\partial v}{b}_{ij}(x,v)$, $\alpha >2$, $\beta >2$, $\alpha +\beta <2\cdot 2^{\ast }$, $2^{\ast }=\frac{2N}{N-2}$ is the critical Sobolev exponent and $\mathrm{\Omega }\subset {\mathbb{R}}^N$ ($N\ge 3$) is a bounded smooth domain. By using the perturbation method, we establish the existence of both positive and negative solutions for this system.

MSC:35J60, 35B33.

1 Introduction

Let us consider the following quasilinear elliptic system:

$$\{\begin{array}{cc}-{\sum }_{i,j=1}^N{D}_j(a_{ij}(x,u)D_{i}u)+\frac{1}{2}{\sum }_{i,j=1}^N{D}_s{a}_{ij}(x,u)D_{i}u{D}_{j}u=\frac{2\alpha }{\alpha +\beta }{|u|}^{\alpha -2}{|v|}^{\beta }u,\hfill & x\in \mathrm{\Omega },\hfill \\ -{\sum }_{i,j=1}^N{D}_j(b_{ij}(x,v)D_{i}v)+\frac{1}{2}{\sum }_{i,j=1}^N{D}_s{b}_{ij}(x,v)D_{i}v{D}_{j}v=\frac{2\beta }{\alpha +\beta }{|u|}^{\alpha }{|v|}^{\beta -2}v,\hfill & x\in \mathrm{\Omega },\hfill \\ u=0,v=0,\hfill & x\in \partial \mathrm{\Omega },\hfill \end{array}$$

(1.1)

where $D_{i}u=\frac{\partial u}{\partial x_i}$, $D_s{a}_{ij}(x,u)=\frac{\partial }{\partial u}{a}_{ij}(x,u)$, $D_s{b}_{ij}(x,v)=\frac{\partial }{\partial v}{b}_{ij}(x,v)$, $\alpha >2$, $\beta >2$, $\alpha +\beta <2\cdot 2^{\ast }$, $2^{\ast }=\frac{2N}{N-2}$ is the critical Sobolev exponent and $\mathrm{\Omega }\subset {\mathbb{R}}^N$ ($N\ge 3$) is a bounded smooth domain. This system includes the following special class of system with $a_{ij}(x,u)=(1+u^2){\delta }_{ij}$, $b_{ij}(x,v)=(1+v^2){\delta }_{ij}$, i.e.,

$$\{\begin{array}{cc}-\mathrm{△}u-\frac{1}{2}u\mathrm{△}(u^2)=\frac{2\alpha }{\alpha +\beta }{|u|}^{\alpha -2}{|v|}^{\beta }u,\hfill & x\in \mathrm{\Omega },\hfill \\ -\mathrm{△}v-\frac{1}{2}v\mathrm{△}(v^2)=\frac{2\beta }{\alpha +\beta }{|u|}^{\alpha }{|v|}^{\beta -2}v,\hfill & x\in \mathrm{\Omega },\hfill \\ u=0,v=0,\hfill & x\in \partial \mathrm{\Omega },\hfill \end{array}$$

which is referred to as the so-called modified nonlinear Schrödinger system.

Our assumptions on the functions $a_{ij}$ and $b_{ij}$ are as follows.

• (A₁) The functions $a_{ij}\in C^1(\overline{\mathrm{\Omega }}\times \mathbb{R},\mathbb{R})$, $b_{ij}\in C^1(\overline{\mathrm{\Omega }}\times \mathbb{R},\mathbb{R})$, $a_{ij}=a_{ji}$, $b_{ij}=b_{ji}$, $i,j=1,2,\dots ,N$.

• (A₂) There exist constants $a_0$, $a_1$, $b_0$, $b_1$ satisfying $a_1\ge a_0>0$, $b_1\ge b_0>0$, $(\alpha +\beta -2)a_0>2a_1$ and $(\alpha +\beta -2)b_0>2b_1$ such that

  $$\begin{array}{c}{a}_0(1+s^2){|\xi |}^2\le \sum _{i,j=1}^N{a}_{ij}(x,s){\xi }_i{\xi }_j\le a_1(1+s^2){|\xi |}^2,\hfill \\ b_0(1+s^2){|\xi |}^2\le \sum _{i,j=1}^N{b}_{ij}(x,s){\xi }_i{\xi }_j\le b_1(1+s^2){|\xi |}^2\hfill \end{array}$$

  for $x\in \overline{\mathrm{\Omega }}$, $\xi \in {\mathbb{R}}^N$, $s\in \mathbb{R}$.

• (A₃)

  $$\begin{array}{c}0\le \sum _{i,j=1}^N{D}_s{a}_{ij}(x,s)s{\xi }_i{\xi }_j\le 2\sum _{i,j=1}^N{a}_{ij}(x,s){\xi }_i{\xi }_j,\hfill \\ 0\le \sum _{i,j=1}^N{D}_s{b}_{ij}(x,s)s{\xi }_i{\xi }_j\le 2\sum _{i,j=1}^N{b}_{ij}(x,s){\xi }_i{\xi }_j\hfill \end{array}$$

  for $x\in \overline{\mathrm{\Omega }}$, $\xi \in {\mathbb{R}}^N$, $s\in \mathbb{R}$.

In recent years, much attention has been devoted to the quasilinear Schrödinger equation of the following form:

$$-\mathrm{△}u+\lambda V(x)u-k\mathrm{△}\left(u^2\right)u={|u|}^{p-2}u,x\in {\mathbb{R}}^N.$$

(1.2)

See, for example, [1] where Poppenberg et al. proved the existence of a positive ground state solution by using a constrained minimization argument. Using a change of variables, Liu et al. [2] used an Orlicz space to prove the existence of a soliton solution for equation (1.2) via the mountain pass theorem. Colin and Jeanjean [3] also made use of a change of variables but worked in the Sobolev space $H^1({\mathbb{R}}^N)$. They proved the existence of a positive solution for equation (1.2) from the classical results given by Berestycki and Lions [4]. Liu et al. [5] established the existence of both one-sign and nodal ground states of soliton-type solutions for equation (1.2) by the Nehari method. By using the Nehari manifold method and the concentration compactness principle (see [6]) in the Orlicz space, Guo and Tang [7] considered the following quasilinear Schrödinger system:

$$\{\begin{array}{cc}-\mathrm{△}u+(\lambda a(x)+1)u-\frac{1}{2}(\mathrm{△}{|u|}^2)u=\frac{2\alpha }{\alpha +\beta }{|u|}^{\alpha -2}{|v|}^{\beta }u,\hfill & x\in {\mathbb{R}}^N,\hfill \\ -\mathrm{△}u+(\lambda b(x)+1)u-\frac{1}{2}(\mathrm{△}{|u|}^2)u=\frac{2\beta }{\alpha +\beta }{|u|}^{\alpha }{|v|}^{\beta -2}v,\hfill & x\in {\mathbb{R}}^N,\hfill \\ u(x)\to 0,v(x)\to 0,\hfill & |x|\to \mathrm{\infty },\hfill \end{array}$$

(1.3)

with $a(x)\ge 0$, $b(x)\ge 0$ having a potential well and $\alpha >2$, $\beta >2$, $\alpha +\beta <2\cdot 2^{\ast }$, and they proved the existence of a ground state solution for system (1.3) which localizes near the potential well $\text{int }{a}^{-1}(0)$ for λ large enough. Guo and Tang [8] considered also ground state solutions of the single quasilinear Schrödinger equation corresponding to system (1.3) by the same methods and obtained similar results. In particular, by the perturbation method, Liu et al. [9] considered the existence and multiplicity of solutions for the following quasilinear equation of the form

$$\{\begin{array}{cc}{\sum }_{i,j=1}^N{D}_j(a_{ij}(x,u)D_{i}u)-\frac{1}{2}{\sum }_{i,j=1}^N{D}_s{a}_{ij}(x,u)D_{i}u{D}_{j}u+f(x,u)=0,\hfill & x\in \mathrm{\Omega },\hfill \\ u=0,\hfill & x\in \partial \mathrm{\Omega }\hfill \end{array}$$

(1.4)

under suitable assumptions.

It is worth pointing out that the existence of one-bump or multi-bump bound state solutions for the related semilinear Schrödinger equation (1.2) for $k=0$ has been extensively studied. One can see Bartsch and Wang [10], Ambrosetti et al. [11], Ambrosetti et al. [12], Byeon and Wang [13], Cingolani and Lazzo [14], Cingolani and Nolasco [15], Del Pino and Felmer [16, 17], Floer and Weinstein [18], Oh [19, 20] and the references therein.

Motivated by the single equation (1.4), the purpose of this paper is to study the existence of both positive and negative solutions for the coupled quasilinear system (1.1). We mainly follow the idea of Liu et al. [9] to perturb the functional and obtain our main results. We point out that the procedure to system (1.1) is not trivial at all. Since the appearance of the quasilinear terms ${\sum }_{i,j=1}^N{D}_j(a_{ij}(x,u)D_{i}u)-\frac{1}{2}{\sum }_{i,j=1}^N{D}_s{a}_{ij}(x,u)D_{i}u{D}_{j}u$ and ${\sum }_{i,j=1}^N{D}_j(b_{ij}(x,v)D_{i}v)-\frac{1}{2}{\sum }_{i,j=1}^N{D}_s{b}_{ij}(x,v)D_{i}v{D}_{j}v$, we need more delicate estimates.

The paper is organized as follows. In Section 2, we introduce a perturbation of the functional and give our main results (Theorem 2.1 and Theorem 2.2). In Section 3, we verify the Palais-Smale condition for the perturbed functional. Section 4 is devoted to some asymptotic behavior of the sequences $\{(u_n,v_n)\}\subset W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })$ and $\{{\mu }_n\}\subset (0,1]$ satisfying some conditions. Finally, our main results will be proved in Section 5.

Throughout this paper, we will use the same C to denote various generic positive constants, and we will use $o(1)$ to denote quantities that tend to 0.

2 Perturbation of the functional and main results

In order to obtain the desired existence of solutions for system (1.1), in this section, we introduce a perturbation of the functional and give our main results.

The weak form of system (1.1) is

$$\begin{array}{r}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_j\phi +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{a}_{ij}(x,u)D_{i}u{D}_{j}u\phi \\ +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_j\psi +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{b}_{ij}(x,v)D_{i}v{D}_{j}v\psi \\ -\frac{2\alpha }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha -2}{|v|}^{\beta }u\phi -\frac{2\beta }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha }{|v|}^{\beta -2}v\psi =0\end{array}$$

(2.1)

for all $(\phi ,\psi )\in C_0^{\mathrm{\infty }}(\mathrm{\Omega })\times C_0^{\mathrm{\infty }}(\mathrm{\Omega })$, which is formally the variational formulation of the following functional:

$$I_0(u,v)=\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_{j}u+\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_{j}v-\frac{2}{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha }{|v|}^{\beta }.$$

(2.2)

We may define the derivative of $I_0$ at $(u,v)$ in the direction of $(\phi ,\psi )\in C_0^{\mathrm{\infty }}(\mathrm{\Omega })\times C_0^{\mathrm{\infty }}(\mathrm{\Omega })$ as follows:

$$\begin{array}{rcl}〈I_0^{\prime }(u,v),(\phi ,\psi )〉& =& {\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_j\phi +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{a}_{ij}(x,u)D_{i}u{D}_{j}u\phi \\ +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_j\psi +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{b}_{ij}(x,v)D_{i}v{D}_{j}v\psi \\ -\frac{2\alpha }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha -2}{|v|}^{\beta }u\phi -\frac{2\beta }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha }{|v|}^{\beta -2}v\psi .\end{array}$$

(2.3)

We call $(u,v)$ a critical point of $I_0$ if $(u,v)\in W_0^{1,2}(\mathrm{\Omega })\times W_0^{1,2}(\mathrm{\Omega })$, ${\int }_{\mathrm{\Omega }}{u}^2{|\mathrm{\nabla }u|}^2<\mathrm{\infty }$, ${\int }_{\mathrm{\Omega }}{v}^2{|\mathrm{\nabla }v|}^2<\mathrm{\infty }$ and $〈I_0^{\prime }(u,v),(\phi ,\psi )〉=0$ for all $(\phi ,\psi )\in C_0^{\mathrm{\infty }}(\mathrm{\Omega })\times C_0^{\mathrm{\infty }}(\mathrm{\Omega })$. That is, $(u,v)$ is a weak solution for system (1.1).

When we consider system (1.1) by using the classical critical point theory, we encounter the difficulties due to the lack of an appropriate working space. In general, it seems that there is no suitable space in which the variational functional $I_0$ possesses both smoothness and compactness properties. For smoothness, one would need to work in a space smaller than $W_0^{1,2}(\mathrm{\Omega })$ to control the term involving the quasilinear term in system (1.1), but it seems impossible to obtain bounds for ${(\mathit{PS})}_c$ sequence in this setting. Several ideas and approaches, such as minimizations [1, 21], the Nehari method [5] and change of variables [2, 3], have been used in recent years to overcome the difficulties. In this paper, we consider the perturbed functional

$$\begin{array}{rcl}{I}_{\mu }(u,v)& =& \frac{1}{4}\mu {\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }u|}^4+{|\mathrm{\nabla }v|}^4)+I_0(u,v)\\ =& \frac{1}{4}\mu {\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }u|}^4+{|\mathrm{\nabla }v|}^4)+\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_{j}u\\ +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_{j}v-\frac{2}{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha }{|v|}^{\beta },\end{array}$$

(2.4)

where $\mu \in (0,1]$ is a parameter. Then it is easy to see that $I_{\mu }$ is a $C^1$-functional on $W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })$. We can define also the derivative of $I_{\mu }$ at $(u,v)$ in the direction of $(\phi ,\psi )$ as follows:

$$\begin{array}{rcl}〈I_{\mu }^{\prime }(u,v),(\phi ,\psi )〉& =& \mu {\int }_{\mathrm{\Omega }}{|\mathrm{\nabla }u|}^2\mathrm{\nabla }u\mathrm{\nabla }\phi +\mu {\int }_{\mathrm{\Omega }}{|\mathrm{\nabla }v|}^2\mathrm{\nabla }v\mathrm{\nabla }\psi \\ +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_j\phi +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{a}_{ij}(x,u)D_{i}u{D}_{j}u\phi \\ +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_j\psi +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{b}_{ij}(x,v)D_{i}v{D}_{j}v\psi \\ -\frac{2\alpha }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha -2}{|v|}^{\beta }u\phi -\frac{2\beta }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha }{|v|}^{\beta -2}v\psi \end{array}$$

(2.5)

for all $(\phi ,\psi )\in C_0^{\mathrm{\infty }}(\mathrm{\Omega })\times C_0^{\mathrm{\infty }}(\mathrm{\Omega })$. The idea of this paper is to obtain the existence of the critical points of $I_{\mu }$ for $\mu >0$ small and establish suitable estimates for the critical points as $\mu \to 0$ so that we may pass to the limit to get the solutions for the original system (1.1).

Our main results are as follows.

Theorem 2.1 Assume that (A₁)-(A₃) hold, $\alpha >2$, $\beta >2$ and $\alpha +\beta <2\cdot 2^{\ast }$. Let ${\mu }_n\to 0$ and let $\{(u_n,v_n)\}\subset W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })$ be a sequence of critical points of $I_{{\mu }_n}$ satisfying $I_{{\mu }_n}^{\prime }(u_n,v_n)=0$ and $I_{{\mu }_n}(u_n,v_n)\le C$ for some C independent of n. Then, up to a subsequence,

$$\begin{array}{c}{u}_n\to u,v_n\to v\mathit{\text{in }}{W}_0^{1,2}(\mathrm{\Omega }),\hfill \\ u_n\mathrm{\nabla }{u}_n\to u\mathrm{\nabla }u,v_n\mathrm{\nabla }{v}_n\to v\mathrm{\nabla }v\mathit{\text{in }}{L}^2(\mathrm{\Omega }),\hfill \\ {\mu }_n{\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }{u}_n|}^4+{|\mathrm{\nabla }{v}_n|}^4)\to 0,\hfill \\ I_{{\mu }_n}^{\prime }(u_n,v_n)\to I_0^{\prime }(u,v)\hfill \end{array}$$

as $n\to \mathrm{\infty }$, and $(u,v)$ is a critical point of $I_0$.

Theorem 2.2 Assume that (A₁)-(A₃) hold, $\alpha >2$, $\beta >2$ and $\alpha +\beta <2\cdot 2^{\ast }$. Then $I_{\mu }$ has a positive critical point $(u_{\mu },v_{\mu })$ and a negative critical point $({\tilde{u}}_{\mu },{\tilde{v}}_{\mu })$, and $(u_{\mu },v_{\mu })$ (resp.,$({\tilde{u}}_{\mu },{\tilde{v}}_{\mu })$) converges to a positive (resp., negative) solution for system (1.1) as $\mu \to 0$.

Notation We denote by $\parallel \cdot \parallel$ the norm of $W_0^{1,4}(\mathrm{\Omega })$ and by ${|\cdot |}_s$ the norm of $L^s(\mathrm{\Omega })$ ($1\le s<+\mathrm{\infty }$).

3 Compactness of the perturbed functional

In this section, we verify the Palais-Smale condition (${(\mathit{PS})}_c$ condition in short) for the perturbed functional $I_{\mu }(u,v)$. We have the following proposition.

Proposition 3.1 For $\mu >0$ fixed, the functional $I_{\mu }(u,v)$ satisfies ${(\mathit{PS})}_c$ condition for all $c\in \mathbb{R}$. That is, any sequence $\{(u_n,v_n)\}\subset W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })$ satisfying, for $c\in \mathbb{R}$,

$$I_{\mu }(u_n,v_n)\to c,$$

(3.1)

$$I_{\mu }^{\prime }(u_n,v_n)\to 0\mathit{\text{strongly in }}{(W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega }))}^{\ast }$$

(3.2)

has a strongly convergent subsequence in $W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })$, where ${(W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega }))}^{\ast }$ is the dual space of $W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })$.

To give the proof of Proposition 3.1, we need the following lemma firstly.

Lemma 3.2 Suppose that a sequence $\{(u_n,v_n)\}\subset W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })$ satisfies (3.1) and (3.2). Then

$$\underset{n\to \mathrm{\infty }}{lim\hspace{0.17em}sup}{\parallel (u_n,v_n)\parallel }^4\le {(\frac{1}{4}-\frac{1}{\alpha +\beta })}^{-1}{\mu }^{-1}c.$$

Proof It follows from (3.1) and (3.2) that

$$\begin{array}{r}c+o(1)-\frac{1}{\alpha +\beta }o(1)\parallel (u_n,v_n)\parallel \\ =I_{\mu }(u_n,v_n)-\frac{1}{\alpha +\beta }〈I_{\mu }^{\prime }(u_n,v_n),(u_n,v_n)〉\\ =(\frac{1}{4}-\frac{1}{\alpha +\beta })\mu {\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }{u}_n|}^4+{|\mathrm{\nabla }{v}_n|}^4)\\ +(\frac{1}{2}-\frac{1}{\alpha +\beta }){\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_{j}u+(\frac{1}{2}-\frac{1}{\alpha +\beta }){\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_{j}v\\ -\frac{1}{2(\alpha +\beta )}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{a}_{ij}(x,u)D_{i}u{D}_{j}u-\frac{1}{2(\alpha +\beta )}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{b}_{ij}(x,v)D_{i}v{D}_{j}v\\ \ge (\frac{1}{4}-\frac{1}{\alpha +\beta })\mu {\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }u|}^4+{|\mathrm{\nabla }v|}^4)+\frac{(\alpha +\beta -2)a_0-2a_1}{2(\alpha +\beta )}{\int }_{\mathrm{\Omega }}(1+u_n^2){|\mathrm{\nabla }{u}_n|}^2\\ +\frac{(\alpha +\beta -2)b_0-2b_1}{2(\alpha +\beta )}{\int }_{\mathrm{\Omega }}(1+v_n^2){|\mathrm{\nabla }{v}_n|}^2\\ \ge (\frac{1}{4}-\frac{1}{\alpha +\beta })\mu {\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }u|}^4+{|\mathrm{\nabla }v|}^4).\end{array}$$

Thus we have

$$\underset{n\to \mathrm{\infty }}{lim\hspace{0.17em}sup}{\parallel (u_n,v_n)\parallel }^4\le {(\frac{1}{4}-\frac{1}{\alpha +\beta })}^{-1}{\mu }^{-1}c.$$

This completes the proof of Lemma 3.2. □

Now we give the proof of Proposition 3.1.

Proof of Proposition 3.1 From Lemma 3.2 , we know that $\{(u_n,v_n)\}$ is bounded in $W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })$. So there exists a subsequence of $\{(u_n,v_n)\}$, still denoted by $\{(u_n,v_n)\}$, such that

$$\begin{array}{c}(u_n,v_n)\rightharpoonup (u,v)\text{weakly in }{W}_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })\text{ as }n\to \mathrm{\infty },\hfill \\ u_n\to u,v_n\to v\text{strongly in }{L}^s(\mathrm{\Omega })\text{ as }n\to \mathrm{\infty }\text{ for any }2<s<2\cdot 2^{\ast }.\hfill \end{array}$$

Now we prove that $(u_n,v_n)\to (u,v)$ in $W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })$. In (2.5), choosing $(\phi ,\psi )=(u_n-u_m,v_n-v_m)$, we have

$$\begin{array}{r}o(1)\parallel (u_n-u_m,v_n-v_m)\parallel \\ =〈I_{\mu }^{\prime }(u_n,v_n)-I_{\mu }^{\prime }(u_m,v_m),(u_n-u_m,v_n-v_m)〉\\ =\mu {\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }{u}_n|}^2\mathrm{\nabla }{u}_n-{|\mathrm{\nabla }{u}_m|}^2\mathrm{\nabla }{u}_m)(\mathrm{\nabla }{u}_n-\mathrm{\nabla }{u}_m)\\ +\mu {\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }{v}_n|}^2\mathrm{\nabla }{v}_n-{|\mathrm{\nabla }{v}_m|}^2\mathrm{\nabla }{v}_m)(\mathrm{\nabla }{v}_n-\mathrm{\nabla }{v}_m)\\ +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N(a_{ij}(x,u_n)D_i{u}_n-a_{ij}(x,u_m)D_i{u}_m)(D_j{u}_n-D_j{u}_m)\\ +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N(b_{ij}(x,v_n)D_i{v}_n-b_{ij}(x,v_m)D_i{v}_m)(D_j{v}_n-D_j{v}_m)\\ +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N(D_s{a}_{ij}(x,u_n)D_i{u}_n{D}_j{u}_n-D_s{a}_{ij}(x,u_m)D_i{u}_m{D}_j{u}_m)(u_n-u_m)\\ +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N(D_s{b}_{ij}(x,v_n)D_i{v}_n{D}_j{v}_n-D_s{b}_{ij}(x,v_m)D_i{v}_m{D}_j{v}_m)(v_n-v_m)\\ -\frac{2\alpha }{\alpha +\beta }{\int }_{\mathrm{\Omega }}({|u_n|}^{\alpha -2}{|v_n|}^{\beta }{u}_n-{|u_m|}^{\alpha -2}{|v_m|}^{\beta }{u}_m)(u_n-u_m)\\ -\frac{2\beta }{\alpha +\beta }{\int }_{\mathrm{\Omega }}({|u_n|}^{\alpha }{|v_n|}^{\beta -2}{v}_n-{|u_m|}^{\alpha }{|v_m|}^{\beta -2}{v}_m)(v_n-v_m).\end{array}$$

(3.3)

We may estimate the terms involved as follows:

$$\begin{array}{r}\mu {\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }{u}_n|}^2\mathrm{\nabla }{u}_n-{|\mathrm{\nabla }{u}_m|}^2\mathrm{\nabla }{u}_m)(\mathrm{\nabla }{u}_n-\mathrm{\nabla }{u}_m)\ge \frac{1}{4}\mu {\int }_{\mathrm{\Omega }}{|\mathrm{\nabla }{u}_n-\mathrm{\nabla }{u}_m|}^4,\\ \mu {\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }{v}_n|}^2\mathrm{\nabla }{v}_n-{|\mathrm{\nabla }{v}_m|}^2\mathrm{\nabla }{v}_m)(\mathrm{\nabla }{v}_n-\mathrm{\nabla }{v}_m)\ge \frac{1}{4}\mu {\int }_{\mathrm{\Omega }}{|\mathrm{\nabla }{v}_n-\mathrm{\nabla }{v}_m|}^4,\\ {\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N(a_{ij}(x,u_n)D_i{u}_n-a_{ij}(x,u_m)D_i{u}_m)(D_j{u}_n-D_j{u}_m)\\ ={\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u_n)D_i(u_n-u_m)D_j(u_n-u_m)\\ +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N(a_{ij}(x,u_n)-a_{ij}(x,u_m))D_i{u}_m{D}_j(u_n-u_m)\\ \ge {\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{a}_{ij}(x,t{u}_n+(1-t)u_m)(u_n-u_m)D_i{u}_m{D}_j(u_n-u_m)\\ \ge -C{|u_n-u_m|}_4\parallel u_m\parallel (\parallel u_n\parallel +\parallel u_m\parallel )\\ \to 0\text{as }m,n\to \mathrm{\infty }\text{ for some }t\in (0,1),\\ {\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N(b_{ij}(x,v_n)D_i{v}_n-b_{ij}(x,v_m)D_i{v}_m)(D_j{v}_n-D_j{v}_m)\\ ={\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v_n)D_i(v_n-v_m)D_j(v_n-v_m)\\ +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N(b_{ij}(x,v_n)-b_{ij}(x,v_m))D_i{v}_m{D}_j(v_n-v_m)\\ \ge {\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{b}_{ij}(x,\tau v_n+(1-\tau )v_m)(v_n-v_m)D_i{v}_m{D}_j(v_n-v_m)\\ \ge -C{|v_n-v_m|}_4\parallel v_m\parallel (\parallel v_n\parallel +\parallel v_m\parallel )\\ \to 0\text{as }m,n\to \mathrm{\infty }\text{ for some }\tau \in (0,1),\\ \frac{1}{2}|{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N(D_s{a}_{ij}(x,u_n)D_i{u}_n{D}_j{u}_n-D_s{a}_{ij}(x,u_m)D_i{u}_m{D}_j{u}_m)(u_n-u_m)|\\ \le \frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N|D_s{a}_{ij}(x,u_n)D_i{u}_n{D}_j{u}_n(u_n-u_m)|\\ +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N|D_s{a}_{ij}(x,u_n)D_i{u}_n{D}_j{u}_n(u_n-u_m)|\\ \le C({\parallel u_n\parallel }^2+{\parallel u_m\parallel }^2){|u_n-u_m|}_4\\ \to 0\text{as }m,n\to \mathrm{\infty },\\ \frac{1}{2}|{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N(D_s{b}_{ij}(x,v_n)D_i{v}_n{D}_j{v}_n-D_s{b}_{ij}(x,v_m)D_i{v}_m{D}_j{v}_m)(v_n-v_m)|\\ \le \frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N|D_s{b}_{ij}(x,v_n)D_i{v}_n{D}_j{v}_n(v_n-v_m)|\\ +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N|D_s{b}_{ij}(x,v_n)D_i{v}_n{D}_j{v}_n(v_n-v_m)|\\ \le C({\parallel v_n\parallel }^2+{\parallel v_m\parallel }^2){|v_n-v_m|}_4\\ \to 0\text{as }m,n\to \mathrm{\infty },\\ \frac{2\alpha }{\alpha +\beta }|{\int }_{\mathrm{\Omega }}({|u_n|}^{\alpha -2}{|v_n|}^{\beta }{u}_n-{|u_m|}^{\alpha -2}{|v_m|}^{\beta }{u}_m)(u_n-u_m)|\\ \le \frac{2\alpha }{\alpha +\beta }{\int }_{\mathrm{\Omega }}({|u_n|}^{\alpha -1}{|v_n|}^{\beta }+{|u_m|}^{\alpha -1}{|v_m|}^{\beta })|u_n-u_m|\\ \le \frac{2\alpha }{\alpha +\beta }({|u_n|}_{\alpha +\beta }^{\alpha -1}{|v_n|}_{\alpha +\beta }^{\beta }+{|u_m|}_{\alpha +\beta }^{\alpha -1}{|v_m|}_{\alpha +\beta }^{\beta }){|u_n-u_m|}_{\alpha +\beta }\\ \to 0\text{as }m,n\to \mathrm{\infty },\\ \frac{2\beta }{\alpha +\beta }|{\int }_{\mathrm{\Omega }}({|u_n|}^{\alpha }{|v_n|}^{\beta -2}{v}_n-{|u_m|}^{\alpha }{|v_m|}^{\beta -2}{v}_m)(v_n-v_m)|\\ \le \frac{2\beta }{\alpha +\beta }{\int }_{\mathrm{\Omega }}({|u_n|}^{\alpha }{|v_n|}^{\beta -1}+{|u_m|}^{\alpha }{|v_m|}^{\beta -1})|v_n-v_m|\\ \le \frac{2\beta }{\alpha +\beta }({|u_n|}_{\alpha +\beta }^{\alpha }{|v_n|}_{\alpha +\beta }^{\beta -1}+{|u_m|}_{\alpha +\beta }^{\alpha }{|v_m|}_{\alpha +\beta }^{\beta -1}){|v_n-v_m|}_{\alpha +\beta }\\ \to 0\text{as }m,n\to \mathrm{\infty }.\end{array}$$

Returning to (3.3), we have

$$\frac{1}{4}\mu {\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }{u}_n-\mathrm{\nabla }{u}_m|}^4+{|\mathrm{\nabla }{v}_n-\mathrm{\nabla }{v}_m|}^4)\le o(1)\parallel (u_n-u_m,v_n-v_m)\parallel +o(1),$$

which implies that $\parallel (u_n-u_m,v_n-v_m)\parallel \to 0$, i.e., $(u_n,u_m)\to (u,v)$ in $W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })$. This completes the proof of Proposition 3.1. □

4 Some asymptotic behavior

Proposition 3.1 enables us to apply minimax argument to the functional $I_{\mu }(u,v)$. In this section, we also study the behavior of the sequences $\{(u_n,v_n)\}\subset W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })$ and $\{{\mu }_n\}\subset (0,1]$ satisfying

$${\mu }_n\to 0,$$

(4.1)

$$I_{{\mu }_n}(u_n,v_n)\to c,$$

(4.2)

$${\parallel I_{{\mu }_n}^{\prime }(u_n,v_n)\parallel }^{\ast }\to 0.$$

(4.3)

The following proposition is the key of this section.

Proposition 4.1 Assume that the sequences $\{(u_n,v_n)\}\subset W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })$ and $\{{\mu }_n\}\subset (0,1]$ satisfy (4.1)-(4.3). Then, after extracting a sequence, still denoted by n, we have

$$\begin{array}{c}(u_n,v_n)\rightharpoonup (u,v)\mathit{\text{in }}{W}_0^{1,2}(\mathrm{\Omega })\times W_0^{1,2}(\mathrm{\Omega }),\hfill \\ (u_n\mathrm{\nabla }{u}_n,v_n\mathrm{\nabla }{v}_n)\rightharpoonup (u\mathrm{\nabla }u,v\mathrm{\nabla }v)\mathit{\text{in }}{L}^2(\mathrm{\Omega })\times L^2(\mathrm{\Omega })\hfill \end{array}$$

and

$$(u_n(x),v_n(x))\to (u(x),v(x))\mathit{\text{a.e. }}x\in \mathrm{\Omega }$$

as $n\to \mathrm{\infty }$.

Proof Similar to the proof of Lemma 3.2, by (4.1)-(4.3), we have

$$\begin{array}{rcl}C& \ge & I_{{\mu }_n}(u_n,v_n)-\frac{1}{\alpha +\beta }〈I_{{\mu }_n}^{\prime }(u_n,v_n),(u_n,v_n)〉\\ \ge & (\frac{1}{4}-\frac{1}{\alpha +\beta }){\mu }_n{\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }{u}_n|}^4+{|\mathrm{\nabla }{v}_n|}^4)\\ +\frac{(\alpha +\beta -2)a_0-2a_1}{2(\alpha +\beta )}{\int }_{\mathrm{\Omega }}(1+u_n^2){|\mathrm{\nabla }{u}_n|}^2\\ +\frac{(\alpha +\beta -2)b_0-2b_1}{2(\alpha +\beta )}{\int }_{\mathrm{\Omega }}(1+v_n^2){|\mathrm{\nabla }{v}_n|}^2.\end{array}$$

(4.4)

Thus

$${\mu }_n{\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }{u}_n|}^4+{|\mathrm{\nabla }{v}_n|}^4)+{\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }{u}_n|}^2+{|\mathrm{\nabla }{v}_n|}^2)+{\int }_{\mathrm{\Omega }}(u_n^2{|\mathrm{\nabla }{u}_n|}^2+v_n^2{|\mathrm{\nabla }{v}_n|}^2)\le C$$

(4.5)

for some C independent of n. Then, up to a subsequence, we have

$$\begin{array}{c}(u_n,v_n)\rightharpoonup (u,v)\text{in }{W}_0^{1,2}(\mathrm{\Omega })\times W_0^{1,2}(\mathrm{\Omega }),\hfill \\ (u_n\mathrm{\nabla }{u}_n,v_n\mathrm{\nabla }{v}_n)\rightharpoonup (u\mathrm{\nabla }u,v\mathrm{\nabla }v)\text{in }{L}^2(\mathrm{\Omega })\times L^2(\mathrm{\Omega })\hfill \end{array}$$

and

$$(u_n(x),v_n(x))\to (u(x),v(x))\text{a.e. }x\in \mathrm{\Omega }$$

as $n\to \mathrm{\infty }$. This completes the proof of Proposition 4.1. □

5 Proof of main results

In this section, we give the proof of our main results. Firstly, we prove Theorem 2.1.

Proof of Theorem 2.1 Note that $(u_n,v_n)$ satisfies the following equation:

$$\begin{array}{r}{\mu }_n{\int }_{\mathrm{\Omega }}{|\mathrm{\nabla }{u}_n|}^2\mathrm{\nabla }{u}_n\mathrm{\nabla }\phi +{\mu }_n{\int }_{\mathrm{\Omega }}{|\mathrm{\nabla }{v}_n|}^2\mathrm{\nabla }{v}_n\mathrm{\nabla }\psi \\ +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u_n)D_i{u}_n{D}_j\phi +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{a}_{ij}(x,u_n)D_i{u}_n{D}_j{u}_n\phi \\ +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v_n)D_i{v}_n{D}_j\psi +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{b}_{ij}(x,v_n)D_i{v}_n{D}_j{v}_n\psi \\ -\frac{2\alpha }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u_n|}^{\alpha -2}{|v_n|}^{\beta }{u}_n\phi -\frac{2\beta }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u_n|}^{\alpha }{|v_n|}^{\beta -2}{v}_n\psi =0\end{array}$$

(5.1)

for all $(\phi ,\psi )\in W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })$. Since

$${\left({\int }_{\mathrm{\Omega }}{|u_n|}^{\frac{4N}{N-2}}\right)}^{\frac{N-2}{N}}\le C{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u_n)D_i{u}_n{D}_j{u}_n\le C$$

and

$${\left({\int }_{\mathrm{\Omega }}{|v_n|}^{\frac{4N}{N-2}}\right)}^{\frac{N-2}{N}}\le C{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v_n)D_i{v}_n{D}_j{v}_n\le C.$$

By Moser’s iteration, we have

$${|u_n|}_{L^{\mathrm{\infty }}}\le C,{|v_n|}_{L^{\mathrm{\infty }}}\le C.$$

(5.2)

Hence

$${|u|}_{L^{\mathrm{\infty }}}\le C,{|v|}_{L^{\mathrm{\infty }}}\le C$$

(5.3)

for some C independent of n. To show that $(u,v)$ is a critical point of $I_0$, we use some arguments in [22, 23] (see more references therein). In (5.1), we choose $\phi =\xi exp(-M{u}_n)$, $\psi =\eta exp(-M{v}_n)$, where $\xi \in C_0^{\mathrm{\infty }}(\mathrm{\Omega })$, $\xi \ge 0$, $\eta \in C_0^{\mathrm{\infty }}(\mathrm{\Omega })$, $\eta \ge 0$ and $M>0$ is a constant. Substituting $(\phi ,\psi )$ into (5.1), we have

$$\begin{array}{rcl}0& =& {\mu }_n{\int }_{\mathrm{\Omega }}{|\mathrm{\nabla }{u}_n|}^2\mathrm{\nabla }{u}_n(\mathrm{\nabla }\xi exp(-M{u}_n)-\xi \mathrm{\nabla }{u}_{n}exp(-M{u}_n))\\ +{\mu }_n{\int }_{\mathrm{\Omega }}{|\mathrm{\nabla }{v}_n|}^2\mathrm{\nabla }{v}_n(\mathrm{\nabla }\eta exp(-M{v}_n)-\eta \mathrm{\nabla }{v}_{n}exp(-M{v}_n))\\ +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u_n)D_i{u}_n(D_j\xi exp(-M{u}_n)-M\xi D_j{u}_{n}exp(-M{u}_n))\\ +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v_n)D_i{v}_n(D_j\eta exp(-M{v}_n)-M\eta D_j{v}_{n}exp(-M{v}_n))\\ +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{a}_{ij}(x,u_n)D_i{u}_n{D}_j{u}_n\xi exp(-M{u}_n)\\ +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{b}_{ij}(x,v_n)D_i{v}_n{D}_j{v}_n\eta exp(-M{v}_n)\\ -\frac{2\alpha }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u_n|}^{\alpha -2}{|v_n|}^{\beta }{u}_n\xi exp(-M{u}_n)-\frac{2\beta }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u_n|}^{\alpha }{|v_n|}^{\beta -2}{v}_n\eta exp(-M{v}_n)\\ \le & {\mu }_n{\int }_{\mathrm{\Omega }}{|\mathrm{\nabla }{u}_n|}^2\mathrm{\nabla }{u}_n\mathrm{\nabla }\xi exp(-M{u}_n)+{\mu }_n{\int }_{\mathrm{\Omega }}{|\mathrm{\nabla }{v}_n|}^2\mathrm{\nabla }{v}_n\mathrm{\nabla }\eta exp(-M{v}_n)\\ +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u_n)D_i{u}_n{D}_j\xi exp(-M{u}_n)+{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v_n)D_i{v}_n{D}_j\eta exp(-M{v}_n)\\ -{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N(M{a}_{ij}(x,u_n)-\frac{1}{2}{D}_s{a}_{ij}(x,u_n))D_i{u}_n{D}_j{u}_n\xi exp(-M{u}_n)\\ -{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N(M{b}_{ij}(x,v_n)-\frac{1}{2}{D}_s{b}_{ij}(x,v_n))D_i{v}_n{D}_j{v}_n\eta exp(-M{v}_n)\\ -\frac{2\alpha }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u_n|}^{\alpha -2}{|v_n|}^{\beta }{u}_n\xi exp(-M{u}_n)\\ -\frac{2\beta }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u_n|}^{\alpha }{|v_n|}^{\beta -2}{v}_n\eta exp(-M{v}_n).\end{array}$$

(5.4)

Note that $M{a}_{ij}(x,u_n)-\frac{1}{2}{D}_s{a}_{ij}(x,u_n)$, $M{b}_{ij}(x,v_n)-\frac{1}{2}{D}_s{b}_{ij}(x,v_n)$ are positive for M large enough. By Fatou’s lemma, the weak convergence of $\{(u_n,v_n)\}$ and the fact that ${\mu }_n{\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }{u}_n|}^4+{|\mathrm{\nabla }{v}_n|}^4)$ is bounded, we have

$$\begin{array}{rcl}0& \le & {\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_j\xi exp(-Mu)+{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_j\eta exp(-Mv)\\ -{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N(M{a}_{ij}(x,u)-\frac{1}{2}{D}_s{a}_{ij}(x,u))D_{i}u{D}_{j}u\xi exp(-Mu)\\ -{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N(M{b}_{ij}(x,v)-\frac{1}{2}{D}_s{b}_{ij}(x,v))D_{i}v{D}_{j}v\eta exp(-Mv)\\ -\frac{2\alpha }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha -2}{|v|}^{\beta }u\xi exp(-Mu)-\frac{2\beta }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha }{|v|}^{\beta -2}v\eta exp(-Mv)\\ =& {\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_j(\xi exp(-Mu))+{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,u)D_{i}v{D}_j(\eta exp(-Mv))\\ +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{a}_{ij}(x,u)D_{i}u{D}_{j}u\xi exp(-Mu)\\ +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{b}_{ij}(x,v)D_{i}v{D}_{j}v\eta exp(-Mv)\\ -\frac{2\alpha }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha -2}{|v|}^{\beta }u\xi exp(-Mu)-\frac{2\beta }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha }{|v|}^{\beta -2}v\eta exp(-Mv).\end{array}$$

(5.5)

Let $(\chi ,\omega )\ge (0,0)$, $(\chi ,\omega )\in C_0^{\mathrm{\infty }}(\mathrm{\Omega })\times C_0^{\mathrm{\infty }}(\mathrm{\Omega })$. We may choose $\xi =\chi exp(Mu)$, $\eta =\omega exp(Mv)$ such that $(\xi ,\eta )\in W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })$, ${|\xi |}_{L^{\mathrm{\infty }}(\mathrm{\Omega })}\le C$ and ${|\eta |}_{L^{\mathrm{\infty }}(\mathrm{\Omega })}\le C$. Then we obtain

$$\begin{array}{r}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_j\chi +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_j\omega \\ +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{a}_{ij}(x,u)D_{i}u{D}_{j}u\chi +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{b}_{ij}(x,v)D_{i}v{D}_{j}v\omega \\ -\frac{2\alpha }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha -2}{|v|}^{\beta }u\chi -\frac{2\beta }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha }{|v|}^{\beta -2}v\omega \ge 0\end{array}$$

(5.6)

for all $(\chi ,\omega )\ge (0,0)$, $(\chi ,\omega )\in C_0^{\mathrm{\infty }}(\mathrm{\Omega })\times C_0^{\mathrm{\infty }}(\mathrm{\Omega })$.

Similarly, we may obtain an opposite inequality. Thus we have

$$\begin{array}{r}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_j\chi +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_j\omega \\ +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{a}_{ij}(x,u)D_{i}u{D}_{j}u\chi +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{b}_{ij}(x,v)D_{i}v{D}_{j}v\omega \\ -\frac{2\alpha }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha -2}{|v|}^{\beta }u\chi -\frac{2\beta }{\alpha +\beta }{\int }_{\mathrm{\Omega }}{|u|}^{\alpha }{|v|}^{\beta -2}v\omega =0\end{array}$$

(5.7)

for all $(\chi ,\omega )\in C_0^{\mathrm{\infty }}(\mathrm{\Omega })\times C_0^{\mathrm{\infty }}(\mathrm{\Omega })$. That is, $(u,v)$ is a critical point of $I_0$ and a solution for system (1.1). By doing approximations, we have $(u,v)$ in the place of $(\chi ,\omega )$ of (5.7)

$$\begin{array}{r}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_{j}u+{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_{j}v\\ +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{a}_{ij}(x,u)u{D}_{i}u{D}_{j}u\\ +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{b}_{ij}(x,v)v{D}_{i}v{D}_{j}v-2{\int }_{\mathrm{\Omega }}{|u|}^{\alpha }{|v|}^{\beta }=0.\end{array}$$

(5.8)

Setting $(\phi ,\psi )=(u_n,v_n)$ in (5.1), we have

$$\begin{array}{r}{\mu }_n{\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }{u}_n|}^4+{|\mathrm{\nabla }{v}_n|}^4)+{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u_n)D_i{u}_n{D}_j{u}_n\\ +{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v_n)D_i{v}_n{D}_j{v}_n+\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{a}_{ij}(x,u_n)u_n{D}_i{u}_n{D}_j{u}_n\\ +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{D}_s{b}_{ij}(x,v_n)v_n{D}_i{v}_n{D}_j{v}_n-2{\int }_{\mathrm{\Omega }}{|u_n|}^{\alpha }{|v_n|}^{\beta }=0.\end{array}$$

(5.9)

Using ${\int }_{\mathrm{\Omega }}{|u_n|}^{\alpha }{|v_n|}^{\beta }\to {\int }_{\mathrm{\Omega }}{|u|}^{\alpha }{|v|}^{\beta }$ as $n\to \mathrm{\infty }$, (5.8), (5.9) and lower semi-continuity, we obtain

$$\begin{array}{c}{\mu }_n{\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }{u}_n|}^4+{|\mathrm{\nabla }{v}_n|}^4)\to 0,\hfill \\ {\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u_n)D_i{u}_n{D}_j{u}_n\to {\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_{j}u,\hfill \\ {\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v_n)D_i{v}_n{D}_j{v}_n\to {\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_{j}v\hfill \end{array}$$

as $n\to \mathrm{\infty }$.

In particular, we have

$$\begin{array}{c}{u}_n\to u,v_n\to v\text{in }{W}_0^{1,2}(\mathrm{\Omega }),\hfill \\ u_n\mathrm{\nabla }{u}_n\to u\mathrm{\nabla }u,v_n\mathrm{\nabla }{v}_n\to v\mathrm{\nabla }v\text{in }{L}^2(\mathrm{\Omega })\hfill \end{array}$$

and

$$I_{{\mu }_n}^{\prime }(u_n,v_n)\to I_0^{\prime }(u,v)$$

as $n\to \mathrm{\infty }$. This completes the proof of Theorem 2.1. □

Next, we apply the mountain pass theorem to obtain the existence of critical points of $I_{\mu }$. Set

$$\begin{array}{rl}{\mathrm{\Sigma }}_{\rho }=& \{(u,v)\in W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })|\\ {\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_{j}u+{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_{j}v\le {\rho }^2\}\end{array}$$

for $\rho >0$.

Let us consider the functional

$$\begin{array}{rcl}{I}_{\mu }^{+}(u,v)& =& \frac{1}{4}\mu {\int }_{\mathrm{\Omega }}({|\mathrm{\nabla }u|}^4+{|\mathrm{\nabla }v|}^4)+\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_{j}u\\ +\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_{j}v-\frac{2}{\alpha +\beta }{\int }_{\mathrm{\Omega }}{\left(u^{+}\right)}^{\alpha }{\left(v^{+}\right)}^{\beta }.\end{array}$$

(5.10)

Here and in what follows, we denote $u^{+}=max\{u,0\}$. The functional $I_{\mu }$ satisfies ${(\mathit{PS})}_c$ condition. Similarly, we may verify that $I_{\mu }^{+}$ satisfies ${(\mathit{PS})}_c$ condition. By the ε-Young inequality, for any $\epsilon >0$, there exists $C_{\epsilon }>0$ such that

$${\left(u^{+}\right)}^{\alpha }{\left(v^{+}\right)}^{\beta }\le \epsilon {\left(u^{+}\right)}^{\alpha +\beta }+C_{\epsilon }{\left(v^{+}\right)}^{\alpha +\beta }$$

and

$$\begin{array}{c}{\int }_{\mathrm{\Omega }}{|u|}^{\alpha +\beta }\le C{\left({\int }_{\mathrm{\Omega }}{u}^2{|\mathrm{\nabla }u|}^2\right)}^{\frac{\alpha +\beta }{4}}\le C{({\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_{j}u)}^{\frac{\alpha +\beta }{4}},\hfill \\ {\int }_{\mathrm{\Omega }}{|v|}^{\alpha +\beta }\le C{\left({\int }_{\mathrm{\Omega }}{v}^2{|\mathrm{\nabla }v|}^2\right)}^{\frac{\alpha +\beta }{4}}\le C{({\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_{j}v)}^{\frac{\alpha +\beta }{4}}.\hfill \end{array}$$

Then

$$\begin{array}{r}-\frac{2}{\alpha +\beta }{\int }_{\mathrm{\Omega }}{\left(u^{+}\right)}^{\alpha }{\left(v^{+}\right)}^{\beta }\\ \ge -\frac{2}{\alpha +\beta }\epsilon {\int }_{\mathrm{\Omega }}{\left(u^{+}\right)}^{\alpha +\beta }-\frac{2}{\alpha +\beta }{C}_{\epsilon }{\int }_{\mathrm{\Omega }}{\left(u^{+}\right)}^{\alpha +\beta }\\ \ge -\frac{2C}{\alpha +\beta }\epsilon {({\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_{j}u)}^{\frac{\alpha +\beta }{4}}-\frac{2C_{\epsilon }}{\alpha +\beta }{({\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_{j}v)}^{\frac{\alpha +\beta }{4}}\\ \ge -\frac{2C}{\alpha +\beta }\epsilon {\rho }^{\frac{\alpha +\beta }{2}}-\frac{2C_{\epsilon }}{\alpha +\beta }{\rho }^{\frac{\alpha +\beta }{2}}\\ \ge -\frac{1}{\alpha +\beta }{\rho }^2\end{array}$$

for ε, ρ small. Thus we have

$$\begin{array}{rcl}{I}_{\mu }^{+}(u,v)& \ge & \frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,u)D_{i}u{D}_{j}u+\frac{1}{2}{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,v)D_{i}v{D}_{j}v-\frac{2}{\alpha +\beta }{\int }_{\mathrm{\Omega }}{\left(u^{+}\right)}^{\alpha }{\left(v^{+}\right)}^{\beta }\\ \ge & \frac{1}{2}{\rho }^2-\frac{1}{\alpha +\beta }{\rho }^2=(\frac{1}{2}-\frac{1}{\alpha +\beta }){\rho }^2\end{array}$$

for $(u,v)\in \partial {\mathrm{\Sigma }}_{\rho }$ and for $\rho >0$ small enough. Choose $(\phi ,\psi )\ge (0,0)$, $(\chi ,\omega )\in C_0^{\mathrm{\infty }}(\mathrm{\Omega })\times C_0^{\mathrm{\infty }}(\mathrm{\Omega })$ and $T>0$. Define a path $(g,h):[0,1]\to W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega })$ by $(g(t),h(t))=(tT\phi ,tT\psi )$. When T is large enough, we have

$$\begin{array}{c}{I}_{\mu }^{+}(g(1),h(1))<0,\hfill \\ {\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{a}_{ij}(x,g(1))D_{i}g(1)D_{j}g(1)+{\int }_{\mathrm{\Omega }}\sum _{i,j=1}^N{b}_{ij}(x,h(1))D_{i}h(1)D_{j}h(1)>{\rho }^2\hfill \end{array}$$

and

$$\underset{t\in [0,1]}{sup}{I}_{\mu }^{+}(g(t),h(t))\le m$$

for some m independent of $\mu \in (0,1]$.

Define

$$c_{\mu }=\underset{(g,h)\in \mathrm{\Gamma }}{inf}\underset{t\in [0,1]}{sup}{I}_{\mu }^{+}(g(t),h(t)),$$

where

$$\begin{array}{rl}\mathrm{\Gamma }=& \{(g,h)\in C([0,1],W_0^{1,4}(\mathrm{\Omega })\times W_0^{1,4}(\mathrm{\Omega }))|\\ (g(0),h(0))=(0,0),(g(1),h(1))=(T\phi ,T\psi )\}.\end{array}$$

From the mountain pass theorem we obtain that

$$c_{\mu }\ge (\frac{1}{2}-\frac{1}{\alpha +\beta }){\rho }^2$$

is a critical value of $I_{\mu }^{+}$.

Let $(u_{\mu },v_{\mu })$ be a critical point corresponding to $c_{\mu }$. We have $(u_{\mu },v_{\mu })\ge (0,0)$. Thus $(u_{\mu },v_{\mu })$ is a positive critical point of $I_{\mu }$ by the strong maximum principle. In summary, we have the following.

Proposition 5.1 There exist positive constants ρ and m independent of μ such that $I_{\mu }$ has a positive critical point $(u_{\mu },v_{\mu })$ satisfying

$$(\frac{1}{2}-\frac{1}{\alpha +\beta }){\rho }^2\le I_{\mu }(u_{\mu },v_{\mu })\le m.$$

Finally, we give the proof of Theorem 2.2.

Proof of Theorem 2.2 For a positive solution of system (1.1), the proof follows from Proposition 5.1 and Theorem 2.1. A similar argument gives a negative solution of system (1.1). This completes the proof of Theorem 2.2. □