Existence of positive ground state solutions of Schrödinger–Poisson system involving negative nonlocal term and critical exponent on bounded domain

Abstract

In this paper, we prove the existence of positive ground state solutions of the Schrödinger–Poisson system involving a negative nonlocal term and critical exponent on a bounded domain. The main tools are the mountain pass theorem and the concentration compactness principle.

1 Introduction

In this paper, we consider the following Schrödinger–Poisson system:

$$ \textstyle\begin{cases} -\triangle u-\phi u=\lambda u^{q-1}+u^{5} &\text{in } \varOmega , \\ -\triangle \phi =u^{2} &\text{in } \varOmega , \\ u=\phi =0 &\text{on }\partial \varOmega , \end{cases} $$

(1.1)

where $\lambda >0$ is a parameter, $2< q<6$, and Ω is a smooth bounded domain in ${\mathbb{R}}^{3}$.

System (1.1) is related to the stationary analogue of the nonlinear parabolic Schrödinger–Poisson system

$$ \textstyle\begin{cases} -i\frac{\partial \psi }{\partial t}=-\Delta \psi +\phi (x)\psi - \vert \psi \vert ^{p-2}\psi &\text{in } \varOmega , \\ -\Delta \phi = \vert \psi \vert ^{2} & \text{in } \varOmega , \\ \psi =\phi =0 & \text{on }\partial \varOmega . \end{cases} $$

(1.2)

The first equation in (1.2) is called the Schrödinger–Poisson equation, which describes quantum particles interacting with the electromagnetic field generated by a motion. Similar problems have been widely investigated, and it is well known that they have a strong physical meaning because they appear in quantum mechanics models (see e.g. [3]) and in semiconductor theory [10, 12]. Variational methods and critical point theory are always powerful tools in studying nonlinear differential equations. For more details as regards the physical relevance of the Schrödinger–Poisson system, we refer to [1, 13] and some related results [14, 16,17,18, 20].

The Schrödinger–Poisson system on whole space ${\mathbb{R}}^{N}$ has attracted a lot of attention. Few works concern the existence of solutions for the Schrödinger–Poisson system on a bounded domain, particularly, critical nonlinearity except [2, 7, 8]. Up to now, Schrödinger–Poisson system (1.1) has never been studied by variational methods. Lei and Suo [8] studied the following Schrödinger–Poisson system:

$$ \textstyle\begin{cases} -\triangle u+\kappa \phi u=\kappa \vert u \vert ^{p-2}u+ \vert u \vert ^{4}u & \text{in }\varOmega , \\ -\triangle \phi =u^{2} & \text{in }\varOmega , \\ \phi =u=0 & \text{on }\partial \varOmega , \end{cases} $$

(1.3)

where Ω is a smooth bounded domain in ${\mathbb{R}}^{3}$, $\kappa >0$ is a real parameter, and $1< p<2$. There exists $\kappa ^{*}>0$ such that there at least two positive solutions, and one of them is a positive ground state solution for $\kappa \in (0, \kappa ^{*})$. Zhang [19] considered the negative nonlocal Schrödinger–Poisson system on a bounded domain and obtained thtat there are at least two solutions involving a singularity term by using the Nehari method. Li and Tang [9] obtained at least two positive solutions $(u,\phi _{u})\in D^{1,2}({\mathbb{R}}^{3})\times D ^{1,2}({\mathbb{R}}^{3})$ involving a negative nonlocal term in ${\mathbb{R}}^{3}$.

Our paper is motivated by all the results mentioned [2, 7,8,9, 19]. Up to now, there was no information about system (1.1) on a bounded domain Ω; this is what we are interested in. To deal with our system (1.1), we should estimate the critical value as regards the difficulty caused by the critical exponent.

Now our main results can be stated as follows.

Theorem 1.1

Let$2< q\leq 4$. Then there exists$\lambda ^{*}>0$such that system (1.1) has at least one positive ground state solution for all$\lambda >\lambda ^{*}$.

Theorem 1.2

Let$4< q<6$. Then system (1.1) has at least one positive ground state solution for all$\lambda >0$.

2 Preliminaries

Let X be the usual Sobolev space $H_{0}^{1}(\varOmega )$ with the inner product $(u,v)=\int _{ \varOmega }\nabla u\nabla v \,dx$ and norm $\|u\|=\sqrt{(u,u)}$; $|u|_{s}$ denotes the norm of the space $L^{s}( \varOmega )$, $2\leq s\leq 6$. For any $r>0$ and $x\in \varOmega$, $B_{r}(x)$ denotes the ball of radius r centered at x. C and $C_{i}$ ($i=1,2,3,\dots$) denote various positive constants, which may vary from line to line.

It is well known that system (1.1) can be reduced to a nonlinear Schrödinger equation with nonlocal term. Indeed, the Lax–Milgram theorem implies that for all $u\in X$, there exists a unique $\phi _{u} \in X$ such that

$$ -\Delta \phi _{u}=u^{2}. $$

It is standard to see that system (1.1) is variational and its solutions are the critical points of the functional defined in X by

$$ I(u)=\frac{1}{2} \Vert u \Vert ^{2}- \frac{1}{4} \int _{\varOmega }\phi _{u}u^{2}\,dx - \frac{ \lambda }{q} \int _{\varOmega } \vert u \vert ^{q}\,dx- \frac{1}{6} \int _{ \varOmega } \vert u \vert ^{6}\,dx. $$

For simplicity, in many cases, we just say that $u\in X$, instead of $(u,\phi _{u})\in X\times X$, is a weak solution of system (1.1). It is easy to see that $I\in C^{1}(X,{\mathbb{R}})$ (see [8, 9]) and

$$ \bigl\langle I'(u),\varphi \bigr\rangle = \int _{ \varOmega }\nabla u\nabla \varphi \,dx- \int _{ \varOmega }\phi _{u}u\varphi \,dx-\lambda \int _{\varOmega } \vert u \vert ^{q-2}u \varphi \,dx- \int _{ \varOmega }u^{5}\varphi \,dx, \quad \forall u, \varphi \in X. $$

Let S be the best Sobolev constant, namely

$$ S:=\inf_{u\in H_{0}^{1}( \varOmega )\backslash \{0\}}\frac{ \int _{ \varOmega } \vert \nabla u \vert ^{2}\,dx}{(\int _{ \varOmega } \vert u \vert ^{6}\,dx)^{ \frac{1}{3}}}. $$

(2.1)

As it is well known, the function

$$ U(x)=\frac{({3\epsilon ^{2}})^{\frac{1}{4}}}{(\epsilon ^{2}+ \vert x \vert ^{2})^{ \frac{1}{2}}}, \quad x\in {\mathbb{R}}^{3}, $$

(2.2)

is an extremal function for the minimum problem (2.1), that is, it is a positive solution of the equation

$$ -\Delta u=u^{5},\quad \forall x\in x\in {\mathbb{R}}^{3}, $$

(2.3)

and

$$ \Vert U \Vert ^{2}= \vert U \vert _{6}^{6}=S^{\frac{3}{2}}; $$

(2.4)

see [11].

Before proving our Theorem 1.1, we need the following lemma.

Lemma 2.1

(see [6])

For every$u\in H_{0}^{1}(\varOmega )$, there exists a unique solution$\phi _{u}\in H_{0}^{1}(\varOmega )$of

$$ \textstyle\begin{cases} -\triangle \phi =u^{2} &\textit{in }\varOmega , \\ \phi =0 &\textit{on }\partial \varOmega , \end{cases} $$

and

• (1) $\phi _{u}\geq 0$;moreover, $\phi _{u}> 0$when$u\neq 0$;

• (2) for each$t\neq 0$, $\phi _{tu}=t^{2}\phi _{u}$;

• (3) $\int _{\varOmega } \phi _{u} u^{2}\,dx=\int _{\varOmega }|\nabla \phi _{u}|^{2}\,dx \leq S^{-1}|u|_{ \frac{12}{5}}^{4}\leq C\|u\|^{4}$;

• (4) if$F(u)=\int _{\varOmega } \phi _{u} u^{2}\,dx$, then$F: H_{0}^{1}( \varOmega )\rightarrow H_{0}^{1}(\varOmega )$is$C^{1}$, and

  $$ \bigl\langle F'(u),v\bigr\rangle =4 \int _{\varOmega } \phi _{u} uv\,dx, \quad \forall v\in H_{0}^{1}(\varOmega ). $$

3 The Palais–Smale condition

First, we prove the following mountain-pass geometry of the functional I.

Lemma 3.1

Let$2< q<6$and$\lambda >0$. Then the functionalIsatisfies the following conditions:

• (i) There exist two constants$\alpha , \rho >0$such that

  $$ I(u)\geq \alpha >0 \quad \textit{with } \Vert u \Vert =\rho . $$

• (ii) There exists$e\in X$with$\|e\|>\rho$such that$I(e)<0$.

Proof

(i). We have

$$\begin{aligned} I(u) &=\frac{1}{2} \Vert u \Vert ^{2}- \frac{1}{4} \int _{\varOmega }\phi _{u}u^{2}\,dx- \frac{ \lambda }{q} \int _{\varOmega } \vert u \vert ^{q}\,dx - \frac{1}{6} \int _{\varOmega } \vert u \vert ^{6}\,dx \\ &\geq \frac{1}{2} \Vert u \Vert ^{2}- \frac{1}{4}C \Vert u \Vert ^{4}-C_{1} \Vert u \Vert ^{q}-\frac{1}{6S ^{3}} \Vert u \Vert ^{6}. \end{aligned}$$

Therefore, since $q>2$, there exist $\alpha , \rho >0$ such that $I(u)\geq \alpha >0$ with $\|u\|=\rho$.

(ii). For $u\in X\setminus \{0\}$, we have

$$ I(tu)\leq \frac{1}{2}t^{2} \Vert u \Vert ^{2}-\frac{1}{6}t^{6} \int _{ \varOmega } \vert u \vert ^{6}\,dx \rightarrow -\infty $$

as $t\rightarrow +\infty$. Then we can find $e\in X$ such that $\|e\|>\rho$ and $I(e)<0$. This completes the proof. □

Therefore by using the mountain pass theorem without $(PS)_{c}$ condition (see [15]) it follows that there exists a $(PS)_{c}$ sequence $\{u_{n}\}\subset X$ such that

$$ I(u_{n})\rightarrow c =\inf_{\gamma \in \varGamma }\max _{t\in [0,1]}I\bigl(\gamma (t)\bigr) \quad \text{and}\quad I'(u_{n})\rightarrow 0, $$

where

$$ \varGamma = \bigl\{ \gamma \in C \bigl([0,1],X \bigr): \gamma (0)=0, \gamma (1)=e \bigr\} . $$

Lemma 3.2

Let$2< q<6$and$\lambda >0$. Let$\{u_{n}\}\subset X$be a$(PS)_{c}$sequence ofIwith$0< c<\frac{1}{3}S^{\frac{3}{2}}$. Then there exists$u\in X$such that$u_{n}\rightarrow u$inX.

Proof

Let $\{u_{n}\}\subset X$ be a $(PS)_{c}$ for I, that is,

$$ I(u_{n})\rightarrow c\quad \text{and}\quad I'(u_{n})\rightarrow 0 \quad \text{as }n \rightarrow \infty . $$

(3.1)

We claim that $\{u_{n}\}$ is bounded in X. In the case $2< q\leq 4$, we deduce that

$$\begin{aligned} 1+c+o\bigl( \Vert u_{n} \Vert \bigr) &\geq I(u_{n})- \frac{1}{q}\bigl\langle I'(u_{n}),u_{n} \bigr\rangle \\ &= \biggl(\frac{1}{2}-\frac{1}{q} \biggr) \Vert u_{n} \Vert ^{2}+ \biggl(\frac{1}{q}- \frac{1}{4} \biggr) \int _{\varOmega }\phi _{u_{n}}{u_{n}}^{2}\,dx+ \biggl(\frac{1}{q}- \frac{1}{6} \biggr) \int _{ \varOmega } \vert u_{n} \vert ^{6}\,dx \\ &\geq \biggl(\frac{1}{2}-\frac{1}{q} \biggr) \Vert u_{n} \Vert ^{2}; \end{aligned}$$

in the case $4< q<6$, we have

$$\begin{aligned} 1+c+o\bigl( \Vert u_{n} \Vert \bigr) &\geq I(u_{n})- \frac{1}{4}\bigl\langle I'(u_{n}),u_{n} \bigr\rangle \\ &= \frac{1}{4} \Vert u_{n} \Vert ^{2}+ \biggl(\frac{1}{4}-\frac{1}{q} \biggr) \lambda \int _{\varOmega } \vert u_{n} \vert ^{q}\,dx+\frac{1}{12} \int _{ \varOmega } \vert u_{n} \vert ^{6}\,dx \\ &\geq \frac{1}{4} \Vert u_{n} \Vert ^{2}, \end{aligned}$$

which implies that $\{u_{n}\}$ is bounded in X. Going if necessary to a subsequence, still denoted by $\{u_{n}\}$, we can assume that for n large enough,

$$ \textstyle\begin{cases} u_{n}\rightharpoonup u & \text{in }X; \\ u_{n}\rightarrow u & \text{in }L^{p}(\varOmega ), p\in [1,6); \\ u_{n}\rightarrow u & \text{for a.e. }x\in \varOmega . \end{cases} $$

(3.2)

By the concentration compactness principle (see [5, 11]) there exists at most countable set J, points $\{x_{j}\}_{j\in J} \subset \varOmega$, and values $\{v_{j}\}_{j\in J},\{\mu _{j}\}_{j\in J} \subset R^{+}$ such that

$$\begin{aligned} &\vert \nabla u_{n} \vert ^{2}\rightharpoonup \mu \geq \vert \nabla u \vert ^{2}+\sum_{j\in J} \mu _{j}\delta _{x_{j}}, \end{aligned}$$

(3.3)

$$\begin{aligned} &\vert u_{n} \vert ^{6}\rightharpoonup \nu= \vert u \vert ^{6}+\sum_{j\in J}\nu _{j} \delta _{x_{j}}, \end{aligned}$$

(3.4)

where $\delta _{x_{j}}$ is the Dirac mass at $x_{j}$. Moreover, we have

$$ \mu _{j}, \nu _{j}\geq 0, \qquad \mu _{j} \geq S\nu _{j}^{\frac{1}{3}}. $$

(3.5)

We claim that $J=\emptyset$. Suppose, on the contrary, that $J\neq \emptyset$, that is, there exists $j_{0}\in J$ such that $\mu _{j_{0}}\neq 0$.

On the one hand, for any $\varepsilon >0$ small, assume that $\psi _{\varepsilon ,j}(x)\in C_{0}^{\infty }(\mathbb{R}^{3})$ is such that $\psi _{\varepsilon ,j}(x)\in [0,1]$,

$$ \psi _{\varepsilon ,j}(x)=1, \quad \text{in }B\biggl(x_{j}, \frac{\varepsilon }{2}\biggr); \qquad \psi _{\varepsilon ,j}(x)=0, \quad \text{in }X\setminus B(x_{j},\varepsilon ); \qquad \bigl\vert \nabla \psi _{\varepsilon ,j}(x) \bigr\vert \leq \frac{4}{\varepsilon }. $$

Since $\{u_{n}\}\subset X$ is bounded and $\{\psi _{\varepsilon ,j}u _{n}\}$ is also bounded, we have

$$\begin{aligned} o(1) &=\bigl\langle I'(u_{n}),\psi _{\varepsilon ,j}u_{n}\bigr\rangle \\ &= \biggl( \int _{ \varOmega }u_{n}\nabla u_{n}\nabla \psi _{\varepsilon ,j}\,dx + \int _{ \varOmega } \vert \nabla u_{n} \vert ^{2}\psi _{\varepsilon ,j}\,dx \biggr) \\ &\quad {}- \int _{ \varOmega }\phi _{u_{n}}u_{n}^{2} \psi _{\varepsilon ,j}\,dx-\lambda \int _{ \varOmega } \vert u_{n} \vert ^{q}\psi _{\varepsilon ,j}\,dx- \int _{ \varOmega }u _{n}^{6}\psi _{\varepsilon ,j}\,dx, \end{aligned}$$

(3.6)

and by the Hölder inequality we get

$$\begin{aligned} & \lim_{\varepsilon \rightarrow 0}\limsup_{n\rightarrow \infty } \biggl\vert \int _{ \varOmega }u_{n}\nabla u_{n} \nabla \psi _{\varepsilon ,j}\,dx \biggr\vert \\ &\quad \leq \lim_{\varepsilon \rightarrow 0}\limsup_{n\rightarrow \infty } \biggl( \int _{B_{\varepsilon }(x_{j})} \vert \nabla u_{n} \vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggl( \int _{B_{\varepsilon }(x _{j})} \vert \nabla \psi _{\varepsilon ,j} \vert ^{2} \vert u_{n} \vert ^{2}\,dx \biggr)^{ \frac{1}{2}} \\ &\quad \leq \lim_{\varepsilon \rightarrow 0} C_{2} \biggl( \int _{B_{ \varepsilon }(x_{j})} \vert \nabla \psi _{\varepsilon ,j} \vert ^{2} \vert u \vert ^{2}\,dx \biggr) ^{\frac{1}{2}} \\ &\quad \leq \lim_{\varepsilon \rightarrow 0} C_{2} \biggl( \int _{B_{ \varepsilon }(x_{j})} \vert \nabla \psi _{\varepsilon ,j} \vert ^{3}\,dx \biggr)^{ \frac{1}{3}} \biggl( \int _{B_{\varepsilon }(x_{j})} \vert u \vert ^{6}\,dx \biggr) ^{\frac{1}{6}} \\ &\quad \leq \lim_{\varepsilon \rightarrow 0}C_{3} \biggl( \int _{B_{\varepsilon }(x_{j})} \vert u \vert ^{6}\,dx \biggr)^{\frac{1}{6}}=0. \end{aligned}$$

(3.7)

From (3.2)–(3.4) we have

$$\begin{aligned} & \lim_{\varepsilon \rightarrow 0}\limsup_{n\rightarrow \infty } \int _{ \varOmega } \vert u_{n} \vert ^{6} \psi _{\varepsilon ,j}\,dx \\ &\quad =\lim_{\varepsilon \rightarrow 0} \int _{ \varOmega } \vert u \vert ^{6} \psi _{\varepsilon ,j}\,dx+\nu _{j} \\ &\quad =\nu _{j}, \end{aligned}$$

(3.8)

and by Lemma 2.1 and (3.2) we obtain

$$\begin{aligned}& \lim_{\varepsilon \rightarrow 0}\limsup_{n\rightarrow \infty } \int _{ \varOmega }\phi _{u_{n}}u_{n}^{2} \psi _{\varepsilon ,j}\,dx =0, \end{aligned}$$

(3.9)

$$\begin{aligned}& \lim_{\varepsilon \rightarrow 0}\limsup_{n\rightarrow \infty } \int _{ \varOmega } \vert u_{n} \vert ^{q} \psi _{\varepsilon ,j}\,dx=0, \end{aligned}$$

(3.10)

and

$$\begin{aligned} &\lim_{\varepsilon \rightarrow 0}\limsup_{n\rightarrow \infty } \int _{ \varOmega } \vert \nabla u_{n} \vert ^{2} \psi _{\varepsilon ,j}\,dx \geq \lim_{\varepsilon \rightarrow 0} \int _{ \varOmega } \vert \nabla u \vert ^{2}\psi _{\varepsilon ,j}\,dx+\mu _{j} =\mu _{j}. \end{aligned}$$

(3.11)

By (3.6)–(3.11) we obtain

$$ \nu _{j}\geq \mu _{j}, $$

which, combined with $\mu _{j_{0}}\neq 0$ and (3.5), gives

$$ \nu _{j_{0}}\geq S^{\frac{3}{2}}. $$

(3.12)

From (3.3)–(3.5) and (3.12) in the case $2< q\leq 4$, we have

$$\begin{aligned} c &=\lim_{n\rightarrow \infty } \biggl\{ I(u_{n})- \frac{1}{q} \bigl\langle I'(u_{n}),u_{n} \bigr\rangle \biggr\} \\ &=\lim_{n\rightarrow \infty } \biggl\{ \biggl(\frac{1}{2}- \frac{1}{q} \biggr) \Vert u_{n} \Vert ^{2}+ \biggl(\frac{1}{q}-\frac{1}{4} \biggr) \int _{\varOmega }\phi _{u_{n}}{u_{n}}^{2}\,dx+ \biggl(\frac{1}{q}- \frac{1}{6} \biggr) \int _{ \varOmega } \vert u_{n} \vert ^{6}\,dx \biggr\} \\ &\geq \biggl(\frac{1}{2}-\frac{1}{q} \biggr) \biggl( \Vert u \Vert ^{2}+\sum_{j\in J}\mu _{j} \biggr)+ \biggl(\frac{1}{q}-\frac{1}{6} \biggr) \biggl( \int _{ \varOmega } \vert u \vert ^{6}\,dx+\sum _{j\in J}\nu _{j} \biggr) \\ &\geq \biggl(\frac{1}{2}-\frac{1}{q} \biggr)\mu _{j_{0}}+ \biggl(\frac{1}{q}- \frac{1}{6} \biggr)\nu _{j_{0}} \geq \frac{1}{3}S^{\frac{3}{2}}, \end{aligned}$$

and in the case $4< q<6$, we have

$$\begin{aligned} c &=\lim_{n\rightarrow \infty } \biggl\{ I(u_{n})- \frac{1}{4} \bigl\langle I'(u_{n}),u_{n} \bigr\rangle \biggr\} \\ &=\lim_{n\rightarrow \infty } \biggl\{ \frac{1}{4} \Vert u_{n} \Vert ^{2}+ \biggl(\frac{1}{4} - \frac{1}{q} \biggr)\lambda \int _{\varOmega } \vert u_{n} \vert ^{q}\,dx+ \frac{1}{12} \int _{ \varOmega } \vert u_{n} \vert ^{6}\,dx \biggr\} \\ &\geq \frac{1}{4} \biggl( \Vert u \Vert ^{2}+\sum _{j\in J}\mu _{j} \biggr)+ \frac{1}{12} \biggl( \int _{ \varOmega } \vert u \vert ^{6}\,dx+\sum _{j\in J}\nu _{j} \biggr) \\ &\geq \frac{1}{4}\mu _{j_{0}}+\frac{1}{12}\nu _{j_{0}} \geq \frac{1}{3}S^{\frac{3}{2}}, \end{aligned}$$

where we use $\nu _{j}\geq \mu _{j}$ and $\nu _{j}\geq S^{\frac{3}{2}}$. Therefore by $c<\frac{1}{3}S^{\frac{3}{2}}$ it is a contradiction. This implies that J is empty, which means that $\int _{ \varOmega }|u_{n}|^{6}\,dx \rightarrow \int _{ \varOmega }|u|^{6}\,dx$. We can also get $u_{n}\rightarrow u$ in X (see Lemma 2.2 in [8]). So Lemma 3.2 holds. □

Lemma 3.3

If$2< q\leq 4$, then there exist$\lambda ^{*}>0$and${\overline{v}} _{0}\in H_{0}^{1}(\varOmega )$such that

$$ \sup_{s\geq 0}I(s{\overline{v}}_{0})< \frac{1}{3}S^{\frac{3}{2}} \quad \textit{for all }\lambda >\lambda ^{*}. $$

If$4< q<6$, then there exists${\overline{v}}_{1}\in H_{0}^{1}(\varOmega )$such that

$$ \sup_{s\geq 0}I(s{\overline{v}}_{1})< \frac{1}{3}S^{\frac{3}{2}} \quad \textit{for all }\lambda >0. $$

Proof

We choose a function $\eta \in C_{0}^{\infty }(\varOmega )$ such that $0\leq \eta (x)\leq 1$, $|\nabla \eta |\leq C$ in Ω. $\eta (x)=1$ for $|x|< 2r_{0}$, and $\eta (x)=0$ for $|x|>3r_{0}$. Define

$$ u_{\epsilon }(x)=\eta (x)U(x). $$

It is known (see [15]) that

$$\begin{aligned} \begin{aligned} & \vert u_{\epsilon } \vert _{6}^{6}=S^{\frac{3}{2}}+O \bigl(\epsilon ^{3}\bigr), \\ & \Vert u_{\epsilon } \Vert ^{2}=S^{\frac{3}{2}}+O( \epsilon ), \\ & \Vert u_{\epsilon } \Vert ^{4}\leq S^{3}+O( \epsilon ), \\ &C_{4}\epsilon ^{\frac{p}{2}}\leq \int _{\varOmega }u_{\epsilon }^{p}\,dx \leq C_{5}\epsilon ^{\frac{p}{2}}, \quad 1\leq p< 3, \\ &C_{6}\epsilon ^{\frac{p}{2}} \vert \ln \epsilon \vert \leq \int _{\varOmega }u_{ \epsilon }^{p}\,dx\leq C_{7}\epsilon ^{\frac{p}{2}} \vert \ln \epsilon \vert , \quad p=3, \\ &C_{8}\epsilon ^{\frac{6-p}{2}}\leq \int _{\varOmega }u_{\epsilon }^{p}\,dx \leq C_{9}\epsilon ^{\frac{6-p}{2}}, \quad 3< p< 6. \end{aligned} \end{aligned}$$

(3.13)

Set

$$ h(su_{\epsilon })=\frac{s^{2}}{2} \Vert u_{\epsilon } \Vert ^{2}-\frac{s^{q} \lambda }{q} \int _{\varOmega } \vert u_{\epsilon } \vert ^{q}\,dx-\frac{ s^{6}}{6} \int _{ \varOmega } \vert u_{\epsilon } \vert ^{6}\,dx. $$

We can also prove that $\max_{s\geq 0}h(su_{\epsilon })$ is attained at $s_{0}$ for $0< s_{1}< s_{0}< s_{2}$, that is,

$$ \max_{s\geq 0}h(su_{\epsilon })=h(s_{0}u_{\epsilon }). $$

(3.14)

Combining (3.13) with (3.14), $4< q<6$, we deduce

$$\begin{aligned} \sup_{t\geq 0}I(su_{\epsilon }) &=\frac{s^{2}}{2} \Vert u_{\epsilon } \Vert ^{2}-\frac{s^{4}}{4} \int _{\varOmega }\phi _{u_{\epsilon }}{u_{\epsilon }}^{2}\,dx -\frac{s^{q}\lambda }{q} \int _{\varOmega } \vert u_{\epsilon } \vert ^{q}\,dx-\frac{ s ^{6}}{6} \int _{ \varOmega } \vert u_{\epsilon } \vert ^{6}\,dx \\ &\leq \frac{s^{2}}{2} \Vert u_{\epsilon } \Vert ^{2}- \frac{s^{q}\lambda }{q} \int _{\varOmega } \vert u_{\epsilon } \vert ^{q}\,dx-\frac{ s^{6}}{6} \int _{ \varOmega } \vert u _{\epsilon } \vert ^{6}\,dx \\ &\leq \frac{s_{0}^{2}}{2} \Vert u_{\epsilon } \Vert ^{2}- \frac{s_{0}^{q}\lambda }{q} \int _{\varOmega } \vert u_{\epsilon } \vert ^{q}\,dx-\frac{ s_{0}^{6}}{6} \int _{ \varOmega } \vert u_{\epsilon } \vert ^{6}\,dx \\ &\leq \frac{1}{2}s_{0}^{2} \bigl(S^{\frac{3}{2}}+O(\epsilon ) \bigr)-C _{10}\epsilon ^{\frac{6-q}{2}}- \frac{s_{0}^{6}}{6} \bigl(S^{ \frac{3}{2}}+O\bigl(\epsilon ^{3}\bigr) \bigr) \\ &={\frac{1}{2}s_{0}^{2}S^{\frac{3}{2}}- \frac{s_{0}^{6}}{6}S^{ \frac{3}{2}}}+O(\epsilon ) -O\bigl(\epsilon ^{3}\bigr)-C_{10}\epsilon ^{ \frac{6-q}{2}} \\ &\leq \sup_{k\geq 0} \biggl\{ {\frac{k^{2}}{2}S^{\frac{3}{2}}- \frac{k ^{6}}{6}S^{\frac{3}{2}}} \biggr\} +O(\epsilon ) -C_{10} \epsilon ^{\frac{6-q}{2}} < \frac{1}{3}S^{\frac{3}{2}} \quad \text{as } \epsilon \rightarrow 0^{+}. \end{aligned}$$

(3.15)

Similarly, in the case $2< q\leq 4$, by (3.13) and (3.15) we have

$$\begin{aligned} \sup_{t\geq 0}I(su_{\epsilon }) &\leq \frac{s_{0}^{2}}{2} \Vert u _{\epsilon } \Vert ^{2}-\frac{s_{0}^{q}\lambda }{q} \int _{\varOmega } \vert u_{ \epsilon } \vert ^{q}\,dx-\frac{ s_{0}^{6}}{6} \int _{ \varOmega } \vert u_{\epsilon } \vert ^{6}\,dx \\ &\leq \frac{1}{2}s_{0}^{2} \bigl(S^{\frac{3}{2}}+O(\epsilon ) \bigr)-C _{11}\lambda \epsilon ^{\frac{6-q}{2}}- \frac{s_{0}^{6}}{6} \bigl(S ^{\frac{3}{2}}+O\bigl( \epsilon ^{3}\bigr) \bigr) \\ &\leq \frac{1}{3}S^{\frac{3}{2}}+O(\epsilon )-C_{11} \lambda \epsilon ^{\frac{6-q}{2}}< \frac{1}{3}S^{\frac{3}{2}} \quad \text{as } \epsilon \rightarrow 0^{+}, \end{aligned}$$

(3.16)

provided that λ is large enough. Thus there exists $\lambda ^{*}>0$ such that $I(su_{\epsilon })<\frac{1}{3}S^{ \frac{3}{2}}$ for all $\lambda >\lambda ^{*}$. This completes the proof. □

4 Proof of theorems

Proof of Theorems 1.1 and 1.2

Due to Lemma 3.1, $I(u)$ satisfies the mountain pass geometry. From Lemmas 3.2 and 3.3 we obtain the $(PS)_{c}$ condition with $0< c<\frac{1}{3}S^{\frac{3}{2}}$. Therefore system (1.1) has a nontrivial solution $u_{0}$, and $I(u_{0})=c>0$, which is a mountain pass solution. Since $I(|u|)=I(u)$, by a result due to Brézis and Nirenberg (Theorem 10 in [4]) we conclude that $u_{0}\geq 0$. By the strong maximum principle we have $u_{0}>0$ in Ω. Therefore $u_{0}$ is a positive solution of system (1.1) with $I(u_{0})>0$.

Next, we show that system (1.1) has a positive ground state solution in X when $2< p\leq 4$ or $4< p<6$.

Define

$$ m:=\inf_{v\in \mathcal{M}}I(v), \mathcal{M}=\bigl\{ v\in X\setminus \{0\} \mid I'(v)=0\bigr\} . $$

There exists $\{u_{n}\}\subset X$ such that $u_{n}\neq 0$. Since $u_{0}$ is a solution of system (1.1), by the definition of m we have

$$ I(u_{n})\rightarrow m, m< \frac{1}{3}S^{\frac{3}{2}}, \qquad I'(u_{n})\rightarrow 0 \quad \text{as }n \rightarrow \infty . $$

(4.1)

Obviously, from Lemma 3.2 we can easily deduce that $\{u_{n}\}$ is bounded in X. Then there exist a nonnegative subsequence of $\{u_{n}\}$ (still denoted by $\{u_{n}\}$) and $u_{1}\in X$ such that $u_{n}\rightharpoonup u_{1}$ in X. We can obtain that $u_{n}\rightarrow u_{1}$ in X and $I(u_{1})=m$ with $u_{1}>0$ by the last section in [8], that is, $u_{1}$ is a positive ground state solution to system (1.1). This completes the proof. □