Existence of multiple solutions for a quasilinear Neumann problem with critical exponent

Abstract

The main purpose of this paper is to establish the existence and multiplicity of nontrivial solutions for a quasilinear Neumann problem with critical exponent. It is shown, by the methods of the Lions concentration-compactness principle and the mountain pass lemma, that under certain conditions, the existence of nontrivial solutions are obtained.

1 Introduction

In this paper, we consider the following quasilinear elliptic problem with critical Sobolev exponent:

$$ \textstyle\begin{cases} -\varepsilon^{p}\Delta_{p}u+V(x)\vert u\vert ^{p-2}u=Q(x)\vert u\vert ^{p^{*}-2}u +P(x)\vert u\vert ^{ {q-2}}u, & x\in \Omega , \\ \vert \nabla u\vert ^{p-2}\frac{\partial u}{\partial \nu }=0, &x\in \partial \Omega , \end{cases} $$

(1.1)

where $\Omega \subset R^{N}$ is a bounded domain with smooth boundary, $\Delta_{p}u=\operatorname{div}(\vert \nabla u\vert ^{p-2}\nabla u)$, $\varepsilon >0$, $1< p<N$, $p < q < p^{*}=\frac{Np}{N-p}$, ν denotes the unit outward normal vector with respect to ∂Ω. The weight functions $V(x)$, $Q(x)$ and $P(x)$ are continuous on Ω. Such problems arise in the theory of quasiregular and quasiconformal mapping or in the study of non-Newtonian fluids. In the latter case, the p is a characteristic of the medium. Media with $p > 2$ are called dilatant fluids and those with $p < 2$ are called pseudoplastics. If $p = 2$, they are Newtonian fluids.

The early study of Laplacian elliptic equation with critical Sobolev exponent was Pohozaev [1], the author established the nonexistence of nontrivial solution to the Dirichlet problems when Ω is a star-shaped domain with respect to the origin. Later, Brézis and Nirenberg [2] showed the existence of positive solutions by introducing the low-order perturbation terms, and Struwe [3] also obtained the global compactness result. Since then, the study of these elliptic problems with critical growth terms have been paid wide attentions in recent years (see [4–7]). Set $p=2$, $\varepsilon =1$, $P(x)=0$, $V(x)=\lambda$, then Problem (1.1) reduces to the following semilinear elliptic problem:

$$ \textstyle\begin{cases} -\Delta u+\lambda u=Q(x)\vert u\vert ^{2^{*}-2}u,& x\in \Omega , \\ \frac{\partial u}{\partial v}=0,& x\in \Omega . \end{cases} $$

(1.2)

Comte and Knaap [8] proved that there exists a nontrivial solution of problem (1.2) by variational method if $Q(x)=1$ and $\lambda =-\mu$. Chabrowski and Willem [9] studied this problem with the assumption that the function $Q(x)$ is nonnegative and Hölder continuous, they obtained the existence of least energy solutions by solving minimization problem corresponding to

$$ \begin{aligned} S_{\lambda }= \inf_{u\in H^{1}(\Omega ), \int_{\Omega }Q(x)\vert u\vert ^{2^{*}}\,\mathrm{d}x \neq 0} \frac{\int_{\Omega }(\vert \nabla u\vert ^{2}+\lambda u^{2})\,\mathrm{d}x}{( \int_{\Omega }Q(x)\vert u\vert ^{2^{*}}\,\mathrm{d}x)^{\frac{2}{2^{*}}}}. \end{aligned} $$

Subsequently, Chabrowski and Girão [10] investigated the existence and nonexistence of least energy solutions when the function $Q(x)$ has some symmetry properties. For more relevant information as regards the corresponding problems, the interested reader may refer to [11–21] and the references therein.

As for quasilinear elliptic problems with critical Sobolev exponent, the existence and multiplicity of solutions have also been studied extensively. Abreu et al. [22] studied the following nonhomogeneous Neumann boundary problems:

$$ \textstyle\begin{cases} -\Delta_{p} u+\lambda u^{p-1}=u^{q}, &x\in \Omega , \\ u>0, &x\in \Omega , \\ \vert \nabla u\vert ^{p-2}\frac{\partial u}{\partial v}=\varphi , &x\in \partial \Omega , \end{cases} $$

(1.3)

where $p-1< q\leq p^{*}-1$, $\varphi \in C^{\alpha }(\overline{\Omega })$, $0<\alpha <1$, $\varphi \not \equiv 0$. They proved that there exists a $\lambda^{*}>0$ such that problem (1.3) has at least two positive solutions if $\lambda >\lambda^{*}$, has at least one positive solution if $\lambda =\lambda^{*}$ and has no positive solution if $\lambda <\lambda^{*}$ relying on the lower and upper solutions method and variational approach. Zhao et al. [23] discussed the quasilinear elliptic problem of the form

$$ \textstyle\begin{cases} -\Delta_{p} u+\lambda (x) \vert u\vert ^{p-2}u=\vert u\vert ^{p^{*}-2}u + \vert u\vert ^{r-2}u, &x \in \Omega , \\ \vert \nabla u\vert ^{p-2}\frac{\partial u}{\partial v}=\eta \vert u\vert ^{p-2}u , &x \in \partial \Omega , \end{cases} $$

(1.4)

they showed that there exists at least a nontrivial solution when $p < r < p^{*}$ and there exist infinitely many solutions when $1 < r < p$ by using the Mountain pass theorem and the concentration-compactness principle. Some authors also studied the critical Sobolev exponent for quasilinear equations and the corresponding evolution problems with Neumann boundary conditions, the reader may also refer to [24–37].

Motivated by the results of the above papers, we discuss the existence of nontrivial nonnegative solutions to Problem (1.1) by a variational method. The special features of this problem are the following. Firstly, due to the lack of compactness of the embedding of $W^{1,p}(\Omega ) \hookrightarrow L^{p^{*}}(\Omega )$, we cannot use the standard variational argument directly. In order to overcome this difficulty and obtain the existence of solutions, we have to add restrictions on the weight functions $Q(x)$ and $P(x)$ to prove the corresponding functional of Problem (1.1) satisfies $(PS)_{c}$-condition in a suitable range by the Lions concentration-compactness principle. Secondly, the weight function $V(x)$ may be unbounded near the boundary ∂Ω, which leads to the space $W^{1,p}(\Omega )$ is not suitable for our problem. To solve such problem, we have to introduce a suitable weighted Sobolev space.

For the sake of convenience, we introducing a new parameter $\lambda =\varepsilon^{-p}$, then Problem (1.1) may be rewritten as the following problem:

$$ \textstyle\begin{cases} -\Delta_{p} u+\lambda V(x)\vert u\vert ^{p-2}u=\lambda Q(x)\vert u\vert ^{p^{*}-2}u+ \lambda P(x)\vert u\vert ^{q-2}u,&x\in \Omega , \\ \vert \nabla u\vert ^{p-2}\frac{\partial u}{\partial \nu }=0,&x\in \partial \Omega . \end{cases} $$

(1.5)

Throughout this paper, we make some assumptions on the weight functions $Q(x)$, $P(x)$, $V(x)$ as the following:

• (A1) $Q(x)$, $P(x)$ are continuous on Ω̅, and $Q(x)>0$, $P(x)\geq 0$ for $x\in \overline{\Omega }$;

• (A2) $V(x)$ is continuous in Ω, and $V(x)\geq 0$, $V(x)\not \equiv 0$ for $x\in \Omega$.

Set $Q_{m}=\max_{x\in \partial \Omega }Q(x)$, $Q_{M}= \max_{x\in \overline{\Omega }}Q(x)$, $P_{M}=\max_{x\in \overline{ \Omega }}P(x)$. The main results of this paper are the following.

Theorem 1.1

Suppose that (A1), (A2) hold, $H(0)>0$ and $Q_{m}=Q(0)$. If functions $Q(x)$, $V(x)$ satisfy

• (A3) $Q_{M}\leq 2^{\frac{p}{N-p}}Q_{m}$, and $\vert Q(x)-Q(0)\vert =o(\vert x\vert ^{ \alpha })$ as $x\rightarrow 0$, where $1<\alpha <\frac{N}{p-1}$;

• (A4) $\int_{\Omega \cap B(0,\delta )}V^{r^{\prime }}\,\mathrm{d}x< \infty$, where $\frac{1}{r}+\frac{1}{r^{\prime }}=1$, $1< r<\frac{N(p-1)}{Np+2p-N-p ^{2}-1}$, $\delta >0$.

Then Problem (1.5) has at least one nontrivial solution for every $\lambda >0$ and $N\geq 2p$, where $H(0)$ will be later determined.

Theorem 1.2

Suppose that (A1), (A2) hold. If $Q_{M}>2^{\frac{p}{N-p}}Q_{m}$ and functions $P(x)$, $V(x)$ satisfy

• (A5) $P(x)\not \equiv 0$ for $x\in \Omega$, and $V\in L^{1}(\Omega )$.

Then there exists a $\lambda_{*}>0$ such that Problem (1.5) has at least one nontrivial solution for $0<\lambda <\lambda_{*}$.

Theorem 1.3

Suppose that (A1), (A2) hold. If $Q_{M}>2^{\frac{p}{N-p}}Q_{m}$ and functions $P(x)$, $V(x)$ satisfy

• (A6) $P(x)>0$ for $x\in \overline{\Omega }$;

• (A7) there exist $x_{0}\in \Omega$ and constant $\delta >0$ such that $V(x)=0$ for $x\in B(x_{0},\delta )\subset \Omega$.

Then there exists a $\lambda^{*}>0$ such that Problem (1.5) has at least one nontrivial solution for $\lambda >\lambda^{*}$.

Theorem 1.4

Suppose that (A1), (A2) hold. If $Q_{M}>2^{\frac{p}{N-p}}Q_{m}$ and functions $P(x)$, $V(x)$ satisfy the conditions (A6) and (A7). Then, for every integer n, there exists a constant $\Lambda_{n}>0$ such that Problem (1.5) has at least n pairs of nontrivial solutions for $\lambda >\Lambda_{n}$.

2 Preliminaries

Firstly, we define the weighted Sobolev space

$$\begin{aligned} W_{\lambda ,V}^{1,p}(\Omega )= \biggl\{ u; D_{i}u\in L^{p}(\Omega ), i=1, 2, \ldots , N, \int_{\Omega }V(x)\vert u\vert ^{p}\,\mathrm{d}x< + \infty \biggr\} \end{aligned}$$

with norm $\Vert u\Vert _{\lambda ,V}=(\int_{\Omega }(\vert \nabla u\vert ^{p}+\lambda V(x)\vert u\vert ^{p})\,\mathrm{d}x)^{\frac{1}{p}}$. Obviously, norms $\Vert u\Vert _{ \lambda ,V}$ and $\Vert u\Vert _{V}$ are equivalent, $W^{1,p}_{\lambda ,V}( \Omega )\hookrightarrow W^{1,p}(\Omega )$, where $\Vert u\Vert _{V}= ( \int_{\Omega }(\vert \nabla u\vert ^{p}+ V(x)\vert u\vert ^{p})\,\mathrm{d}x )^{ \frac{1}{p}}$.

By using the Sobolev embedding theorem, we know that there exists a constant $C_{q}>0$ such that

$$ \begin{aligned} \vert u\vert _{q}\leq C_{q}\Vert u\Vert _{V}\leq C_{q}\Vert u \Vert _{\lambda V},\quad \mbox{for } \lambda \geq 1, u\in W_{\lambda ,V}^{1,p}( \Omega ), \end{aligned} $$

(2.1)

and

$$ \begin{aligned} \vert u\vert _{q}\leq C_{q}\Vert u\Vert _{V}\leq \lambda^{-\frac{1}{p}}C_{q} \Vert u\Vert _{ \lambda V},\quad \mbox{for }0< \lambda < 1, u\in W_{\lambda ,V}^{1,p}( \Omega ), \end{aligned} $$

(2.2)

where $\vert u\vert _{q}= (\int_{\Omega }\vert u\vert ^{q}\,\mathrm{d}x )^{ \frac{1}{q}}$, $q\in (p, p^{*})$.

Next, we give the definition of weak solution to Problem (1.5).

Definition 2.1

A function $u\in W_{\lambda ,V}^{1,p}(\Omega )$ is said to be a weak solution of Problem (1.5) if it satisfies

$$\begin{aligned}& \int_{\Omega }\vert \nabla u\vert ^{p-2}\nabla u\nabla \psi \,\mathrm{d}x+ \lambda \int_{\Omega }V(x)\vert u\vert ^{p-2}u \psi \, \mathrm{d}x \\& \quad =\lambda \int_{\Omega }Q(x)\vert u\vert ^{p^{*}-2}u \psi \, \mathrm{d}x+\lambda \int_{\Omega }P(x)\vert u\vert ^{q-2} u \psi \, \mathrm{d}x,\quad \forall \psi \in W_{\lambda ,V}^{1,p}(\Omega ). \end{aligned}$$

Thus, the corresponding energy functional of Problem (1.5) is defined in $W_{\lambda ,V}^{1,p}(\Omega )$ by

$$\begin{aligned} J_{\lambda }(u)=\frac{1}{p} \int_{\Omega } \bigl(\vert \nabla u\vert ^{p}+\lambda V(x)\vert u\vert ^{p} \bigr) \,\mathrm{d}x - \frac{\lambda }{p^{*}} \int_{\Omega }Q(x)\vert u\vert ^{p^{*}} \,\mathrm{d}x - \frac{\lambda }{q} \int_{\Omega }P(x)\vert u\vert ^{q} \,\mathrm{d}x. \end{aligned}$$

Let S be the best Sobolev constants, namely

$$ \begin{aligned} S=\inf_{D^{1,p}(R^{N})\setminus \{0\}}\frac{\int_{\Omega }\vert \nabla u\vert ^{p} \,\mathrm{d}x}{(\int_{\Omega }\vert u\vert ^{p^{*}}\,\mathrm{d}x)^{\frac{p}{p^{*}}}}, \end{aligned} $$

(2.3)

where $D^{1,p}(R^{N})=\{u\in L^{p^{*}}(R^{N}): \vert \nabla u\vert \in L^{p}(R ^{N})\}$. This constant S is achieved by the functional $u_{\varepsilon }$ given by

$$\begin{aligned} u_{\varepsilon }(x)=C_{Np}\varepsilon^{\frac{N-p}{p^{2}}} \bigl(\varepsilon +\vert x\vert ^{\frac{p}{p-1}} \bigr)^{\frac{p-N}{p}}, \end{aligned}$$

where the constant $C_{Np}$ is chosen such that $-\Delta_{p} u_{ \varepsilon }=\vert u_{\varepsilon }\vert ^{p^{*}-1}$ in $R^{N}$ (see [22] for details).

In order to obtain the existence of solutions to Problem (1.5), we need the following lemma.

Lemma 2.1

For each $\lambda >0$,

• (i) there exist constants $\beta_{\lambda }, \rho_{\lambda }>0$ such that $J_{\lambda }(u)\geq \beta_{\lambda }$ for $\Vert u\Vert _{\lambda V}= \rho_{\lambda }$;

• (ii) there exists an $u_{0}\in W_{\lambda ,V}^{1,p}(\Omega )$ with $u_{0}\not \equiv 0$ such that $J_{\lambda }(u_{0})<0$ for $\Vert u_{0}\Vert _{\lambda V}>\rho_{\lambda }$.

Proof

(i) Firstly, we consider the case $\lambda \geq 1$. let $\Vert u\Vert _{V} ^{p}=\rho^{p}$, then $\rho_{\lambda }^{p}=\Vert u\Vert _{\lambda V}^{p}\leq \lambda \rho^{p}$. Using (2.1) and (2.3), we have

$$\begin{aligned} J_{\lambda }(u) \geq &\frac{1}{p}\Vert u\Vert _{\lambda V}^{p}-\frac{\lambda }{p^{*}}Q_{M}S^{-\frac{p^{*}}{p}} \biggl( \int_{\Omega }\vert \nabla u\vert ^{p} \,\mathrm{d}x \biggr)^{\frac{p^{*}}{p}}-\frac{\lambda }{q}P_{M}C_{q}^{q} \Vert u\Vert _{\lambda V}^{q} \\ \geq &\frac{1}{p}\Vert u\Vert _{\lambda V}^{p}- \frac{\lambda }{p^{*}}Q_{M}S ^{-\frac{p^{*}}{p}}\Vert u\Vert _{\lambda V}^{p^{*}} -\frac{\lambda }{q}P_{M}C _{q}^{q}\Vert u\Vert _{\lambda V}^{q} \\ \geq &\rho_{\lambda }^{p} \biggl(\frac{1}{p}- \frac{\lambda^{ \frac{p^{*}}{p}}}{p^{*}}Q_{M}S^{-\frac{p^{*}}{p}}\rho^{p^{*}-p}- \frac{ \lambda^{\frac{q}{p}}}{q}P_{M}C_{q}^{q} \rho^{q-p} \biggr). \end{aligned}$$

Since $p< q< p^{*}$, taking $\rho >0$ small enough, there exists a $\beta_{\lambda }>0$ such that $J_{\lambda }(u)\geq \beta_{\lambda }$ for $\Vert u\Vert _{\lambda V}=\rho_{\lambda }$.

If $0<\lambda <1$, let $\Vert u\Vert _{V}^{p}=\rho^{p}$, then $\rho > \rho_{\lambda }>\lambda^{\frac{1}{p}} \rho$. Combining (2.2) with (2.3), we see that

$$\begin{aligned} J_{\lambda }(u) \geq &\frac{1}{p}\Vert u\Vert _{\lambda V}^{p}-\frac{\lambda }{p^{*}}Q_{M}S^{-\frac{p^{*}}{p}} \Vert u\Vert _{V}^{p^{*}}-\frac{\lambda }{q}P _{M}C_{q}^{q}\Vert u\Vert _{V}^{q} \\ >&\lambda \rho^{p} \biggl(\frac{1}{p}-\frac{1}{p^{*}}Q_{M}S^{- \frac{p^{*}}{p}} \rho^{p^{*}-p}-\frac{1}{q}P_{M}C_{q}^{q} \rho^{q-p} \biggr). \end{aligned}$$

Since $p< q< p^{*}$, taking $\rho >0$ small enough, there exists a $\beta_{\lambda }>0$ such that $J_{\lambda }(u)\geq \beta_{\lambda }$ for $\Vert u\Vert _{\lambda V}=\rho_{\lambda }$.

(ii) For $u\in W_{\lambda ,V}^{1,p}(\Omega )$ and $u\not \equiv 0$, we define

$$\begin{aligned} J_{\lambda }(t u)=\frac{t^{p}}{p}\Vert u\Vert _{\lambda V}^{p}- \frac{t^{p^{*}}}{p ^{*}}\lambda \int_{\Omega }Q(x)\vert u\vert ^{p^{*}}\,\mathrm{d}x - \frac{t^{q}}{q}\lambda \int_{\Omega }P(x)\vert u\vert ^{q}\,\mathrm{d}x,\quad t>0 , \end{aligned}$$

it follows from $\lim_{t\rightarrow +\infty }J_{\lambda }(t u)=- \infty$ that there exists a $t_{0}>0$ such that $\Vert t_{0}u\Vert _{\lambda V}>\rho_{\lambda }$ and $J_{\lambda }(t_{0}u)<0$. Letting $u_{0}=t _{0}u$, then condition (ii) holds. The proof of Lemma 2.1 is completed. □

Define

$$\begin{aligned} c=\inf_{h\in \Gamma }\sup_{t\in [0,1]}J_{\lambda } \bigl(h(t) \bigr), \end{aligned}$$

where $\Gamma =\{h\in C([0,1], W_{\lambda ,V}^{1,p}(\Omega ))\mid h(0)=0, h(1)=t_{0}u=u_{0}\}$. Using Lemma 2.1, we know that the energy functional $J_{\lambda }(u)$ satisfies the geometry of the mountain pass lemma, then there exists a $(PS)_{c}$-sequence $\{u_{n}\}\subset W _{\lambda ,V}^{1,p}(\Omega )$ such that $J_{\lambda }(u_{n})\rightarrow c$, $J_{\lambda }^{\prime }(u_{n})\rightarrow 0$ as $n\rightarrow \infty$.

Lemma 2.2

Assume (A1), (A2) hold, and $\{u_{n}\}$ be a $(PS)_{c}$-sequence at the level of c for $J_{\lambda }$ with $c< c^{*}=\min \{\frac{S ^{\frac{N}{p}}}{N\lambda^{\frac{N-p}{p}}Q_{M}^{\frac{N-p}{p}}}, \frac{S ^{\frac{N}{p}}}{2N\lambda^{\frac{N-p}{p}}Q_{m}^{\frac{N-p}{p}}} \}$, then $\{u_{n}\}$ is relatively compact in $W_{\lambda ,V}^{1,p}( \Omega )$.

Proof

Firstly, we prove that $\{u_{n}\}$ is bounded. Since $J_{\lambda }(u _{n})\rightarrow c$, $J_{\lambda }^{\prime }(u_{n})\rightarrow 0$ as $n\rightarrow \infty$, we have

$$\begin{aligned}& J_{\lambda }(u_{n}) =\frac{1}{p} \int_{\Omega } \bigl(\vert \nabla u_{n}\vert ^{p}+ \lambda V(x)\vert u_{n}\vert ^{p} \bigr) \,\mathrm{d}x - \frac{\lambda }{p^{*}} \int_{ \Omega }Q(x)\vert u_{n}\vert ^{p^{*}} \,\mathrm{d}x -\frac{\lambda }{q} \int_{ \Omega }P(x)\vert u_{n}\vert ^{q} \,\mathrm{d}x \\& \hphantom{J_{\lambda }(u_{n})}=c+o(1)\Vert u_{n}\Vert , \\& \int_{\Omega } \bigl(\vert \nabla u_{n}\vert ^{p}+\lambda V(x)\vert u_{n}\vert ^{p} \bigr) \,\mathrm{d}x- \lambda \int_{\Omega }Q(x)\vert u_{n}\vert ^{p^{*}} \,\mathrm{d}x -\lambda \int_{\Omega }P(x)\vert u_{n}\vert ^{q} \,\mathrm{d}x \\& \quad =o(1)\Vert u_{n}\Vert . \end{aligned}$$

Combining (A1) and (A2), one has

$$\begin{aligned} c+o(1)\Vert u_{n}\Vert =& \biggl(\frac{1}{p}- \frac{1}{q} \biggr) \int_{\Omega } \bigl(\vert \nabla u_{n}\vert ^{p}+\lambda V(x)\vert u_{n}\vert ^{p} \bigr) \,\mathrm{d}x +\lambda \biggl(\frac{1}{q}-\frac{1}{p ^{*}} \biggr)\int_{\Omega } Q(x) \vert u_{n}\vert ^{p^{*}}\,\mathrm{d}x \\ \geq & \biggl(\frac{1}{p}-\frac{1}{q} \biggr)\Vert u_{n}\Vert _{\lambda V}^{p}. \end{aligned}$$

Thus, we can find that $\{u_{n}\}$ is bounded in $W_{\lambda ,V}^{1,p}( \Omega )$.

Next, we prove that $\{u_{n}\}$ is relatively compact in $W_{\lambda ,V}^{1,p}(\Omega )$. Since $\{u_{n}\}$ is bounded in $W_{\lambda ,V} ^{1,p}(\Omega )$, there exists a subsequence, still denoted by $\{u_{n}\}$ and $u \in W_{\lambda ,V}^{1,p}(\Omega )$ such that

$$\begin{aligned}& u_{n}\rightharpoonup u\quad \mbox{weakly in } W_{\lambda ,V}^{1,p}( \Omega ), \\& u_{n}\rightharpoonup u \quad \mbox{weakly in } L^{p^{*}}( \Omega ), \\& u_{n}\rightarrow u \quad \mbox{strongly in } L^{q}(\Omega ), p \leq q< p ^{*} , \\& u_{n}\rightarrow u\quad \mbox{a.e. in } \Omega . \end{aligned}$$

By the Lions concentration-compactness principle [38], there exists at most set J, a set of different points $\{x_{j}\}_{j\in J}\subset \overline{ \Omega }$, sets of nonnegative real numbers $\{\mu_{j}\}_{j\in J}$, $\{\nu_{j}\}_{j\in J}$ such that

$$ \begin{aligned} &\vert \nabla u_{n}\vert ^{p}\rightharpoonup \,\mathrm{d}\mu \geq \vert \nabla u\vert ^{p}+ \sum_{j\in J}\mu_{j} \delta_{x_{j}}, \\ &\vert u_{n}\vert ^{p^{*}}\rightharpoonup \,\mathrm{d}\nu = \vert u\vert ^{p^{*}}+\sum_{j\in J} \nu_{j}\delta_{x_{j}}, \end{aligned} $$

(2.4)

where $\delta_{x}$ is the Dirac mass at x, and the constants $\mu_{j}$, $\nu_{j}$ satisfying

$$\begin{aligned}& S\nu_{j}^{\frac{p}{p^{*}}}\leq \mu_{j},\quad \mbox{where } x_{j} \in \Omega , \end{aligned}$$

(2.5)

$$\begin{aligned}& \frac{S}{2^{\frac{p}{N}}}\nu_{j}^{\frac{p}{p^{*}}} \leq \mu_{j}, \quad \mbox{where } x_{j}\in \partial \Omega . \end{aligned}$$

(2.6)

Next, we prove $\mu_{j}=0$ and $\nu_{j}=0$, where $j\in J$. In fact, choosing $\varepsilon >0$ sufficiently small such that $B_{\varepsilon }(x_{i})\cap B_{\varepsilon }(x_{j})=\varnothing$ for $i\neq j$, $i,j \in J$. Let $\phi_{\varepsilon }^{j}(x)$ be a smooth cut off function centered at $x_{j}$ such that

$$\begin{aligned} 0\leq \phi_{\varepsilon }^{j}(x)\leq 1 \quad \mbox{for } \vert x-x_{j}\vert < \varepsilon ,\quad \phi_{\varepsilon }^{j}(x)= \textstyle\begin{cases} 1, \vert x-x_{j}\vert \leq \frac{\varepsilon }{2}, \\ 0, \vert x-x_{j}\vert \geq \varepsilon , \end{cases}\displaystyle \mbox{and}\quad \bigl\vert \nabla \phi_{\varepsilon }^{j} \bigr\vert \leq \frac{4}{\varepsilon }. \end{aligned}$$

Noting that

$$\begin{aligned}& \bigl\langle J_{\lambda }^{\prime }(u_{n}),u_{n} \phi_{\varepsilon }^{j}(x) \bigr\rangle \\& \quad = \int_{\Omega }\vert \nabla u_{n}\vert ^{p} \phi_{\varepsilon }^{j}(x) \,\mathrm{d}x+ \int_{\Omega }\vert \nabla u_{n}\vert ^{p-2} \nabla u_{n}\nabla \phi_{\varepsilon }^{j}(x)u_{n} \,\mathrm{d}x \\& \qquad {}+\lambda \int_{\Omega }V(x)\vert u_{n}\vert ^{p} \phi_{\varepsilon }^{j}(x) \,\mathrm{d}x -\lambda \int_{\Omega }Q(x)\vert u_{n}\vert ^{p^{*}} \phi_{\varepsilon }^{j}(x)\,\mathrm{d}x \\& \qquad {}-\lambda \int_{\Omega }P(x)\vert u_{n}\vert ^{q} \phi_{\varepsilon }^{j}(x) \,\mathrm{d}x, \end{aligned}$$

and by (2.4), we have

$$\begin{aligned}& \lim_{\varepsilon \rightarrow 0}\lim_{n\rightarrow \infty } \int_{\Omega }\vert \nabla u_{n}\vert ^{p} \phi_{\varepsilon }^{j}(x)\,\mathrm{d}x \geq \lim _{\varepsilon \rightarrow 0} \biggl[ \int_{\Omega }\vert \nabla u\vert ^{p} \phi_{\varepsilon }^{j}(x)\,\mathrm{d}x+ \int_{\Omega }\sum_{j\in J}\mu _{j}\delta_{x_{j}}\phi_{\varepsilon }^{j}(x) \, \mathrm{d}x \biggr] \geq \mu _{j}, \\& \lim_{\varepsilon \rightarrow 0}\lim_{n\rightarrow \infty } \int_{\Omega }\vert \nabla u_{n}\vert ^{p-2} \nabla u_{n}\nabla \phi_{\varepsilon }^{j}(x)u_{n} \,\mathrm{d}x=0, \\& \lim_{\varepsilon \rightarrow 0}\lim_{n\rightarrow \infty } \int_{\Omega }V(x)\vert u_{n}\vert ^{p} \phi_{\varepsilon }^{j}(x)\,\mathrm{d}x=0, \\ & \lim_{\varepsilon \rightarrow 0}\lim_{n\rightarrow \infty } \int_{\Omega }Q(x)\vert u_{n}\vert ^{p^{*}} \phi_{\varepsilon }^{j}(x) \,\mathrm{d}x=Q(x_{j}) \nu_{j}, \\ & \lim_{\varepsilon \rightarrow 0}\lim_{n\rightarrow \infty } \int_{\Omega }P(x)\vert u_{n}\vert ^{q} \phi_{\varepsilon }^{j}(x)\,\mathrm{d}x=0. \end{aligned}$$

Thus,

$$ 0=\lim_{\varepsilon \rightarrow 0}\lim_{n\rightarrow \infty } \bigl\langle J_{\lambda }^{\prime }(u_{n}),u_{n} \phi_{\varepsilon }^{j}(x) \bigr\rangle \geq \mu_{j}- \lambda Q(x_{j})\nu_{j}. $$

If $\nu_{j}\neq 0$, by (2.5) and (2.6), we find that

$$\begin{aligned}& \nu_{j}\geq \frac{S^{\frac{N}{p}}}{\lambda^{\frac{N}{p}}Q^{ \frac{N}{p}}(x_{j})},\quad x_{j}\in \Omega , \\ & \nu_{j}\geq \frac{S^{\frac{N}{p}}}{2\lambda^{\frac{N}{p}}Q^{ \frac{N}{p}}(x_{j})},\quad x_{j}\in \partial \Omega . \end{aligned}$$

On the other hand,

$$\begin{aligned} c =&\lim_{n\rightarrow \infty } \biggl(J_{\lambda }(u_{n})- \frac{1}{p} \bigl\langle J_{\lambda }^{\prime }(u_{n}),u_{n} \bigr\rangle \biggr) \\ =& \biggl(\frac{1}{p}-\frac{1}{p^{*}} \biggr)\lambda \int_{\Omega }Q(x)\vert u\vert ^{p^{*}} \,\mathrm{d}x + \biggl(\frac{1}{p}-\frac{1}{q} \biggr)\lambda \int_{\Omega }P(x) \vert u\vert ^{q} \,\mathrm{d}x+ \biggl(\frac{1}{p}-\frac{1}{p^{*}} \biggr)\lambda \sum _{j\in J}Q(x_{j}) \nu_{j} \\ \geq &\frac{1}{N}\lambda \sum_{j\in J}Q(x_{j}) \nu_{j}, \end{aligned}$$

consequently,

$$\begin{aligned}& c\geq \frac{1}{N}\lambda Q(x_{j})\nu_{j}\geq \frac{S^{\frac{N}{p}}}{N \lambda^{\frac{N-p}{p}}Q^{\frac{N-p}{p}}_{M}},\quad x_{j}\in \Omega , \\ & c\geq \frac{1}{N}\lambda Q(x_{j})\nu_{j}\geq \frac{S^{\frac{N}{p}}}{2N \lambda^{\frac{N-p}{p}}Q^{\frac{N-p}{p}}_{m}}, \quad x_{j}\in \partial \Omega , \end{aligned}$$

which is a contradiction. Hence, $\mu_{j}=0$, $\nu_{j}=0$ and we find that $u_{n}\rightarrow u$ strongly in $L^{p^{*}}(\Omega )$.

Now, we prove that $u_{n}\rightarrow u$ strongly in $W_{\lambda ,V} ^{1,p}(\Omega )$. We have

$$\begin{aligned}& \bigl\langle J_{\lambda }^{\prime }(u_{n})-J_{\lambda }^{\prime }(u),u_{n}-u \bigr\rangle \\& \quad =\Vert u_{n}\Vert _{\lambda ,V}^{p}+ \Vert u \Vert _{\lambda ,V}^{p}- \int_{ \Omega } \bigl(\vert \nabla u_{n}\vert ^{p-2}\nabla u_{n}\nabla u+\lambda V(x)\vert u_{n}\vert ^{p-2}u _{n}u \bigr)\,\mathrm{d}x \\& \qquad {}- \int_{\Omega } \bigl(\vert \nabla u\vert ^{p-2}\nabla u \nabla u_{n}+\lambda V(x)\vert u\vert ^{p-2}uu _{n} \bigr)\,\mathrm{d}x-I-II, \end{aligned}$$

where

$$\begin{aligned}& I =\lambda \int_{\Omega }Q(x) \bigl(\vert u_{n}\vert ^{p^{*}-2}u_{n}-\vert u\vert ^{p^{*}-2}u \bigr) (u _{n}-u)\,\mathrm{d}x, \\& II =\lambda \int_{\Omega }P(x) \bigl(\vert u_{n}\vert ^{q-2}u_{n}-\vert u\vert ^{q-2}u \bigr) (u_{n}-u) \,\mathrm{d}x. \end{aligned}$$

By the Hölder inequality and Jensen’s inequality

$$\begin{aligned} (a+b)^{\alpha }(c+d)^{1-\alpha }\geq a^{\alpha }c^{1-\alpha }+b^{ \alpha }d^{1-\alpha }, \end{aligned}$$

where $\alpha \in (0,1)$, $a>0$, $b>0$, $c>0$, $d>0$, we have

$$\begin{aligned}& \int_{\Omega } \bigl(\vert \nabla u_{n}\vert ^{p-2}\nabla u_{n}\nabla u+\lambda V(x)\vert u_{n}\vert ^{p-2}u_{n}u \bigr)\,\mathrm{d}x \\& \quad \leq \biggl( \int_{\Omega }\vert \nabla u_{n}\vert ^{p} \,\mathrm{d}x \biggr)^{ \frac{p-1}{p}} \biggl( \int_{\Omega }\vert \nabla u\vert ^{p}\,\mathrm{d}x \biggr)^{ \frac{1}{p}} \\& \qquad {}+ \biggl(\lambda \int_{\Omega }V(x)\vert u_{n}\vert ^{p} \, \mathrm{d}x \biggr)^{\frac{p-1}{p}} \biggl(\lambda \int_{\Omega }V(x)\vert u\vert ^{p} \,\mathrm{d}x \biggr)^{\frac{1}{p}} \\& \quad \leq \biggl( \int_{\Omega }\vert \nabla u_{n}\vert ^{p}+\lambda V(x)\vert u_{n}\vert ^{p} \, \mathrm{d}x \biggr)^{\frac{p-1}{p}} \biggl( \int_{\Omega }\vert \nabla u\vert ^{p}+ \lambda V(x) \vert u\vert ^{p}\,\mathrm{d}x \biggr)^{\frac{1}{p}} =\Vert u_{n}\Vert _{\lambda V}^{p-1}\Vert u \Vert _{\lambda V}. \end{aligned}$$

Similarly, we get

$$\begin{aligned}& \int_{\Omega } \bigl(\vert \nabla u\vert ^{p-2}\nabla u \nabla u_{n}+\lambda V(x)\vert u\vert ^{p-2}uu _{n} \bigr)\,\mathrm{d}x\leq \Vert u\Vert _{\lambda V}^{p-1} \Vert u_{n}\Vert _{\lambda V}, \\& \vert I\vert \leq \lambda Q_{M} \biggl[ \int_{\Omega }\vert u_{n}\vert ^{p^{*}-1} \vert u_{n}-u\vert \,\mathrm{d}x+ \int_{\Omega }\vert u\vert ^{p^{*}-1}\vert u_{n}-u\vert \,\mathrm{d}x \biggr] \\& \hphantom{\vert I\vert } \leq \lambda Q_{M} \biggl( \int_{\Omega }\vert u_{n}\vert ^{p^{*}} \, \mathrm{d}x \biggr)^{\frac{p^{*}-1}{p^{*}}} \biggl( \int_{\Omega }\vert u_{n}-u\vert ^{p^{*}} \, \mathrm{d}x \biggr)^{\frac{1}{p^{*}}} \\& \hphantom{\vert I\vert \leq} {}+\lambda Q_{M} \biggl( \int_{\Omega }\vert u\vert ^{p^{*}}\,\mathrm{d}x \biggr)^{\frac{p ^{*}-1}{p^{*}}} \biggl( \int_{\Omega }\vert u_{n}-u\vert ^{p^{*}} \, \mathrm{d}x \biggr)^{\frac{1}{p ^{*}}}, \\& \vert II\vert \leq \lambda P_{M} \biggl[ \int_{\Omega }\vert u_{n}\vert ^{q-1} \vert u_{n}-u\vert \,\mathrm{d}x+ \int_{\Omega }\vert u\vert ^{q-1}\vert u_{n}-u\vert \,\mathrm{d}x \biggr] \\& \hphantom{\vert II\vert } \leq \lambda P_{M} \biggl( \int_{\Omega }\vert u_{n}\vert ^{q} \, \mathrm{d}x \biggr)^{ \frac{q-1}{q}} \biggl( \int_{\Omega }\vert u_{n}-u\vert ^{q} \, \mathrm{d}x \biggr)^{ \frac{1}{q}} \\& \hphantom{\vert II\vert \leq} {}+\lambda P_{M} \biggl( \int_{\Omega }\vert u\vert ^{q}\,\mathrm{d}x \biggr)^{ \frac{q-1}{q}} \biggl( \int_{\Omega }\vert u_{n}-u\vert ^{q} \, \mathrm{d}x \biggr)^{ \frac{1}{q}}. \end{aligned}$$

We have

$$\begin{aligned} 0 =&\lim_{n\rightarrow \infty } \bigl\langle J_{\lambda }^{\prime }(u_{n})-J_{\lambda }^{\prime }(u),u_{n}-u \bigr\rangle \geq \lim_{n\rightarrow \infty } \bigl(\Vert u_{n}\Vert _{\lambda V}^{p-1}-\Vert u\Vert _{\lambda V}^{p-1} \bigr) \bigl(\Vert u_{n}\Vert _{\lambda V}-\Vert u\Vert _{\lambda V} \bigr) \geq 0. \end{aligned}$$

Hence, $u_{n}\rightarrow u$ strongly in $W_{\lambda ,V}^{1,p}(\Omega )$. □

Since $0\in \partial \Omega$ and $\partial \Omega \in C^{2}$, the boundary ∂Ω near the origin can be represented $x_{N}=h(x^{\prime })=\frac{1}{2}\sum_{i=1}^{N-1}\lambda_{i}x_{i}^{2}+o(\vert x^{\prime }\vert ^{2})$, where $x^{\prime }=(x_{1},x_{2},\ldots ,x_{N-1}) \in D(0,\delta )=B(0,\delta )\cap \{x_{N}=0\}, \lambda_{i}$ ($i=1,2, \ldots ,N-1$) are the principal curvatures of ∂Ω at 0 and the mean curvatures $H(0)=\frac{1}{N-1}\sum_{i=1}^{N-1}\lambda _{i}>0$. Then the following lemma holds.

Lemma 2.3

([22])

(1) For $N> 2p-1$ and $\varepsilon >0$ small enough,

$$\begin{aligned}& \int_{\Omega }\vert \nabla u_{\varepsilon }\vert ^{p}\,\mathrm{d}x = \int_{R_{+} ^{N}}\vert \nabla u_{\varepsilon }\vert ^{p}\,\mathrm{d}x-K_{1}(\varepsilon )+o \bigl( \varepsilon^{\frac{p-1}{p}} \bigr), \\& \int_{\Omega }\vert u_{\varepsilon }\vert ^{p ^{*}} \, \mathrm{d}x= \int_{R_{+}^{N}}\vert u_{\varepsilon }\vert ^{p^{*}} \, \mathrm{d}x-K_{2}(\varepsilon )+o \bigl(\varepsilon^{\frac{p-1}{p}} \bigr), \end{aligned}$$

where $K_{1}(\varepsilon )$, $K_{2}(\varepsilon )$ satisfy

$$\begin{aligned}& \lim_{\varepsilon \rightarrow 0}\varepsilon^{-\frac{p-1}{p}}K_{1}( \varepsilon ) =\frac{1}{2}H(0)C_{Np}^{p} \biggl( \frac{N-p}{p-1} \biggr)^{p} \int_{R^{N-1}} \bigl(1+ \bigl\vert x^{\prime } \bigr\vert ^{\frac{p}{p-1}} \bigr)^{-N} \bigl\vert x^{\prime } \bigr\vert ^{ \frac{3p-2}{p-1}}\,\mathrm{d}x^{\prime }=K_{1}, \\& \lim_{\varepsilon \rightarrow 0}\varepsilon^{-\frac{p-1}{p}}K_{2}( \varepsilon ) =\frac{1}{2}H(0)C_{Np}^{p^{*}} \int_{R^{N-1}} \bigl(1+ \bigl\vert x^{\prime } \bigr\vert ^{\frac{p}{p-1}} \bigr)^{-N} \bigl\vert x^{\prime } \bigr\vert ^{2}\,\mathrm{d}x^{\prime } =K_{2}. \end{aligned}$$

(2)

$$\begin{aligned}& \int_{\Omega }\vert u_{\varepsilon }\vert ^{p} \, \mathrm{d}x= \textstyle\begin{cases} O(\varepsilon^{\frac{N-p}{p}}), & N< p^{2}, \\ O(\varepsilon^{\frac{N-p}{p}}\vert \ln \varepsilon \vert ), & N=p^{2}, \\ O(\varepsilon^{p-1}), & N>p^{2}. \end{cases}\displaystyle \end{aligned}$$

(3)

$$\begin{aligned}& \int_{\Omega }\vert u_{\varepsilon }\vert ^{q} \, \mathrm{d}x= \textstyle\begin{cases} O(\varepsilon^{\frac{q(N-p)}{p^{2}}}), & q< \frac{N(p-1)}{N-p}, \\ O(\varepsilon^{\frac{q(N-p)}{p^{2}}}\vert \ln \varepsilon \vert ), & q= \frac{N(p-1)}{N-p}, \\ O(\varepsilon^{\frac{(p-1)(Np-q(N-p))}{p^{2}}}), & q> \frac{N(p-1)}{N-p}. \end{cases}\displaystyle \end{aligned}$$

3 Proof of main results

Let $\varphi (x)\in C_{0}^{\infty }(R^{N})$ be a smooth cut off function such that

$$\begin{aligned}& 0\leq \varphi (x)\leq 1,\quad \frac{\delta }{2}\leq \vert x\vert \leq \delta ; \\& \varphi (x)=1, \quad \vert x\vert < \frac{\delta }{2}; \\& \varphi (x)=0,\quad \vert x\vert >\delta . \end{aligned}$$

Define $\omega_{\varepsilon }=\varphi u_{\varepsilon }$, then we have the following lemma about $\omega_{\varepsilon }$.

Lemma 3.1

Suppose $N\geq 2p$, $0\in \partial \Omega$. If the function $V(x)$ satisfies $\int_{\Omega \cap B(0,\delta )}V^{r^{\prime }}\,\mathrm{d}x<\infty$, then

$$\begin{aligned}& \int_{\Omega \cap B(0,\delta )}V\omega_{\varepsilon }^{p}\,\mathrm{d}x=O \bigl( \varepsilon^{\frac{N-p}{p}+p-N+\frac{N(p-1)}{pr}} \bigr), \end{aligned}$$

where $\frac{1}{r}+\frac{1}{r^{\prime }}=1$, $1< r<\frac{N(p-1)}{Np+2p-N-p ^{2}-1}$.

Proof

According to the Hölder inequality and the definition of $\omega_{\varepsilon }$, we have

$$\begin{aligned} \int_{\Omega \cap B(0,\delta )}V\omega_{\varepsilon }^{p}\,\mathrm{d}x \leq & \biggl( \int_{\Omega \cap B(0,\delta )}V^{r^{\prime }}\,\mathrm{d}x \biggr)^{\frac{1}{r^{\prime }}} \biggl( \int_{\Omega \cap B(0,\delta )} \omega_{\varepsilon }^{pr}\,\mathrm{d}x \biggr)^{\frac{1}{r}} \\ \leq & \varepsilon^{\frac{(N-p)}{p}}C_{Np}^{p} \biggl( \int_{\Omega \cap B(0,\delta )}V^{r^{\prime }}\,\mathrm{d}x \biggr)^{\frac{1}{r ^{\prime }}} \biggl( \int_{ B(0,\delta )} \bigl(\varepsilon +\vert x\vert ^{\frac{p}{p-1}} \bigr)^{r(p-N)} \,\mathrm{d}x \biggr)^{\frac{1}{r}} \\ =&\varepsilon^{{\frac{N-p}{p}+p-N+\frac{N(p-1)}{pr}}}C_{Np}^{p} \biggl( \int_{\Omega \cap B(0,\delta )} V^{r^{\prime }}\,\mathrm{d}x \biggr)^{\frac{1}{r^{\prime }}} \\ &{}\times \biggl( \int_{ B(0,\delta \varepsilon^{-\frac{p-1}{p}})} \bigl(1+\vert x\vert ^{ \frac{p}{p-1}} \bigr)^{r(p-N)}\,\mathrm{d}x \biggr)^{\frac{1}{r}} . \end{aligned}$$

Noting that $\frac{N(p-1)}{(N-p)p}\leq 1< r$, a series of computations yield

$$\begin{aligned} & \int_{\Omega \cap B(0,\delta )}V\omega_{\varepsilon }^{p}\,\mathrm{d}x = O \bigl(\varepsilon^{\frac{N-p}{p}+p-N+\frac{N(p-1)}{pr}} \bigr). \end{aligned}$$

 □

Lemma 3.2

Suppose that (A1), (A2) hold and $0\in \partial \Omega$, $H(0)>0$, $Q _{m}=Q(0)$. If the functions $Q(x)$, $V(x)$ satisfy the conditions (A3), (A4), then there exists a nonnegative function $v\in W_{\lambda ,V}^{1,p}(\Omega )$, $v\not \equiv 0$, such that

$$\begin{aligned} \sup_{t\geq 0}J_{\lambda }(tv)< c^{*} \end{aligned}$$

(3.1)

for each $\lambda >0$, $N\geq 2p$.

Proof

We divide the proof into three steps.

(i) We consider the functional

$$\begin{aligned} g(t) =&J_{\lambda }(t\omega_{\varepsilon }) \\ =&\frac{t^{p}}{p} \int_{\Omega } \bigl(\vert \nabla \omega_{\varepsilon }\vert ^{p}+ \lambda V(x)\vert \omega_{\varepsilon }\vert ^{p} \bigr)\,\mathrm{d}x-\frac{t^{p*}}{p ^{*}}\lambda \int_{\Omega }Q(x)\vert \omega_{\varepsilon }\vert ^{p^{*}} \,\mathrm{d}x \\ &{}-\frac{t^{q}}{q}\lambda \int_{\Omega }P(x)\vert \omega_{\varepsilon }\vert ^{q}\,\mathrm{d}x, \quad t>0. \end{aligned}$$

Noting that $\lim_{t\rightarrow \infty }g(t)=-\infty$, $g(0)=0$, $g(t)>0$ for $t\rightarrow 0^{+}$, we know that there exists a $t_{\varepsilon }>0$ such that $\sup_{t>0}g(t)$ is attained for $t_{\varepsilon }$ and $t_{\varepsilon }$ is uniformly bounded for $\varepsilon >0$ sufficiently small. Thus,

$$\begin{aligned} g(t_{\varepsilon }) =&\sup_{t\geq 0} J_{\lambda }(t\omega_{\varepsilon }) \\ \leq &\sup_{t\geq 0} \biggl[\frac{t^{p}}{p} \int_{\Omega }\vert \nabla\omega_{\varepsilon }\vert ^{p}\,\mathrm{d}x-\frac{t^{p*}}{p^{*}}\lambda \int_{\Omega }Q(x)\vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x \biggr] \\ &{}+\frac{t_{\varepsilon }^{p}}{p} \int_{\Omega }\lambda V(x)\vert \omega_{\varepsilon }\vert ^{p}\,\mathrm{d}x-\frac{t_{\varepsilon }^{q}}{q} \lambda \int_{\Omega }P(x)\vert \omega_{\varepsilon }\vert ^{q}\,\mathrm{d}x \\ =&\frac{1}{N} \biggl[\frac{\int_{\Omega }\vert \nabla \omega_{\varepsilon }\vert ^{p} \,\mathrm{d}x}{(\lambda \int_{\Omega }Q(x)\vert \omega_{\varepsilon }\vert ^{p ^{*}}\,\mathrm{d}x)^{\frac{N-p}{N}}} \biggr]^{\frac{N}{p}}+ \frac{t_{\varepsilon }^{p}}{p} \int_{\Omega }\lambda V(x)\vert \omega_{\varepsilon }\vert ^{p} \,\mathrm{d}x \\ &{}-\frac{t_{\varepsilon }^{q}}{q}\lambda \int_{\Omega }P(x)\vert \omega_{\varepsilon }\vert ^{q}\,\mathrm{d}x. \end{aligned}$$

(3.2)

(ii) When $\varepsilon >0$ is sufficiently small, we have

$$\begin{aligned}& \begin{aligned} & \int_{\Omega }Q(x)\vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x =Q_{m} \int_{\Omega }\vert u_{\varepsilon }\vert ^{p^{*}} \, \mathrm{d}x+o \bigl( \varepsilon^{\frac{p-1}{p}} \bigr), \\ & \int_{\Omega }\vert \nabla \omega_{\varepsilon }\vert ^{p}\,\mathrm{d}x\leq \int_{\Omega }\vert \nabla u_{\varepsilon }\vert ^{p}\,\mathrm{d}x+o \bigl(\varepsilon^{ \frac{p-1}{p}} \bigr), \\ &\int_{\Omega }\vert \omega_{\varepsilon }\vert ^{q} \,\mathrm{d}x= \int_{\Omega }\vert u_{\varepsilon }\vert ^{q} \, \mathrm{d}x+o \bigl(\varepsilon^{\frac{p-1}{p}} \bigr), \\ &\int_{\Omega }\vert \omega_{\varepsilon }\vert ^{p^{*}} \,\mathrm{d}x= \int_{ \Omega }\vert u_{\varepsilon }\vert ^{p^{*}} \, \mathrm{d}x+o \bigl( \varepsilon^{\frac{p-1}{p}} \bigr). \end{aligned} \end{aligned}$$

(3.3)

We firstly prove the first formula. Since $\vert Q(x)-Q(0)\vert =o(\vert x\vert ^{\alpha })$ for $x\rightarrow 0$, there exists a $0<\delta_{0}\leq \delta$ such that $\vert Q(x)-Q(0)\vert \leq C\vert x\vert ^{\alpha }$ for $\vert x\vert <\delta_{0}$, where $C>0$ is constant. Moreover

$$\begin{aligned}& \int_{\Omega } \bigl\vert Q(x)-Q(0) \bigr\vert \vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x \\& \quad \leq \int_{\Omega \cap \vert x\vert \leq \delta_{0}} \bigl\vert Q(x)-Q(0) \bigr\vert \vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x+ \int_{\Omega \cap \vert x\vert \geq \delta_{0}} \bigl\vert Q(x)-Q(0) \bigr\vert \vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x \\& \quad \leq C \int_{\vert x\vert \leq \delta_{0}} \vert x\vert ^{\alpha }\vert \omega_{\varepsilon} \vert ^{p^{*}}\,\mathrm{d}x +2Q_{M} \int_{\Omega \cap \vert x\vert \geq \delta_{0}} \vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x \\& \quad \leq C C_{Np}^{p^{*}}\varepsilon^{\frac{(p-1)\alpha }{p}} \int_{\vert x\vert \leq \frac{\delta_{0}}{\varepsilon^{\frac{p-1}{p}}}}\vert x\vert ^{ \alpha } \bigl(1+\vert x \vert ^{\frac{p}{p-1}} \bigr)^{-N}\,\mathrm{d}x \\& \qquad {}+2Q_{M}C_{Np}^{p^{*}}\varepsilon^{\frac{N}{p}} \int_{\Omega \cap \vert x\vert \geq \delta_{0}} \bigl(\varepsilon +\vert x\vert ^{\frac{p}{p-1}} \bigr)^{-N} \,\mathrm{d}x \\& \quad =O \bigl(\varepsilon^{\frac{(p-1)\alpha }{p}} \bigr)+O \bigl(\varepsilon^{\frac{N}{p}} \bigr). \end{aligned}$$

Since $N\geq 2p$, $1<\alpha <\frac{N}{p-1}$, $\int_{\Omega }\vert Q(x)-Q(0)\vert \vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x=o(\varepsilon^{ \frac{p-1}{p}})$, which implies

$$\begin{aligned}& \int_{\Omega }Q(x)\vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x \\& \quad =Q_{m} \int_{\Omega }\vert \omega_{\varepsilon }\vert ^{p^{*}} \,\mathrm{d}x+ \int_{\Omega } \bigl(Q(x)-Q(0) \bigr)\vert \omega_{\varepsilon } \vert ^{p^{*}}\,\mathrm{d}x \\& \quad =Q_{m} \int_{\Omega }\vert \omega_{\varepsilon }\vert ^{p^{*}} \,\mathrm{d}x+o \bigl( \varepsilon^{\frac{p-1}{p}} \bigr). \end{aligned}$$

Similarly, we can evaluate the rest of formulas and omit the details here.

(iii) $\sup_{t\geq 0}J_{\lambda }(t\omega_{\varepsilon })< c^{*}$.

Combining (3.3) with Lemma 2.3, one has

$$\begin{aligned}& \int_{\Omega }\vert \nabla \omega_{\varepsilon }\vert ^{p}\,\mathrm{d}x \le M _{1}\bigl(1-M_{1}^{-1}K_{1}\bigl( \varepsilon )+o \bigl(\varepsilon^{\frac{p-1}{p}}\bigr) \bigr), \\& \int_{\Omega }\vert \omega_{\varepsilon }\vert ^{p^{*}} \,\mathrm{d}x = M_{2}\bigl(1-M _{2}^{-1}K_{2}\bigl( \varepsilon )+o \bigl(\varepsilon^{\frac{p-1}{p}}\bigr)\bigr), \end{aligned}$$

where $M_{1}=\frac{1}{2}\int_{R^{N}}\vert \nabla u_{\varepsilon }\vert ^{p} \,\mathrm{d}x$, $M_{2}=\frac{1}{2}\int_{R^{N}}\vert u_{\varepsilon }\vert ^{p^{*}} \,\mathrm{d}x$. Then, using (3.2), (3.3), Lemma 2.3 and Lemma 3.1, we see that

$$\begin{aligned} \sup_{t\geq 0}J_{\lambda }(t\omega_{\varepsilon }) \leq & \frac{S^{ \frac{N}{p}}}{2N(\lambda Q_{m})^{\frac{N-p}{p}}} \biggl[1+\frac{N-p}{p}M _{2}^{-1}K_{2}( \varepsilon ) -\frac{N}{p}M_{1}^{-1}K_{1}( \varepsilon )+o \bigl(\varepsilon^{\frac{p-1}{p}} \bigr) \biggr] \\ &{}+O \bigl( \varepsilon^{\frac{N-p}{p}+p-N+\frac{(p-1)N}{pr}} \bigr). \end{aligned}$$

Next, we claim that

$$ \begin{aligned} \lim_{\varepsilon \rightarrow 0} \varepsilon^{\frac{p-1}{p}} \biggl[ \frac{N-p}{p}M_{2}^{-1}K_{2}( \varepsilon ) -\frac{N}{p}M_{1}^{-1}K _{1}( \varepsilon ) \biggr]< 0 \end{aligned} $$

(3.4)

for $\varepsilon > 0$ small enough, which implies (3.1) holds. According to $\lim_{\varepsilon \rightarrow 0} \varepsilon^{\frac{p-1}{p}}K_{1}(\varepsilon ) =K_{1}$, $\lim_{\varepsilon \rightarrow 0}\varepsilon^{\frac{p-1}{p}}K_{2}( \varepsilon ) =K_{2}$, we know that (3.4) is equivalent to $\frac{K_{1}}{K_{2}} >\frac{N-p}{N}\frac{M_{1}}{M_{2}}$.

From the expressions of $K_{1}$, $K_{2}$, $M_{1}$, $M_{2}$ and $u_{\varepsilon }$, a series of computations yield

$$\begin{aligned}& \frac{K_{1}}{K_{2}} = \frac{\frac{1}{2}H(0)C_{Np}^{p}(\frac{N-p}{p-1})^{p} \int_{R^{N-1}} (1+\vert x^{\prime }\vert ^{\frac{p}{p-1}})^{-N}\vert x^{\prime }\vert ^{ \frac{3p-2}{p-1}}\,\mathrm{d}x^{\prime }}{\frac{1}{2}H(0)C_{Np}^{p^{*}} \int_{R^{N-1}} (1+\vert x^{\prime }\vert ^{\frac{p}{p-1}})^{-N}\vert x^{\prime }\vert ^{2} \,\mathrm{d}x^{\prime }} \\& \hphantom{\frac{K_{1}}{K_{2}}}=C_{Np}^{p-p^{*}} \biggl(\frac{N-p}{p-1} \biggr)^{p} \frac{\int_{0}^{\infty }(1+r ^{2})^{-N}r^{\frac{2Np+3p-2N-2}{p}}\,\mathrm{d}r}{\int_{0}^{\infty }(1+r ^{2})^{-N}r^{\frac{2Np+p-2N-2}{p}}\,\mathrm{d}r}, \\& \frac{N-p}{N}\frac{M_{1}}{M_{2}} =\frac{N-p}{N}\frac{\int_{R^{N}}\vert \nabla u_{\varepsilon }\vert ^{p}\,\mathrm{d}x}{\int_{R^{N}}\vert u_{\varepsilon} \vert ^{p^{*}}\,\mathrm{d}x} \\& \hphantom{\frac{N-p}{N}\frac{M_{1}}{M_{2}}}=\frac{N-p}{N}C_{Np}^{p-p^{*}} \biggl(\frac{N-p}{p-1} \biggr)^{p}\frac{\int_{0} ^{\infty }(1+r^{2})^{-N}r^{\frac{2Np+p-2N}{p}}\,\mathrm{d}r}{\int_{0} ^{\infty }(1+r^{2})^{-N}r^{\frac{2Np-p-2N}{p}}\,\mathrm{d}r}. \end{aligned}$$

Integrating by parts, we have

$$\int_{0}^{\infty }\frac{r^{\beta }}{(1+r^{2})^{n}}\,\mathrm{d}r= \frac{ \beta -1}{2n-\beta -1} \int_{0}^{\infty } \frac{r^{\beta -2}}{(1+r^{2})^{n}}\,\mathrm{d}r\quad \mbox{for }2\leq \beta < 2n-1 . $$

Then

$$\begin{aligned}& \frac{K_{1}}{K_{2}} =C_{Np}^{p-p^{*}} \biggl( \frac{N-p}{p-1} \biggr)^{p} \frac{(p-1)(N+1)}{N-2p+1}, \\& \frac{N-p}{N}\frac{M_{1}}{M_{2}}=C_{Np}^{p-p^{*}} \biggl( \frac{N-p}{p-1} \biggr)^{p}(p-1). \end{aligned}$$

This implies $\frac{K_{1}}{K_{2}} >\frac{N-p}{N}\frac{M_{1}}{M_{2}}$. Thus

$$ \begin{aligned} \sup_{t\geq 0}J_{\lambda }(t \omega_{\varepsilon })< \frac{S^{ \frac{N}{p}}}{2N(\lambda Q_{m})^{\frac{N-p}{p}}}=c^{*}. \end{aligned} $$

The proof of Lemma 3.2 is complete. □

Proof of Theorem 1.1

Applying Lemma 2.1 and Lemma 3.2, we obtain

$$c=\inf_{h\in \Gamma }\max _{t\in [0,1]}J_{\lambda } \bigl(h(t) \bigr) \leq \sup _{t\geq 0}J_{\lambda }(t \omega_{\varepsilon })< c^{*}. $$

From Lemma 2.2 and the mountain pass theorem, we know that there exists at least one nontrivial solution to Problem (1.5). Since $J_{\lambda }(u)\geq J_{\lambda }(\vert u\vert )$, Problem (1.5) has at least one nonnegative nontrivial solution. The proof of Theorem 1.1 is complete. □

Proof of Theorem 1.2

Consider the following function:

$$\begin{aligned} h(t) =&J_{\lambda }(t u ) \\ =&\frac{t^{p}}{p} \int_{\Omega } \bigl(\vert \nabla u\vert ^{p}+\lambda V(x)\vert u\vert ^{p} \bigr) \,\mathrm{d}x-\frac{t^{p*}}{p^{*}} \lambda \int_{\Omega }Q(x)\vert u\vert ^{p^{*}} \,\mathrm{d}x \\ &{}- \frac{t^{q}}{q}\lambda \int_{\Omega }P(x)\vert u\vert ^{q} \,\mathrm{d}x,\quad t>0. \end{aligned}$$

Since $V\in L^{1}(\Omega )$, we find that

$$\begin{aligned} \sup_{t\geq 0}h(t) =&\sup_{t\geq 0} \biggl[ \frac{t^{p}}{p} \int_{\Omega }\lambda V(x)\vert A\vert ^{p} \, \mathrm{d}x-\frac{t^{p*}}{p^{*}}\lambda \int_{ \Omega }Q(x)\vert A\vert ^{p^{*}}\,\mathrm{d}x- \frac{t^{q}}{q}\lambda \int_{ \Omega }P(x)\vert A\vert ^{q}\,\mathrm{d}x \biggr] \\ \leq &\frac{\lambda }{N} \biggl[\frac{\int_{\Omega }V(x)\,\mathrm{d}x}{( \int_{\Omega }Q(x)\,\mathrm{d}x)^{\frac{N-p}{N}}} \biggr]^{\frac{N}{p}} \quad \mbox{for } u=A. \end{aligned}$$

Then $\sup_{t\geq 0}J_{\lambda }(tA)< c^{*}$ for $\lambda <\frac{S( \int_{\Omega }Q(x)\,\mathrm{d}x)^{\frac{N-p}{N}}}{Q_{M}^{\frac{N-p}{N}} \int_{\Omega }V(x)\,\mathrm{d}x}$.

Similarly,

$$\begin{aligned} J_{\lambda }(tA) =&\sup_{t\geq 0}h(t) \\ =&\sup_{t\geq 0} \biggl[ \frac{t^{p}}{p} \int_{\Omega }\lambda V(x)\vert A\vert ^{p} \, \mathrm{d}x-\frac{t ^{p*}}{p^{*}}\lambda \int_{\Omega }Q(x)\vert A\vert ^{p^{*}}\,\mathrm{d}x - \frac{t ^{q}}{q}\lambda \int_{\Omega }P(x)\vert A\vert ^{q}\,\mathrm{d}x \biggr] \\ \leq& \lambda \biggl(\frac{q-p}{pq} \biggr)\frac{(\int_{\Omega }V(x)\,\mathrm{d}x)^{ \frac{q}{q-p}}}{(\int_{\Omega }P(x)\,\mathrm{d}x)^{\frac{p}{q-p}}}< c ^{*} \end{aligned}$$

for $\lambda <(\frac{pq}{q-p})^{\frac{p}{N}}\frac{S}{N^{\frac{p}{N}}Q _{M}^{\frac{N-p}{N}}}\frac{(\int_{\Omega }P(x)\,\mathrm{d}x)^{\frac{p ^{2}}{N(q-p)}}}{(\int_{\Omega }V(x)\,\mathrm{d}x)^{\frac{pq}{N(q-p)}}}$.

Set

$$\lambda_{*}=\max \biggl\{ \frac{S(\int_{\Omega }Q(x)\,\mathrm{d}x)^{ \frac{N-p}{N}}}{Q_{M}^{\frac{N-p}{N}}\int_{\Omega }V(x)\,\mathrm{d}x}, \biggl( \frac{pq}{q-p} \biggr)^{\frac{p}{N}}\frac{S}{N^{\frac{p}{N}}Q_{M}^{ \frac{N-p}{N}}} \frac{(\int_{\Omega }P(x)\,\mathrm{d}x)^{ \frac{p^{2}}{N(q-p)}}}{(\int_{\Omega }V(x)\,\mathrm{d}x)^{ \frac{pq}{N(q-p)}}} \biggr\} , $$

then we have $\sup_{t\geq 0}J_{\lambda }(tA)< c^{*}$ for $0<\lambda <\lambda_{*}$. Similar to the proof of Theorem 1.1, Problem (1.5) has at least one nonnegative nontrivial solution. The proof of Theorem 1.2 is complete. □

Proof of Theorem 1.3

Define

$$ \begin{aligned} K=\inf_{u \in W_{0}^{1,p}(B(x_{0} ,\delta ))\setminus \{0\}}\frac{ \int_{B(x_{0},\delta )}\vert \nabla u\vert ^{p}\,\mathrm{d}x}{( \int_{B( x_{0},\delta )}\vert u\vert ^{q}\,\mathrm{d}x)^{\frac{p}{q}}}. \end{aligned} $$

Since $p< q< p^{*}$, as is well known, there exists a function $w\in W _{0}^{1,p}(B(x_{0} ,\delta ))$ such that

$$ \begin{aligned} K=\frac{\int_{B(x_{0},\delta )}\vert \nabla w\vert ^{p}\,\mathrm{d}x}{( \int_{B( x_{0},\delta )}\vert w\vert ^{q}\,\mathrm{d}x)^{\frac{p}{q}}}. \end{aligned} $$

Thus,

$$\begin{aligned} \sup_{t\geq 0}J_{\lambda }(t w) \leq &\sup _{t\geq 0} \biggl[\frac{t^{p}}{p} \int_{B(x_{0} ,\delta )} \bigl(\vert \nabla w\vert ^{p}+\lambda V(x)\vert w\vert ^{p} \bigr) \,\mathrm{d}x-\frac{t^{q}}{q} \lambda \int_{B(x_{0} ,\delta )}P(x)\vert w\vert ^{q} \,\mathrm{d}x \biggr] \\ \leq &\frac{q-p}{pq}\frac{(\int_{B(x_{0} ,\delta )}\vert \nabla w\vert ^{p} \,\mathrm{d}x)^{\frac{q}{q-p}}}{P_{m}^{\frac{p}{q-p}}( \int_{B(x_{0} ,\delta )}\lambda \vert w\vert ^{q}\,\mathrm{d}x)^{\frac{p}{q-p}}} \\ =&\frac{q-p}{pq}\frac{K^{\frac{q}{q-p}}}{\lambda^{\frac{p}{q-p}}P _{m}^{\frac{p}{q-p}}}. \end{aligned}$$

Let $\lambda^{*}= (\frac{N(q-p)K^{\frac{q}{q-p}}Q_{M}^{ \frac{N-p}{p}}}{pqS^{\frac{N}{p}}P_{m}^{\frac{p}{q-p}}} )^{ \frac{p(q-p)}{Np+pq-Nq}}$, where $P_{m}=\min_{x\in B(x_{0} ,\delta )}P(x)$, then $\sup_{t\geq 0}J _{\lambda }(tw)< c^{*}$ for $\lambda >\lambda^{*}$. Similar to the proof of Theorem 1.1, Problem (1.5) has at least one nonnegative nontrivial solution for $\lambda >\lambda^{*}$. The proof of Theorem 1.3 is complete. □

Proof of Theorem 1.4

Fix $n\in N$, let $\varphi_{1}$, $\varphi_{2},\ldots ,\varphi_{n}\in C_{0}^{\infty }(R^{N})$ be smooth functions such that $\operatorname{supp}\varphi_{j}\subset B(x_{0},\delta )$, $j=1, 2, \ldots , n$, $\operatorname{supp}\varphi_{i}\cap \operatorname{supp}\varphi_{j}= \varnothing$, $i\neq j$.

We define $E_{n}=\operatorname{Span}\{\varphi_{1}, \varphi_{2}, \ldots , \varphi _{n}\}$, Σ is the set of all symmetric and closed subsets of $W _{V}^{1,p}(\Omega )$, $\gamma (A)$ is the Krasnoselski genus,

$$i(A)=\min_{h\in \Gamma }\gamma \bigl(h(A) \bigr) \cap \partial B_{ \beta_{\lambda }} ),\quad A\in \Sigma , $$

where Γ is the set of all odd homomorphisms $C^{1}(W_{V}^{1,p}( \Omega ), W_{V}^{1,p}(\Omega ))$.

Set

$$ \begin{aligned} c_{j}=\inf_{i(A)\geq j}\sup _{u\in A}J_{\lambda }(u),\quad j=1, 2, \ldots , n. \end{aligned} $$

Since $i(E_{n})=\dim E_{n}=n$ and $J_{\lambda }(u)\geq \beta_{\lambda }$ for $\Vert u\Vert _{\lambda V}=\rho_{\lambda }$ in Lemma 2.1, we find that

$$ \begin{aligned} \beta_{\lambda }\leq c_{1}\leq c_{2}\leq \cdots \leq c_{n}\leq \sup_{u\in E_{n}}J_{\lambda }(u). \end{aligned} $$

We now estimate $\sup_{u\in E_{n}}J_{\lambda }(u)$. If $u\in E_{n}$, one has $u=\sum_{j=1}^{n}\tau_{j}\varphi_{j}$ for $\tau_{j}\in R$. From the properties of $\varphi_{j}$, we obtain

$$\begin{aligned} \sup_{u\in E_{n}}J_{\lambda }(u) =&\sup_{u\in E_{n}} \Biggl(\sum_{j=1} ^{n}J_{\lambda }( \tau_{j}\varphi_{j}) \Biggr) \\ \leq &\sup_{u\in E_{n}}\sum_{j=1}^{n} \biggl[\frac{\tau_{j}^{p}}{p} \int_{B(x_{0} ,\delta )} \bigl(\vert \nabla \varphi_{j}\vert ^{p}+\lambda V(x)\vert \varphi_{j}\vert ^{p} \bigr)\,\mathrm{d}x-\frac{\tau_{j}^{q}}{q}\lambda \int_{B(x_{0} ,\delta )}P(x)\vert \varphi_{j}\vert ^{q}\,\mathrm{d}x \biggr] \\ \leq &\frac{q-p}{pq}\sum_{j=1}^{n} \frac{(\int_{B(x_{0} ,\delta )}\vert \nabla \varphi_{j}\vert ^{p}\,\mathrm{d}x)^{\frac{q}{q-p}}}{ \lambda^{\frac{p}{q-p}}P_{m}^{\frac{p}{q-p}}(\int_{B(x_{0} ,\delta )}\vert \varphi_{j}\vert ^{q}\,\mathrm{d}x)^{\frac{p}{q-p}}}. \end{aligned}$$

Consequently, there exists a $\Lambda_{n}>0$ such that $\sup_{u\in E_{n}}J_{\lambda }(u)< c^{*}$ for $\lambda >\Lambda_{n}$. Similar to the proof of Theorem 1.1, Problem (1.5) has at least n pairs of nonnegative nontrivial solutions. The proof of Theorem 1.4 is complete. □

4 Conclusion

In this paper, we study the following quasilinear Neumann problem with critical Sobolev exponent:

$$ \textstyle\begin{cases} -\varepsilon^{p}\Delta_{p}u+V(x)\vert u\vert ^{p-2}u=Q(x)\vert u\vert ^{p^{*}-2}u +P(x)\vert u\vert ^{ {q-2}}u, & x\in \Omega , \\ \vert \nabla u\vert ^{p-2}\frac{\partial u}{\partial \nu }=0, &x\in \partial \Omega , \end{cases} $$

where the weight functions $V(x)$ is continuous in Ω and $Q(x)$, $P(x)$ are continuous on Ω̅. Due to the lack of compactness of the embedding of $W^{1,p}(\Omega )\hookrightarrow L ^{p^{*}}(\Omega )$ and the fact that the weight function $V(x)$ may be unbounded close to the boundary ∂Ω, some classical methods may not directly be applied to our problem. We introduce a suitable weighted Sobolev space and add restrictions on the weight functions $Q(x)$ and $P(x)$ to prove the corresponding functional of problem satisfies $(PS)_{c}$-condition in a suitable range by the Lions concentration-compactness principle, then apply the mountain pass lemma, the existence and multiplicity of nontrivial solutions are obtained.