Let \(\mathcal{J}:W^{1,p}_{0}(\varOmega )\rightarrow {\mathbb{R}} \) be the variational functional defined by
$$ \mathcal{J}(u)=\frac{1}{2} \int _{\varOmega }\bigl(A\bigl( \vert \nabla u \vert ^{2} \bigr)-\lambda A\bigl(u^{2}\bigr)\bigr) \,dx-\frac{1}{q} \int _{\varOmega } \vert u \vert ^{q} \,dx. $$
By hypothesis (7) and the Sobolev embedding theorem, \(\mathcal{J}\) is well defined. Moreover, \(\mathcal{J}\in C^{1}(W^{1,p}_{0}(\varOmega ),{\mathbb{R}} )\) and its Gâteux directional derivative is given by
$$ \bigl\langle \mathcal{J}'(u),v\bigr\rangle = \int _{\varOmega }\bigl(a\bigl( \vert \nabla u \vert ^{2} \bigr) \nabla u\nabla v-\lambda a\bigl(u^{2}\bigr)uv\bigr) \,dx - \int _{\varOmega } \vert u \vert ^{q-2}uv \,dx $$
for any u, \(v\in W^{1,p}_{0}(\varOmega )\).
Since the problem has a variational structure, then solutions of problem (6) are critical points of the energy functional \(\mathcal{J}\).
We define the truncation
$$ h(t)= \textstyle\begin{cases} t^{q-1}& \text{if $t> 0$}, \\ 0& \text{if $t\geq 0$}, \end{cases} $$
and we set \(H(t):=\int _{0}^{t}h(s)\,ds\).
Consider the variational functional
$$ \mathcal{E}(u)=\frac{1}{2} \int _{\varOmega }\bigl(A\bigl( \vert \nabla u \vert ^{2} \bigr)-\lambda A\bigl(u^{2}\bigr)\bigr) \,dx- \int _{\varOmega }H(u)\,dx. $$
Then \(\mathcal{E}\) is well defined, \(\mathcal{E}\in C^{1}(W_{0}^{1,p}(\varOmega ))\) and, for all \(u,v\in W_{0}^{1,p}(\varOmega )\),
$$ \bigl\langle \mathcal{E}'(u),v\bigr\rangle = \int _{\varOmega }\bigl[a\bigl( \vert \nabla u \vert ^{2} \bigr) \nabla u\nabla v-\lambda a\bigl(u^{2}\bigr)uv\bigr] \,dx- \int _{\varOmega }h(u)v\,dx. $$
3.1 Verification of the Palais–Smale condition
Let \((u_{n})\subset W^{1,p}_{0}(\varOmega )\) be a Palais–Smale sequence of \(\mathcal{E}\), that is,
$$ \mathcal{E}(u_{n})=O(1) \quad\text{and}\quad \bigl\Vert \mathcal{E}'(u_{n}) \bigr\Vert _{W^{-1,p'}( \varOmega )}=o(1) \quad\text{as } n\rightarrow \infty. $$
It follows that
$$ \frac{1}{2} \int _{\varOmega }A\bigl( \vert \nabla u_{n} \vert ^{2}\bigr)\,dx-\frac{\lambda }{2} \int _{\varOmega }A\bigl(u_{n}^{2}\bigr)\,dx- \int _{\varOmega }H(u_{n})\,dx=O(1)$$
(10)
and, for all \(v\in W^{1,p}_{0}(\varOmega )\),
$$ \int _{\varOmega }a\bigl( \vert \nabla u_{n} \vert ^{2}\bigr)\nabla u_{n}\nabla v\,dx-\lambda \int _{\varOmega }a\bigl(u_{n}^{2} \bigr)u_{n}v\,dx- \int _{\varOmega }h(u_{n})v\,dx=o(1) \Vert v \Vert .$$
(11)
Taking \(v=u_{n}\) in (11), we obtain
$$ \int _{\varOmega }a\bigl( \vert \nabla u_{n} \vert ^{2}\bigr)|\nabla u_{n}^{2}\,dx-\lambda \int _{\varOmega }a\bigl(u_{n}^{2} \bigr)u_{n}^{2}\,dx- \int _{\varOmega }h(u_{n})u_{n}\,dx=o(1) \Vert u_{n} \Vert .$$
(12)
But, for all \(n\geq 1\),
$$ \int _{\varOmega }h(u_{n})u_{n}\,dx=q \int _{\varOmega }H(u_{n})\,dx.$$
(13)
Relations (10), (11), and (13) yield
$$\begin{aligned} (1-q) \int _{\varOmega }H(u_{n})\,dx={}& \int _{\varOmega } \biggl[ \frac{1}{2} A\bigl( \vert \nabla u_{n} \vert ^{2}\bigr)-a\bigl( \vert \nabla u_{n} \vert ^{2}\bigr) \vert \nabla u_{n} \vert ^{2} \biggr]\,dx \\ &{}-\lambda \int _{\varOmega } \biggl[\frac{1}{2} A\bigl( u_{n}^{2}\bigr)-a\bigl( u_{n}^{2}\bigr) u_{n}^{2} \biggr]\,dx+O(1) \quad\text{as } n\rightarrow \infty. \end{aligned}$$
(14)
But \(A(t^{2})=2\int _{0}^{t}sa(s^{2})\,ds\) and the mapping \((0,+\infty )\ni s\mapsto sa(s^{2})\) is increasing. Therefore
$$ A\bigl(t^{2}\bigr)\leq 2t^{2}a \bigl(t^{2}\bigr) \quad\text{for all } t\geq 0.$$
(15)
Claim. The sequence \((u_{n})\subset W^{1,p}_{0}(\varOmega )\) is bounded.
For this purpose, we first show that
$$ \text{the sequence $(u_{n})$ is bounded in $L^{q}(\varOmega )$}.$$
(16)
Assume first that \(\lambda \leq 0\). Thus, from (14) and (15) we deduce that
$$ 0\leq (q-1) \int _{\varOmega }H(u_{n})\,dx\leq O(1) \quad\text{as } n \rightarrow \infty. $$
hence \((u_{n})\) is bounded in \(L^{q}(\varOmega )\).
If \(\lambda >0\), relations (14) and (15) yield
$$ 0\leq (q-1) \int _{\varOmega }H(u_{n})\,dx\leq \lambda \int _{\varOmega } \biggl[ \frac{1}{2} A\bigl( u_{n}^{2}\bigr)-a\bigl( u_{n}^{2}\bigr) u_{n}^{2} \biggr]\,dx.$$
(17)
Since \(1< p< q\), relation (17) shows that \((u_{n})\) is bounded in \(L^{q}(\varOmega )\).
By (8) and since \(\lambda <\varLambda \), there exists \(c_{0}>0\) such that, for all \(u\in W^{1,p}_{0}(\varOmega )\),
$$ \int _{\varOmega }A\bigl( \vert \nabla u \vert ^{2}\bigr) \,dx-\lambda \int _{\varOmega }A\bigl(u^{2}\bigr)\,dx\geq c_{0} \int _{\varOmega }A\bigl( \vert \nabla u \vert ^{2}\bigr) \,dx.$$
(18)
Returning to (10) and using (16) in combination with (18) and the hypothesis \(\lambda <\varLambda \), we obtain the claim.
Next, we prove that \((u_{n})\subset W^{1,p}_{0}(\varOmega )\) contains a strongly convergent subsequence.
Relation (11) yields, for all \(v\in W^{1,p}_{0}(\varOmega )\),
$$ \int _{\varOmega }a\bigl( \vert \nabla u_{n} \vert ^{2}\bigr)\nabla u_{n}\nabla v\,dx= \int _{\varOmega }\psi (u_{n})v\,dx+o(1) \Vert v \Vert \quad\text{as } n\rightarrow \infty,$$
(19)
where
$$ \psi (w)=\lambda a\bigl(w^{2}\bigr)w+h(w) \quad\text{for all } w\in W^{1,p}_{0}( \varOmega ). $$
Obviously, ψ is a continuous function.
Assume that \(1< p< N\) (similar arguments work if \(p\geq N\)). It follows that
$$ \bigl\vert \psi (w) \bigr\vert \leq C\bigl(1+ \vert w \vert ^{(Np-N+p)/(N-p)}\bigr) \quad\text{for all } w\in W^{1,p}_{0}( \varOmega )$$
(20)
and
$$ \psi (w)=o\bigl( \vert w \vert ^{Np/(N-p)}\bigr) \quad\text{as } \vert w \vert \rightarrow \infty.$$
(21)
A crucial idea in the proof is to show that the sequence \(\{\psi (u_{n})\}\subset W^{-1,p'}(\varOmega )\) contains a strongly convergent subsequence. Indeed, in this case, relation (19) combined with the Sobolev embedding theorem implies that, up to a subsequence, \(\{\psi (u_{n})\}\) converges strongly in \((L^{Np/(N-p)}(\varOmega ))^{*}= L^{Np/(Np-N+p)}(\varOmega )\).
By our claim and the Rellich–Kondrachov embedding theorem, we can assume, up to a subsequence, that
$$ u_{n}\rightarrow u \quad\text{in } L^{Np/(N-p)}(\varOmega ). $$
Fix \(\delta >0\). Thus, by the Egorov theorem, there exists \(\omega \subset \varOmega \) such that \(|\omega | <\delta \) and
$$ u_{n}\rightarrow u \quad\text{uniformly in $\varOmega \setminus \omega $}. $$
Fix \(\eta >0\) small enough. Thus, to conclude the proof, it is enough to show that
$$ \int _{\omega } \bigl\vert \psi (u_{n})-\psi (u) \bigr\vert ^{Np/(Np-N+p)} \,dx\leq \eta\quad \text{for all $n$ big enough}. $$
Relation (20) implies that
$$ \int _{\omega } \bigl\vert \psi (u) \bigr\vert ^{Np/(Np-N+p)} \,dx\leq C \int _{\omega }\bigl(1+ \vert u \vert ^{Np/(N-p)}\bigr)\,dx $$
and the right-hand side can be made smaller than any positive constant if we choose \(\delta >0\) small enough.
Next, by (21),
$$ \int _{\omega } \bigl\vert \psi (u_{n})-\psi (u) \bigr\vert ^{Np/(Np-N+p)}\,dx\leq \varepsilon \int _{\omega } \vert u_{n}-u \vert ^{Np/(N-p)} \,dx+C_{\varepsilon } \vert \omega \vert , $$
and the right-hand side can be made as small as we wish. This follows by combining our claim with the Sobolev embedding theorem.
We conclude that the energy functional \(\mathcal{E}\) satisfies the Palais–Smale condition.
3.2 Proof of Theorem 1(i) concluded
We first prove that \(\mathcal{E}\) satisfies the geometric hypotheses of the mountain pass theorem.
Fix \(\lambda <\varLambda \). We have for all \(u\in W^{1,p}_{0}(\varOmega )\)
$$\begin{aligned} \mathcal{E}(u)&\geq \frac{c_{0}}{2} \int _{\varOmega }A\bigl( \vert \nabla u \vert ^{2}\bigr) \,dx- \int _{\varOmega }H(u)\,dx \\ &\geq \frac{c_{0}}{2} \int _{\varOmega }A\bigl( \vert \nabla u \vert ^{2}\bigr) \,dx- \frac{1}{q} \int _{\varOmega } \vert u \vert ^{q}\,dx, \end{aligned}$$
where \(c_{0}\) is as in (18).
Next, by (3), we obtain for all \(u\in W^{1,p}_{0}(\varOmega )\)
$$ \mathcal{E}(u)\geq \frac{c_{0}\gamma '}{2} \int _{\varOmega } \vert \nabla u \vert ^{p}\,dx- \frac{1}{q} \int _{\varOmega } \vert u \vert ^{q}\,dx. $$
Fix \(\varepsilon >0\) small enough. Thus, by (7) and the Sobolev embedding theorem, there exists \(r>0\) such that
$$ \mathcal{E}(u)\geq \varepsilon \quad\text{for all $u\in W^{1,p}_{0}( \varOmega )$ with $ \Vert u \Vert =r$}. $$
This establishes the existence of a “mountain” near the origin.
Next, we show the existence of a “valley” far from the origin.
Let \(\varphi _{1}>0\) be the first eigenfunction of the Laplace operator, hence \(\varphi _{1}\in W^{1,p}_{0}(\varOmega )\). For all \(t>0\), we have
$$\begin{aligned} \mathcal{E}(t\varphi _{1})={}& \frac{1}{2} \int _{\varOmega }\bigl(A\bigl(t^{2} \vert \nabla \varphi _{1} \vert ^{2}\bigr)-\lambda A\bigl(t^{2}\varphi _{1}^{2}\bigr)\bigr) \,dx- \frac{t^{q}}{q} \int _{\varOmega }\varphi _{1}^{q}\,dx \\ \leq{}& \frac{1}{2} \int _{[t \vert \nabla \varphi _{1} \vert \geq t_{0}]}A\bigl(t^{2} \vert \nabla \varphi _{1} \vert ^{2}\bigr)\,dx + \frac{1}{2} \int _{[t \vert \nabla \varphi _{1} \vert < t_{0}]}A\bigl(t^{2} \vert \nabla \varphi _{1} \vert ^{2}\bigr)\,dx \\ &{}-\frac{\lambda }{2} \int _{[t\varphi _{1}\geq t_{0}]} A\bigl(t^{2} \varphi _{1}^{2} \bigr)\,dx -\frac{\lambda }{2} \int _{[t\varphi _{1}< t_{0}]} A\bigl(t^{2} \varphi _{1}^{2} \bigr)\,dx-\frac{t^{q}}{q} \int _{\varOmega }\varphi _{1}^{q}\,dx, \end{aligned}$$
(22)
where \(t_{0}\) is defined in (3).
Next, we evaluate the terms arising in (22). By (3) we have
$$ \frac{1}{2} \int _{[t \vert \nabla \varphi _{1} \vert \geq t_{0}]}A\bigl(t^{2} \vert \nabla \varphi _{1} \vert ^{2}\bigr)\,dx\leq c_{1}t^{p}, \quad\text{where $c_{1}=\frac{\varGamma '}{2} \int _{\varOmega } \vert \nabla \varphi _{1} \vert ^{p}\,dx>0$}, $$
and
$$ -\frac{\lambda }{2} \int _{[t\varphi _{1}\geq t_{0}]} A\bigl(t^{2}\varphi _{1}^{2} \bigr)\,dx \leq c_{2}t^{p}, \quad\text{where $c_{2}=\frac{ \vert \lambda \vert \varGamma '}{2} \int _{\varOmega }\varphi _{1}^{p}\,dx>0$}. $$
Recall that \(A(t^{2})=2\int _{0}^{t}sa(s^{2})\,ds\). Thus, by (1),
$$ A(t)\leq 2p^{-1}\varGamma \bigl[(\kappa +t)^{p}-\kappa ^{p}\bigr]. $$
It follows that
$$\begin{aligned} \frac{1}{2} \int _{[t \vert \nabla \varphi _{1} \vert < t_{0}]}A\bigl(t^{2} \vert \nabla \varphi _{1} \vert ^{2}\bigr)\,dx\leq{}& \frac{\varGamma }{p} \int _{[t \vert \nabla \varphi _{1} \vert < t_{0}]} \bigl(\kappa +t \vert \nabla \varphi _{1} \vert \bigr)^{p}\,dx-c_{4} \\ &{}\times \frac{\varGamma }{p} \int _{\varOmega }(\kappa +t_{0})^{p} \,dx-c_{4}=O(1). \end{aligned}$$
Similarly, we deduce that
$$ -\frac{\lambda }{2} \int _{[t\varphi _{1}< t_{0}]} A\bigl(t^{2}\varphi _{1}^{2} \bigr)\,dx \leq O(1). $$
Returning now to (22) we obtain
$$ \mathcal{E}(t\varphi _{1})\leq (c_{1}+c_{2})t^{p}-c_{3}t^{q}+O(1), $$
where \(c_{3}=q^{-1}\int _{\varOmega }\varphi _{1}^{q}\,dx>0\). Using (7) we deduce that \(\mathcal{E}(t\varphi _{1})<0\) for t large enough.
We have verified all the hypotheses of the mountain pass theorem. It follows that \(\mathcal{E}\) has a nontrivial critical point u. Thus, for all \(v\in W^{1,p}_{0}(\varOmega )\),
$$ \int _{\varOmega }a\bigl( \vert \nabla u \vert ^{2}\bigr) \nabla u\nabla v\,dx=\lambda \int _{\varOmega }a\bigl(u^{2}\bigr)uv\,dx + \int _{\varOmega }h(u)v\,dx. $$
Taking \(v=u^{-}\) as a test function, we obtain
$$ \int _{\varOmega }a\bigl( \vert \nabla u \vert ^{2}\bigr) \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx-\lambda \int _{\varOmega }a\bigl(u^{2}\bigr) \bigl\vert u^{-} \bigr\vert ^{2}\,dx=0. $$
Finally, by (8) and since \(\lambda <\varLambda \), we conclude that \(u^{-}=0\), hence \(u\geq 0\). This means that \(h(u)=u^{q-1}\), so u is a solution of problem (6).
It remains to show that \(u>0\) in Ω. For this purpose, we observe that relations (4) and (7) imply that the hypotheses of the generalized maximum principle of Pucci and Serrin [13] are fulfilled. We conclude that \(u>0\) in Ω.