# Multiplicity of solutions for a class of fractional p-Kirchhoff system with sign-changing weight functions

## Abstract

In this paper, we investigate the fractional p-Kirchhoff -type system:

\begin{aligned} \textstyle\begin{cases} M (\int_{{ \mathbb {R} }^{2N}}\frac{\vert u(x)-u(y) \vert ^{p}}{\vert x-y \vert ^{N+ps}}\,dx\,dy )(- \Delta )^{s}_{p}u=\mu g(x)\vert u \vert ^{\beta -2}u+\frac{a}{a+b}h(x)\vert u \vert ^{a-2}u\vert v \vert ^{b},&\mbox{in } \Omega , \\ M (\int_{{ \mathbb {R} }^{2N}}\frac{\vert v(x)-v(y) \vert ^{p}}{\vert x-y \vert ^{N+ps}}\,dx\,dy )(- \Delta )^{s}_{p}v=\sigma f(x)\vert v \vert ^{\beta -2}v+\frac{b}{a+b}h(x)\vert v \vert ^{b-2}v\vert u \vert ^{a},&\mbox{in } \Omega , \\ u=v=0,&\mbox{in } { \mathbb {R} }^{N}\setminus \Omega , \end{cases}\displaystyle \end{aligned}

where $$\Omega \subset \mathbb{R}^{N}$$ is a smooth bounded domain, $$(-\Delta )^{s}_{p}$$ is the fractional p-Laplacian operator with $$0< s<1<p$$ and $$ps< N$$. $$a>1$$, $$b>1$$ satisfy $$2< a+b< p_{s}^{*}$$. $$1<\beta <p_{s}^{*}$$, $$p_{s}^{*}=\frac{Np}{N-ps}$$ is the fractional critical exponent. μ, σ are two real parameters. $$M(t)=k+\lambda t^{\tau }$$, $$k>0$$, λ, $$\tau \geq 0$$, $$\tau =0$$ if and only if $$\lambda =0$$. The weight functions g, f, h change sign in Ω and satisfy suitable conditions. By using the Nehari manifold method, it is proved that the system has at least two solutions provided that $$2< a+b< p\leq p(\tau +1)<\beta <p_{s}^{*}$$ and $$(\mu ,\sigma )$$ belongs to a certain subset of $$\mathbb {R} ^{2}$$. Also, by using the mountain pass theorem, we prove that there exist $$\lambda _{1}\geq \lambda_{0}$$ such that the system admits at least a nontrivial solution for $$\lambda \in (0,\lambda_{0})$$ and no nontrivial solution for $$\lambda >\lambda_{1}$$ under the assumptions $$\mu =\sigma =0$$ and $$p< a+b<\min \{p(\tau +1),p_{s}^{*}\}$$.

## 1 Introduction

In this paper, we investigate the multiplicity of solutions to the following fractional elliptic system:

\begin{aligned} \textstyle\begin{cases} M (\int_{{ \mathbb {R} }^{2N}}\frac{\vert u(x)-u(y) \vert ^{p}}{\vert x-y \vert ^{N+ps}}\,dx\,dy )(-\Delta )^{s}_{p}u=\mu g(x)\vert u \vert ^{\beta -2}u+\frac{a}{a+b}h(x)\vert u \vert ^{a-2}u\vert v \vert ^{b } &\mbox{in } \Omega , \\ M (\int_{{ \mathbb {R} }^{2N}}\frac{\vert v(x)-v(y) \vert ^{p}}{\vert x-y \vert ^{N+ps}}\,dx\,dy )(-\Delta )^{s}_{p}v=\sigma f(x)\vert v \vert ^{\beta -2}v+\frac{b}{a+b}h(x)\vert v \vert ^{b-2}v\vert u \vert ^{a } &\mbox{in } \Omega , \\ u=v=0&\mbox{in } { \mathbb {R} }^{N}\setminus \Omega , \end{cases}\displaystyle \end{aligned}
(1.1)

where $$\Omega \subset \mathbb{R}^{N}$$ is a smooth bounded domain, $$0< s<1<p$$ and $$ps< N$$. $$a>1$$, $$b>1$$ satisfy $$2< a+b< p^{\ast }_{s}$$. $$1<\beta <p_{s}^{*}$$, $$p_{s}^{*}=\frac{Np}{N-ps}$$ is the fractional critical exponent. μ, σ are two real parameters. $$M(t)=k+\lambda t^{\tau }$$, $$k>0$$, λ, $$\tau \geq 0$$, $$\tau =0$$ if and only if $$\lambda =0$$. The weight functions g, f, h change sign in Ω and satisfy further assumption which will be given later. $$(-\Delta )^{s}_{p}$$ is the fractional p-Laplacian operator defined on smooth functions by

$$(-\Delta )^{s}_{p}m(x)=2\lim_{\varepsilon \rightarrow 0^{+}} \int_{{ \mathbb {R} }^{N}\setminus B_{\varepsilon }(x)} \frac{\vert m(x)-m(y) \vert ^{p-2}(m(x)-m(y))}{\vert x-y \vert ^{N+ps}}\,dy,\quad x\in { \mathbb {R} } ^{N}.$$

Problem (1.1) is related to the stationary analogue of the following Kirchhoff model:

$$\rho u_{tt}- \biggl(\frac{p_{0}}{h}+\frac{E}{2L} \int_{0}^{L}u_{x}^{2}\,dx \biggr)u _{xx}=0$$

which was proposed by Kirchhoff in 1883 as a generalization of the well-known D’Alembert wave equation for free vibrations of elastic strings, where ρ, $$p_{0}$$, h, E, L are constants which represent some physical meanings, respectively. Indeed, Kirchhoff’s model takes into account the changes in length of the string produced by transverse vibrations. In particular, Kirchhoff’s equation models several physical and biological systems, we refer to  for more details. The Kirchhoff type equation and system have attracted attention and have been discussed by many authors, we refer to [9, 14, 2123, 29] and the references therein.

Up to now, a great attention has been paid to the study of the fractional Laplacian equation and system, see, for example, [4, 5, 10, 11, 13, 18, 22, 23, 29, 30, 33]. In particular, the fractional and nonlocal operators of elliptic type arise in a quite natural way in many different applications, such as continuum mechanics, phase transition phenomena, population dynamics, and game theory, as they are the typical outcome of stochastic stabilization of Lévy processes, see [2, 8]. The literature on fractional nonlocal operators and their applications is very interesting and quite large, see, for example, [4, 20, 25, 28]. For the basic properties of fractional Sobolev spaces, we refer the readers to .

During the past ten years, by using the Nehari manifold and Fibering maps, several authors have solved semilinear and quasilinear elliptic problems with critical nonlinearity and subcritical nonlinearity, see [6, 7, 1517, 19, 24, 26, 31, 32, 34] and the references therein. Particularly, in , Chen and Deng considered the following fractional p-Laplacian system:

\begin{aligned} \textstyle\begin{cases} (-\Delta )^{s}_{p}u=\mu \vert u \vert ^{\beta -2}u+\frac{2a}{a+b}\vert u \vert ^{a-2}u\vert v \vert ^{b } &\mbox{in } \Omega , \\ (-\Delta )^{s}_{p}v=\sigma \vert v \vert ^{\beta -2}v+\frac{2b}{a+b}\vert v \vert ^{b-2}v\vert u \vert ^{a } &\mbox{in } \Omega , \\ u=v=0&\mbox{in } { \mathbb {R} }^{n}\setminus \Omega , \end{cases}\displaystyle \end{aligned}
(1.2)

where $$a>1$$, $$b>1$$, $$1<\beta <p<a+b<p_{s}^{*}$$, $$\mu >0$$, $$\sigma >0$$. Using the Nehari manifold method, they proved (1.2) has at least two nontrivial solutions when $$0<\mu^{\frac{p}{p-q}}+\sigma^{ \frac{p}{p-q}}<C$$ for some $$C>0$$.

In , Rasouli and Afrouzi investigated the following elliptic system:

\begin{aligned} \textstyle\begin{cases} -\Delta_{q}u+n(x)\vert u \vert ^{q-2}u=\mu g(x)\vert u \vert ^{\beta -2}u&x\in \Omega , \\ -\Delta_{q}v+n(x)\vert v \vert ^{q-2}v=\sigma f(x)\vert v \vert ^{\beta -2}v&x\in \Omega, \\ \vert \nabla u \vert ^{q-2}\frac{\partial u}{\partial n}=\frac{a}{a+b}\vert u \vert ^{a-2}u\vert v \vert ^{b},\qquad \vert \nabla v \vert ^{q-2}\frac{\partial v}{\partial n}=\frac{b}{a+b}\vert u \vert ^{a}\vert v \vert ^{b-2}v,& x\in \partial \Omega , \end{cases}\displaystyle \end{aligned}
(1.3)

where $$a>1$$, $$b>1$$, $$2< a+b< q<\beta <p^{*}$$, $$(\mu ,\sigma )\in { \mathbb {R} }^{2} \setminus \{(0,0)\}$$, $$g(x)$$, $$f(x)\in C(\Omega )$$ are functions which change sign in Ω. Using the Nehari manifold method, they proved (1.3) admits at least two solutions when $$(\mu , \sigma )$$ belongs to a certain subset of $${ \mathbb {R} }^{2}$$.

However, to our best knowledge, there are few results on a fractional p-Kirchhoff system with sign-changing concave-convex nonlinearity, especially for parameters meeting $$a+b< p\leq p(\tau +1)<\beta <p_{s} ^{*}$$ or $$p< a+b<\min \{p(\tau +1),p_{s}^{*}\}$$. Motivated by the above and the idea of [6, 9, 10, 24], in this paper, we are concerned with the multiplicity of solutions for system (1.1).

To state our main result precisely, we introduce some notations. Let $$0< s<1<p$$ with $$ps< N$$. Define

\begin{aligned} W= \biggl\{ v| v:{ \mathbb {R} }^{N}\rightarrow { \mathbb {R} }\mbox{ is measurable}, v | _{\Omega }\in L^{p}(\Omega ), \mbox{and} { \int_{\mathcal{Q}}\frac{\vert v(x)-v(y) \vert ^{p}}{\vert x-y \vert ^{N+ps}}\,dx\,dy}< +\infty \biggr\} , \end{aligned}

where $$\mathcal{Q}={ \mathbb {R} }^{2N}\setminus (\Omega^{c}\times \Omega^{c})$$ with $$\Omega^{c}={ \mathbb {R} }^{N}\setminus \Omega$$, this space is endowed with the norm

\begin{aligned} \Vert v \Vert _{W}=\Vert v \Vert _{L^{p}(\Omega )}+ \biggl( \int_{\mathcal{Q}} \frac{\vert v(x)-v(y) \vert ^{p}}{\vert x-y \vert ^{N+ps}}\,dx\,dy \biggr)^{1/p}. \end{aligned}
(1.4)

We denote the space $$W_{0}=\{v\mid v\in W, v=0 \mbox{ a.e. in } \Omega^{c}\}$$ or equivalently the closure of $$C^{\infty }_{0}( \Omega )$$ in W and introduce the norm

\begin{aligned} \Vert v \Vert =\Vert v \Vert _{W_{0}}= \biggl( \int_{\mathcal{Q}} \frac{\vert v(x)-v(y) \vert ^{p}}{\vert x-y \vert ^{N+ps}}\,dx\,dy \biggr)^{1/p}, \quad \forall v \in W_{0}. \end{aligned}
(1.5)

Then $$(W_{0},\Vert \cdot \Vert _{W_{0}})$$ is a uniformly convex Banach space, see [33, Theorem 2.4]. By results of [12, 15, 33], $$W_{0}$$ is continuously embedded in $$L^{r}(\Omega )$$ for any $$1\leq r\leq p_{s}^{*}$$ and compact for whenever $$1\leq r< p_{s}^{*}$$, then there exists $$C_{r}>0$$ such that, for $$r\in [1,p_{s}^{*}]$$,

\begin{aligned} \Vert v \Vert _{r}=\Vert v \Vert _{L^{r}(\Omega )}\leq C_{r}\Vert v \Vert _{W_{0}}=C_{r} \Vert v \Vert ,\quad \forall v\in W_{0}. \end{aligned}
(1.6)

For convenience, for some $$\beta \in (1,p_{s}^{*})$$, we denote $$C_{\ast }=C_{\beta }$$.

For the product space $$X=W_{0}\times W_{0}$$, we introduce the norm

\begin{aligned} \bigl\Vert (u,v) \bigr\Vert = \bigl\Vert (u,v) \bigr\Vert _{X}= \bigl(\Vert u \Vert ^{p}_{W_{0}}+\Vert v \Vert ^{p}_{W_{0}} \bigr)^{1/p}= \bigl( \Vert u \Vert ^{p}+\Vert v \Vert ^{p} \bigr)^{1/p}, \quad \forall (u,v)\in X. \end{aligned}
(1.7)

Then $$(X, \Vert \cdot \Vert _{X})$$ is a reflexive Banach space.

Our main results are as follows.

### Theorem 1.1

Let $$0< s<1<p$$ with $$ps< N$$, $$\Omega \subset { \mathbb {R} }^{N}$$ is a smooth bounded domain, $$(\mu ,\sigma )\in { \mathbb {R} }^{2}\backslash \{(0,0)\}$$, $$M(t)=k+\lambda t^{\tau }$$, $$k>0$$, λ, $$\tau \geq 0$$, $$\tau =0$$ if and only if $$\lambda =0$$. Assume

$$\mathrm{(H_{0})}$$ :

$$a+b< p\leq p(\tau +1)<\beta <p^{*}_{s}=\frac{Np}{N-ps}$$;

$$\mathrm{(H_{1})}$$ :

$$g,f\in C(\overline{\Omega })$$ with $$g^{\pm }=\max \{\pm g,0\}\not \equiv 0$$ and $$f^{\pm }=\max \{\pm f,0\} \not \equiv 0$$;

$$\mathrm{(H_{2})}$$ :

$$h\in L^{\frac{\beta }{\beta -a-b}}(\Omega )$$ with $$\operatorname{meas}(\{x\in \Omega : h(x)>0\})>0$$,

then there exists $$\theta^{\ast }>0$$ such that when μ, σ satisfy

$$0< \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert f \Vert _{ \infty } \bigr)^{\frac{p}{\beta -p}}< \theta^{*},$$

problem (1.1) admits at least two solutions in X.

### Theorem 1.2

Let $$0< s<1<p$$ with $$ps< N$$, $$\Omega \subset { \mathbb {R} }^{N}$$ is a smooth bounded domain, $$\mu =\sigma =0$$, $$M(t)=k+\lambda t^{\tau }$$, $$k>0$$, $$\lambda ,\tau >0$$. Assume

$$\mathrm{(H_{0}^{\prime})}$$ :

$$p< a+b<\min\{p(\tau +1),p^{\ast }_{s}\}$$.

In addition, suppose that one of the following holds:

$$\mathrm{(H_{3})}$$ :

$$h\in L^{\frac{p_{s}^{\ast }}{p_{s}^{\ast }-a-b}}(\Omega )$$ with $$\operatorname{meas}(\{x\in \Omega : h(x)>0\})>0$$;

$$\mathrm{(H_{4})}$$ :

$$h\in L^{\infty }(\Omega )$$ with $$\operatorname{meas}(\{x\in \Omega : h(x)>0\})>0$$,

then there exist $$\lambda_{1}\geq \lambda_{0}>0$$ such that problem (1.1) admits at least a nontrivial solution in X for $$\lambda \in (0,\lambda_{0})$$ and no nontrivial solution in X for $$\lambda >\lambda_{1}$$.

The organization of this paper as follows. In Sect. 2, we give some notations and properties of the Nehari manifold. In Sect. 3, we give the proof of Theorem 1.1. In Sect. 4, by applying the mountain pass theorem, we give the proof of Theorem 1.2.

## 2 Nehari manifold

Throughout this section we assume that all the conditions in Theorem 1.1 hold.

To simplify notations, for $$(v,w)\in X$$, we set

\begin{aligned} \mathcal{B}(v,w)=M \bigl(\Vert v \Vert ^{p} \bigr) \int_{\mathcal{Q}} \frac{\vert v(x)-v(y) \vert ^{p-2}(v(x)-v(y))(w(x)-w(y))}{\vert x-y \vert ^{N+ps}}\,dx\,dy. \end{aligned}
(2.1)

### Definition 2.1

We say that the couple $$(u,v)\in X$$ is a weak solution to (1.1) if

\begin{aligned}& \mathcal{B}(u,w_{1})+\mathcal{B}(v,w_{2}) \\& \quad = \int_{\Omega } \bigl(\mu g(x)\vert u \vert ^{\beta -2}uw_{1}+ \sigma f(x)\vert v \vert ^{\beta -2}vw_{2} \bigr)\,dx \\& \quad \quad {}+\frac{a}{a+b} \int_{\Omega }h(x)\vert u \vert ^{a-2}u\vert v \vert ^{b}w_{1}\,dx + \frac{b}{a+b} \int_{\Omega }h(x)\vert v \vert ^{b-2}v\vert u \vert ^{a}w_{2}\,dx \end{aligned}

for any $$(w_{1},w_{2})\in X$$.

Clearly, the weak solutions to (1.1) are exactly the critical points of the following functional:

\begin{aligned} \mathcal{J}_{\mu ,\sigma }(u,v)=\frac{k}{p} \bigl\Vert (u,v) \bigr\Vert ^{p}+\frac{\lambda }{\alpha }G(u,v)-\frac{1}{\beta }L(u,v)- \frac{1}{q}H(u,v), \end{aligned}
(2.2)

where

\begin{aligned}& \begin{aligned} &\alpha =p(\tau +1), \\ & G(u,v)=\Vert u \Vert ^{\alpha }+\Vert v \Vert ^{\alpha }, \\ & q=a+b, \\ &L(u,v)= \int_{\Omega } \bigl(\mu g(x)\vert u \vert ^{\beta }+ \sigma f(x)\vert v \vert ^{ \beta } \bigr)\,dx, \\ & H(u,v)= \int_{\Omega }h(x)\vert u \vert ^{a}\vert v \vert ^{b}\,dx. \end{aligned} \end{aligned}
(2.3)

A direct computation shows that $$\mathcal{J}_{\mu ,\sigma }\in C^{1}(X, { \mathbb {R} })$$ and

\begin{aligned}& \bigl\langle \mathcal{J}'_{\mu ,\sigma }(u,v),(w_{1},w_{2}) \bigr\rangle \\& \quad = \mathcal{B}(u,w_{1})+\mathcal{B}(v,w_{2})- \int_{\Omega } \bigl( \mu g(x)\vert u \vert ^{\beta -2}uw_{1}+ \sigma f(x)\vert v \vert ^{\beta -2}vw_{2} \bigr)\,dx \\& \quad \quad {}-\frac{a}{q} \int_{\Omega }h(x)\vert u \vert ^{a-2}u\vert v \vert ^{b}w_{1}\,dx -\frac{b}{q} \int_{\Omega }h(x)\vert v \vert ^{b-2}v\vert u \vert ^{a}w_{2}\,dx \end{aligned}
(2.4)

for all $$(u,v), (w_{1},w_{2})\in X$$.

We consider the Nehari manifold

$$\mathcal{S}_{\mu ,\sigma }= \bigl\{ (u,v)\in X\backslash \bigl\{ (0,0) \bigr\} \mid \bigl\langle \mathcal{J}'_{\mu ,\sigma }(u,v), (u,v) \bigr\rangle =0 \bigr\} .$$

Thus, $$(u,v)\in \mathcal{S}_{\mu ,\sigma }$$ if and only if

\begin{aligned} \bigl\langle \mathcal{J}'_{\mu ,\sigma }(u,v),(u,v) \bigr\rangle =k \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda G(u,v)-L(u,v)-H(u,v)=0. \end{aligned}
(2.5)

Denote

\begin{aligned} \Psi_{\mu ,\sigma }(u,v) &= \bigl\langle \mathcal{J}'_{\mu ,\sigma }(u, v),(u, v) \bigr\rangle \\ &=k \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda G(u,v)-L(u,v)-H(u,v). \end{aligned}

Then, for $$(u,v)\in \mathcal{S}_{\mu ,\sigma }$$,

\begin{aligned} \bigl\langle \Psi '_{\mu ,\sigma }(u,v),(u,v) \bigr\rangle =&kp \bigl\Vert (u,v) \bigr\Vert ^{p}+ \lambda \alpha G(u,v)-\beta L(u,v)-qH(u,v) \\ =&k(p-q) \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda (\alpha -q) G(u,v)-(\beta -q) L(u,v) \end{aligned}
(2.6)
\begin{aligned} =&k(p-\beta ) \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda (\alpha - \beta ) G(u,v)+(\beta -q) H(u,v). \end{aligned}
(2.7)

Obviously, $$\mathcal{S}_{\mu ,\sigma }$$ can be divided into the following three parts:

\begin{aligned} \mathcal{S}^{+}_{\mu ,\sigma }= \bigl\{ (u,v)\in \mathcal{S}_{\mu ,\sigma } \mid \bigl\langle \Psi '_{\mu ,\sigma }(u,v),(u,v) \bigr\rangle >0 \bigr\} , \\ \mathcal{S}^{0}_{\mu ,\sigma }= \bigl\{ (u,v)\in \mathcal{S}_{\mu ,\sigma } \mid \bigl\langle \Psi '_{\mu ,\sigma }(u,v),(u,v) \bigr\rangle =0 \bigr\} , \\ \mathcal{S}^{-}_{\mu ,\sigma }= \bigl\{ (u,v)\in \mathcal{S}_{\mu ,\sigma } \mid \bigl\langle \Psi '_{\mu ,\sigma }(u,v),(u,v) \bigr\rangle < 0 \bigr\} . \end{aligned}

Set

$$\theta_{0}= \biggl[\frac{k(p-q)}{(\beta -q)C_{\ast }^{\beta }} \biggr]^{\frac{p}{ \beta -p}} \biggl[ \frac{k(\beta -p)}{(\beta -q)C_{\ast }^{q}\Vert h \Vert _{ {\frac{\beta }{\beta -q}}}} \biggr]^{\frac{p}{p-q}}$$

and

$$\Lambda_{\theta_{0}}= \bigl\{ (\mu ,\sigma )\in \mathbb{R}^{2} \backslash \bigl\{ (0,0) \bigr\} : 0< \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert f \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}< \theta_{0} \bigr\} .$$

### Lemma 2.2

For any $$(\mu ,\sigma )\in \Lambda_{\theta_{0}}$$, we have $$\mathcal{S}^{0}_{\mu ,\sigma }=\emptyset$$.

### Proof

We argue by contradiction, then there exists $$(\mu ,\sigma )\in \Lambda_{\theta_{0}}$$ such that $$\mathcal{S}^{0}_{\mu ,\sigma }\neq \emptyset$$. For $$(u,v)\in \mathcal{S}^{0}_{\mu ,\sigma }$$, from (2.6) and (2.7), we can deduce that

\begin{aligned} k(p-q) \bigl\Vert (u,v) \bigr\Vert ^{p}\leq k(p-q) \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda (\alpha -q) G(u,v)=( \beta -q) L(u,v) \end{aligned}
(2.8)

and

\begin{aligned} k(\beta -p) \bigl\Vert (u,v) \bigr\Vert ^{p} \leq k( \beta -p) \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda ( \beta - \alpha ) G(u,v) =(\beta -q) H(u,v). \end{aligned}
(2.9)

By the Sobolev embedding theorem and Hölder’s inequality, we get

\begin{aligned} L(u,v) =& \int_{\Omega } \bigl(\mu g(x)\vert u \vert ^{\beta }+ \sigma f(x)\vert v \vert ^{\beta } \bigr)\,dx \\ \leq &\vert \mu \vert \Vert g \Vert _{\infty }\Vert u \Vert ^{\beta }_{\beta }+\vert \sigma \vert \Vert f \Vert _{\infty }\Vert v \Vert ^{\beta }_{\beta } \\ \leq &C_{\ast }^{\beta } \bigl(\vert \mu \vert \Vert g \Vert _{\infty }\Vert u \Vert ^{\beta }+\vert \sigma \vert \Vert f \Vert _{\infty }\Vert v \Vert ^{\beta } \bigr) \\ \leq &C_{\ast }^{\beta } \bigl[ \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{ \beta -p}}+ \bigl(\vert \sigma \vert \Vert f \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}} \bigr]^{\frac{ \beta -p}{p}} \bigl\Vert (u,v) \bigr\Vert ^{\beta }. \end{aligned}
(2.10)

From (2.8) and (2.10), it follows that

\begin{aligned} \bigl\Vert (u,v) \bigr\Vert \geq \biggl[\frac{k(p-q)}{(\beta -q)C_{\ast }^{\beta }} \biggr]^{\frac{1}{ \beta -p}} { \bigl[ \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert f \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}} \bigr]^{-\frac{1}{p}}}. \end{aligned}
(2.11)

By the Sobolev embedding theorem and Hölder’s inequality, we obtain

\begin{aligned} H(u,v)&= \int_{\Omega } h(x)\vert u \vert ^{a}\vert v \vert ^{b}\,dx \leq \Vert h \Vert _{\frac{\beta }{\beta -q}} \Vert u \Vert ^{a}_{\beta }\Vert v \Vert ^{b}_{\beta } \leq \Vert h \Vert _{{\frac{ \beta }{\beta -q}}}C_{\ast }^{q} \bigl\Vert (u,v) \bigr\Vert ^{q}. \end{aligned}
(2.12)

From (2.9) and (2.12), it follows that

\begin{aligned} \bigl\Vert (u,v) \bigr\Vert \leq \biggl[\frac{\beta -q}{k(\beta -p)} \Vert h \Vert _{\frac{\beta }{ \beta -q}}C_{\ast }^{q} \biggr]^{\frac{1}{p-q}}. \end{aligned}
(2.13)

Combining (2.11) with (2.13), it yields that

\begin{aligned} \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert f \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}} \geq \biggl[\frac{k(p-q)}{(\beta -q)C_{\ast } ^{\beta }} \biggr]^{\frac{p}{\beta -p}} \biggl[\frac{k(\beta -p)}{( \beta -q)C_{\ast }^{q}\Vert h \Vert _{\frac{\beta }{\beta -q}}} \biggr]^{ \frac{p}{p-q}} = \theta_{0}, \end{aligned}

### Lemma 2.3

The functional $$\mathcal{J}_{\mu ,\sigma }$$ is coercive and bounded below on $$\mathcal{S}_{\mu ,\sigma }$$.

### Proof

For every $$(u,v)\in \mathcal{S}_{\mu ,\sigma }$$, using (2.2), (2.5), and (2.12) yields

\begin{aligned} \mathcal{J}_{\mu ,\sigma }(u,v) =&k \biggl(\frac{1}{p}- \frac{1}{\beta } \biggr) \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda \biggl(\frac{1}{\alpha } -\frac{1}{\beta } \biggr)G(u,v)- \biggl( \frac{1}{q}-\frac{1}{\beta } \biggr)H(u,v) \\ \geq &k \biggl(\frac{1}{p}-\frac{1}{\beta } \biggr) \bigl\Vert (u,v) \bigr\Vert ^{p}- \biggl(\frac{1}{q} -\frac{1}{ \beta } \biggr) \Vert h \Vert _{\frac{\beta }{\beta -q}}C_{\ast }^{q} \bigl\Vert (u,v) \bigr\Vert ^{q}. \end{aligned}
(2.14)

Hence, $$\mathcal{J}_{\mu ,\sigma }$$ is coercive and bounded below on $$\mathcal{S}_{\mu ,\sigma }$$. □

### Lemma 2.4

Assume that $$(u_{0},v_{0})$$ is a local minimizer of $$\mathcal{J}_{ \mu ,\sigma }$$ on $$\mathcal{S}_{\mu ,\sigma }$$ and $$(u_{0},v_{0}) \notin \mathcal{S}^{0}_{\mu ,\sigma }$$, then $$\mathcal{J}_{\mu , \sigma }'(u_{0},v_{0})=0$$.

### Proof

The proof is similar to that of Theorem 2.3 in . □

### Lemma 2.5

We have

1. (1)

if $$(u,v)\in \mathcal{S}^{+}_{\mu ,\sigma }$$, then $$H(u,v)>0$$;

2. (2)

if $$(u,v)\in \mathcal{S}^{0}_{\mu ,\sigma }$$, then $$H(u,v)>0$$ and $$L(u,v)>0$$;

3. (3)

if $$(u,v)\in \mathcal{S}^{-}_{\mu ,\sigma }$$, then $$L(u,v)>0$$.

### Proof

By using (2.6) and (2.7), we arrive at the conclusion immediately. □

By Lemmas 2.22.3, for any $$(\mu ,\sigma )\in\Lambda_{\theta_{0}}$$, we obtain $$\mathcal{S}_{\mu ,\sigma }= \mathcal{S}^{+}_{\mu ,\sigma }\cup \mathcal{S}^{-}_{\mu ,\sigma }$$ and $$\mathcal{J}_{\mu ,\sigma }$$ is coercive and bounded below on $$\mathcal{S}^{+}_{\mu ,\sigma }$$ and $$\mathcal{S}^{-}_{\mu ,\sigma }$$.

Define

$$\varepsilon_{\mu ,\sigma }=\inf_{(u,v)\in \mathcal{S}_{\mu ,\sigma }} \mathcal{J}_{\mu ,\sigma }(u,v), \qquad \varepsilon^{+}_{\mu ,\sigma }= \inf_{(u,v)\in \mathcal{S}^{+}_{\mu ,\sigma }} \mathcal{J}_{\mu , \sigma }(u,v), \qquad \varepsilon^{-}_{\mu ,\sigma }= \inf_{(u,v)\in \mathcal{S}^{-}_{\mu ,\sigma }}\mathcal{J}_{\mu , \sigma }(u,v),$$

and set

$$\Lambda_{\theta^{\ast }}= \bigl\{ (\mu ,\sigma )\in \mathbb{R}^{2} \backslash \bigl\{ (0,0) \bigr\} : 0< \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert f \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}< \theta^{\ast } \bigr\} ,$$

where $$\theta^{\ast }=(\frac{q}{p})^{\frac{p}{p-q}}\theta_{0}<\theta _{0}$$. Obviously, $$\Lambda_{\theta^{\ast }}\subset \Lambda_{\theta _{0}}$$. Then the following result is established.

### Lemma 2.6

If $$(\mu ,\sigma )\in \Lambda_{\theta^{\ast }}$$, then

1. (i)

$$\varepsilon_{\mu ,\sigma }\leq \varepsilon^{+}_{\mu , \sigma }<0$$;

2. (ii)

There exists $$\eta_{0}=\eta_{0}(a,b,\beta ,p,\mu ,\sigma )>0$$ such that $$\varepsilon^{-}_{\mu ,\sigma }\geq \eta_{0}$$.

### Proof

(i) For $$(u,v)\in \mathcal{S}^{+}_{\mu ,\sigma }$$, using (2.6), we get

$$\frac{k(p-q)}{\beta -q} \bigl\Vert (u,v) \bigr\Vert ^{p}+ \frac{\lambda (\alpha -q)}{\beta -q}G(u,v)> L(u,v).$$

This combined with (2.5) yields

\begin{aligned} \mathcal{J}_{\mu ,\sigma }(u,v) =&k \biggl(\frac{1}{p}- \frac{1}{q} \biggr) \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda \biggl( \frac{1}{\alpha } -\frac{1}{q} \biggr)G(u,v)+ \biggl( \frac{1}{q}- \frac{1}{ \beta } \biggr)L(u,v) \\ < & \frac{k(q-p)(\beta -p)}{pq\beta } \bigl\Vert (u,v) \bigr\Vert ^{p} + \frac{\lambda (q- \alpha )(\beta -\alpha )}{\alpha q\beta }G(u,v) \\ < &0. \end{aligned}

Thus, $$\varepsilon_{\mu ,\sigma }\leq \varepsilon^{+}_{\mu ,\sigma }<0$$.

(ii) For $$(u,v)\in \mathcal{S}^{-}_{\mu ,\sigma }$$, by (2.6) and (2.10), we obtain

\begin{aligned} \bigl\Vert (u,v) \bigr\Vert ^{p} \leq & \bigl\Vert (u,v) \bigr\Vert ^{p}+\frac{\lambda (\alpha -q)}{k(p-q)}G(u,v) < \frac{(\beta -q)}{k(p-q)}L(u,v) \\ < &\frac{(\beta -q)}{k(p-q)}C_{\ast }^{\beta } \bigl[ \bigl(\vert \mu \vert \Vert g \Vert _{ \infty } \bigr)^{\frac{p}{\beta -p}} + \bigl(\vert \sigma \vert \Vert f \Vert _{\infty } \bigr)^{\frac{p}{ \beta -p}} \bigr]^{\frac{\beta -p}{p}} \bigl\Vert (u,v) \bigr\Vert ^{\beta }. \end{aligned}

That is,

\begin{aligned} \bigl\Vert (u,v) \bigr\Vert > \biggl(\frac{k(p-q)}{(\beta -q)C_{\ast }^{\beta }} \biggr)^{\frac{1}{ \beta -p}} \bigl[ \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert f \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}} \bigr]^{-\frac{1}{p}}. \end{aligned}
(2.15)

By using (2.15) and (2.14) of Lemma 2.3, it follows that

\begin{aligned}& \mathcal{J}_{\mu ,\sigma }(u,v) \\& \quad \geq k \biggl(\frac{1}{p}-\frac{1}{\beta } \biggr) \bigl\Vert (u,v) \bigr\Vert ^{p}- \biggl(\frac{1}{q}-\frac{1}{ \beta } \biggr) \Vert h \Vert _{\frac{\beta }{\beta -q}}C_{\ast }^{q} \bigl\Vert (u,v) \bigr\Vert ^{q} \\& \quad = \bigl\Vert (u,v) \bigr\Vert ^{q} \biggl[k \biggl( \frac{1}{p}-\frac{1}{\beta } \biggr) \bigl\Vert (u,v) \bigr\Vert ^{p-q} - \biggl( \frac{1}{q}-\frac{1}{\beta } \biggr)\Vert h \Vert _{\frac{\beta }{\beta -q}}C_{\ast } ^{q} \biggr] \\& \quad > \biggl(\frac{k(p-q)}{(\beta -q)C_{\ast }^{\beta }} \biggr)^{\frac{q}{ \beta -p}} { \bigl[ \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert h \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}} \bigr]^{-\frac{q}{p}}} \biggl[\frac{k(\beta -p)}{p\beta } \\& \quad \quad {} \times \biggl(\frac{k(p-q)}{(\beta -q)C_{\ast }^{\beta }} \biggr)^{\frac{p-q}{ \beta -p}} { \bigl( \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert h \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}} \bigr)^{-\frac{p-q}{p}}} - \biggl(\frac{1}{q}- \frac{1}{\beta } \biggr)\Vert h \Vert _{\frac{\beta }{\beta -q}}C_{ \ast }^{q} \biggr] \\& \quad \geq \eta_{0}>0, \end{aligned}

due to $$(\mu ,\sigma )\in \Lambda_{\theta^{\ast }}$$. □

Fix $$(u,v)\in X$$ with $$L(u,v)>0$$, define

\begin{aligned} \varphi (t)=kt^{p-q} \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda t^{\alpha -q}G(u,v)-t^{ \beta -q}L(u,v), \quad t\geq 0. \end{aligned}
(2.16)

Obviously, $$\varphi (0)=0$$, $$\lim_{t\rightarrow +\infty }\varphi (t)=-\infty$$, $$\varphi '(t)=t^{p-q-1}g(t)$$, where

\begin{aligned} g(t)=k(p-q) \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda ( \alpha-q)t^{\alpha -p}G(u,v) -( \beta -q)t^{\beta -p}L(u,v). \end{aligned}

When $$\lambda >0$$, denote

$$t^{*}= \biggl[\frac{\lambda (\alpha -p)(\alpha -q)G(u,v)}{(\beta -q)( \beta -p)L(u,v)} \biggr]^{\frac{1}{\beta -\alpha }}>0.$$

It is easy to see that $$g(t)$$ is increasing on $$[0,t^{*})$$ and decreasing on $$(t^{*},+\infty )$$. Note that $$g(0)=k(p-q)\Vert (u,v) \Vert ^{p}>0$$ and $$\lim_{t\rightarrow +\infty }g(t)=-\infty$$, so there exists a unique $$t_{\lambda }>t^{*}$$ such that $$g(t_{\lambda })=0$$. Moreover, $$\varphi (t)$$ reaches the maximum at $$t_{\lambda }$$, is increasing on $$[0,t_{\lambda })$$ and decreasing on $$(t_{\lambda },+\infty )$$.

When $$\lambda =0$$, we have

\begin{aligned} t_{\lambda }=t_{0}= \biggl[\frac{k(p-q)\Vert (u,v) \Vert ^{p}}{(\beta -q)L(u,v)} \biggr]^{\frac{1}{\beta -p}}>0. \end{aligned}
(2.17)

It is easy to show that $$t_{\lambda }\geq t_{0}$$ for $$\lambda \geq 0$$, thus

\begin{aligned} \varphi (t_{\lambda })\geq \varphi (t_{0})\geq t^{p-q}_{0}\frac{k( \beta -p)}{\beta -q} \bigl\Vert (u,v) \bigr\Vert ^{p}. \end{aligned}
(2.18)

Then the following lemma holds.

### Lemma 2.7

Assume $$(u,v)\in X$$ with $$L(u,v)>0$$ and $$(\mu ,\sigma )\in \Lambda_{\theta^{\ast }}$$, we obtain

(i) if $$H(u,v)\leq 0$$, then there exists a unique $$0< t_{\lambda }< t^{-}$$ such that $$(t^{-}u,t^{-}v)\in \mathcal{S}^{-}_{\mu ,\sigma }$$ and

$$\mathcal{J}_{\mu ,\sigma } \bigl(t^{-}u,t^{-}v \bigr)=\sup _{t\geq 0} \mathcal{J} _{\mu ,\sigma }(tu,tv);$$

(ii) if $$H(u,v)>0$$, then there exist unique $$0< t^{+}< t_{\lambda }< t^{-}$$ such that $$(t^{+}u,t^{+}v)\in \mathcal{S}^{+}_{\mu ,\sigma }$$, $$(t^{-}u,t^{-}v)\in \mathcal{S}^{-}_{\mu ,\sigma }$$ and

$$\mathcal{J}_{\mu ,\sigma } \bigl(t^{+}u,t^{+}v \bigr)=\inf _{0\leq t\leq t_{\lambda }}\mathcal{J}_{\mu ,\sigma }(tu,tv),\qquad \mathcal{J}_{\mu ,\sigma } \bigl(t ^{-}u,t^{-}v \bigr)=\sup _{t\geq 0} \mathcal{J}_{\mu ,\sigma }(tu,tv).$$

### Proof

(i) If $$H(u,v)\leq 0$$, by (2.16), there exists a unique $$0< t_{\lambda }< t^{-}$$ such that $$\varphi (t^{-})=H(u,v)$$ and $$\varphi '(t^{-})<0$$. Note that

\begin{aligned} \Psi_{\mu ,\sigma } \bigl(t^{-}u,t^{-}v \bigr) =&k \bigl(t^{-} \bigr)^{p} \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda \bigl(t^{-} \bigr)^{\alpha }G(u,v)- \bigl(t^{-} \bigr)^{\beta }L(u,v)- \bigl(t^{-} \bigr)^{q}H(u,v) \\ =& \bigl(t^{-} \bigr)^{q} \bigl[\varphi \bigl(t^{-} \bigr)-H(u,v) \bigr]=0 \end{aligned}

and

\begin{aligned} k(p-q) \bigl(t^{-} \bigr)^{p} \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda (\alpha -q) \bigl(t^{-} \bigr)^{\alpha }G(u,v)-( \beta -q) \bigl(t^{-} \bigr)^{\beta }L(u,v) = \bigl(t^{-} \bigr)^{q+1}\varphi ' \bigl(t^{-} \bigr)< 0. \end{aligned}

Thus, $$(t^{-}u,t^{-}v)\in \mathcal{S}^{-}_{\mu ,\sigma }$$. It is easy to derive

$$\frac{d}{\,dt}\mathcal{J}_{\mu ,\sigma }(tu,tv)=t^{q-1} \bigl[ \varphi (t)-H(u,v) \bigr].$$

Hence, $$\mathcal{J}_{\mu ,\sigma }(tu,tv)$$ increases for $$t\in [0,t ^{-})$$ and decreases for $$t\in (t^{-},+\infty )$$. This implies $$\mathcal{J}_{\mu ,\sigma }(t^{-}u,t^{-}v)=\sup_{t\geq 0} \mathcal{J}_{\mu ,\sigma }(tu,tv)$$.

(ii) If $$H(u,v)>0$$, it follows from (2.10), (2.12), (2.17), and (2.18) that

$$\varphi (0)=0< H(u,v)\leq \Vert h \Vert _{\frac{\beta }{\beta -q}}C_{\ast }^{q} \bigl\Vert (u,v) \bigr\Vert ^{q}< \varphi (t_{0})\leq \varphi (t_{\lambda })$$

for $$(\mu ,\sigma )\in \Lambda_{\theta_{0}}$$. Hence, there exist unique $$t^{+}$$, $$t^{-}>0$$ such that $$t^{+}< t_{\lambda }< t^{-}$$, $$\varphi (t ^{+})=H(u,v)=\varphi (t^{-})$$, $$\varphi '(t^{-})<0<\varphi '(t^{+})$$. Similar to the argument in (i), we get $$(t^{+}u,t^{+}v)\in \mathcal{S}^{+}_{\mu ,\sigma }$$, $$(t^{-}u,t^{-}v)\in \mathcal{S}^{-} _{\mu ,\sigma }$$, and

\begin{aligned}& \mathcal{J}_{\mu ,\sigma }(tu,tv)\geq \mathcal{J}_{\mu ,\sigma } \bigl(t^{+}u,t^{+}v \bigr),\quad \forall t\in \bigl[0,t^{-} \bigr], \\& \mathcal{J}_{\mu ,\sigma } \bigl(t^{-}u,t^{-}v \bigr)\geq \mathcal{J}_{\mu ,\sigma }(tu,tv),\quad \forall t \in [t^{+},+\infty ). \end{aligned}

Note that $$\mathcal{J}_{\mu ,\sigma }(t^{-}u,t^{-}v)\geq \varepsilon ^{-}_{\mu ,\sigma }>0$$. Thus

$$\mathcal{J}_{\mu ,\sigma } \bigl(t^{+}u,t^{+}v \bigr)=\inf _{0\leq t\leq t_{\lambda }}\mathcal{J}_{\mu ,\sigma }(tu,tv),\qquad \mathcal{J}_{\mu ,\sigma } \bigl(t ^{-}u,t^{-}v \bigr)=\sup _{t\geq 0}\mathcal{J}_{\mu ,\sigma }(tu,tv).$$

So we arrive at the conclusion. □

Fix $$(u,v)\in X$$ with $$H(u,v)>0$$, define

\begin{aligned} \bar{\varphi }(t)=kt^{p-\beta } \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda t^{\alpha - \beta }G(u,v)-t^{q-\beta }H(u,v), \quad t>0. \end{aligned}
(2.19)

Obviously, $$\lim_{t\rightarrow 0^{+}}\bar{\varphi }(t)=-\infty$$ and $$\lim_{t\rightarrow +\infty }\bar{\varphi }(t)=0$$, $$\bar{\varphi }'(t)=t ^{q-\beta -1}\bar{g}(t)$$, where

\begin{aligned} \bar{g}(t)=k(p-\beta )t^{p-q} \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda (\alpha -\beta )t ^{\alpha -q}G(u,v) +(\beta -q)H(u,v). \end{aligned}

Clearly, $$\lim_{t\rightarrow 0^{+}}\bar{g}(t)=(\beta -q)H(u,v)>0$$, $$\lim_{t\rightarrow +\infty }\bar{g}(t)=-\infty$$ and $$\bar{g}(t)$$ decreases on $$(0,+\infty )$$. Then there exists a unique $$\bar{t}_{ \lambda }>0$$ such that $$\bar{g}(\bar{t}_{\lambda })=0$$. Moreover, $$\bar{\varphi }(t)$$ reaches the maximum at $$\bar{t}_{\lambda }$$, is increasing on $$(0,\bar{t}_{\lambda })$$ and decreasing on $$(\bar{t} _{\lambda },+\infty )$$. In particular, when $$\lambda =0$$, we have

\begin{aligned} \bar{t}_{\lambda }=\bar{t}_{0}= \biggl[ \frac{(\beta -q)H(u,v)}{k(\beta -p)\Vert (u,v) \Vert ^{p}} \biggr]^{\frac{1}{p-q}}>0. \end{aligned}
(2.20)

It is easy to prove that $$\bar{t}_{0}\geq \bar{t}_{\lambda }$$ for $$\lambda \geq 0$$, thus

\begin{aligned} \bar{\varphi }(\bar{t}_{\lambda })\geq \bar{\varphi }( \bar{t}_{0}) \geq (\bar{t}_{0})^{q-\beta } \frac{p-q}{\beta -p}H(u,v). \end{aligned}
(2.21)

Then the following lemma holds.

### Lemma 2.8

Assume $$(u,v)\in X$$ with $$H(u,v)>0$$ and $$(\mu ,\sigma )\in \Lambda_{\theta^{\ast }}$$, we obtain

(i) if $$L(u,v)\leq 0$$, then there exists a unique $$0<\bar{t} ^{+}<\bar{t}_{\lambda }$$ such that $$(\bar{t}^{+}u,\bar{t}^{+}v)\in \mathcal{S}^{+}_{\mu ,\sigma }$$ and

$$\mathcal{J}_{\mu ,\sigma } \bigl(\bar{t}^{+}u,\bar{t}^{+}v \bigr)=\inf_{t\geq 0} \mathcal{J}_{\mu ,\sigma }(tu,tv);$$

(ii) if $$L(u,v)>0$$, then there exist unique $$0<\bar{t}^{+}< \bar{t}_{\lambda }<\bar{t}^{-}$$ such that $$(\bar{t}^{+}u,\bar{t}^{+}v) \in \mathcal{S}^{+}_{\mu ,\sigma }$$, $$(\bar{t}^{-}u,\bar{t}^{-}v) \in \mathcal{S}^{-}_{\mu ,\sigma }$$ and

$$\mathcal{J}_{\mu ,\sigma } \bigl(\bar{t}^{+}u,\bar{t}^{+}v \bigr)= \inf_{0\leq t\leq \bar{t}_{\lambda }}\mathcal{J}_{\mu ,\sigma }(tu,tv),\qquad \mathcal{J}_{\mu ,\sigma } \bigl(\bar{t}^{-}u,\bar{t}^{-}v \bigr)=\sup_{t \geq 0} \mathcal{J}_{\mu ,\sigma }(tu,tv).$$

### Proof

Using (2.19), (2.20), and (2.21), similar to the proof of Lemma 2.7, we can get the conclusion of Lemma 2.8. □

## 3 Proof of Theorem 1.1

Throughout this section, we still assume that all the conditions in Theorem 1.1 hold.

To prove Theorem 1.1, we first prove the following two propositions.

### Proposition 3.1

Assume $$(\mu ,\sigma )\in \Lambda_{\theta^{*}}$$, then $$\mathcal{J} _{\mu ,\sigma }$$ has a minimizer $$(u^{+}_{0},v^{+}_{0})$$ in $$\mathcal{S}^{+}_{\mu ,\sigma }$$ and satisfies

(1) $$\mathcal{J}_{\mu ,\sigma }(u^{+}_{0},v^{+}_{0})=\varepsilon ^{+}_{\mu ,\sigma }$$;

(2) $$(u^{+}_{0},v^{+}_{0})$$ is a solution of problem (1.1) such that $$u^{+}_{0}\not \equiv 0, v^{+}_{0}\not \equiv 0$$.

### Proof

By Lemma 2.3, we have $$\mathcal{J}_{\mu ,\sigma }$$ is coercive and bounded below on $$\mathcal{S}^{+}_{\mu ,\sigma }$$. Hence, there exists a minimizing sequence $$\{(u_{n},v_{n})\}\subset \mathcal{S}^{+}_{ \mu ,\sigma }$$, bounded in X. Since X is reflexive, there is a subsequence, still denoted by $$\{(u_{n},v_{n})\}$$ and $$(u^{+}_{0},v ^{+}_{0})\in X$$ such that, as $$n\rightarrow \infty$$,

$$u_{n}\rightharpoonup u^{+}_{0} \quad \mbox{ in } W_{0}; \qquad u_{n}\rightarrow u ^{+}_{0} \quad \mbox{ in } L^{r}(\Omega );$$
$$v_{n}\rightharpoonup v^{+}_{0} \quad \mbox{ in } W_{0}; \qquad v_{n}\rightarrow v ^{+}_{0} \quad \mbox{ in } L^{r}(\Omega )$$

for all $$r\in [1,p_{s}^{*})$$. By [3, Theorem 1.2.7], there exists $$\rho (x)\in L^{r}(\Omega )$$ such that

\begin{aligned}& \mbox{for all } n, \quad \bigl\vert u_{n}(x) \bigr\vert \leq \rho(x), \qquad \bigl\vert v_{n}(x) \bigr\vert \leq \rho (x)\quad \mbox{a.e. in } \Omega ; \\& u_{n}\rightarrow u^{+}_{0}, v_{n} \rightarrow v^{+}_{0 } \quad \mbox{a.e. in } \Omega \mbox{ as }n\rightarrow \infty \end{aligned}

for all $$r\in [1,p_{s}^{*})$$. By the dominated convergence theorem, we obtain

$$\lim_{n\rightarrow \infty }H(u_{n},v_{n})=H \bigl(u^{+}_{0},v^{+}_{0} \bigr), \qquad \lim_{n\rightarrow \infty }L(u_{n},v_{n})= L \bigl(u^{+}_{0},v ^{+}_{0} \bigr).$$

In view of $$(u_{n},v_{n})\in \mathcal{S}^{+}_{\mu ,\sigma }$$, then

\begin{aligned} \mathcal{J}_{\mu ,\sigma }(u_{n},v_{n}) =&k \biggl( \frac{1}{p}-\frac{1}{ \beta } \biggr) \bigl\Vert (u_{n},v_{n}) \bigr\Vert ^{p}+\lambda \biggl(\frac{1}{\alpha } -\frac{1}{ \beta } \biggr)G(u_{n},v_{n})+ \biggl(\frac{1}{\beta }- \frac{1}{q} \biggr)H(u_{n},v_{n}) \\ >& \biggl(\frac{1}{\beta }-\frac{1}{q} \biggr)H(u_{n},v_{n}), \end{aligned}

and

$$\lim_{n\rightarrow \infty }\mathcal{J}_{\mu ,\sigma }(u_{n},v_{n})= \varepsilon^{+}_{\mu ,\sigma }< 0.$$

It follows $$H(u^{+}_{0},v^{+}_{0})>0$$, in particular, $$u^{+}_{0} \not \equiv 0$$, $$v^{+}_{0}\not \equiv 0$$. Next, we show that $$u_{n}\rightarrow u^{+}_{0}$$ in $$W_{0}$$, $$v_{n}\rightarrow v^{+}_{0}$$ in $$W_{0}$$. If not, then either

\begin{aligned} \bigl\Vert u^{+}_{0} \bigr\Vert < \liminf_{n\rightarrow \infty }\Vert u_{n} \Vert \quad \mbox{or} \quad \bigl\Vert v^{+}_{0} \bigr\Vert < \liminf _{n\rightarrow \infty }\Vert v_{n} \Vert . \end{aligned}
(3.1)

Fix $$(u,v)\in X$$ with $$H(u,v)>0$$, denote

$$\psi_{(u,v)}(t)=\bar{\varphi }(t)-L(u,v),\quad t>0,$$

where $$\bar{\varphi }(t)$$ is given by (2.19). Obviously, $$\lim_{t\rightarrow 0^{+}}\psi_{(u,v)}(t)=-\infty$$, $$\lim_{t\rightarrow +\infty }\psi_{(u,v)}(t)=-L(u,v)$$, $$\psi '_{(u,v)}(t)=\bar{ \varphi }'(t)$$, then we get that $$\varphi_{(u,v)}(t)$$ reaches the maximum at $$\bar{t}_{\lambda }$$, is increasing on $$(0,\bar{t}_{\lambda })$$ and decreasing on $$(\bar{t}_{\lambda },+\infty )$$. Note that $$H(u^{+}_{0},v^{+}_{0})>0$$, by Lemma 2.8, there exists a unique $$0< t^{+}_{0}<\bar{t}_{\lambda }(u^{+}_{0},v^{+}_{0})$$ such that $$(t^{+}_{0}u^{+}_{0},t^{+}_{0}v^{+}_{0})\in \mathcal{S}^{+}_{\mu , \sigma }$$ and

\begin{aligned} \mathcal{J}_{\mu ,\sigma } \bigl(t^{+}_{0}u^{+}_{0},t^{+}_{0}v^{+}_{0} \bigr)= \inf_{0\leq t\leq \bar{t}_{\lambda }(u^{+}_{0},v^{+}_{0})} \mathcal{J}_{\mu ,\sigma } \bigl(tu^{+}_{0},tv^{+}_{0} \bigr). \end{aligned}
(3.2)

Note that $$(t^{+}_{0}u^{+}_{0},t^{+}_{0}v^{+}_{0})\in \mathcal{S}^{+} _{\mu ,\sigma }\subset \mathcal{S}_{\mu ,\sigma }$$ and the definition of $$\bar{\varphi }(t)$$, it is easy to derive

\begin{aligned} \psi_{(u^{+}_{0},v^{+}_{0})} \bigl(t^{+}_{0} \bigr) =&\bar{\varphi } \bigl(t^{+}_{0} \bigr)-L \bigl(u ^{+}_{0},v^{+}_{0} \bigr)=0. \end{aligned}
(3.3)

It follows from (3.1) and (3.3) that

$$\psi_{(u_{n},v_{n})} \bigl(t^{+}_{0} \bigr)>0 \quad \mbox{ for large enough } n.$$

Since $$(u_{n},v_{n})\in \mathcal{S}^{+}_{\mu ,\sigma }$$, we get $$\bar{t}_{\lambda }(u_{n},v_{n})>1$$. Moreover, we can deduce that $$\psi_{(u_{n},v_{n})}(1)=\bar{\varphi }(1)-L(u_{n},v_{n})=0$$ and $$\psi_{(u_{n},v_{n})}(t)$$ increases on $$(0,\bar{t}_{\lambda }(u_{n},v _{n}))$$. This implies, for all n, $$\psi_{(u_{n},v_{n})}(t)\leq 0, \forall t\in (0,1]$$. Thus

\begin{aligned} 1< t^{+}_{0}< \bar{t}_{\lambda } \bigl(u^{+}_{0},v^{+}_{0} \bigr). \end{aligned}
(3.4)

It follows from (3.1), (3.2), and (3.4) that

$$\mathcal{J}_{\mu ,\sigma } \bigl(t^{+}_{0}u^{+}_{0},t^{+}_{0}v^{+}_{0} \bigr) \leq \mathcal{J}_{\mu ,\sigma } \bigl(u^{+}_{0},v^{+}_{0} \bigr)< \lim_{n\rightarrow \infty }\mathcal{J}_{\mu ,\sigma }(u_{n},v_{n}) = \varepsilon^{+}_{\mu ,\sigma },$$

$$u_{n}\rightarrow u^{+}_{0}\quad \mbox{in } W_{0},\qquad v_{n}\rightarrow v ^{+}_{0} \quad \mbox{in } W_{0}.$$

This implies

$$\mathcal{J}_{\mu ,\sigma }(u_{n},v_{n})\rightarrow\mathcal{J}_{\mu ,\sigma } \bigl(u^{+}_{0},v^{+}_{0}\bigr)=\varepsilon^{+}_{\mu ,\sigma }\quad \mbox{as } n\rightarrow \infty .$$

Namely, $$(u^{+}_{0},v^{+}_{0})$$ is a minimizer of $$\mathcal{J}_{ \mu ,\sigma }$$ on $$\mathcal{S}^{+}_{\mu ,\sigma }$$. By Lemma 2.4, $$(u^{+}_{0},v^{+}_{0})$$ is a solution of problem (1.1) such that $$u^{+}_{0}\not \equiv 0, v^{+}_{0}\not \equiv 0$$. □

### Proposition 3.2

Assume $$(\mu ,\sigma )\in \Lambda_{\theta^{*}}$$, then $$\mathcal{J} _{\mu ,\sigma }$$ has a minimizer $$(u^{-}_{0},v^{-}_{0})$$ in $$\mathcal{S}^{-}_{\mu ,\sigma }$$ and satisfies

(1) $$\mathcal{J}_{\mu ,\sigma }(u^{-}_{0},v^{-}_{0})=\varepsilon ^{-}_{\mu ,\sigma }$$;

(2) $$(u^{-}_{0},v^{-}_{0})$$ is a solution of problem (1.1) such that $$(u^{-}_{0},v^{-}_{0})\not \equiv (0,0)$$.

### Proof

Since $$\mathcal{J}_{\mu ,\sigma }$$ is coercive and bounded below on $$\mathcal{S}^{-}_{\mu ,\sigma }$$, there exists a minimizing sequence $$\{(u_{n},v_{n})\}\subset \mathcal{S}^{-}_{\mu ,\sigma }$$, bounded in X. Note that X is reflexive, then there is a subsequence, still denoted by $$\{(u_{n},v_{n})\}$$ and $$(u^{-}_{0},v^{-}_{0})\in X$$ such that, as $$n\rightarrow \infty$$,

\begin{aligned}& u_{n}\rightharpoonup u^{-}_{0} \quad \mbox{in }W_{0};\qquad u_{n}\rightarrow u ^{-}_{0}\quad \mbox{in } L^{r}(\Omega ); \\& v_{n}\rightharpoonup v^{-}_{0} \quad \mbox{in }W_{0}; \qquad v_{n}\rightarrow v ^{-}_{0}\quad \mbox{in } L^{r}(\Omega ) \end{aligned}

for all $$r\in [1,p_{s}^{*})$$. By [3, Theorem 1.2.7], there exists $$\varrho (x)\in L^{r}(\Omega )$$ such that

\begin{aligned}& \mbox{for all } n, \quad \bigl\vert u_{n}(x) \bigr\vert \leq \varrho (x),\qquad \bigl\vert v_{n}(x) \bigr\vert \leq \varrho (x)\quad \mbox{a.e. in } \Omega ; \\& u_{n}\rightarrow u^{-}_{0}, \qquad v_{n}\rightarrow v^{-}_{0 } \quad\mbox{a.e. in } \Omega \mbox{ as }n\rightarrow \infty \end{aligned}

for all $$r\in [1,p_{s}^{*})$$. By the dominated convergence theorem, we obtain

$$\lim_{n\rightarrow \infty }H(u_{n},v_{n})=H \bigl(u^{-}_{0},v^{-}_{0} \bigr), \qquad \lim_{n\rightarrow \infty }L(u_{n},v_{n})= L \bigl(u^{-}_{0},v ^{-}_{0} \bigr).$$

Moreover, by (2.6), we have

\begin{aligned} L(u_{n},v_{n})>\frac{k(p-q)}{\beta -q} \bigl\Vert (u_{n},v_{n}) \bigr\Vert ^{p}. \end{aligned}
(3.5)

Using (2.10) and (3.5), there exists a positive constant c such that

$$L({u_{n},v_{n}})>c>0.$$

This implies

$$L \bigl(u^{-}_{0},v^{-}_{0} \bigr)>0.$$

In particular, $$(u^{-}_{0},v^{-}_{0})\not \equiv (0,0)$$. Next, we show that $$u_{n}\rightarrow u^{-}_{0}$$ in $$W_{0}$$, $$v_{n}\rightarrow v^{-} _{0}$$ in $$W_{0}$$. If not, then either

\begin{aligned} \bigl\Vert u^{-}_{0} \bigr\Vert < \liminf_{n\rightarrow \infty }\Vert u_{n} \Vert \quad \mbox{or} \quad \bigl\Vert v^{-}_{0} \bigr\Vert < \liminf _{n\rightarrow \infty }\Vert v_{n} \Vert . \end{aligned}
(3.6)

By Lemma 2.7, there exists a unique $$t^{-}_{0}>t_{\lambda }(u ^{-}_{0},v^{-}_{0})>0$$ such that $$(t^{-}_{0}u^{-}_{0},t^{-}_{0}v^{-} _{0})\in \mathcal{S}^{-}_{\mu ,\sigma }$$. In view of $$(u_{n},v_{n}) \in \mathcal{S}^{-}_{\mu ,\sigma }$$, it follows $$\mathcal{J}_{\mu , \sigma }(tu_{n},tv_{n})\leq \mathcal{J}_{\mu ,\sigma }(u_{n},v_{n})$$ for $$t\geq 0$$. By (3.6), we have

$$\mathcal{J}_{\mu ,\sigma } \bigl(t^{-}_{0}u^{-}_{0},t^{-}_{0}v^{-}_{0} \bigr)< \liminf_{n\rightarrow \infty }\mathcal{J}_{\mu ,\sigma } \bigl(t^{-}_{0}u _{n},t^{-}_{0}v_{n} \bigr)\leq \lim_{n\rightarrow \infty }\mathcal{J}_{ \mu ,\sigma }(u_{n},v_{n}) =\varepsilon^{-}_{\mu ,\sigma },$$

$$u_{n}\rightarrow u^{-}_{0}\quad \mbox{in }W_{0},\qquad v_{n}\rightarrow v ^{-}_{0}\quad \mbox{in } W_{0}.$$

This implies

$$\mathcal{J}_{\mu ,\sigma }(u_{n},v_{n})\rightarrow \mathcal{J}_{ \mu ,\sigma } \bigl(u^{-}_{0},v^{-}_{0} \bigr)=\varepsilon^{-}_{\mu ,\sigma }\quad \mbox{as } n\rightarrow \infty .$$

Namely, $$(u^{-}_{0},v^{-}_{0})$$ is a minimizer of $$\mathcal{J}_{ \mu ,\sigma }$$ on $$\mathcal{S}^{-}_{\mu ,\sigma }$$. By Lemma 2.4, $$(u^{-}_{0},v^{-}_{0})$$ is a solution of problem (1.1) such that $$(u^{-}_{0},v^{-}_{0})\not \equiv (0,0)$$. □

### Proof of Theorem 1.1

By Propositions 3.1 and 3.2, we obtain that when $$(\mu ,\sigma )\in \Lambda_{\theta^{\ast }}$$, problem (1.1) has at least two solutions $$(u^{+}_{0},v^{+}_{0})$$ and $$(u^{-}_{0},v^{-}_{0})$$ such that $$(u^{+}_{0},v^{+}_{0})\in \mathcal{S}^{+}_{\mu ,\sigma }, (u^{-}_{0},v ^{-}_{0})\in \mathcal{S}^{-}_{\mu ,\sigma }$$, where $$u^{+}_{0}\not \equiv 0,v^{+}_{0}\not \equiv 0$$ and $$(u^{-}_{0},v^{-}_{0})\not \equiv (0,0)$$. Note that $$\mathcal{S}^{+}_{\mu ,\sigma }\cap \mathcal{S}^{-}_{\mu ,\sigma }=\emptyset$$, then these two solutions are distinct. This finishes the proof. □

## 4 Proof of Theorem 1.2

Throughout this section, we assume that all the conditions in Theorem 1.2 hold.

Since $$\mu =\sigma =0$$, then (2.2) becomes

\begin{aligned} \mathcal{J}(u,v)=\mathcal{J}_{0,0}(u,v)= \frac{k}{p} \bigl\Vert (u,v) \bigr\Vert ^{p}+ \frac{ \lambda }{\alpha }G(u,v)-\frac{1}{q}H(u,v), \end{aligned}
(4.1)

where $$\alpha , G(u,v), q, H(u,v)$$ are as in (2.3).

If $$\mathrm{(H_{3})}$$ holds, we derive from Hölder’s inequality and (1.6) that

\begin{aligned} H(u,v)= \int_{\Omega } h(x)\vert u \vert ^{a}\vert v \vert ^{b}\,dx \leq \Vert h \Vert _{\frac{p_{s} ^{\ast }}{p_{s}^{\ast }-q}}C_{p_{s}^{\ast }}^{q} \bigl\Vert (u,v) \bigr\Vert ^{q}. \end{aligned}
(4.2)

If $$\mathrm{(H_{4})}$$ holds, similarly, we have

\begin{aligned} H(u,v)= \int_{\Omega } h(x)\vert u \vert ^{a}\vert v \vert ^{b}\,dx\leq \Vert h \Vert _{\infty }C_{q} ^{q} \bigl\Vert (u,v) \bigr\Vert ^{q}. \end{aligned}
(4.3)

The proof of Theorem 1.2 is mainly dependent on the following mountain pass theorem.

### Lemma 4.1

()

Let X be a real Banach space, suppose that $$\mathcal{I}\in C^{1}(X, \mathbb {R} )$$ satisfies $$(\mathit{PS})$$ with $$\mathcal{I}(0)=0$$. In addition,

$$\mathrm{(A_{1})}$$ :

there exist positive numbers δ and d such that $$\mathcal{I}(u)\geq d$$ if $$\Vert u \Vert _{X}=\delta$$;

$$\mathrm{(A_{2})}$$ :

there exists $$v\in X$$ such that $$\Vert v \Vert _{X}>\delta$$ and $$\mathcal{I}(v)<0$$.

Then there exists a critical value $$e\geq d$$ for $$\mathcal{I}$$. Moreover, e can be characterized as

$$e=\inf_{\Phi \in \Gamma }\max_{0\leq t\leq 1}\mathcal{I} \bigl( \Phi (t) \bigr),$$

where

$$\Gamma = \bigl\{ \Phi \in C \bigl([0,1],X \bigr) \mid \Phi (0)=0, \Phi (1)=v \bigr\} .$$

Next, we will prove that the functional $$\mathcal{J}$$ defined by (4.1) satisfies $$(\mathit{PS})$$.

Recall that we say $$\mathcal{J}$$ satisfies the $$(\mathit{PS})$$ condition at the level $$c\in \mathbb {R}$$ (shortly: $$\mathcal{J}$$ satisfies $$(PS)_{c}$$) if every sequence $$\{(u_{n},v_{n})\}\subset X$$ along with $$\mathcal{J}(u_{n},v _{n})\rightarrow c$$ and $$\mathcal{J}'(u_{n},v_{n})\rightarrow 0$$ as $$n\rightarrow \infty$$ has a converging subsequence (in X). We say $$\mathcal{J}$$ satisfies the $$(\mathit{PS})$$ condition (shortly: $$\mathcal{J}$$ satisfies $$(\mathit{PS})$$) if $$\mathcal{J}$$ satisfies $$(PS)_{{c}}$$ for each $$c\in \mathbb {R}$$.

### Lemma 4.2

Any $$(PS)_{c}$$ sequence $$\{(u_{n},v_{n})\}$$ for $$\mathcal{J}$$ is bounded in X.

### Proof

Let the sequence $$\{(u_{n},v_{n})\}\subset X$$ satisfy

\begin{aligned} \mathcal{J}(u_{n},v_{n})\rightarrow c,\qquad \mathcal{J}'(u_{n},v_{n}) \rightarrow 0\quad \mbox{as }n\rightarrow \infty . \end{aligned}
(4.4)

If $$(H_{0}^{\prime})$$ and $$\mathrm{(H_{3})}$$ are true, by (4.2), we take $$s>\alpha$$ and obtain that, for large n,

\begin{aligned}& c+1+ \bigl\Vert (u_{n},v_{n}) \bigr\Vert \\& \quad \geq \mathcal{J}(u_{n},v_{n})-s^{-1} \mathcal{J}'(u_{n},v_{n}) (u_{n},v_{n}) \geq \lambda \biggl(\frac{1}{\alpha }-\frac{1}{s} \biggr)G(u_{n},v_{n})+ \biggl( \frac{1}{s}-\frac{1}{q} \biggr)H(u_{n}, v_{n}) \\& \quad \geq \frac{\lambda }{2^{\tau }} \biggl(\frac{1}{\alpha }-\frac{1}{s} \biggr) \bigl\Vert (u _{n},v_{n}) \bigr\Vert ^{\alpha } + \biggl( \frac{1}{s}-\frac{1}{q} \biggr)\Vert h \Vert _{\frac{p_{s} ^{\ast }}{p_{s}^{\ast }-q}}C_{p_{s}^{\ast }}^{q} \bigl\Vert (u_{n},v_{n}) \bigr\Vert ^{q}. \end{aligned}
(4.5)

This implies that $$\{(u_{n},v_{n})\}$$ is bounded in X.

If $$(H_{0}^{\prime})$$ and $$\mathrm{(H_{4})}$$ hold, substituting $$H(u_{n},v _{n})$$ in (4.5) by (4.3), we conclude that $$\{(u_{n},v _{n})\}$$ is bounded in X. This finishes the proof. □

In view of the sequence $$\{(u_{n},v_{n})\}$$ given by (4.4) is a bounded sequence in X, there is a subsequence, still denoted by $$\{(u_{n},v_{n})\}$$ and $$(u,v)\in X$$ such that $$\Vert (u_{n},v_{n}) \Vert \leq M, \Vert (u,v) \Vert \leq M$$ with some constant $$M>0$$ and, as $$n\rightarrow \infty$$,

\begin{aligned} \textstyle\begin{cases} u_{n}\rightharpoonup u\quad \mbox{in } W_{0}, \qquad u_{n}\rightarrow u\quad \mbox{ in } L^{r}(\Omega ), \forall r\in [1,p_{s}^{*}); \\ v_{n}\rightharpoonup v \quad \mbox{in } W_{0},\qquad v_{n}\rightarrow v\quad \mbox{ in } L^{r}(\Omega ), \forall r\in [1,p_{s}^{*}); \\ u_{n}\rightarrow u, \qquad v_{n}\rightarrow v \quad \mbox{a.e. in } \Omega . \end{cases}\displaystyle \end{aligned}
(4.6)

### Lemma 4.3

Let $$\{(u_{n},v_{n})\}$$ be a $$(PS)_{c}$$ sequence and satisfy (4.6). Then the following statements hold:

1. (i)

$$(u_{n},v_{n})\rightarrow (u,v)$$ in X as $$n\rightarrow \infty$$, that is, $$\mathcal{J}$$ satisfies $$(\mathit{PS})$$.

2. (ii)

$$(u,v)\in X$$ is a critical point for $$\mathcal{J}$$.

### Proof

(i)

$$\bigl\langle \mathcal{J}'(u_{n},v_{n}) (u_{n}-u,0) \bigr\rangle =P_{n}- \frac{a}{q} \int_{\Omega }h(x)\vert u_{n} \vert ^{a-2}u_{n}(u_{n}-u)\vert v_{n} \vert ^{b}\,dx,$$

where

\begin{aligned} P_{n} =M \bigl(\Vert u_{n} \Vert ^{p} \bigr) \int_{\mathcal{Q}}\frac{\vert \omega_{n} \vert ^{p-2} \omega_{n}(\omega_{n}-\omega )}{\vert x-y \vert ^{N+ps}}\,dx\,dy,\qquad \omega_{n}=u_{n}(x)-u_{n}(y), \omega =u(x)-u(y). \end{aligned}

Obviously, $$\langle \mathcal{J}'(u_{n},v_{n})(u_{n}-u,0)\rangle \rightarrow 0$$ as $$n\rightarrow \infty$$ since $$\mathcal{J}'(u_{n},v _{n})\rightarrow 0$$. By the dominated convergence theorem, we obtain

\begin{aligned} \int_{\Omega }h(x)\vert u_{n} \vert ^{a-2}u_{n}(u_{n}-u)\vert v_{n} \vert ^{b}\,dx\rightarrow 0\quad \mbox{as }n\rightarrow \infty . \end{aligned}

This implies $$P_{n}\rightarrow 0 as n\rightarrow \infty$$. Moreover, the fact $$u_{n}\rightharpoonup u$$ in $$W_{0}$$ implies $$Q_{n}\rightarrow 0$$, where $$Q_{n}=M(\Vert u_{n} \Vert ^{p})\int_{\mathcal{Q}}\frac{\vert \omega \vert ^{p-2} \omega (\omega_{n}-\omega )}{\vert x-y \vert ^{N+ps}}\,dx\,dy$$.

Therefore,

\begin{aligned} P_{n}-Q_{n} =M \bigl(\Vert u_{n} \Vert ^{p} \bigr) \int_{\mathcal{Q}}\frac{[\vert \omega_{n} \vert ^{p-2} \omega_{n}-\vert \omega \vert ^{p-2}\omega ](\omega_{n}-\omega )}{\vert x-y \vert ^{N+ps}}\,dx\,dy \rightarrow 0\quad \mbox{as } n\rightarrow \infty . \end{aligned}
(4.7)

Notice that there are the well-known vector inequalities given by

\begin{aligned}& \begin{aligned} &\bigl\langle \vert \eta \vert ^{p-2}\eta -\vert \xi \vert ^{p-2}\xi ,\eta -\xi \bigr\rangle \geq S _{p}\vert \eta -\xi \vert ^{p},\quad p\geq 2, \\ &\bigl\langle \vert \eta \vert ^{p-2}\eta -\vert \xi \vert ^{p-2}\xi ,\eta -\xi \bigr\rangle \geq \widetilde{S}_{p} \vert \eta -\xi \vert ^{2}\bigl(\vert \eta \vert +\vert \xi \vert \bigr)^{p-2},\quad 1< p< 2, \end{aligned} \end{aligned}
(4.8)

for all η, $$\xi \in { \mathbb {R} }^{N}$$, where $$S_{p}$$, $$\widetilde{S}_{p}>0$$ are constants depending only on p. Combining (4.7) with (4.8), it follows

$$u_{n}\rightarrow u \quad \mbox{in } W_{0} \mbox{ as }n\rightarrow\infty .$$

Similarly, we can also prove

$$v_{n}\rightarrow v \quad \mbox{in } W_{0} \mbox{ as }n \rightarrow\infty .$$

Therefore,

$$(u_{n},v_{n})\rightarrow (u,v) \quad \mbox{in } X \mbox{ as } n\rightarrow\infty .$$

This implies that $$\mathcal{J}$$ satisfies $$(\mathit{PS})$$.

(ii) Note that $$\mathcal{J}\in C^{1}(X, \mathbb {R} )$$, using $$(u_{n},v _{n})$$ to be a $$(PS)_{c}$$ sequence and (i) of Lemma 4.3, we can prove that $$(u,v)\in X$$ is a critical point for $$\mathcal{J}$$. □

### Proof of Theorem 1.2

By Lemmas 4.24.3, we have that the functional $$\mathcal{J}$$ satisfies $$(\mathit{PS})$$. Next, we prove that $$\mathcal{J}$$ satisfies $$\mathrm{(A1)}$$ and $$\mathrm{(A2)}$$. By (4.2) and (4.3), we get

\begin{aligned} \mathcal{J}(u,v) \geq &\frac{k}{p} \bigl\Vert (u,v) \bigr\Vert ^{p}+\frac{\lambda }{ \alpha }G(u,v)-\frac{l}{q} \bigl\Vert (u,v) \bigr\Vert ^{q} \\ \geq &\frac{k}{p} \bigl\Vert (u,v) \bigr\Vert ^{p}+ \frac{\lambda }{2^{\tau }\alpha } \bigl\Vert (u,v) \bigr\Vert ^{\alpha }- \frac{l}{q} \bigl\Vert (u,v) \bigr\Vert ^{q}, \end{aligned}

where $$l=\max \{C^{q}_{p_{s}^{\ast }}\Vert h \Vert _{\frac{p_{s}^{\ast }}{p _{s}^{\ast }-q}}, C_{q}^{q}\Vert h \Vert _{\infty }\}$$. Denote

$$\phi (z)=z^{p} \biggl(\frac{k}{p}+\frac{b}{2^{\tau }\alpha }z^{\alpha -p}- \frac{l}{q}z^{q-p} \biggr),\quad z>0.$$

It is easy to verify that there exist $$z_{1}, d>0$$ such that $$\phi (z_{1})\geq d$$. Let $$\delta =z_{1}$$, we get $$\mathcal{J}(u,v) \geq d$$ if $$\Vert (u,v) \Vert =\delta$$. Thus, condition $$(A_{1})$$ is satisfied.

Furthermore, we verify $$(A_{2})$$. By $$\mathrm{(H_{3})}$$ or $$\mathrm{(H_{4})}$$, we can choose $$u_{0}, v_{0}\in C_{0}^{\infty }(\Omega ), u_{0}\not \equiv 0, v_{0}\not \equiv 0$$, such that $$\Vert u_{0} \Vert =\Vert v_{0} \Vert =1$$ and $$\int_{\Omega }h(x)\vert u_{0} \vert ^{a}\vert v_{0} \vert ^{b}\,dx>0$$. Let

\begin{aligned} \mathcal{J}(tu_{0},tv_{0})=t^{p} \gamma (t),\qquad \gamma (t)=B_{1}+\lambda B_{2}t^{\alpha -p}-B_{3}t^{q-p}, \quad t\geq 0, \end{aligned}
(4.9)

where

$$B_{1}=\frac{2k}{p}>0, \qquad B_{2}= \frac{2}{\alpha }>0, \qquad B_{3}= \int_{\Omega}h(x)\vert u_{0} \vert ^{a} \vert v_{0} \vert ^{b}\,dx>0.$$

Then there exist $$\lambda_{0}>0$$ and large $$t_{\lambda }>2^{- \frac{1}{p}}\delta$$ such that $$\mathcal{J}(t_{\lambda }u_{0},t_{ \lambda }v_{0})<0$$ for $$\lambda \in (0,\lambda_{0})$$, where δ is in $$\mathrm{(A1)}$$. Let $$q_{0}=t_{\lambda }u_{0}, w_{0}=t _{\lambda }v_{0}$$, then $$\Vert (q_{0},w_{0}) \Vert >\delta$$ and $$\mathcal{J}(q _{0},w_{0})<0$$. Hence, by Lemma 4.1, there exists $$(u_{1},v_{1}) \in X$$ with $$\mathcal{(}u_{1},v_{1})\geq d>0$$ which is a solution of problem (1.1) under the assumptions in Theorem 1.2. In addition, it is easy to verify $$u_{1}\not \equiv 0, v_{1}\not \equiv 0$$. We now prove the second part of Theorem 1.2. If $$(u,v)$$ is a nontrivial solution of problem (1.1), combining (4.2)–(4.3) with Young’s inequality, we get

\begin{aligned} k \bigl\Vert (u,v) \bigr\Vert ^{p}+ \frac{\lambda }{2^{\tau }} \bigl\Vert (u,v) \bigr\Vert ^{\alpha } \leq & k \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda G(u,v) = \int_{\Omega }h(x)\vert u \vert ^{a}\vert v \vert ^{b}\,dx \\ \leq & l \bigl\Vert (u,v) \bigr\Vert ^{q}\leq k \bigl\Vert (u,v) \bigr\Vert ^{p}+\frac{\lambda_{1}}{2^{ \tau }} \bigl\Vert (u,v) \bigr\Vert ^{\alpha }, \end{aligned}
(4.10)

where $$l=\max \{C^{q}_{p_{s}^{\ast }}\Vert h \Vert _{\frac{p_{s}^{\ast }}{p _{s}^{\ast }-q}}, C_{q}^{q}\Vert h \Vert _{\infty }\}$$, $$\lambda_{1}=2^{\tau }(\frac{q-p}{ \alpha -p})(\frac{\alpha -q}{\alpha -p})^{\frac{\alpha -q}{q-p}}l^{\frac{ \alpha -p}{q-p}}k^{-\frac{\alpha -q}{q-p}}$$. This implies $$0<\lambda \leq \lambda_{1}$$. In other words, if $$\lambda >\lambda_{1}$$, there is no nontrivial solution of problem (1.1). This finishes the proof. □

## References

1. Alves, C.O., Corrêa, F.J.S.A., Ma, T.F.: Positive solutions for a quasilinear elliptic equation of Kirchhoff type. Comput. Math. Appl. 49(1), 85–93 (2005)

2. Applebaum, D.: Lévy processes-from probability of finance quantum groups. Not. Am. Math. Soc. 51, 1336–1347 (2004)

3. Badiale, M., Serra, E.: Semilinear Elliptic Equations for Beginners: Existence Results Via the Variational Approach. Springer, London (2010) https://doi.org/10.1007/978-0-85729-227-8

4. Barrios, B., Colorado, E., De Pablo, A., Sanchez, U.: On some critical problems for the fractional Laplacian operator. J. Differ. Equ. 252, 6133–6162 (2012)

5. Brändle, C., Colorado, E., de Pablo, A., Sánchez, U.: A concave-convex elliptic problem involving the fractional Laplacian. Proc. R. Soc. Edinb. A 143, 39–71 (2013)

6. Brown, K.J., Wu, T.F.: A semilinear elliptic system involving nonlinear boundary condition and sign-changing weight function. J. Math. Anal. Appl. 337, 1326–1336 (2008)

7. Brown, K.J., Zhang, Y.: The Nehari manifold for a semilinear elliptic equation with a sign-changing weight function. J. Differ. Equ. 193, 481–499 (2003)

8. Caffarelli, L.: Nonlocal equations, drifts and games. In: Nonlinear Partial Differential Equations. Abel Symposia, vol. 7, pp. 37–52 (2012)

9. Chen, C.S., Chen, Q.: Infinitely many solutions for p-Kirchhoff equation with concave-convex nonlinearities in $$\mathbb {R} ^{N}$$. Math. Methods Appl. Sci. 39, 1493–1504 (2016)

10. Chen, J.W., Deng, B.S.: The Nehari manifold for non-local elliptic operators involving concave-convex nonlinearities. Z. Angew. Math. Phys. 66, 1387–1400 (2015)

11. Chen, J.W., Deng, B.S.: The Nehari manifold for a fractional p-Laplacian system involving concave-convex nonlinearities. Nonlinear Anal., Real World Appl. 27, 80–92 (2016)

12. Di Nezza, E., Palatucci, G., Valdinoci, E.: Hitchhiker’s guide to the fractional Sobolev spaces. Bull. Sci. Math. 136, 521–573 (2012)

13. Dong, X.Y., Bai, Z.B., Zhang, S.Q.: Positive solutions to boundary value problems of p-Laplacian with fractional derivative. Bound. Value Probl. 2017, Article ID 5 (2017). https://doi.org/10.1186/s13661-016-0735-z

14. Figueiredo, G.M.: Existence of a positive solutions for a Kirchhoff equation with critical growth via truncation argument. J. Math. Anal. Appl. 401(2), 706–713 (2013)

15. Goyal, S., Sreenadh, K.: A Nehari manifold for non-local elliptic operator with concave-convex non-linearities and sign-changing weight function. Proc. Math. Sci. 125(4), 545–558 (2015)

16. Hsu, T.S.: Multiple positive solutions for a critical quasilinear elliptic system with concave-convex nonlinearities. Nonlinear Anal. 71, 2688–2698 (2009)

17. Hsu, T.S., Lin, H.L.: Multiple positive solutions for a critical elliptic system with concave-convex nonlinearities. Proc. R. Soc. Edinb. A 139(6), 1163–1177 (2009)

18. Lu, C.N., Fu, C., Yang, H.W.: Time-fractional generalized Boussinesq equation for Rossby solitary waves with dissipation effect in stratified fluid and conservation laws as well as exact solutions. Appl. Math. Comput. 327, 104–116 (2018)

19. Lu, F.Y.: The Nehari manifold and application to a semilinear elliptic system. Nonlinear Anal. 71, 3425–3433 (2009)

20. Molica Bisci, G., Repovs, D.: Higher nonlocal problems with bounded potential. J. Math. Anal. Appl. 420, 591–601 (2014)

21. Pucci, P., Saldi, S.: Critical stationary Kirchhoff equations in $$\mathbb {R} ^{N}$$ involving nonlocal operators. Rev. Mat. Iberoam. 32(1), 1–22 (2016)

22. Pucci, P., Xiang, Q.M., Zhang, B.L.: Multiple solutions for nonhomogeneous Schrödinger–Kirchhoff type equations involving the fractional p-Laplacian in $${ \mathbb {R} }^{N}$$. Calc. Var. 54, 2785–2806 (2015)

23. Pucci, P., Xiang, Q.M., Zhang, B.L.: Existence and multiplicity of entire solutions for fractional p-Kirchhoff equations. Adv. Nonlinear Anal. 5(1), 27–55 (2016)

24. Rasouli, S.H., Afrouzi, G.A.: The Nehari manifold for a class of concave-convex elliptic systems involving the p-Laplacian and nonlinear boundary condition. Nonlinear Anal. 73, 3390–3401 (2010)

25. Servadei, R., Valdinoci, E.: Mountain pass solutions for non-local elliptic operators. J. Math. Anal. Appl. 389, 887–898 (2012)

26. Song, H.X., Chen, C.S., Yan, Q.L.: Existence of multiple solutions for a p-Laplacian system in $${ \mathbb {R} }^{N}$$ with sign-changing weight functions. Can. Math. Bull. 59(2), 417–434 (2016)

27. Struwe, M.: Variational methods. Applications to Nonlinear Partial Differential Equations and Hamiltonian Systems, 3rd edn. A Series of Modern Surveys in Mathematics, vol. 34. Springer, New York (2000). https://doi.org/10.1007/978-3-662-04194-9

28. Teng, K.M.: Two nontrivial solutions for hemivariational inequalities driven by nonlocal elliptic operators. Nonlinear Anal., Real World Appl. 14, 867–874 (2013)

29. Wang, L., Zhang, B.L.: Infinitely many solutions for Schrödinger–Kirchhoff type equations involving the fractional p-Laplacian and critical exponent. Electron. J. Differ. Equ. 2016, Article ID 339 (2016)

30. Wei, Y., Su, X.: Multiplicity of solutions for non-local elliptic equations driven by the fractional Laplacian. Calc. Var. Partial Differ. Equ. 52(1–2), 95–124 (2015)

31. Wu, T.F.: The Nehari manifold for a semilinear elliptic system involving sign-changing weight functions. Nonlinear Anal. 68, 1733–1745 (2008)

32. Wu, T.F.: On semilinear elliptic equations involving critical Sobolev exponents and sign-changing. Commun. Pure Appl. Anal. 7, 383–405 (2008)

33. Xiang, Q.M., Zhang, B.L., Ferrara, M.: Existence of solutions for Kirchhoff type problem involving the nonlocal fractional p-Laplacian. J. Math. Anal. Appl. 424, 1021–1041 (2015)

34. Yuan, Q., Chen, C.S., Yang, H.W.: Existence of positive solutions for a Schrödinger–Poisson system with bounded potential and weighted functions in $$\mathbb{R}^{3}$$. Bound. Value Probl. 2017, Article ID 151 (2017). https://doi.org/10.1186/s13661-017-0886-6

### Acknowledgements

The authors thank the anonymous referees for invaluable comments and suggestions which improved the presentation of this manuscript.

Not applicable.

## Funding

This work is supported by the National Natural Science Foundation of China (No. 11571092) and China Postdoctoral Science Foundation (No. 2017M611664).

## Author information

Authors

### Contributions

Each of the authors contributed to each part of this study equally, all authors read and approved the final manuscript.

### Corresponding author

Correspondence to Yunfeng Wei.

## Ethics declarations

### Competing interests

The authors declare that they have no competing interests.

Not applicable.

### Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions 