Classification of nonnegative solutions to static Schrödinger–Hartree–Maxwell system involving the fractional Laplacian

Abstract

This paper is mainly concerned with the following semi-linear system involving the fractional Laplacian:

$$ \textstyle\begin{cases} (-\Delta )^{\frac{\alpha }{2}}u(x)= (\frac{1}{ \vert \cdot \vert ^{\sigma }} \ast v^{p_{1}} )v^{p_{2}}(x), \quad x\in \mathbb{R}^{n}, \\ (-\Delta )^{\frac{\alpha }{2}}v(x)= (\frac{1}{ \vert \cdot \vert ^{\sigma }} \ast u^{q_{1}} )u^{q_{2}}(x), \quad x\in \mathbb{R}^{n}, \\ u(x)\geq 0,\quad\quad v(x)\geq 0, \quad x\in \mathbb{R}^{n}, \end{cases} $$

where $0<\alpha \leq 2$, $n\geq 2$, $0<\sigma <n$, and $0< p_{1}, q_{1}\leq \frac{2n-\sigma }{n-\alpha }$, $0< p_{2}, q_{2}\leq \frac{n+\alpha -\sigma }{n-\alpha }$. Applying a variant (for nonlocal nonlinearity) of the direct method of moving spheres for fractional Laplacians, which was developed by W. Chen, Y. Li, and R. Zhang (J. Funct. Anal. 272(10):4131–4157, 2017), we derive the explicit forms for positive solution $(u,v)$ in the critical case and nonexistence of positive solutions in the subcritical cases.

1 Introduction

In this paper, we consider the following semi-linear system involving the fractional Laplacian:

$$ \textstyle\begin{cases} (-\Delta )^{\frac{\alpha }{2}}u(x)= (\frac{1}{ \vert \cdot \vert ^{\sigma }} \ast v^{p_{1}} )v^{p_{2}}(x), \quad x\in \mathbb{R}^{n}, \\ (-\Delta )^{\frac{\alpha }{2}}v(x)= (\frac{1}{ \vert \cdot \vert ^{\sigma }} \ast u^{q_{1}} )u^{q_{2}}(x), \quad x\in \mathbb{R}^{n}, \\ u(x)\geq 0,\quad\quad v(x)\geq 0, \quad x\in \mathbb{R}^{n}, \end{cases} $$

(1.1)

where $0<\alpha \leq 2$, $n\geq 2$, $0<\sigma <n$, and $0< p_{1}, q_{1}\leq \frac{2n-\sigma }{n-\alpha }$, $0< p_{2}, q_{2}\leq \frac{n+\alpha -\sigma }{n-\alpha }$.

We assume $u,v\in C^{1,1}_{\mathrm{loc}}\cap \mathcal{L}_{\alpha }(\mathbb{R}^{n})$ if $0<\alpha <2$ and $u,v\in C^{2}(\mathbb{R}^{n})$ if $\alpha =2$, where

$$ \mathcal{L}_{\alpha } \bigl(\mathbb{R}^{n} \bigr):= \biggl\{ u:\mathbb{R}^{n} \rightarrow \mathbb{R} \Big\vert \int _{\mathbb{R}^{n}} \frac{ \vert u(y) \vert }{1+ \vert y \vert ^{n+\alpha }}\,dy< \infty \biggr\} . $$

(1.2)

The nonlocal fractional Laplacians $(-\Delta )^{\frac{\alpha }{2}}$ with $0<\alpha <2$ are defined by (see [9, 15, 19, 43, 47])

$$ (-\Delta )^{\frac{\alpha }{2}}u(x)=C_{\alpha ,n} \,P.V. \int _{ \mathbb{R}^{n}}\frac{u(x)-u(y)}{ \vert x-y \vert ^{n+\alpha }}\,dy:=C_{\alpha ,n} \lim _{\epsilon \rightarrow 0} \int _{ \vert y-x \vert \geq \epsilon } \frac{u(x)-u(y)}{ \vert x-y \vert ^{n+\alpha }}\,dy, $$

(1.3)

for functions $u,v\in C^{1,1}_{\mathrm{loc}}\cap \mathcal{L}_{\alpha }(\mathbb{R}^{n})$, where $C_{\alpha ,n}= (\int _{\mathbb{R}^{n}} \frac{1-\cos (2\pi \zeta _{1})}{ \vert \zeta \vert ^{n+\alpha }}\,d\zeta )^{-1}$ is the normalization constant. The fractional Laplacians $(-\Delta )^{\frac{\alpha }{2}}$ can also be defined equivalently (see [18]) by Caffarelli and Silvestre’s extension method (see [5]) for $u,v\in C^{1.1}_{\mathrm{loc}}\cap \mathcal{L}_{\alpha }(\mathbb{R}^{n})$.

The fractional Laplacian can be seen as the infinitesimal generator of a stable Lévy process and has several applications in probability, optimization, and finance (see [1, 3]). It has also been widely used to model diverse physical phenomena, such as anomalous diffusion and quasi-geostrophic flows, turbulence and water waves, molecular dynamics and relativistic quantum mechanics of stars. However, the nonlocal feature of the fractional Laplacians makes them difficult to study. In order to overcome this difficulty, Chen, Li, and Ou [17] developed the method of moving planes in integral forms. Subsequently, Caffarelli and Silvestre [5] introduced an extension method to overcome this difficulty, which reduced this nonlocal problem into a local one in higher dimensions. This extension method provides a powerful tool and leads to very active studies in equations involving the fractional Laplacians, and a series of fruitful results have been obtained (see [2, 20] and the references therein).

In [15], Chen, Li, and Li developed a direct method of moving planes for the fractional Laplacians (see also [22]). Instead of using the extension method of Caffarelli and Silvestre [5], they worked directly on the nonlocal operator to establish strong maximum principles for anti-symmetric functions and narrow region principles, and then obtained classification and Liouville type results for nonnegative solutions. The direct method of moving planes introduced in [15] has been applied to study more general nonlocal operators with general nonlinearities (see [14, 22]). The method of moving planes was initially invented by Alexanderoff in the early 1950s. Later, it was further developed by Serrin [43], Gidas, Ni, and Nirenberg [28, 30], Caffarelli, Gidas, and Spruck [4], Chen and Li [10], Li and Zhu [33], Lin [34], Chen, Li, and Ou [17], Chen, Li, and Li [15], and many others. For more literature works on the classification of solutions and Liouville type theorems for various PDE and IE problems via the methods of moving planes or spheres, please refer to [6, 8, 9, 13, 19, 21, 24, 26, 27, 29, 35–40, 45] and the references therein.

Chen, Li, and Zhang introduced in [19] another direct method i.e. the method of moving spheres on the fractional Laplacians, which is more convenient than the method of moving planes. The method of moving spheres was initially used by Padilla [42], Chen and Li [11], and Li and Zhu [33]. It can be applied to capture the explicit form of solutions directly rather than going through the procedure of deriving radial symmetry of solutions and then classifying radial solutions.

There are lots of literature works on the qualitative properties of solutions to Hartree and Choquard equations of fractional or higher order, please see e.g. Cao and Dai [6], Chen and Li [12], Dai, Fang et al. [21], Dai and Qin [26], Dai and Liu [23], Lei [31], Liu [36], Moroz and Schaftingen [41], Ma and Zhao [40], Xu and Lei [46], and the references therein. Liu proved in [36] the classification results for positive solutions to (1.1) with $\alpha =2$, $\sigma =4\in (0,n)$, $p_{1}=q_{1}=2$, $p_{2}=q_{2}=1$, $u=v$ by using the idea of considering the equivalent systems of integral equations instead, which was initially used by Ma and Zhao [40]. In [6], Cao and Dai considered the differential equations directly and classified all the positive $C^{4}$ solutions to the $\dot{H}^{2}$-critical bi-harmonic equation (1.1) with $\alpha =4$, $\sigma =8\in (0,n)$, $p_{1}=q_{1}=2$, $0< p_{2},q_{2}\leq 1$, $u=v$. They also derived Liouville theorem in the subcritical cases. One should observe that system (1.1) can be written as the integral system

$$ \textstyle\begin{cases} u(y)= {\int _{\mathbb{R}^{n}} \frac{R_{\alpha ,n}}{ \vert y-z \vert ^{n-\alpha }} ( {\int _{ \mathbb{R}^{n}}\frac{v^{p_{1}}(\xi )}{ \vert z-\xi \vert ^{\sigma }}\,d\xi } )v^{p_{2}}(z)\,dz}, \\ v(y)= {\int _{\mathbb{R}^{n}} \frac{R_{\alpha ,n}}{ \vert y-z \vert ^{n-\alpha }} ( {\int _{ \mathbb{R}^{n}}\frac{u^{q_{1}}(\zeta )}{ \vert z-\zeta \vert ^{\sigma }}\,d\zeta } )u^{q_{2}}(z)\,dz}, \end{cases} $$

(1.4)

where the Riesz potential’s constants $R_{\alpha ,n}:= \frac{\Gamma (\frac{n-\alpha }{2} )}{\pi ^{\frac{n}{2}}2^{\alpha }\Gamma (\frac{\alpha }{2})}$ (see [44]).

When $\sigma =2\alpha$, $\alpha \in (0,\frac{n}{2})$, $p_{1}=q_{1}=2$, $p_{2}=q_{2}=1$, $u=v$, Dai, Fang et al. [21] classified all the positive $H^{\frac{\alpha }{2}}(\mathbb{R}^{n})$ weak solutions to (1.1) by using the method of moving planes in integral forms for the equivalent integral equation system (1.4) due to Chen, Li, and Ou [16, 17], in which they established the equivalence between a PDE system and an integral system, and also classified all the $L^{\frac{2n}{n-\alpha }}(\mathbb{R}^{n})$ integrable solutions to the equivalent integral equation. For $0<\alpha <\min \{2,\frac{n}{2}\}$, Dai, Fang, and Qin [22] classified all the $C^{1,1}_{\mathrm{loc}}\cap \mathcal{L}_{\alpha }$ solutions to (1.1) with $\sigma =2\alpha$, $p_{1}=q_{1}=2$, $p_{2}=q_{2}=1$, $u=v$ by applying a variant (for nonlocal nonlinearity) of the direct method of moving planes for fractional Laplacians. The qualitative properties of solutions to general fractional order or higher order elliptic equations have also been extensively studied, for instance, see Chen, Fang, and Yang [9], Chen, Li, and Li [15], Chen, Li, and Ou [17], Caffarelli and Silvestre [5], Chang and Yang [8], Dai and Qin [26], Cao, Dai, and Qin [7], Dai, Liu, and Qin [25], Fang and Chen [27], Lin [34], Wei and Xu [45] and the references therein.

Our main theorem is the following complete classification theorem for PDE system (1.1).

Theorem 1.1

Let $n\geq 2$, $0<\sigma <n$, $0<\alpha \leq 2$, and $0< p_{1}\leq \frac{2n-\sigma }{n-\alpha }$, $0< p_{2}\leq \frac{n+\alpha -\sigma }{n-\alpha }$, $0< q_{1}\leq \frac{2n-\sigma }{n-\alpha }$, $0< q_{2}\leq \frac{n+\alpha -\sigma }{n-\alpha }$. Suppose that $(u,v)$ is a pair of nonnegative classical solutions of (1.1). If $p_{1}=\frac{2n-\sigma }{n-\sigma }$, $p_{2}=\frac{n+\alpha -\sigma }{n-\alpha }$, $q_{1}=\frac{2n-\sigma }{n-\alpha }$, and $q_{2}=\frac{n+\alpha -\sigma }{n-\alpha }$, then either $(u,v)\equiv (0,0)$ or u, v must assume the following form:

$$ u(x)=C_{1} \biggl(\frac{\mu }{1+\mu ^{2} \vert x-x_{0} \vert ^{2}} \biggr)^{ \frac{n-\alpha }{2}},\quad \quad v(x)=C_{2} \biggl( \frac{\mu }{1+\mu ^{2} \vert x-x_{0} \vert ^{2}} \biggr)^{\frac{n-\alpha }{2}} $$

for some $\mu >0$ and $x_{0}\in \mathbb{R}^{n}$, where the constants $C_{1}$, $C_{2}$ depend on n, α, σ. If $c_{i}\geq 0$, $\sum_{i=1}^{4} c_{i}>0$, $c_{1}(\frac{2n-\sigma }{n-\alpha }-p_{1})+c_{2}( \frac{n+\alpha -\sigma }{n-\alpha }-p_{2})+c_{3}( \frac{2n-\sigma }{n-\alpha }-q_{1})+c_{4}( \frac{n+\alpha -\sigma }{n-\alpha }-q_{2})>0$, then $(u,v)\equiv (0,0)$ in $\mathbb{R}^{n}$.

Remark 1.2

We apply a variant (for nonlocal nonlinearity) of the direct method of moving spheres for fractional Laplacians developed by Chen, Li, and Zhang [19] to prove Theorem 1.1, in which we extended the classification results by Dai and Liu [23], and Dai, Liu, and Qin [25] for a single equation. However, since the nonlinearities in our PDE system (1.1) are nonlocal, the difference between two nonlinearities will become much more complicated and subtle.

The rest of our paper is organized as follows. In Sect. 2, we carry out our proof of Theorem 1.1. In the following, we use C to denote a general positive constant that may depend on n, α, $p_{1}$, $p_{2}$, $q_{1}$, $q_{2}$, σ, u, and v, and whose value may differ from line to line.

2 Proof of Theorem 1.1

In this section, we use a direct method of moving spheres for nonlocal nonlinearity with the help of the narrow region principle to classify the nonnegative solutions of PDE system (1.1).

2.1 The direct method of moving spheres for nonlocal nonlinearity

Let $n\geq 2$, $0<\sigma <n$, $0<\alpha \leq 2$ with $0< p_{1}\leq \frac{2n-\sigma }{n-\alpha }$, $0< p_{2}\leq \frac{n+\alpha -\sigma }{n-\alpha }$, $0< q_{1}\leq \frac{2n-\sigma }{n-\alpha }$, and $0< q_{2}\leq \frac{n+\alpha -\sigma }{n-\alpha }$. Suppose that $(u,v)$ is a pair of nonnegative classical solutions of (1.1) which is not identically zero.

If there exists some point $x^{0}\in \mathbb{R}^{n}$ such that $u(x^{0})=0$, then we have

$$\begin{aligned} (-\Delta )^{\frac{\alpha }{2}}u \bigl(x^{0} \bigr)=C_{\alpha ,n}\,P.V. \int _{ \mathbb{R}^{n}}\frac{-u(y)}{ \vert x^{0}-y \vert ^{n+\alpha }}\,dy< 0. \end{aligned}$$

(2.1)

On the other hand, we can deduce from system (1.1) that

$$\begin{aligned} \int _{\mathbb{R}^{n}}\frac{v^{p_{2}}(\xi )}{ \vert x-\xi \vert }\,d\xi v^{p_{2}}(x) \geq 0, \end{aligned}$$

(2.2)

then we can derive a contradiction from (2.1), (2.2) for $u,v\geq 0$, $u,v\not \equiv 0$. Thus, one can deduce immediately that $u,v>0$ in $\mathbb{R}^{n}$ and $\int _{\mathbb{R}^{n}}\frac{u^{q_{1}}(x)}{ \vert x \vert ^{\sigma }}\,dx<+\infty$, $\int _{\mathbb{R}^{n}}\frac{v^{p_{1}}(x)}{ \vert x \vert ^{\sigma }}\,dx<+\infty$. From now onwards we shall assume that $(u,v)$ is a positive solution.

For any $x\in \mathbb{R}^{n}$ and $\lambda >0$, denote

$$\begin{aligned}& u_{x,\lambda }(y):= \biggl(\frac{\lambda }{ \vert y-x \vert } \biggr)^{n-\alpha }u \bigl(y^{x, \lambda } \bigr),\quad \forall y\in \mathbb{R}^{n} \setminus \{x\}, \\& v_{x,\lambda }(y):= \biggl(\frac{\lambda }{ \vert y-x \vert } \biggr)^{n-\alpha }v \bigl(y^{x, \lambda } \bigr),\quad \forall y\in \mathbb{R}^{n} \setminus \{x\}, \end{aligned}$$

where

$$ y^{x,\lambda }=\frac{\lambda ^{2}(y-x)}{ \vert y-x \vert ^{2}}+x. $$

Then, since $(u,v)$ is a pair of positive classical solutions of (1.1), one can verify that $u_{x,\lambda },v_{x,\lambda }\in \mathcal{L}_{\alpha }(\mathbb{R}^{n}) \cap C^{1,1}_{\mathrm{loc}}(\mathbb{R}^{n}\setminus \{x\})$ if $0<\alpha <2$ ($u_{x, \lambda }, v_{x,\lambda }\in C^{2}(\mathbb{R}^{n}\setminus \{x\})$ if $\alpha =2$) and satisfies the integrability property

$$\begin{aligned}& \int _{\mathbb{R}^{n}} \frac{u_{x,\lambda }^{q_{1}}(y)}{\lambda ^{\sigma }}\,dy= \int _{ \mathbb{R}^{n}}\frac{u^{q_{1}}(x)}{ \vert x \vert ^{\sigma }}\,dx< +\infty , \\& \int _{\mathbb{R}^{n}} \frac{v_{x,\lambda }^{p_{1}}(y)}{\lambda ^{\sigma }}\,dy= \int _{ \mathbb{R}^{n}}\frac{v^{p_{1}}(x)}{ \vert x \vert ^{\sigma }}\,dx< +\infty , \end{aligned}$$

and a similar equation as u, v for any $x\in \mathbb{R}^{n}$ and $\lambda >0$. In fact, without loss of generality, we may assume $x=0$ for simplicity and get, for $0<\alpha <2$ ($\alpha =2$ is similar),

$$\begin{aligned}& (-\Delta )^{\frac{\alpha }{2}}u_{0,\lambda }(y) \\& \quad = C_{\alpha ,n}\,P.V. \int _{\mathbb{R}^{n}} \frac{ ( (\frac{\lambda }{ \vert y \vert } )^{n-\alpha } - (\frac{\lambda }{ \vert z \vert } )^{n-\alpha } )u (\frac{\lambda ^{2}y}{ \vert y \vert ^{2}} )+ (\frac{\lambda }{ \vert z \vert } )^{n-\alpha } (u (\frac{\lambda ^{2}y}{ \vert y \vert ^{2}} )- u (\frac{\lambda ^{2}z}{ \vert z \vert ^{2}} ) )}{ \vert y-z \vert ^{n+\alpha }}\,dz \\& \quad = u \biggl(\frac{\lambda ^{2}y}{ \vert y \vert ^{2}} \biggr) (-\Delta )^{ \frac{\alpha }{2}} \biggl[ \biggl(\frac{\lambda }{ \vert y \vert } \biggr)^{n-\alpha } \biggr] +C_{\alpha ,n}\,P.V. \int _{\mathbb{R}^{n}} \frac{u (\frac{\lambda ^{2}y}{ \vert y \vert ^{2}} )-u(z)}{ \vert y-\frac{\lambda ^{2}z}{ \vert z \vert ^{2}} \vert ^{n+\alpha }}\frac{\lambda ^{n+\alpha }}{ \vert z \vert ^{n+\alpha }}\,dz \end{aligned}$$

(2.3)

$$\begin{aligned}& \quad = \frac{\lambda ^{n+\alpha }}{ \vert y \vert ^{n+\alpha }}(- \Delta )^{\frac{\alpha }{2}}u \biggl( \frac{\lambda ^{2}y}{ \vert y \vert ^{2}} \biggr) \\& \quad = \frac{\lambda ^{n+\alpha }}{ \vert y \vert ^{n+\alpha }} \int _{ \mathbb{R}^{n}} \frac{v^{p_{1}}(z)}{ \vert \frac{\lambda ^{2}y}{ \vert y \vert ^{2}}-z \vert ^{\sigma }}\,dz \cdot v^{p_{2}} \biggl( \frac{\lambda ^{2}y}{ \vert y \vert ^{2}} \biggr) \\& \quad = \frac{\lambda ^{n+\alpha }}{ \vert y \vert ^{n+\alpha }} \int _{ \mathbb{R}^{n}} \frac{\lambda ^{2n} \vert z \vert ^{-2n}}{ \vert \frac{\lambda ^{2}y}{ \vert y \vert ^{2}}-\frac{\lambda ^{2}z}{ \vert z \vert ^{2}} \vert ^{\sigma }} v^{p_{1}} \biggl( \frac{\lambda ^{2}z}{ \vert z \vert ^{2}} \biggr)\,dz\cdot v^{p_{2}} \biggl( \frac{\lambda ^{2}y}{ \vert y \vert ^{2}} \biggr) \\& \quad = \int _{\mathbb{R}^{n}} \frac{v^{p_{1}}_{0,\lambda }(z)}{ \vert y-z \vert ^{\sigma }} \biggl( \frac{\lambda }{ \vert z \vert } \biggr)^{\tau _{1}}\,dz \biggl(\frac{\lambda }{ \vert y \vert } \biggr)^{\tau _{2}}v^{p_{2}}_{0,\lambda }(y), \\& (-\Delta )^{\frac{\alpha }{2}}v_{0,\lambda }(y) \\& \quad = C_{\alpha ,n} \,P.V. \int _{\mathbb{R}^{n}} \frac{ ( (\frac{\lambda }{ \vert y \vert } )^{n-\alpha } - (\frac{\lambda }{ \vert z \vert } )^{n-\alpha } )v (\frac{\lambda ^{2}y}{ \vert y \vert ^{2}} )+ (\frac{\lambda }{ \vert z \vert } )^{n-\alpha } (v (\frac{\lambda ^{2}y}{ \vert y \vert ^{2}} )- v (\frac{\lambda ^{2}z}{ \vert z \vert ^{2}} ) )}{ \vert y-z \vert ^{n+\alpha }}\,dz \\& \quad = v \biggl(\frac{\lambda ^{2}y}{ \vert y \vert ^{2}} \biggr) (-\Delta )^{ \frac{\alpha }{2}} \biggl[ \biggl(\frac{\lambda }{ \vert y \vert } \biggr)^{n-\alpha } \biggr] +C_{\alpha ,n}\,P.V. \int _{\mathbb{R}^{n}} \frac{v (\frac{\lambda ^{2}y}{ \vert y \vert ^{2}} )-v(z)}{ \vert y-\frac{\lambda ^{2}z}{ \vert z \vert ^{2}} \vert ^{n+\alpha }}\frac{\lambda ^{n+\alpha }}{ \vert z \vert ^{n+\alpha }}\,dz \\& \quad = \frac{\lambda ^{n+\alpha }}{ \vert y \vert ^{n+\alpha }}(-\Delta )^{ \frac{\alpha }{2}}v \biggl( \frac{\lambda ^{2}y}{ \vert y \vert ^{2}} \biggr) \\& \quad = \frac{\lambda ^{n+\alpha }}{ \vert y \vert ^{n+\alpha }} \int _{\mathbb{R}^{n}} \frac{u^{q_{1}}(z)}{ \vert \frac{\lambda ^{2}y}{ \vert y \vert ^{2}}-z \vert ^{\sigma }}\,dz \cdot u^{q_{2}} \biggl( \frac{\lambda ^{2}y}{ \vert y \vert ^{2}} \biggr) \\& \quad = \frac{\lambda ^{n+\alpha }}{ \vert y \vert ^{n+\alpha }} \int _{\mathbb{R}^{n}} \frac{\lambda ^{2n} \vert z \vert ^{-2n}}{ \vert \frac{\lambda ^{2}y}{ \vert y \vert ^{2}}-\frac{\lambda ^{2}z}{ \vert z \vert ^{2}} \vert ^{\sigma }} u^{q_{1}} \biggl( \frac{\lambda ^{2}z}{ \vert z \vert ^{2}} \biggr)\,dz\cdot u^{q_{2}} \biggl( \frac{\lambda ^{2}y}{ \vert y \vert ^{2}} \biggr) \\& \quad = \int _{\mathbb{R}^{n}} \frac{u^{q_{1}}_{0,\lambda }(z)}{ \vert y-z \vert ^{\sigma }} \biggl( \frac{\lambda }{ \vert z \vert } \biggr)^{\tau _{3}}\,dz \biggl(\frac{\lambda }{ \vert y \vert } \biggr)^{\tau _{4}}u^{q_{2}}_{0,\lambda }(y). \end{aligned}$$

(2.4)

This means that the conformal transforms $u_{x,\lambda },v_{x,\lambda }\in \mathcal{L}_{\alpha }(\mathbb{R}^{n}) \cap C^{1,1}_{\mathrm{loc}}(\mathbb{R}^{n}\setminus \{x\})$ if $0<\alpha <2$ ($u_{x, \lambda }, v_{x,\lambda }\in C^{2}(\mathbb{R}^{n}\setminus \{x\})$ if $\alpha =2$) satisfy

$$ \textstyle\begin{cases} (-\Delta )^{\frac{\alpha }{2}}u_{x,\lambda }(y)= {\int _{ \mathbb{R}^{n}}\frac{v^{p_{1}}_{x,\lambda }(z)}{ \vert y-z \vert ^{\sigma }} ( \frac{\lambda }{ \vert z-x \vert } )^{\tau _{1}}\,dz} ( \frac{\lambda }{ \vert y-x \vert } )^{\tau _{2}}v^{p_{2}}_{x,\lambda }(y), \\ (-\Delta )^{\frac{\alpha }{2}}v_{x,\lambda }(y)= {\int _{ \mathbb{R}^{n}}\frac{u^{q_{1}}_{x,\lambda }(z)}{ \vert y-z \vert ^{\sigma }} ( \frac{\lambda }{ \vert z-x \vert } )^{\tau _{3}}\,dz} ( \frac{\lambda }{ \vert y-x \vert } )^{\tau _{4}}u^{q_{2}}_{x,\lambda }(y), \end{cases} $$

(2.5)

for every $y\in \mathbb{R}^{n}\setminus \{x\}$, where $\tau _{1}:=2n-\sigma -p_{1}(n-\alpha )\geq 0$, $\tau _{2}:=n+\alpha -\sigma -p_{2}(n-\alpha )\geq 0$, $\tau _{3}:=2n-\sigma -q_{1}(n-\alpha )\geq 0$ and $\tau _{4}:=n+\alpha -\sigma -q_{2}(n-\alpha )\geq 0$. For any $\lambda >0$, we define

$$\begin{aligned}& B_{\lambda }(x):= \bigl\{ y\in \mathbb{R}^{n} \vert \vert y-x \vert < \lambda \bigr\} , \\& P(y):= \biggl(\frac{1}{ \vert \cdot \vert ^{\sigma }}\ast v^{p_{1}} \biggr) (y), \qquad \widetilde{P}_{x,\lambda }(y):= \int _{B_{\lambda }(x)} \frac{v^{p_{1}-1}(z)}{ \vert y-z \vert ^{\sigma }}\,dz, \\& Q(y):= \biggl(\frac{1}{ \vert \cdot \vert ^{\sigma }}\ast u^{q_{1}} \biggr) (y), \qquad \widetilde{Q}_{x,\lambda }(y):= \int _{B_{\lambda }(x)} \frac{u^{q_{1}-1}(z)}{ \vert y-z \vert ^{\sigma }}\,dz. \end{aligned}$$

Define $U_{x,\lambda }(y)=u_{x,\lambda }(y)-u(y)$, $V_{x,\lambda }(y)=v_{x, \lambda }(y)-v(y)$ for any $y\in B_{\lambda }(x)\setminus \{x\}$. By the definition of $u_{x,\lambda }$, $v_{x,\lambda }$ and $U_{x,\lambda }$, $V_{x,\lambda }$, we have

$$\begin{aligned}& \begin{aligned}[t] U_{x,\lambda }(y)&=u_{x,\lambda }(y)-u(y)= \biggl( \frac{\lambda }{ \vert y-x \vert } \biggr)^{n-\alpha }u \bigl(y^{x,\lambda } \bigr)-u(y) \\ &= \biggl(\frac{\lambda }{ \vert y-x \vert } \biggr)^{n-\alpha } \biggl(u \bigl(y^{x, \lambda } \bigr)- \biggl(\frac{\lambda }{ \vert y^{x,\lambda }-x \vert } \biggr)^{n-\alpha }u \bigl( \bigl(y^{x,\lambda } \bigr)^{x,\lambda } \bigr) \biggr) \\ &=- \biggl(\frac{\lambda }{ \vert y-x \vert } \biggr)^{n-\alpha }U_{x,\lambda } \bigl(y^{x, \lambda } \bigr)=- (U_{x,\lambda } )_{x,\lambda }(y), \end{aligned} \end{aligned}$$

(2.6)

$$\begin{aligned}& \begin{aligned}[t] V_{x,\lambda }(y)&=v_{x,\lambda }(y)-v(y)= \biggl( \frac{\lambda }{ \vert y-x \vert } \biggr)^{n-\alpha }v \bigl(y^{x,\lambda } \bigr)-v(y) \\ &= \biggl(\frac{\lambda }{ \vert y-x \vert } \biggr)^{n-\alpha } \biggl(v \bigl(y^{x, \lambda } \bigr)- \biggl(\frac{\lambda }{ \vert y^{x,\lambda }-x \vert } \biggr)^{n-\alpha }v \bigl( \bigl(y^{x,\lambda } \bigr)^{x,\lambda } \bigr) \biggr) \\ &=- \biggl(\frac{\lambda }{ \vert y-x \vert } \biggr)^{n-\alpha }V_{x,\lambda } \bigl(y^{x, \lambda } \bigr)=- (V_{x,\lambda } )_{x,\lambda }(y) \end{aligned} \end{aligned}$$

(2.7)

for every $y\in B_{\lambda }(x)\setminus \{x\}$.

We will first show that there exists $\epsilon _{0}>0$ (depending on x) sufficiently small such that, for any $0<\lambda \leq \epsilon _{0}$, it holds that $U_{x,\lambda }(y)\geq 0$, $V_{x,\lambda }(y)\geq 0$ for every $y\in B_{\lambda }(x)\setminus \{x\}$.

We first need to show that the nonnegative solution $(u,v)$ to PDE system (1.1) also satisfies the equivalent integral system (1.4).

Lemma 2.1

Assume that $(u,v)$ is a pair of nonnegative solutions to PDE system (1.1), then $(u,v)$ also satisfies the equivalent integral system (1.4), and vice versa.

Proof

Recall that $G(y,z)=\frac{R_{n,\alpha }}{ \vert y-z \vert ^{n-\alpha }}$ is the fundamental solution for $(-\Delta )^{\frac{\alpha }{2}}$ on $\mathbb{R}^{n}$. If $(u,v)$ is a pair of positive solutions of (1.4), then

$$\begin{aligned}& \begin{aligned} (-\Delta )^{\frac{\alpha }{2}}u(y)&= \int _{\mathbb{R}^{n}}(-\Delta )^{ \frac{\alpha }{2}}\frac{R_{n,\alpha }}{ \vert y-z \vert ^{n-\alpha }} \biggl( \frac{1}{ \vert \cdot \vert ^{\sigma }}\ast v^{p_{1}} \biggr) (z)v^{p_{2}}(z)\,dz \\ &= \int _{\mathbb{R}^{n}}\delta _{y}(z) \biggl( \frac{1}{ \vert \cdot \vert ^{\sigma }} \ast v^{p_{1}} \biggr) (z)v^{p_{2}}(z)\,dz \\ &= \biggl(\frac{1}{ \vert \cdot \vert ^{\sigma }}\ast v^{p_{1}} \biggr) (y)v^{p_{2}}(y), \end{aligned} \\& \begin{aligned} (-\Delta )^{\frac{\alpha }{2}}v(y)&= \int _{\mathbb{R}^{n}}(-\Delta )^{ \frac{\alpha }{2}}\frac{R_{n,\alpha }}{ \vert y-z \vert ^{n-\alpha }} \biggl( \frac{1}{ \vert \cdot \vert ^{\sigma }}\ast u^{q_{1}} \biggr) (z)u^{q_{2}}(z)\,dz \\ &= \int _{\mathbb{R}^{n}}\delta _{y}(z) \biggl( \frac{1}{ \vert \cdot \vert ^{\sigma }} \ast u^{q_{1}} \biggr) (z)u^{q_{2}}(z)\,dz \\ &= \biggl(\frac{1}{ \vert \cdot \vert ^{\sigma }}\ast u^{q_{1}} \biggr) (y)u^{q_{2}}(y), \end{aligned} \end{aligned}$$

this is, $(u,v)$ satisfies system (1.1).

Conversely, assume that $(u,v)$ is a pair of positive solutions of (1.1). For any $R>0$, let

$$\begin{aligned}& u_{1,R}(y)= \int _{B_{R}}G^{\alpha }_{R}(y,z) \biggl( \frac{1}{ \vert \cdot \vert ^{\sigma }}\ast v^{p_{1}} \biggr) (z)v^{p_{2}}(z)\,dz, \\& v_{1,R}(y)= \int _{B_{R}}G^{\alpha }_{R}(y,z) \biggl( \frac{1}{ \vert \cdot \vert ^{\sigma }}\ast u^{q_{1}} \biggr) (z)u^{q_{2}}(z)\,dz, \end{aligned}$$

where $G^{\alpha }_{R}$ is Green’s function for $(-\Delta )^{\frac{\alpha }{2}}$ on $B_{R}(0)$ which is given by

$$ G^{\alpha }_{R}(y,z)= \textstyle\begin{cases} \frac{C_{n,\alpha }}{ \vert y-z \vert ^{n-\alpha }} {\int ^{ \frac{t_{R}}{s_{R}}}_{0} \frac{b^{\frac{\alpha }{2}-1}}{(1+b)^{\frac{n}{2}}}\,db}, & \text{for all } y,z\in B_{R}(0), \\ 0, &\text{if } y \text{ or } z\in R^{n}\setminus {B_{R}(0)}, \end{cases} $$

with $s_{R}=\frac{ \vert y-z \vert ^{2}}{R^{2}}$ and $t_{R}= (1-\frac{ \vert y \vert ^{2}}{R^{2}} ) (1- \frac{ \vert z \vert ^{2}}{R^{2}} )$.

Using the properties of Green’s function, we can deduce

$$\begin{aligned}& \textstyle\begin{cases} (-\Delta )^{\frac{\alpha }{2}}u_{1,R}(y)= ( \frac{1}{ \vert \cdot \vert ^{\sigma }}\ast v^{p_{1}} )(y)v^{p_{2}}(y), \quad y\in B_{R}(0), \\ u_{1,R}(y)=0, \quad y\in \mathbb{R}^{n}\setminus {B_{R}(0)}{ ,} \end{cases}\displaystyle \end{aligned}$$

(2.8)

$$\begin{aligned}& \textstyle\begin{cases} (-\Delta )^{\frac{\alpha }{2}}v_{1,R}(y)= ( \frac{1}{ \vert \cdot \vert ^{\sigma }}\ast u^{q_{1}} )(y)u^{q_{2}}(y), \quad y\in B_{R}(0), \\ v_{1,R}(y)=0, \quad y\in \mathbb{R}^{n}\setminus {B_{R}(0)}. \end{cases}\displaystyle \end{aligned}$$

(2.9)

Let $U_{R}=u-u_{1,R}$, $V_{R}=v-v_{1,R}$, by (1.1), (2.8), and (2.9), we have

$$\begin{aligned}& \textstyle\begin{cases} (-\Delta )^{\frac{\alpha }{2}}U_{R}(y)=0, \quad y\in B_{R}(0), \\ U_{R}(y)\geq 0, \quad y\in \mathbb{R}^{n}\setminus {B_{R}(0)}{ ,} \end{cases}\displaystyle \\& \textstyle\begin{cases} (-\Delta )^{\frac{\alpha }{2}}V_{R}(y)=0, \quad y\in B_{R}(0), \\ V_{R}(y)\geq 0, \quad y\in \mathbb{R}^{n}\setminus {B_{R}(0)}{ ,} \end{cases}\displaystyle \end{aligned}$$

for any $R>0$, it follows from the maximum principle that

$$ U_{R}(y)=u(y)-u_{1,R}(y)\geq 0, \quad\quad V_{R}(y)=v(y)-v_{1,R}(y) \geq 0 \quad \text{for all } y\in \mathbb{R}^{n}. $$

Now, for each fixed $y\in \mathbb{R}^{n}$, letting $R\rightarrow \infty$, we have

$$\begin{aligned}& u(y)\geq u_{1}(y):= \int _{\mathbb{R}^{n}} \frac{R_{n,\alpha }}{ \vert y-z \vert ^{n-\alpha }} \biggl( \frac{1}{ \vert \cdot \vert ^{\sigma }}\ast v^{p_{1}} \biggr) (z)v^{p_{2}}(z)\,dz, \\& v(y)\geq v_{1}(y):= \int _{\mathbb{R}^{n}} \frac{R_{n,\alpha }}{ \vert y-z \vert ^{n-\alpha }} \biggl( \frac{1}{ \vert \cdot \vert ^{\sigma }}\ast u^{q_{1}} \biggr) (z)u^{q_{2}}(z)\,dz. \end{aligned}$$

On the other hand, $(u_{1}, v_{1})$ is a pair of solutions of the following system:

$$ \textstyle\begin{cases} (-\Delta )^{\frac{\alpha }{2}}u_{1}(y)= ( \frac{1}{ \vert \cdot \vert ^{\sigma }}\ast v^{p_{1}} )(y)v^{p_{2}}(y), \quad y\in \mathbb{R}^{n}, \\ (-\Delta )^{\frac{\alpha }{2}}v_{1}(y)= ( \frac{1}{ \vert \cdot \vert ^{\sigma }}\ast u^{q_{1}} )(y)u^{q_{2}}(y), \quad y\in \mathbb{R}^{n}, \end{cases} $$

define $U(y)=u(y)-u_{1}(y)$, $V(y)=v(y)-v_{1}(y)$, then

$$\begin{aligned}& \textstyle\begin{cases} (-\Delta )^{\frac{\alpha }{2}}U(y)=0, \quad y\in \mathbb{R}^{n}, \\ U(y)\geq 0, \quad y\in \mathbb{R}^{n}{,} \end{cases}\displaystyle \\& \textstyle\begin{cases} (-\Delta )^{\frac{\alpha }{2}}V(y)=0, \quad y\in \mathbb{R}^{n}, \\ V(y)\geq 0, \quad y\in \mathbb{R}^{n}{.} \end{cases}\displaystyle \end{aligned}$$

By the Liouville theorem, we deduce $U(y)=u(y)-u_{1}(y)\equiv C_{3}\geq 0$, $V(y)=v(y)-v_{1}(y)\equiv C_{4} \geq 0$.

Thus, we have proved that

$$\begin{aligned}& u(y)= \int _{\mathbb{R}^{n}}\frac{R_{n,\alpha }}{ \vert y-z \vert ^{n-\alpha }} \biggl( \frac{1}{ \vert \cdot \vert ^{\sigma }}\ast v^{p_{1}} \biggr) (z)v^{p_{2}}(z)\,dz+C_{3} \geq C_{3}, \\& v(y)= \int _{\mathbb{R}^{n}}\frac{R_{n,\alpha }}{ \vert y-z \vert ^{n-\alpha }} \biggl( \frac{1}{ \vert \cdot \vert ^{\sigma }}\ast u^{q_{1}} \biggr) (z)u^{q_{2}}(z)\,dz+C_{4} \geq C_{4}. \end{aligned}$$

Then we have

$$\begin{aligned}& \begin{aligned} \infty &>u(0)\geq u_{1}(0)= \int _{\mathbb{R}^{n}} \frac{R_{n,\alpha }}{ \vert z \vert ^{n-\alpha }} \biggl( \int _{\mathbb{R}^{n}} \frac{v^{p_{1}}(\xi )}{ \vert z-\xi \vert ^{\sigma }}\,d\xi \biggr)v^{p_{2}}(z) \,dz \\ &\geq C^{p_{1}+p_{2}}_{4} \int _{\mathbb{R}^{n}} \frac{R_{n,\alpha }}{ \vert z \vert ^{n-\alpha }} \int _{\mathbb{R}^{n}} \frac{1}{ \vert z-\xi \vert ^{\sigma }}\,d\xi \,dz, \end{aligned} \\& \begin{aligned} \infty &>v(0)\geq v_{1}(0)= \int _{\mathbb{R}^{n}} \frac{R_{n,\alpha }}{ \vert z \vert ^{n-\alpha }} \biggl( \int _{\mathbb{R}^{n}} \frac{u^{q_{1}}(\zeta )}{ \vert z-\zeta \vert ^{\sigma }}\,d\zeta \biggr)u^{q_{2}}(z) \,dz \\ &\geq C^{q_{1}+q_{2}}_{3} \int _{\mathbb{R}^{n}} \frac{R_{n,\alpha }}{ \vert z \vert ^{n-\alpha }} \int _{\mathbb{R}^{n}} \frac{1}{ \vert z-\zeta \vert ^{\sigma }}\,d\zeta \,dz, \end{aligned} \end{aligned}$$

from which we can infer immediately that $C_{3}=0$, $C_{4}=0$, therefore, we arrive at

$$ \textstyle\begin{cases} u(y)= {\int _{\mathbb{R}^{n}} \frac{R_{n,\alpha }}{ \vert y-z \vert ^{n-\alpha }} ( \frac{1}{ \vert \cdot \vert ^{\sigma }}\ast v^{p_{1}} )(z)v^{p_{2}}(z)\,dz}, \\ v(y)= {\int _{\mathbb{R}^{n}} \frac{R_{n,\alpha }}{ \vert y-z \vert ^{n-\alpha }} ( \frac{1}{ \vert \cdot \vert ^{\sigma }}\ast u^{q_{1}} )(z)u^{q_{2}}(z)\,dz}. \end{cases} $$

Therefore, $(u,v)$ satisfies integral system (1.4). □

Based on Lemma 2.1, we can prove that $U_{x,\lambda }$, $V_{x,\lambda }$ have a strictly positive lower bound in a small neighborhood of x.

Lemma 2.2

For every fixed $x\in \mathbb{R}^{n}$, there exists $\eta _{0}>0$ (depending on x) sufficiently small such that, if $0<\lambda \leq \eta _{0}$, then

$$ U_{x,\lambda }(y)\geq 1,\quad\quad V_{x,\lambda }(y)\geq 1,\quad y\in \overline{B_{\lambda ^{2}}(x)}\setminus \{x\}. $$

Proof

Using a similar argument as that in [19], one can denote

$$\begin{aligned}& f \bigl(v(y) \bigr):=v^{p_{2}}(y) \int _{\mathbb{R}^{n}} \frac{v^{p_{1}}(\xi )}{ \vert y-\xi \vert ^{\sigma }}\,d\xi , \\& g \bigl(u(y) \bigr):=u^{q_{2}}(y) \int _{\mathbb{R}^{n}} \frac{u^{q_{1}}(\zeta )}{ \vert y-\zeta \vert ^{\sigma }}\,d\zeta . \end{aligned}$$

For any $\vert y \vert \geq 1$, since $u,v>0$ also satisfy integral system (1.4), we can deduce that

$$\begin{aligned}& \begin{aligned} u(y)&=R_{\alpha ,n} \int _{\mathbb{R}^{n}} \frac{f(v(z))}{ \vert y-z \vert ^{n-\alpha }}\,dz \\ &\geq R_{\alpha ,n} \int _{B_{\frac{1}{2}}(0)} \frac{f(v(z))}{ \vert y-z \vert ^{n-\alpha }}\,dz \\ &\geq \frac{b_{1}}{ \vert y \vert ^{n-\alpha }} \int _{B_{\frac{1}{2}}(0)}f \bigl(v(z) \bigr)\,dz \\ &\geq \frac{b_{1}}{ \vert y \vert ^{n-\alpha }}, \end{aligned} \\& \begin{aligned} v(y)&=R_{\alpha ,n} \int _{\mathbb{R}^{n}} \frac{g(u(z))}{ \vert y-z \vert ^{n-\alpha }}\,dz \\ &\geq R_{\alpha ,n} \int _{B_{\frac{1}{2}}(0)} \frac{g(u(z))}{ \vert y-z \vert ^{n-\alpha }}\,dz \\ &\geq \frac{b_{2}}{ \vert y \vert ^{n-\alpha }} \int _{B_{\frac{1}{2}}(0)}g \bigl(u(z) \bigr)\,dz \\ &\geq \frac{b_{2}}{ \vert y \vert ^{n-\alpha }}. \end{aligned} \end{aligned}$$

It follows immediately that

$$\begin{aligned}& u_{x,\lambda }(y)= \biggl(\frac{\lambda }{ \vert y-x \vert } \biggr)^{n-\alpha }u \bigl(y^{x, \lambda } \bigr)\geq \biggl(\frac{\lambda }{ \vert y-x \vert } \biggr)^{n-\alpha } \frac{b_{1}}{ \vert y^{x,\lambda } \vert ^{n-\alpha }} = \frac{b_{1}}{\lambda ^{n-\alpha }}, \\& v_{x,\lambda }(y)= \biggl(\frac{\lambda }{ \vert y-x \vert } \biggr)^{n-\alpha }v \bigl(y^{x, \lambda } \bigr)\geq \biggl(\frac{\lambda }{ \vert y-x \vert } \biggr)^{n-\alpha } \frac{b_{2}}{ \vert y^{x,\lambda } \vert ^{n-\alpha }} = \frac{b_{2}}{\lambda ^{n-\alpha }} \end{aligned}$$

for all $y\in \overline{B_{\lambda ^{2}}(x)}\setminus \{x\}$. Therefore, we have if $0<\lambda \leq \eta _{0}$ for some $\eta _{0}(x)>0$ small enough, then

$$\begin{aligned}& U_{x,\lambda }(y)=u_{x,\lambda }(y)-u(y)\geq \frac{b_{1}}{\lambda ^{n-\alpha }}-\max _{ \vert y-x \vert \leq \lambda ^{2}}u(y) \geq 1, \\& V_{x,\lambda }(y)=v_{x,\lambda }(y)-v(y)\geq \frac{b_{2}}{\lambda ^{n-\alpha }}-\max _{ \vert y-x \vert \leq \lambda ^{2}}v(y) \geq 1 \end{aligned}$$

for any $y\in \overline{B_{\lambda ^{2}}(x)}\setminus \{x\}$.

This completes the proof of Lemma 2.2. □

For every fixed $x\in \mathbb{R}^{n}$, define

$$ B_{\lambda }^{-}= \bigl\{ y\in B_{\lambda }(x)\setminus \{x\} \vert U_{x,\lambda }(y)< 0, V_{x,\lambda }(y)< 0 \bigr\} . $$

Now we need the following theorem, which is a variant (for nonlocal nonlinearity) of the narrow region principle (Theorem 2.2 in [19]).

Theorem 2.3

(Narrow region principle)

Assume that $x\in \mathbb{R}^{n}$ is arbitrarily fixed. Let Ω be a narrow region in $B_{\lambda }(x)\setminus \{x\}$ with small thickness $0< l<\lambda$ such that $\Omega \subseteq A_{\lambda ,l}(x):=\{y\in \mathbb{R}^{n}\vert \lambda -l< \vert y-x \vert < \lambda \}$. Suppose $U_{x,\lambda },V_{x,\lambda }\in \mathcal{L}_{\alpha }(\mathbb{R}^{n}) \cap C^{1,1}_{\mathrm{loc}}(\Omega )$ if $0<\alpha <2$ ($U_{x,\lambda }, V_{x, \lambda }\in C^{2}(\Omega )$ if $\alpha =2$) and satisfies

$$\begin{aligned} \textstyle\begin{cases} (-\Delta )^{\frac{\alpha }{2}}U_{x,\lambda }(y)-\mathcal{L}_{1}(y)V_{x, \lambda }(y)-p_{1} ( {\int _{B^{-}_{\lambda }} \frac{v^{p_{1}-1}(z)V_{x,\lambda }(z)}{ \vert y-z \vert ^{\sigma }}\,dz} )v^{p_{2}}(y) \geq 0 \quad \textit{in } \Omega \cap B^{-}_{\lambda }, \\ (-\Delta )^{\frac{\alpha }{2}}V_{x,\lambda }(y)-\mathcal{L}_{2}(y)U_{x, \lambda }(y)-q_{1} ( {\int _{B^{-}_{\lambda }} \frac{u^{q_{1}-1}(z)U_{x,\lambda }(z)}{ \vert y-z \vert ^{\sigma }}\,dz} )u^{q_{2}}(y) \geq 0 \quad \textit{in } \Omega \cap B^{-}_{\lambda }, \\ \textit{negative minimum of } U_{x,\lambda }, V_{x,\lambda } \textit{ is attained in the interior of } B_{\lambda }(x)\setminus \{x \} \\ \quad \textit{if } B^{-}_{\lambda }\neq \emptyset , \\ \textit{negative minimum of } U_{x,\lambda }, V_{x,\lambda } \textit{ cannot be attained in } (B_{\lambda }(x)\setminus \{x\}) \setminus \Omega , \end{cases}\displaystyle \end{aligned}$$

(2.10)

where $\mathcal{L}_{1}(y):=p_{2}v_{x,\lambda }^{p_{2}-1}(y)P(y)$, $\mathcal{L}_{2}(y):=q_{2}u_{x,\lambda }^{q_{2}-1}(y)Q(y)$. Then we have:

• (i) There exists a sufficiently small constant $\gamma _{0}(x)>0$ such that, for all $0<\lambda \leq \gamma _{0}$,

  $$ U_{x,\lambda }(y)\geq 0, V_{x,\lambda }(y)\geq 0, \quad \forall y\in \Omega ; $$

  (2.11)

• (ii) There exists sufficiently small $l_{0}(x,\lambda )>0$ depending on λ continuously such that, for all $0< l\leq l_{0}$,

  $$ U_{x,\lambda }(y)\geq 0, V_{x,\lambda }(y)\geq 0, \quad \forall y\in \Omega . $$

  (2.12)

Proof

Without loss of generality, we may assume $x=0$ here for simplicity. Suppose on the contrary that (2.11) and (2.12) do not hold, we will obtain a contradiction for any $0<\lambda \leq \gamma _{0}$ with constant $\gamma _{0}$ small enough and any $0< l\leq l_{0}(\lambda )$ with $l_{0}(\lambda )$ sufficiently small respectively. By (2.10) and our hypothesis, there exists $\tilde{y}\in (\Omega \cap B^{-}_{\lambda })\subseteq A_{\lambda ,l}(0):= \{y\in \mathbb{R}^{n}\vert \lambda -l< \vert y \vert <\lambda \}$ such that

$$ U_{0,\lambda }(\tilde{y})=\min_{B_{\lambda }(0)\setminus \{0\}}U_{0, \lambda }(y)< 0. $$

(2.13)

We first consider the cases $0<\alpha <2$. Let $\tilde{U}_{0,\lambda }(y)=U_{0,\lambda }(y)-U_{0,\lambda }(\tilde{y})$, then $\tilde{U}_{0,\lambda }(\tilde{y})=0$ and

$$ (-\Delta )^{\alpha /2}\tilde{U}_{0,\lambda }(y)=(-\Delta )^{\alpha /2}U_{0, \lambda }(y). $$

By the anti-symmetry property $U_{x,\lambda }(y)=-(U_{x,\lambda })_{x,\lambda }(y)$, it holds

$$ \begin{aligned} \biggl(\frac{\lambda }{ \vert y \vert } \biggr)^{n-\alpha } \tilde{U}_{0, \lambda } \bigl(y^{0,\lambda } \bigr)&= \biggl(\frac{\lambda }{ \vert y \vert } \biggr)^{n-\alpha }U_{0, \lambda } \bigl(y^{0,\lambda } \bigr)- \biggl( \frac{\lambda }{ \vert y \vert } \biggr)^{n-\alpha }U_{0, \lambda }(\tilde{y}) \\ &=-U_{0,\lambda }(y)+U_{0,\lambda }(\tilde{y})- \biggl(1+ \biggl( \frac{\lambda }{ \vert y \vert } \biggr)^{n-\alpha } \biggr)U_{0,\lambda }(\tilde{y}) \\ &=-\tilde{U}_{0,\lambda }(y)- \biggl(1+ \biggl(\frac{\lambda }{ \vert y \vert } \biggr)^{n-\alpha } \biggr)U_{0,\lambda }(\tilde{y}). \end{aligned} $$

As a consequence, it follows that

$$ \begin{aligned} (-\Delta )^{\alpha /2}\tilde{U}_{0,\lambda }( \tilde{y})&=C_{n, \alpha } \,P.V. \int _{\mathbb{R}^{n}} \frac{\tilde{U}_{0,\lambda }(\tilde{y})-\tilde{U}_{0,\lambda }(z)}{ \vert \tilde{y}-z \vert ^{n+\alpha }}\,dz \\ &=C_{n,\alpha } \,P.V. \int _{B_{\lambda }(0)} \frac{-\tilde{U}_{0,\lambda }(z)}{ \vert \tilde{y}-z \vert ^{n+\alpha }}\,dz+ \int _{ \mathbb{R}^{n}\setminus {B_{\lambda }(0)}} \frac{-\tilde{U}_{0,\lambda }(z)}{ \vert \tilde{y}-z \vert ^{n+\alpha }}\,dz \\ &=C_{n,\alpha } \,P.V. \biggl( \int _{B_{\lambda }(0)} \frac{-\tilde{U}_{0,\lambda }(z)}{ \vert \tilde{y}-z \vert ^{n+\alpha }}\,dz+ \int _{ \mathbb{R}^{n}\setminus {B_{\lambda }(0)}} \frac{ (\frac{\lambda }{ \vert z \vert } )^{n-\alpha }\tilde{U}_{0,\lambda }(z^{0,\lambda })}{ \vert \tilde{y}-z \vert ^{n+\alpha }}\,dz \\ &\quad {} + \int _{\mathbb{R}^{n}\setminus {B_{\lambda }(0)}} \frac{ (1+ (\frac{\lambda }{ \vert z \vert } )^{n-\alpha } )U_{0,\lambda }(\tilde{y})}{ \vert \tilde{y}-z \vert ^{n+\alpha }}\,dz \biggr) \\ &=C_{n,\alpha } \,P.V. \biggl( \int _{B_{\lambda }(0)} \frac{-\tilde{U}_{0,\lambda }(z)}{ \vert \tilde{y}-z \vert ^{n+\alpha }}\,dz+ \int _{B_{\lambda }(0)} \frac{ \tilde{U}_{0,\lambda }(z)}{ \vert \frac{ \vert z \vert \tilde{y}}{\lambda }-\frac{\lambda z}{ \vert z \vert } \vert ^{n+\alpha }}\,dz \\ &\quad {} + \int _{\mathbb{R}^{n}\setminus {B_{\lambda }(0)}} \frac{ (1+ (\frac{\lambda }{ \vert z \vert } )^{n-\alpha } )U_{0,\lambda }(\tilde{y})}{ \vert \tilde{y}-z \vert ^{n+\alpha }}\,dz \biggr). \end{aligned} $$

Notice that, for any $z\in B_{\lambda }(0)\setminus \{0\}$,

$$ \biggl\vert \frac{ \vert z \vert \tilde{y}}{\lambda }-\frac{\lambda z}{ \vert z \vert } \biggr\vert ^{2}- \vert \tilde{y}-z \vert ^{2}= \frac{( \vert \tilde{y} \vert ^{2}-\lambda ^{2})( \vert z \vert ^{2}-\lambda ^{2})}{\lambda ^{2}}>0, $$

together with $U_{0,\lambda }(\tilde{y})<0$, we have

$$ \begin{aligned} (-\Delta )^{\alpha /2}U_{0,\lambda }( \tilde{y})&\leq C_{n,\alpha }U_{0, \lambda }(\tilde{y}) \int _{\mathbb{R}^{n}\setminus {B_{\lambda }(0)}} \frac{1}{ \vert \tilde{y}-z \vert ^{n+\alpha }}\,dz \\ &\leq C_{n,\alpha }U_{0,\lambda }(\tilde{y}) \int _{(\mathbb{R}^{n} \setminus {B_{\lambda }(0)})\cap (B_{4l}(\tilde{y})\setminus {B_{l}( \tilde{y})})}\frac{1}{ \vert \tilde{y}-z \vert ^{n+\alpha }}\,dz \\ &\leq \frac{C}{l^{\alpha }}U_{0,\lambda }(\tilde{y})< 0. \end{aligned} $$

(2.14)

For $\alpha =2$, we can also obtain the same estimate as (2.14) at some point $y_{0}\in \Omega \cap B^{-}_{\lambda }$. To this end, we define

$$ \phi (y):=\cos \frac{ \vert y \vert -\lambda +l}{l}, $$

(2.15)

then it follows that $\phi (y)\in [\cos 1,1]$ for any $y\in \overline{A_{\lambda ,l}(0)}=\{y\in \mathbb{R}^{n} \vert \lambda -l \leq \vert y \vert \leq \lambda \}$ and $-\frac{\Delta \phi (y)}{\phi (y)}\geq \frac{1}{l^{2}}$. Define

$$ \overline{U}_{0,\lambda }(y):=\frac{U_{0,\lambda }(y)}{\phi (y)} $$

(2.16)

for $y\in \overline{A_{\lambda ,l}(0)}$. Then there exists $y_{0}\in \Omega \cap B^{-}_{\lambda }$ such that

$$ \overline{U}_{0,\lambda }(y_{0})=\min _{\overline{A_{\lambda ,l}(0)}} \overline{U}_{0,\lambda }(y)< 0. $$

(2.17)

Since

$$ -\Delta U_{0,\lambda }(y_{0})=-\Delta \overline{U}_{0,\lambda }(y_{0}) \phi (y_{0})-2\nabla \overline{U}_{0,\lambda }(y_{0})\cdot \nabla \phi (y_{0}) -\overline{U}_{0,\lambda }(y_{0})\Delta \phi (y_{0}), $$

(2.18)

it follows immediately that

$$ -\Delta U_{0,\lambda }(y_{0})\leq \frac{1}{l^{2}}U_{0,\lambda }(y_{0}). $$

(2.19)

In conclusion, we have proved that, for both $0<\alpha <2$ and $\alpha =2$, there exists some $\hat{y}\in \Omega \cap B^{-}_{\lambda }$ such that

$$ (-\Delta )^{\frac{\alpha }{2}}U_{0,\lambda }(\hat{y})\leq \frac{C}{l^{\alpha }}U_{0,\lambda }(\hat{y})< 0. $$

(2.20)

Since $\tilde{y}\in \Omega \cap B^{-}_{\lambda }$, we have $V_{0,\lambda }(\tilde{y})<0$, then we know that there exists ȳ such that

$$ V_{0,\lambda }(\bar{y})=\min_{B_{\lambda }(0) \setminus (0)}V_{0,\lambda }(y)< 0. $$

Similar to (2.14), we can derive that

$$ (-\Delta )^{\frac{\alpha }{2}}V_{0,\lambda }(\bar{y})\leq \frac{C}{l^{\alpha }}V_{0,\lambda }(\bar{y})< 0. $$

(2.21)

On the other hand, by (2.10), we have at the point ỹ

$$\begin{aligned} 0 \leq & (-\Delta )^{\frac{\alpha }{2}}U_{0,\lambda }( \tilde{y})- \mathcal{L}_{1}(\tilde{y})V_{0,\lambda }(\tilde{y}) -p_{1} \biggl( \int _{B^{-}_{ \lambda }} \frac{v^{p_{1}-1}(z)V_{0,\lambda }(z)}{ \vert \tilde{y}-z \vert ^{\sigma }}\,dz \biggr)v^{p_{2}}( \tilde{y}) \\ \leq & (-\Delta )^{\frac{\alpha }{2}}U_{0,\lambda }(\tilde{y})- \mathcal{L}_{1}(\tilde{y})V_{0,\lambda }(\bar{y}) -p_{1} \biggl( \int _{B^{-}_{ \lambda }}\frac{v^{p_{1}-1}(z)}{ \vert \tilde{y}-z \vert ^{\sigma }}\,dz \biggr)v^{p_{2}}( \tilde{y})V_{0,\lambda }(\bar{y}) \\ \leq & (-\Delta )^{\frac{\alpha }{2}}U_{0,\lambda }(\tilde{y})-c'_{0, \lambda }( \tilde{y})V_{0,\lambda }(\bar{y}), \end{aligned}$$

(2.22)

where

$$\begin{aligned} c'_{x,\lambda }(y) :=& \mathcal{L}_{1}(y)+p_{1} \widetilde{P}_{x, \lambda }(y)v^{p_{2}}(y) \\ =& p_{2}P(y)v_{x,\lambda }^{p_{2}-1}(y)+p_{1} \widetilde{P}_{x,\lambda }(y)v^{p_{2}}(y)>0. \end{aligned}$$

Since $\lambda -l< \vert y \vert <\lambda$, we derive

$$\begin{aligned} P(y) \leq & \biggl\{ \int _{ \vert y-z \vert < \frac{ \vert z \vert }{2}}+ \int _{ \vert y-z \vert \geq \frac{ \vert z \vert }{2}} \biggr\} \frac{v^{p_{1}}(z)}{ \vert y-z \vert ^{\sigma }}\,dz \\ \leq & \Bigl[\max_{ \vert y \vert \leq 2\lambda }v(y) \Bigr]^{p_{1}} \int _{ \vert y-z \vert < \lambda }\frac{1}{ \vert y-z \vert ^{\sigma }}\,dz +2^{\sigma } \int _{\mathbb{R}^{n}} \frac{v^{p_{1}}(z)}{ \vert z \vert ^{\sigma }}\,dz \\ \leq & C\lambda ^{n-\sigma } \Bigl[\max_{ \vert y \vert \leq 2\lambda }v(y) \Bigr]^{p_{1}}+2^{ \sigma } \int _{\mathbb{R}^{n}}\frac{v^{p_{1}}(x)}{ \vert x \vert ^{\sigma }}\,dx=:C'_{1, \lambda }, \end{aligned}$$

(2.23)

and

$$\begin{aligned} \widetilde{P}_{0,\lambda }(y) \leq & \int _{ \vert y-z \vert < 2\lambda } \frac{1}{ \vert y-z \vert ^{\sigma }}v^{p_{1}-1}(z)\,dz \\ \leq &C\lambda ^{n-\sigma } \Bigl[\max_{ \vert y \vert \leq 4\lambda }v(y) \Bigr]^{p_{1}-1}=:C''_{1, \lambda }. \end{aligned}$$

(2.24)

It is obvious that $C'_{1,\lambda }$ and $C''_{1,\lambda }$ depend on λ continuously and monotone increase with respect to $\lambda >0$.

As a consequence, we can deduce from (2.23) and (2.24) that, for any $\lambda -l\leq \vert y \vert \leq \lambda$,

$$\begin{aligned} 0 < &c'_{0,\lambda }(y)=p_{2}P(y)v_{0,\lambda }^{p_{2}-1}(y)+p_{1} \widetilde{P}_{0,\lambda }(y)v^{p_{2}}(y) \\ \leq &p_{2}C'_{1,\lambda } \Bigl[\min _{ \vert y \vert \leq \lambda }v_{0,\lambda }(y) \Bigr]^{p_{2}-1} +p_{1}C''_{1,\lambda } \Bigl[\max _{ \vert y \vert \leq \lambda }v(y) \Bigr]^{p_{2}}=:C_{1,\lambda }, \end{aligned}$$

(2.25)

where $C_{1,\lambda }$ depends continuously on λ and monotone increases with respect to $\lambda >0$.

From (2.14) and (2.22), we have

$$ U_{0,\lambda }(\tilde{y})\geq c'_{0,\lambda }( \tilde{y})l^{\alpha }V_{0, \lambda }(\bar{y}). $$

(2.26)

By (2.10), we also have at the point ȳ

$$\begin{aligned} 0 \leq & (-\Delta )^{\frac{\alpha }{2}}V_{0,\lambda }( \bar{y})- \mathcal{L}_{2}(\bar{y})U_{0,\lambda }(\bar{y}) -q_{1} \biggl( \int _{B^{-}_{ \lambda }}\frac{u^{q_{1}-1}(z)U_{0,\lambda }(z)}{ \vert \bar{y}-z \vert ^{\sigma }}\,dz \biggr)u^{q_{2}}( \bar{y}) \\ \leq & (-\Delta )^{\frac{\alpha }{2}}V_{0,\lambda }(\bar{y})- \mathcal{L}_{2}(\bar{y})U_{0,\lambda }(\tilde{y}) -q_{1} \biggl( \int _{B^{-}_{ \lambda }}\frac{u^{q_{1}-1}(z)}{ \vert \bar{y}-z \vert ^{\sigma }}\,dz \biggr)u^{q_{2}}( \bar{y})U_{0,\lambda }(\tilde{y}) \\ \leq & (-\Delta )^{\frac{\alpha }{2}}V_{0,\lambda }(\bar{y})-c''_{0, \lambda }( \bar{y})U_{0,\lambda }(\tilde{y}), \end{aligned}$$

(2.27)

where

$$\begin{aligned} c''_{x,\lambda }(y) :=& \mathcal{L}_{2}(y)+q_{1} \widetilde{Q}_{x, \lambda }(y)u^{q_{2}}(y) \\ =& q_{2}Q(y)u_{x,\lambda }^{q_{2}-1}(y)+q_{1} \widetilde{Q}_{x,\lambda }(y)u^{q_{2}}(y)>0. \end{aligned}$$

Since $\lambda -l< \vert y \vert <\lambda$, we have

$$\begin{aligned} Q(y) \leq & \biggl\{ \int _{ \vert y-z \vert < \frac{ \vert z \vert }{2}}+ \int _{ \vert y-z \vert \geq \frac{ \vert z \vert }{2}} \biggr\} \frac{u^{q_{1}}(z)}{ \vert y-z \vert ^{\sigma }}\,dz \\ \leq & \Bigl[\max_{ \vert y \vert \leq 2\lambda }u(y) \Bigr]^{q_{1}} \int _{ \vert y-z \vert < \lambda }\frac{1}{ \vert y-z \vert ^{\sigma }}\,dz +2^{\sigma } \int _{\mathbb{R}^{n}} \frac{u^{q_{1}}(z)}{ \vert z \vert ^{\sigma }}\,dz \\ \leq & C\lambda ^{n-\sigma } \Bigl[\max_{ \vert y \vert \leq 2\lambda }u(y) \Bigr]^{q_{1}}+2^{ \sigma } \int _{\mathbb{R}^{n}}\frac{u^{q_{1}}(x)}{ \vert x \vert ^{\sigma }}\,dx=:C'_{2, \lambda }, \end{aligned}$$

(2.28)

and

$$\begin{aligned} \widetilde{Q}_{0,\lambda }(y) \leq & \int _{ \vert y-z \vert < 2\lambda } \frac{1}{ \vert y-z \vert ^{\sigma }}u^{q_{1}-1}(z)\,dz \\ \leq &C\lambda ^{n-\sigma } \Bigl[\max_{ \vert y \vert \leq 4\lambda }u(y) \Bigr]^{q_{1}-1}=:C''_{2, \lambda }. \end{aligned}$$

(2.29)

It is obvious that $C'_{2,\lambda }$ and $C''_{2,\lambda }$ depend on λ continuously and monotone increase with respect to $\lambda >0$.

Thus, we infer from (2.28) and (2.29) that, for any $\lambda -l\leq \vert y \vert \leq \lambda$,

$$\begin{aligned} 0 < &c''_{0,\lambda }(y)=q_{2}Q(y)u_{0,\lambda }^{q_{2}-1}(y)+q_{1} \widetilde{Q}_{0,\lambda }(y)u^{q_{2}}(y) \\ \leq &q_{2}C'_{2,\lambda } \Bigl[\min _{ \vert y \vert \leq \lambda }u_{0,\lambda }(y) \Bigr]^{q_{2}-1} +q_{1}C''_{2,\lambda } \Bigl[\max _{ \vert y \vert \leq \lambda }u(y) \Bigr]^{q_{2}}=:C_{2,\lambda }, \end{aligned}$$

(2.30)

where $C_{2,\lambda }$ depends continuously on λ and monotone increases with respect to $\lambda >0$.

As a consequence, it follows from (2.21), (2.26), and (2.27) that

$$\begin{aligned} 0 \leq &(-\Delta )^{\frac{\alpha }{2}}V_{0,\lambda }( \bar{y})-c''_{0, \lambda }(\bar{y})U_{0,\lambda }( \tilde{y}) \\ \leq &\frac{C}{l^{\alpha }}V_{0,\lambda }(\bar{y})-c'_{0,\lambda }( \tilde{y})c''_{0,\lambda }(\bar{y})l^{\alpha }V_{0,\lambda }( \bar{y}) \\ \leq &\frac{C}{l^{\alpha }}V_{0,\lambda }(\bar{y})-C_{1,\lambda }C_{2, \lambda }l^{\alpha }V_{0,\lambda }( \bar{y}) \\ =& \biggl(\frac{C}{l^{\alpha }}-C_{\lambda }l^{\alpha } \biggr)V_{0,\lambda }(\bar{y}), \end{aligned}$$

(2.31)

that is,

$$ \frac{C}{{\lambda }^{\alpha }}\leq \frac{C}{l^{\alpha }}\leq C_{\lambda }l^{ \alpha }. $$

(2.32)

We can derive a contradiction from (2.32) directly if $0<\lambda \leq \gamma _{0}$ for some constant $\gamma _{0}$ small enough, or if $0< l\leq l_{0}$ for some sufficiently small $l_{0}$ depending on λ continuously. This implies that (2.11) and (2.12) must hold. Furthermore, by (2.10), we can actually deduce from $U_{x,\lambda }(y)\geq 0$, $V_{x,\lambda }\geq 0$ in Ω that

$$ U_{x,\lambda }(y)\geq 0, \quad\quad V_{x,\lambda }(y)\geq 0, \quad \forall y\in B_{\lambda }(x)\setminus \{x\}. $$

(2.33)

This completes the proof of Theorem 2.3. □

The following lemma provides a starting point for us to move the spheres.

Lemma 2.4

For every $x\in \mathbb{R}^{n}$, there exists $\epsilon _{0}(x)>0$ such that $u_{x,\lambda }(y)\geq u(y)$ and $v_{x,\lambda }(y)\geq v(y)$ for all $\lambda \in (0,\epsilon _{0}(x)]$ and $y\in B_{\lambda }(x)\setminus \{x\}$.

Proof

For every $x\in \mathbb{R}^{n}$, define

$$ B_{\lambda }^{-}= \bigl\{ y\in B_{\lambda }(x)\setminus \{x\} \vert U_{x,\lambda }(y)< 0, V_{x,\lambda }(y)< 0 \bigr\} . $$

Choose $\epsilon _{0}(x):=\min \{\eta _{0}(x),\gamma _{0}(x)\}$, where $\eta _{0}(x)$ and $\gamma _{0}(x)$ are defined the same as in Lemma 2.2 and Theorem 2.3. We will show via contradiction arguments that, for any $0<\lambda \leq \epsilon _{0}$,

$$ B^{-}_{\lambda }=\emptyset . $$

(2.34)

Suppose that (2.34) does not hold, that is, $B^{-}_{\lambda }\neq \emptyset$ and hence $U_{x,\lambda }$, $V_{x,\lambda }$ is negative somewhere in $B_{\lambda }(x)\setminus \{x\}$. For arbitrary $y\in B^{-}_{\lambda }$, one can infer from (1.1) and (2.5) that

$$\begin{aligned} &(-\Delta )^{\frac{\alpha }{2}}U_{x,\lambda }(y) \\ &\quad = \int _{\mathbb{R}^{n}} \frac{v^{p_{1}}_{x,\lambda }(z)}{ \vert y-z \vert ^{\sigma }} \biggl( \frac{\lambda }{ \vert z-x \vert } \biggr)^{\tau _{1}}\,dz \biggl(\frac{\lambda }{ \vert y-x \vert } \biggr)^{\tau _{2}}v^{p_{2}}_{x,\lambda }(y)- \int _{\mathbb{R}^{n}} \frac{v^{p_{1}}(z)}{ \vert y-z \vert ^{\sigma }}\,dz v^{p_{2}}(y) \\ &\quad \geq \int _{\mathbb{R}^{n}} \frac{v^{p_{1}}_{x,\lambda }(z)}{ \vert y-z \vert ^{\sigma }}\,dz v^{p_{2}}_{x, \lambda }(y)- \int _{\mathbb{R}^{n}}\frac{v^{p_{1}}(z)}{ \vert y-z \vert ^{\sigma }}\,dz v^{p_{2}}(y) \\ &\quad \geq p_{2} \int _{\mathbb{R}^{n}}\frac{v^{p_{1}}(z)}{ \vert y-z \vert ^{\sigma }}\,dz v_{x,\lambda }^{p_{2}-1}(y)V_{x,\lambda }(y)+ \int _{\mathbb{R}^{n}} \frac{v^{p_{1}}_{x,\lambda }(z)-v^{p_{1}}(z)}{ \vert y-z \vert ^{\sigma }}\,dz v^{p_{2}}_{x, \lambda }(y) \\ &\quad =\mathcal{L}_{1}(y)V_{x,\lambda }(y)+ \int _{\mathbb{R}^{n}} \frac{v^{p_{1}}_{x,\lambda }(z)-v^{p_{1}}(z)}{ \vert y-z \vert ^{\sigma }}\,dz v^{p_{2}}_{x, \lambda }(y) \\ &\quad \geq \mathcal{L}_{1}(y)V_{x,\lambda }(y)+v_{x,\lambda }^{p_{2}}(y) \int _{B_{\lambda }(x)} \biggl( \frac{1}{ \vert \frac{(y-x) \vert z-x \vert }{\lambda }-\frac{\lambda (z-x)}{ \vert z-x \vert } \vert ^{\sigma }}- \frac{1}{ \vert y-z \vert ^{\sigma }} \biggr) \bigl(v^{p_{1}}(z)-v_{x,\lambda }^{p_{1}}(z) \bigr)\,dz \\ &\quad \geq \mathcal{L}_{1}(y)V_{x,\lambda }(y)+v^{p_{2}}(y) \int _{B^{-}_{\lambda }(x)}\frac{1}{ \vert y-z \vert ^{\sigma }} \bigl(v_{x,\lambda }^{p_{1}}(z)-v^{p_{1}}(z) \bigr)\,dz \\ &\quad \geq \mathcal{L}_{1}(y)V_{x,\lambda }(y)+p_{1} \biggl( \int _{B^{-}_{ \lambda }}\frac{v^{p_{1}-1}(z)V_{x,\lambda }(z)}{ \vert y-z \vert ^{\sigma }}\,dz \biggr)v^{p_{2}}(y), \\ &(-\Delta )^{\frac{\alpha }{2}}V_{x,\lambda }(y) \\ &\quad = \int _{\mathbb{R}^{n}} \frac{u^{q_{1}}_{x,\lambda }(z)}{ \vert y-z \vert ^{\sigma }} \biggl( \frac{\lambda }{ \vert z-x \vert } \biggr)^{\tau _{3}}\,dz \biggl(\frac{\lambda }{ \vert y-x \vert } \biggr)^{\tau _{4}}u^{q_{2}}_{x,\lambda }(y)- \int _{\mathbb{R}^{n}} \frac{u^{q_{1}}(z)}{ \vert y-z \vert ^{\sigma }}\,dz u^{q_{2}}(y) \\ &\quad \geq \int _{\mathbb{R}^{n}} \frac{u^{q_{1}}_{x,\lambda }(z)}{ \vert y-z \vert ^{\sigma }}\,dz u^{q_{2}}_{x, \lambda }(y)- \int _{\mathbb{R}^{n}}\frac{u^{q_{1}}(z)}{ \vert y-z \vert ^{\sigma }}\,dz u^{q_{2}}(y) \\ &\quad \geq q_{2} \int _{\mathbb{R}^{n}}\frac{u^{q_{1}}(z)}{ \vert y-z \vert ^{\sigma }}\,dz u_{x,\lambda }^{q_{2}-1}(y)U_{x,\lambda }(y)+ \int _{\mathbb{R}^{n}} \frac{u^{q_{1}}_{x,\lambda }(z)-u^{q_{1}}(z)}{ \vert y-z \vert ^{\sigma }}\,dz u^{q_{2}}_{x, \lambda }(y) \\ &\quad =\mathcal{L}_{2}(y)U_{x,\lambda }(y)+ \int _{\mathbb{R}^{n}} \frac{u^{q_{1}}_{x,\lambda }(z)-u^{q_{1}}(z)}{ \vert y-z \vert ^{\sigma }}\,dz u^{q_{2}}_{x, \lambda }(y) \\ &\quad \geq \mathcal{L}_{2}(y)U_{x,\lambda }(y)+u_{x,\lambda }^{q_{2}}(y) \int _{B_{\lambda }(x)} \biggl( \frac{1}{ \vert \frac{(y-x) \vert z-x \vert }{\lambda }-\frac{\lambda (z-x)}{ \vert z-x \vert } \vert ^{\sigma }}- \frac{1}{ \vert y-z \vert ^{\sigma }} \biggr) \bigl(u^{q_{1}}(z)-u_{x,\lambda }^{q_{1}}(z) \bigr)\,dz \\ &\quad \geq \mathcal{L}_{2}(y)U_{x,\lambda }(y)+u^{q_{2}}(y) \int _{B^{-}_{\lambda }(x)} \frac{1}{ \vert y-z \vert ^{\sigma }} \bigl(u_{x,\lambda }^{q_{1}}(z)-u^{q_{1}}(z) \bigr)\,dz \\ &\quad \geq \mathcal{L}_{2}(y)U_{x,\lambda }(y)+q_{1} \biggl( \int _{B^{-}_{ \lambda }}\frac{u^{q_{1}-1}(z)U_{x,\lambda }(z)}{ \vert y-z \vert ^{\sigma }}\,dz \biggr)u^{q_{2}}(y). \end{aligned}$$

That is, for all $y\in B^{-}_{\lambda }$,

$$\begin{aligned}& (-\Delta )^{\frac{\alpha }{2}}U_{x,\lambda }(y)-\mathcal{L}_{1}(y)V_{x, \lambda }(y) -p_{1} \biggl( \int _{B^{-}_{\lambda }} \frac{v^{p_{1}-1}(z)V_{x,\lambda }(z)}{ \vert y-z \vert ^{\sigma }}\,dz \biggr)v^{p_{2}}(y) \geq 0, \end{aligned}$$

(2.35)

$$\begin{aligned}& (-\Delta )^{\frac{\alpha }{2}}V_{x,\lambda }(y)-\mathcal{L}_{2}(y)U_{x, \lambda }(y) -q_{1} \biggl( \int _{B^{-}_{\lambda }} \frac{u^{q_{1}-1}(z)U_{x,\lambda }(z)}{ \vert y-z \vert ^{\sigma }}\,dz \biggr)u^{q_{2}}(y) \geq 0. \end{aligned}$$

(2.36)

Since $\epsilon _{0}(x):=\min \{\eta _{0}(x),\gamma _{0}(x)\}$, by Lemma 2.2, we can deduce that, for any $0<\lambda \leq \epsilon _{0}$,

$$ U_{x,\lambda }(y)\geq 1,\quad\quad V_{x,\lambda }(y)\geq 1, \quad \forall y \in \overline{B_{\lambda ^{2}}(x)}\setminus \{x\}. $$

(2.37)

Therefore, by taking $l=\lambda -\lambda ^{2}$ and $\Omega =A_{\lambda ,l}(x)$, then it follows from (2.35), (2.36), and (2.37) that all the conditions in (2.10) in Theorem 2.3 are fulfilled. We can deduce from (i) in Theorem 2.3 that $U_{x,\lambda }\geq 0$, $V_{x,\lambda }\geq 0$ in $\Omega =A_{\lambda ,l}(x)$ for any $0<\lambda \leq \epsilon _{0}(x)$. That is, there exists $\epsilon _{0}(x)>0$ such that, for all $\lambda \in (0,\epsilon _{0}(x)]$,

$$ U_{x,\lambda }(y)\geq 0,\quad\quad V_{x,\lambda }(y)\geq 0, \quad \forall y \in B_{\lambda }(x)\setminus \{x\}. $$

This completes the proof of Lemma 2.4. □

For each fixed $x\in \mathbb{R}^{n}$, we define

$$ \bar{\lambda }(x)=\sup \bigl\{ \lambda >0 \vert u_{x,\mu } \geq u,v_{x,\mu } \geq v\text{ in } B_{\mu }(x)\setminus \{x \}, \forall 0< \mu \leq \lambda \bigr\} , $$

(2.38)

by Lemma 2.4, $\bar{\lambda }(x)$ is well defined and $0<\bar{\lambda }(x)\leq +\infty$ for any $x\in \mathbb{R}^{n}$.

We need the following lemma, which is crucial in our proof.

Lemma 2.5

If $\bar{\lambda }(\bar{x})<+\infty$ for some $\bar{x}\in \mathbb{R}^{n}$, then

$$ u_{\bar{x},\bar{\lambda }(\bar{x})}(y)=u(y), \quad\quad v_{\bar{x},\bar{\lambda }( \bar{x})}(y)=v(y),\quad \forall y\in B_{\bar{\lambda }}( \bar{x})\setminus \{\bar{x}\}. $$

Proof

Without loss of generality, let $\bar{x}=0$. Since $(u,v)$ is a pair of positive solutions to integral system (1.4), one can verify that $u_{0,\lambda }$, $v_{0,\lambda }$ also satisfy a similar integral system as (1.4) in $\mathbb{R}^{n}\setminus \{0\}$. In fact, by (1.4) and direct calculations, we have, for any $y\in \mathbb{R}^{n}\setminus \{0\}$,

$$\begin{aligned}& \begin{aligned} u_{0,\lambda }(y)&= \biggl(\frac{\lambda }{ \vert y \vert } \biggr)^{n-\alpha }u \biggl(\frac{\lambda ^{2}y}{ \vert y^{2} \vert } \biggr) \\ &=\frac{\lambda ^{n-\alpha }}{ \vert y \vert ^{n-\alpha }} \int _{\mathbb{R}^{n}} \frac{R_{\alpha ,n}}{ \vert \frac{\lambda ^{2}y}{ \vert y \vert ^{2}}-z \vert ^{n-\alpha }} \int _{\mathbb{R}^{n}}\frac{v^{p_{1}}(\xi )}{ \vert z-\xi \vert ^{\sigma }}\,d\xi v^{p_{2}}(z)\,dz \\ &= \frac{\lambda ^{n-\alpha }}{ \vert y \vert ^{n-\alpha }} \int _{\mathbb{R}^{n}} \frac{R_{\alpha ,n}}{ \vert \frac{\lambda ^{2}y}{ \vert y \vert ^{2}}-\frac{\lambda ^{2}z}{ \vert z \vert ^{2}} \vert ^{n-\alpha }} \int _{\mathbb{R}^{n}} \frac{v^{p_{1}}(\frac{\lambda ^{2}\xi }{ \vert \xi \vert ^{2}})}{ \vert \frac{\lambda ^{2}z}{ \vert z \vert ^{2}} -\frac{\lambda ^{2}\xi }{ \vert \xi \vert ^{2}} \vert ^{\sigma }} \frac{\lambda ^{2n}}{ \vert \xi \vert ^{2n}}\,d\xi v^{p_{2}} \biggl( \frac{\lambda ^{2}z}{ \vert z \vert ^{2}} \biggr)\frac{\lambda ^{2n}}{ \vert z \vert ^{2n}}\,dz \\ &= \int _{\mathbb{R}^{n}}\frac{R_{\alpha ,n}}{ \vert y-z \vert ^{n-\alpha }} \int _{\mathbb{R}^{n}} \frac{v_{0,\lambda }^{p_{1}}(\xi )}{ \vert z-\xi \vert ^{\sigma }} \biggl( \frac{\lambda }{ \vert \xi \vert } \biggr)^{\tau _{1}}\,d\xi v^{p_{2}}_{0,\lambda }(z) \biggl( \frac{\lambda }{ \vert z \vert } \biggr)^{\tau _{2}}\,dz, \end{aligned} \\& \begin{aligned} v_{0,\lambda }(y)&= \biggl(\frac{\lambda }{ \vert y \vert } \biggr)^{n-\alpha }v \biggl(\frac{\lambda ^{2}y}{ \vert y^{2} \vert } \biggr) \\ &=\frac{\lambda ^{n-\alpha }}{ \vert y \vert ^{n-\alpha }} \int _{\mathbb{R}^{n}} \frac{R_{\alpha ,n}}{ \vert \frac{\lambda ^{2}y}{ \vert y \vert ^{2}}-z \vert ^{n-\alpha }} \int _{\mathbb{R}^{n}}\frac{u^{q_{1}}(\zeta )}{ \vert z-\zeta \vert ^{\sigma }}\,d \zeta u^{q_{2}}(z)\,dz \\ &= \frac{\lambda ^{n-\alpha }}{ \vert y \vert ^{n-\alpha }} \int _{\mathbb{R}^{n}} \frac{R_{\alpha ,n}}{ \vert \frac{\lambda ^{2}y}{ \vert y \vert ^{2}}-\frac{\lambda ^{2}z}{ \vert z \vert ^{2}} \vert ^{n-\alpha }} \int _{\mathbb{R}^{n}} \frac{u^{q_{1}}(\frac{\lambda ^{2}\zeta }{ \vert \zeta \vert ^{2}})}{ \vert \frac{\lambda ^{2}z}{ \vert z \vert ^{2}} -\frac{\lambda ^{2}\zeta }{ \vert \zeta \vert ^{2}} \vert ^{\sigma }} \frac{\lambda ^{2n}}{ \vert \zeta \vert ^{2n}}\,d\zeta u^{q_{2}} \biggl( \frac{\lambda ^{2}z}{ \vert z \vert ^{2}} \biggr)\frac{\lambda ^{2n}}{ \vert z \vert ^{2n}}\,dz \\ &= \int _{\mathbb{R}^{n}}\frac{R_{\alpha ,n}}{ \vert y-z \vert ^{n-\alpha }} \int _{\mathbb{R}^{n}} \frac{u_{0,\lambda }^{q_{1}}(\zeta )}{ \vert z-\zeta \vert ^{\sigma }} \biggl( \frac{\lambda }{ \vert \zeta \vert } \biggr)^{\tau _{3}}\,d\zeta u^{q_{2}}_{0, \lambda }(z) \biggl( \frac{\lambda }{ \vert z \vert } \biggr)^{\tau _{4}}\,dz, \end{aligned} \end{aligned}$$

where $\tau _{1}:=2n-\sigma -p_{1}(n-\alpha )\geq 0$, $\tau _{2}:=n+\alpha - \sigma -p_{2}(n-\alpha )\geq 0$ and $\tau _{3}:=2n-\sigma -q_{1}(n-\alpha )\geq 0$, $\tau _{4}:=n+\alpha - \sigma -q_{2}(n-\alpha )\geq 0$.

Suppose on the contrary that $U_{0,\bar{\lambda }}\geq 0$ but $U_{0,\bar{\lambda }}$ is not identically zero in $B_{\bar{\lambda }}(0)\setminus \{0\}$, then we will get a contradiction with the definition (2.38) of λ̄. We first prove that

$$ U_{0,\bar{\lambda }}(y)>0, \quad\quad V_{0,\bar{\lambda }}(y)>0, \quad \forall y\in B_{\bar{\lambda }}(0)\setminus \{0\}. $$

(2.39)

Indeed, if there exists a point $y^{0}\in B_{\bar{\lambda }}(0)\setminus \{0\}$ such that $U_{0,\bar{\lambda }}(y^{0})>0$, by continuity, there exists small $\gamma >0$ and constant $c_{0}>0$ such that

$$ B_{\gamma } \bigl(y^{0} \bigr)\subset B_{\bar{\lambda }}(0)\setminus \{0\} \quad \text{and} \quad U_{0,\bar{\lambda }}(y)\geq c_{0}>0, \quad \forall y\in B_{\gamma } \bigl(y^{0} \bigr). $$

For any $y\in B_{\bar{\lambda }}(0)\setminus \{0\}$, one can derive that

$$ \begin{aligned} u(y)&= \int _{\mathbb{R}^{n}} \frac{R_{\alpha ,n}}{ \vert y-z \vert ^{n-\alpha }}P(z)v^{p_{2}}(z)\,dz \\ &= \int _{B_{\bar{\lambda }}(0)}\frac{R_{\alpha ,n}}{ \vert y-z \vert ^{n-\alpha }}P(z)v^{p_{2}}(z)\,dz+ \int _{\mathbb{R}^{n}\setminus B_{\bar{\lambda }}(0)} \frac{R_{\alpha ,n}}{ \vert y-z \vert ^{n-\alpha }}P(z)v^{p_{2}}(z)\,dz \\ &= \int _{B_{\bar{\lambda }}(0)}\frac{R_{\alpha ,n}}{ \vert y-z \vert ^{n-\alpha }}P(z)v^{p_{2}}(z)\,dz + \int _{B_{\bar{\lambda }}(0)} \frac{R_{\alpha ,n}}{ \vert \frac{y \vert z \vert }{\bar{\lambda }}-\frac{\bar{\lambda }z}{ \vert z \vert } \vert ^{n-\alpha }} P \bigl(z^{\bar{\lambda }} \bigr) \biggl(\frac{\bar{\lambda }}{ \vert z \vert } \biggr)^{\sigma + \tau _{2}}v^{p_{2}}_{0,\bar{\lambda }}(z) \,dz, \end{aligned} $$

and

$$ \begin{aligned} u_{0,\bar{\lambda }}(y)&= \int _{\mathbb{R}^{n}} \frac{R_{\alpha ,n}}{ \vert y-z \vert ^{n-\alpha }} \int _{\mathbb{R}^{n}} \frac{v_{0,\bar{\lambda }}^{p_{1}}(\xi )}{ \vert z-\xi \vert ^{\sigma }} \biggl( \frac{\bar{\lambda }}{ \vert \xi \vert } \biggr)^{\tau _{1}}\,d\xi v^{p_{2}}_{0, \bar{\lambda }}(z) \biggl( \frac{\bar{\lambda }}{ \vert z \vert } \biggr)^{\tau _{2}}\,dz \\ &= \int _{B_{\bar{\lambda }}(0)}\frac{R_{\alpha ,n}}{ \vert y-z \vert ^{n-\alpha }} \biggl(\frac{\bar{\lambda }}{ \vert z \vert } \biggr)^{\tau _{2}} \bar{P}_{0, \bar{\lambda }}(z)v^{p_{2}}_{0,\bar{\lambda }}(z) \,dz \\ &\quad {} + \int _{B_{\bar{\lambda }}(0)} \frac{R_{\alpha ,n}}{ \vert \frac{y \vert z \vert }{\bar{\lambda }}-\frac{\bar{\lambda }z}{ \vert z \vert } \vert ^{n-\alpha }} \bar{P}_{0,\bar{\lambda }} \bigl(z^{\bar{\lambda }} \bigr) \biggl( \frac{\bar{\lambda }}{ \vert z \vert } \biggr)^{\sigma }v^{p_{2}}(z) \,dz, \end{aligned} $$

where

$$ \bar{P}_{x,\lambda }(y):= \int _{\mathbb{R}^{n}} \frac{v^{p_{1}}_{x,\lambda }(\xi )}{ \vert y-\xi \vert ^{\sigma }} \biggl( \frac{\lambda }{ \vert \xi -x \vert } \biggr)^{\tau _{1}}\,d\xi . $$

Let us define

$$\begin{aligned}& K_{1,\bar{\lambda }}(y,z)=R_{\alpha ,n} \biggl( \frac{1}{ \vert y-z \vert ^{n-\alpha }} - \frac{1}{ \vert \frac{y \vert z \vert }{\bar{\lambda }}-\frac{\bar{\lambda }z}{ \vert z \vert } \vert ^{n-\alpha }} \biggr), \\& K_{2,\bar{\lambda }}(y,z)=R_{\alpha ,n} \biggl( \frac{1}{ \vert y-z \vert ^{\sigma }} - \frac{1}{ \vert \frac{y \vert z \vert }{\bar{\lambda }}-\frac{\bar{\lambda }z}{ \vert z \vert } \vert ^{\sigma }} \biggr). \end{aligned}$$

It is easy to derive that $K_{1,\bar{\lambda }}(y,z)>0$, $K_{2,\bar{\lambda }}(y,z)>0$, and

$$ \bar{P}_{0,\bar{\lambda }}(z)=P \bigl(z^{\bar{\lambda }} \bigr) \biggl( \frac{\bar{\lambda }}{ \vert z \vert } \biggr)^{\sigma },\quad\quad P(z)= \bar{P}_{0,\bar{\lambda }} \bigl(z^{\bar{\lambda }} \bigr) \biggl( \frac{\bar{\lambda }}{ \vert z \vert } \biggr)^{\sigma }, $$

and furthermore,

$$ \bar{P}_{0,\bar{\lambda }}(z)-P(z)= \int _{B_{\bar{\lambda }}(0)}K_{2, \bar{\lambda }}(z,\xi ) \bigl(v^{p_{1}}_{0,\bar{\lambda }}( \xi )-v^{p_{1}}( \xi ) \bigr)\,d\xi >0. $$

As a consequence, it follows immediately that, for any $y\in B_{\bar{\lambda }}(0)\setminus \{0\}$,

$$ \begin{aligned} U_{0,\bar{\lambda }}(y)&= \int _{B_{\bar{\lambda }}(0)}K_{1, \bar{\lambda }}(y,z)P(z) \biggl( \biggl( \frac{\bar{\lambda }}{ \vert z \vert } \biggr)^{\tau _{2}}v^{p_{2}}_{0,\bar{\lambda }}(z)-v^{p_{2}}(z) \biggr)\,dz \\ &\quad {} + \int _{B_{\bar{\lambda }}(0)}K_{1,\bar{\lambda }}(y,z) \bigl(\bar{P}_{0, \bar{\lambda }}(z)-P(z) \bigr) \biggl(\frac{\bar{\lambda }}{ \vert z \vert } \biggr)^{ \tau _{2}}v^{p_{2}}_{0,\bar{\lambda }}(z) \,dz \\ &\geq \int _{B_{\bar{\lambda }}(0)}K_{1,\bar{\lambda }}(y,z)P(z) \biggl( \biggl( \frac{\bar{\lambda }}{ \vert z \vert } \biggr)^{\tau _{2}}v^{p_{2}}_{0, \bar{\lambda }}(z)-v^{p_{2}}(z) \biggr)\,dz \\ &\geq p_{2} \int _{B_{\gamma }(y^{0})}K_{1,\bar{\lambda }}(y,z)P(z)v^{p_{2}-1}_{0, \bar{\lambda }}(z) \bigl(v_{0,\bar{\lambda }}(z)-v(z) \bigr)\,dz>0, \end{aligned} $$

(2.40)

thus we arrive at (2.39). Furthermore, (2.40) also implies that there exists $0<\eta <\bar{\lambda }$ small enough such that, for any $y\in \overline{B_{\eta }(0)}\setminus \{0\}$,

$$ U_{0,\bar{\lambda }}(y)\geq p_{2} \int _{B_{\frac{\gamma }{2}}(y^{0})}c_{9}c_{8}c_{7}^{p_{2}-1}c_{0} \,dz=:\widetilde{c}_{0}>0. $$

(2.41)

Now we define

$$ \tilde{l}_{0}:=\min_{\lambda \in [\bar{\lambda },2\bar{\lambda }]}l_{0}(0, \lambda )>0, $$

(2.42)

where $l_{0}(0,\lambda )$ is given by Theorem 2.3. For fixed small $0< r_{0}<\frac{1}{2}\min \{\tilde{l}_{0},\bar{\lambda }\}$, by (2.39) and (2.41), we can define

$$ m_{0}:=\inf_{y\in \overline{B_{\bar{\lambda }-r_{0}}(0)}\setminus \{0 \}}U_{0,\bar{\lambda }}(y)>0. $$

(2.43)

Similarly, we can also define

$$ n_{0}:=\inf_{y\in \overline{B_{\bar{\lambda }-r_{0}}(0)}\setminus \{0 \}}V_{0,\bar{\lambda }}(y)>0. $$

(2.44)

Then, by the uniform continuity of u on an arbitrary compact set $K\subset \mathbb{R}^{n}$ (say, $K=\overline{B_{4\bar{\lambda }}(0)}$), one can infer from (2.43) that there exists $0<\varepsilon _{0}<\frac{1}{2}\min \{\tilde{l}_{0},\bar{\lambda }\}$ sufficiently small such that, for any $\lambda \in [\bar{\lambda },\bar{\lambda }+\varepsilon _{0}]$,

$$ U_{0,\lambda }(y)\geq \frac{m_{0}}{2}>0, \quad \forall y\in \overline{B_{\bar{\lambda }-r_{0}}(0)}\setminus \{0\}. $$

(2.45)

In order to prove (2.45), one should observe that (2.43) is equivalent to

$$ \vert y \vert ^{n-\alpha }u(y)-\bar{\lambda }^{n-\alpha }u \bigl(y^{0,\bar{\lambda }} \bigr) \geq m_{0}\bar{\lambda }^{n-\alpha }, \quad \forall \vert y \vert \geq \frac{\bar{\lambda }^{2}}{\bar{\lambda }-r_{0}}. $$

(2.46)

Since u is uniformly continuous on $\overline{B_{4\bar{\lambda }}(0)}$, we infer from (2.46) that there exists $0<\varepsilon _{0}<\frac{1}{2}\min \{\tilde{l}_{0},\bar{\lambda }\}$ sufficiently small such that, for any $\lambda \in [\bar{\lambda },\bar{\lambda }+\varepsilon _{0}]$,

$$ \vert y \vert ^{n-\alpha }u(y)-\lambda ^{n-\alpha }u \bigl(y^{0,\lambda } \bigr)\geq \frac{m_{0}}{2}\lambda ^{n-\alpha }, \quad \forall \vert y \vert \geq \frac{\lambda ^{2}}{\lambda -r_{0}}, $$

(2.47)

which is equivalent to (2.45), hence we have proved (2.45).

Similar to (2.45),we can also derive that

$$ V_{0,\lambda }(y)\geq \frac{n_{0}}{2}>0, \quad \forall y\in \overline{B_{\bar{\lambda }-r_{0}}(0)}\setminus \{0\}. $$

(2.48)

For any $\lambda \in [\bar{\lambda },\bar{\lambda }+\varepsilon _{0}]$, let $l:=\lambda -\bar{\lambda }+r_{0}\in (0,\tilde{l}_{0})$ and $\Omega :=A_{\lambda ,l}(0)$, then it follows from (2.35), (2.36), and (2.45) that all conditions (2.10) in Theorem 2.3 are fulfilled, hence we can deduce from (ii) in Theorem 2.3 that

$$ U_{0,\lambda }(y)\geq 0,\quad\quad V_{0,\lambda }(y)\geq 0, \quad \forall y\in \Omega =A_{\lambda ,l}(0). $$

(2.49)

Therefore, one can infer from (2.45) and (2.49) that $B^{-}_{\lambda }=\emptyset$ for all $\lambda \in [\bar{\lambda },\bar{\lambda }+\varepsilon _{0}]$, that is,

$$ U_{0,\lambda }(y)\geq 0,\quad\quad V_{0,\lambda }(y)\geq 0, \quad \forall y\in B_{\lambda }(0)\setminus \{0\}, $$

(2.50)

which contradicts definition (2.38) of $\bar{\lambda }(0)$. As a consequence, in the case $0<\bar{\lambda }(0)<+\infty$, we must have $U_{0,\bar{\lambda }}\equiv 0$, $V_{0,\bar{\lambda }}\equiv 0$ in $B_{\bar{\lambda }}(0)\setminus \{0\}$, that is,

$$ u_{0,\bar{\lambda }(0)}(y)\equiv u(y),\quad\quad v_{0,\bar{\lambda }(0)}(y)\equiv v(y), \quad \forall y\in B_{\bar{\lambda }}(0)\setminus \{0\}. $$

(2.51)

This finishes our proof of Lemma 2.5. □

We also need the following property about the limiting radius $\bar{\lambda }(x)$.

Lemma 2.6

If $\bar{\lambda }(\bar{x})=+\infty$ for some $\bar{x}\in \mathbb{R}^{n}$, then $\bar{\lambda }(x)=+\infty$ for all $x\in \mathbb{R}^{n}$.

Proof

Since $\bar{\lambda }(\bar{x})=+\infty$, recalling the definition of λ̄, we can derive

$$ u_{\bar{x},\lambda }(y)\geq u(y),\quad\quad v_{\bar{x},\lambda }(y)\geq v(y),\quad \forall y\in B_{\lambda }(\bar{x})\setminus \{\bar{x}\}, \forall 0< \lambda < +\infty . $$

That is,

$$ u(y)\geq u_{\bar{x},\lambda }(y),\quad\quad v(y)\geq v_{\bar{x},\lambda }(y),\quad \forall \vert y-\bar{x} \vert \geq \lambda , \forall 0< \lambda < +\infty . $$

It follows immediately that

$$ \lim_{ \vert y \vert \rightarrow \infty } \vert y \vert ^{n-\alpha }u(y)=+\infty ,\quad\quad \lim_{ \vert y \vert \rightarrow \infty } \vert y \vert ^{n-\alpha }v(y)=+\infty . $$

(2.52)

On the other hand, if we assume $\bar{\lambda }(x)<+\infty$ for some $x\in \mathbb{R}^{n}$, then by Lemma 2.5, one arrives at

$$\begin{aligned}& \lim_{ \vert y \vert \rightarrow \infty } \vert y \vert ^{n-\alpha }u(y)=\lim _{ \vert y \vert \rightarrow \infty } \vert y \vert ^{n-\alpha }u_{x,\bar{\lambda }(x)}(y)= \bigl( \bar{\lambda }(x) \bigr)^{n-\alpha }u(x)< +\infty , \\& \lim_{ \vert y \vert \rightarrow \infty } \vert y \vert ^{n-\alpha }v(y)=\lim _{ \vert y \vert \rightarrow \infty } \vert y \vert ^{n-\alpha }v_{x,\bar{\lambda }(x)}(y)= \bigl( \bar{\lambda }(x) \bigr)^{n-\alpha }v(x)< +\infty , \end{aligned}$$

which contradicts (2.52).

This finishes the proof of Lemma 2.6. □

In the following two subsections, we carry out the proof of Theorem 1.1 by discussing the critical cases and subcritical cases separately.

2.2 Classification of positive solutions in the critical case $c_{1}(\frac{2n-\sigma }{n-\alpha }-p_{1})+c_{2}(\frac{n+\alpha -\sigma }{n-\alpha }-p_{2}) +c_{3}(\frac{2n-\sigma }{n-\alpha }-q_{1})+c_{4}(\frac{n+\alpha -\sigma }{n-\alpha }-q_{2})=0$

Without loss of generality, we may assume that $c_{1}>$, $c_{2}>0$, $c_{3}>0$, $c_{4}>0$, that is, $p_{1}=\frac{2n-\sigma }{n-\sigma }$, $p_{2}=\frac{n+\alpha -\sigma }{n-\alpha }$, $q_{1}=\frac{2n-\sigma }{n-\alpha }$, and $q_{2}=\frac{n+\alpha -\sigma }{n-\alpha }$.

We carry out the proof by discussing two different possible cases.

Case (i). $\bar{\lambda }(x)=+\infty$ for all $x\in \mathbb{R}^{n}$. Therefore, for all $x\in \mathbb{R}^{n}$ and $0<\lambda <+\infty$, we have

$$ u_{x,\lambda }(y)\geq u(y),\quad\quad v_{x,\lambda }(y)\geq v(y),\quad \forall y \in B_{\lambda }(x)\setminus \{x\}, \forall 0< \lambda < +\infty . $$

By a calculus lemma (Lemma 11.2 in [32]), we must have $u\equiv d_{1}>0$, $v\equiv d_{2}>0$, which contradicts system (1.1).

Case (ii). By Case (i) and Lemma 2.6, we only need to consider the cases that

$$ \bar{\lambda }(x)< \infty \quad \text{for all } x\in \mathbb{R}^{n}. $$

From Lemma 2.5, we infer that

$$ u_{x,\bar{\lambda }(x)}(y)=u(y),\quad\quad v_{x,\bar{\lambda }(x)}(y)=v(y), \quad \forall y\in B_{\bar{\lambda }(x)}(x)\setminus \{x\}. $$

(2.53)

Since equation (1.1) is conformally invariant, from a calculus lemma (Lemma 11.1 in [32]) and (2.53), we deduce that there exist some $\mu >0$ and $x_{0}\in \mathbb{R}^{n}$ such that

$$ u(x)=C_{1} \biggl(\frac{\mu }{1+\mu ^{2} \vert x-x_{0} \vert ^{2}} \biggr)^{ \frac{n-\alpha }{2}},\quad \quad v(x)=C_{2} \biggl( \frac{\mu }{1+\mu ^{2} \vert x-x_{0} \vert ^{2}} \biggr)^{\frac{n-\alpha }{2}}, \quad \forall x\in \mathbb{R}^{n}, $$

where the constants $C_{1}$, $C_{2}$ depend on n, α, σ.

2.3 Nonexistence of positive solutions in the subcritical case $c_{1}(\frac{2n-\sigma }{n-\alpha }-p_{1})+c_{2}(\frac{n+\alpha -\sigma }{n-\alpha }-p_{2})+c_{3}(\frac{2n-\sigma }{n-\alpha }-q_{1})+c_{4}(\frac{n+\alpha -\sigma }{n-\alpha }-q_{2})>0$

Without loss of generality, we may assume that $c_{1}(\frac{2n-\sigma }{n-\alpha }-p_{1})\geq 0$, $c_{3}(\frac{2n-\sigma }{n-\alpha }-q_{1})\geq 0$ and $c_{2}(\frac{n+\alpha -\sigma }{n-\alpha }-p_{2})>0$, $c_{4}(\frac{n+\alpha -\sigma }{n-\alpha }-q_{2})>0$, that is, $c_{1},c_{3}\geq 0$, $c_{2},c_{4}>0$, $0< p_{1}\leq \frac{2n-\sigma }{n-\alpha }$, $0< p_{2}<\frac{n+\alpha -\sigma }{n-\alpha }$, $0< q_{1}\leq \frac{2n-\sigma }{n-\alpha }$, and $0< q_{2}<\frac{n+\alpha -\sigma }{n-\alpha }$. PDE system (1.1) involves at least one subcritical nonlinearity in such cases.

We will obtain a contradiction in both the following two different possible cases.

Case (i). $\bar{\lambda }(x)=+\infty$ for all $x\in \mathbb{R}^{n}$. Therefore, for all $x\in \mathbb{R}^{n}$ and $0<\lambda <+\infty$, we have

$$ u_{x,\lambda }(y)\geq u(y),\quad\quad v_{x,\lambda }(y)\geq v(y),\quad \forall y \in B_{\lambda }(x)\setminus \{x\}, \forall 0< \lambda < +\infty . $$

By a calculus lemma (Lemma 11.2 in [32]), we must have $u\equiv d_{1}>0$, $v\equiv d_{2}>0$, which contradicts equation (1.1).

Case (ii). By Case (i) and Lemma 2.6, we only need to consider the case that

$$ \bar{\lambda }(x)< \infty \quad \text{for all } x\in \mathbb{R}^{n}. $$

From Lemma 2.5, we infer that

$$ u_{x,\bar{\lambda }(x)}(y)=u(y),\quad\quad v_{x,\bar{\lambda }(x)}(y)=v(y), \quad \forall y\in B_{\bar{\lambda }(x)}(x)\setminus \{x\}. $$

(2.54)

Consider $x=0$, one can derive from (2.40) and (2.54) that

$$ \begin{aligned} 0&=U_{0,\bar{\lambda }}(y)= \int _{B_{\bar{\lambda }}(0)}K_{1, \bar{\lambda }}(y,z)P(z) \biggl( \biggl( \frac{\bar{\lambda }}{ \vert z \vert } \biggr)^{\tau _{2}}v^{p_{2}}_{0,\bar{\lambda }}(z)-v^{p_{2}}(z) \biggr)\,dz \\ &\quad {} + \int _{B_{\bar{\lambda }}(0)}K_{1,\bar{\lambda }}(y,z) \bigl(\bar{P}_{0, \bar{\lambda }}(z)-P(z) \bigr) \biggl(\frac{\bar{\lambda }}{ \vert z \vert } \biggr)^{ \tau _{2}}v^{p_{2}}_{0,\bar{\lambda }}(z) \,dz \\ &= \int _{B_{\bar{\lambda }}(0)}K_{1,\bar{\lambda }}(y,z)P(z) \biggl( \biggl( \frac{\bar{\lambda }}{ \vert z \vert } \biggr)^{\tau _{2}}-1 \biggr)v^{p_{2}}(z)\,dz, \end{aligned} $$

(2.55)

where

$$ \bar{P}_{0,\bar{\lambda }}(z)-P(z)= \int _{B_{\bar{\lambda }}(0)}K_{2, \bar{\lambda }}(z,\xi ) \bigl(v^{p_{1}}_{0,\bar{\lambda }}( \xi )-v^{p_{1}}( \xi ) \bigr)\,d\xi =0, $$

and $\tau _{2}=n+\alpha -p_{2}(n-\alpha )>0$. As a consequence, it follows immediately that

$$ 0\geq \int _{B_{\bar{\lambda }}(0)}K_{1,\bar{\lambda }}(y,z)P(z) \biggl( \biggl( \frac{\bar{\lambda }}{ \vert z \vert } \biggr)^{\tau _{2}}-1 \biggr)v^{p_{2}}(z)\,dz>0, $$

which is absurd.

Thus we have ruled out both Case (i) and Case (ii), and hence system (1.1) does not admit any positive solutions. Therefore, the unique nonnegative solution to (1.1) is $(u,v)\equiv (0,0)$.

This concludes our proof of Theorem 1.1.