Non-existence for a semi-linear fractional system with Sobolev exponents via direct method of moving spheres

Abstract

The aim of this article is to consider the semi-linear fractional system with Sobolev exponents $q = \frac{{n + \alpha}}{{n - \beta}}$ and $p = \frac{{n + \beta}}{{n - \alpha}}$ ($\alpha \ne\beta$):

$$\left \{ \textstyle\begin{array}{l} {( - \Delta)^{\alpha/2}}u(x) = k(x){v^{q}}(x) + f(v(x)),\\ {( - \Delta)^{\beta/2}}v(x) = j(x){u^{p}}(x) + g(u(x)), \end{array}\displaystyle \right . $$

where $0 < \alpha, \beta < 2$. We first establish two maximum principles for narrow regions in the ball and out of the ball by the iteration technique, respectively. Based on these principles, we use the direct method of moving spheres to prove the non-existence of positive solutions to the above system in the whole space and bounded star-shaped domain. As a consequence, the monotonic decreasing properties of $W(x) = { \vert x \vert ^{\frac{{n - \alpha}}{2}}}u(x)$ and ${W_{1}}(x) = { \vert x \vert ^{\frac{{n - \beta}}{2}}}v(x)$ along the radial direction in the whole space are obtained.

1 Introduction

In recent years, the fractional Laplacian has been frequently used in the simulation of various physical phenomena, such as anomalous diffusions and quasi-geotropic flows, turbulence and water waves, molecular dynamics, and relativistic quantum mechanics (see [1–4]). Typically, the standard linear evolution equation involving fractional Laplacian is

$$\frac{{\partial u}}{{\partial t}} + {( - \Delta)^{\alpha/2}}u = 0, $$

which is a model of anomalous diffusion and has been widely used in physics, probability, and finance [5–7]. The fractional Laplacian can be also used to handle Lévy flight, such as the so-called SIS flow progress [8, 9], to derive the well-known Kolmogorov–Fisher equation. At present, the study on the fractional Laplace equations has made many important achievements, including the blow up of positive solutions, Liouville theorems, a priori estimates, and so on [10, 11].

The fractional Laplacian in ${\mathbb{R}}^{n}$ is a nonlocal pseudo-differential operator, which is of the form

$$(-\Delta)^{\frac{\alpha}{2}}u(x)={C_{n, \alpha}}\mathit{P.V.} \int_{{\mathbb {R}}^{n}}{\frac{u(x)-u(y)}{ \vert x-y \vert ^{n+\alpha}}}\,dy, $$

where $0<\alpha<2$ and $\mathit{P.V.}$ stands for the Cauchy principal value. Letting

$$L_{\alpha}= \biggl\{ u:{\mathbb{R}}^{n}\rightarrow\mathbb{R} \Big| \int _{{\mathbb{R}}^{n}}{\frac{ \vert u(x) \vert }{ \vert x-y \vert ^{n+\alpha}}dx< {\infty }} \biggr\} , $$

it is easy to see that, for $u\in{{L_{\alpha}}\cap{C_{\mathrm{loc}}^{1,1}}}$, the integral in the expression of $(-\Delta)^{\frac{\alpha}{2}}u(x)$ is well defined.

Different from the integer order Laplacian, the fractional Laplacian has the so-called nonlocality, which has brought new difficulties in the treatment to equations involving this kind of operators. To circumvent these difficulties, Caffarelli and Silvestre [12] introduced the extension method that reduced the nonlocal problem into a local one in higher dimensions. This method has been extensively applied in the fractional Laplace equations, and a series of results have been obtained (see [13, 14]). But by using this method one needs to require $1 \le\alpha < 2$. One can also consider the method of moving plane in integral forms to investigate the fractional Laplace equations, and this method has been proved to be a powerful tool (see [15–20]). However, for some equations including fully nonlinear nonlocal operators, neither the extension method nor the method of moving plane in integral forms can be applied [21, 22], and this inspires one to use the direct method of moving plane to inspect these equations. Lately, some success on the direct approach was achieved by Jorohs and Weth in [23], where they raised antisymmetric maximum principles and applied them to show the symmetry of solutions to nonlocal problems by using the direct method of moving planes. It needs to note that their maximum principles are only suitable for bounded regions. Chen, Li, and Li [24] have developed a systematic method of moving plane to nonlocal problems either on bounded or unbounded domains. At the same time, Chen, Li, and Zhang [25], Li and Zhu [26] also used the direct method of moving spheres to the fractional Laplace equations, which is sometimes more convenient than the moving planes in some problems. It is noted that in the method of moving planes, one moves parallel planes along a chosen direction to the limiting position to derive symmetry of the solutions about the limiting plane, while in the method of moving spheres, one fixes a center and increases or decreases the radius of the spheres to conclude monotonicity or symmetry of the solutions along the radial directions of the spheres.

In this paper, we use the direct method of moving spheres to establish non-existence of positive solutions to the following semi-linear fractional Laplace system with Sobolev exponents q and p:

$$ \textstyle\begin{cases} {( - \Delta)^{\alpha/2}}u(x) = k(x){v^{q}}(x) + f(v(x)),\\ {( - \Delta)^{\beta/2}}v(x) = j(x){u^{p}}(x) + g(u(x)) \end{cases} $$

(1.1)

in the whole space $R{^{n}}$ and the bounded star-shaped domain in $R{^{n}}$, where

$$q = \frac{{n + \alpha}}{{n - \beta}},\qquad p = \frac{{n + \beta}}{{n - \alpha}} \quad(0 < \alpha,\beta < 2, \alpha \ne \beta). $$

Recall that an open set Ω is called star-shaped with respect to origin provided that, for each $x \in\overline{\varOmega}$, the line segment $\{ \lambda x \vert {0 \le\lambda \le1} \}$ lies in Ω̅.

The elliptic equation with Sobolev exponent $\frac{{n + 2}}{{n - 2}}$

$$ - \Delta u(x) = k(x){u^{\frac{{n + 2}}{{n - 2}}}}(x),\quad x \in R{^{n}} $$

(1.2)

has been extensively researched [27–36]. As we all know, the properties of solutions to (1.2) closely rely on the coefficient $k(x)$. Ding-Ni in [37] derived the properties of solutions to (1.2) when $k(x) = 1 - \eta$, $k(x) = 1$, and $k(x) = 1 + \eta$ respectively, where $\eta(x)$ is a rotational symmetric function about the region. Lin in [38] proved non-existence of smooth positive solutions to the semi-linear elliptic equation with Sobolev exponent

$$ \Delta u(x) + k(x){u^{\frac{{n + 2}}{{n - 2}}}} + f(u) = 0,\quad x \in R{^{n}}, $$

(1.3)

where it is required that

• (1) $k(x)$ is a positive ${C^{1}}$ function with $k(\infty) = \lim_{ \vert x \vert \to\infty} k(x) > 0$, and $k(x) \ne$ constant is non-decreasing along each ray $\{ t\xi \vert {t \ge0} \}$ for any unit vector $\xi \in R{^{n}}$;

• (2) if $t > 0$, then $f(t)$ is a positive ${C^{1}}$ function with $\lim_{ \vert t \vert \to\infty} f(t){t^{ - \frac{{n + 2}}{{n - 2}}}} = 0$, and if $p = \frac{{n + \alpha}}{{n - \alpha}}$, then $\frac {{f(r)}}{{{r^{p}}}}$ is non-increasing about r.

The conclusions to semi-linear Laplace equations have been extended to fractional Laplace equations. Chen, Li, and Zhang in [25] discussed the fractional describing curvature equation

$${( - \Delta)^{\alpha/2}}u(x) = Q(x){u^{p}}(x),\quad x \in{R^{n}}, $$

where $p = \frac{{n + \alpha}}{{n - \alpha}}$. Via the direct method of moving plane, the Liouville theorem to the fractional Lane–Emden system

$$\textstyle\begin{cases} {( - \Delta)^{\alpha/2}}u(x) = {v^{p}}(x),\\ {( - \Delta)^{\alpha/2}}v(x) = {u^{q}}(x), \end{cases} $$

where $0 < \alpha < 2$, $1 < p, q \le\frac{{n + \alpha}}{{n - \alpha }}$, was obtained by Cai and Mei [39]. Using the method of moving plane in integral forms, Dou and Zhou in [40] proved the Liouville theorem of the positive solutions to the following fractional Henon system in ${R^{n}}$:

$$\textstyle\begin{cases} {( - \Delta)^{\alpha/2}}u(x) = { \vert x \vert ^{\beta}}{v^{p}}(x),\\ {( - \Delta)^{\alpha/2}}v(x) = { \vert x \vert ^{\gamma}}{u^{q}}(x), \end{cases} $$

where $0 < \alpha < 2$, $\beta, \gamma > 0$, $1 \le p, q < \infty$.

In this paper, we apply the direct method of moving sphere to obtain non-existence of (1.1). Before describing the main results, let us state some assumptions that are more general than those in the papers mentioned above.

In $R{^{n}}$, we assume that

• (i) $f(v(x))$ and $g(u(x))$ are locally bounded positive functions in x, and for $r \ge0$, $f(r)$ and $g(r)$ are non-decreasing in r;

• (ii) $\frac{{f(r)}}{{{r^{q}}}}$ and $\frac{{g(r)}}{{{r^{p}}}}$ are non-increasing in r;

• (iii) $k(x)$ and $j(x)$ are locally bounded positive functions and non-decreasing along each ray $\{ t\xi \vert {t \ge0} \}$ for any unit vector $\xi \in R{^{n}}$.

Theorem 1.1

Assume that (i), (ii), and (iii) are satisfied. If$u \in{L_{\alpha}} \cap C_{\mathrm{loc}}^{1,1}$, $v \in{L_{\beta}} \cap C_{\mathrm{loc}}^{1,1}$, then system (1.1) has no positive solutions, unless$k(x)$and$j(x)$are constants.

Theorem 1.1 has the following consequence.

Corollary 1.2

Assume that the conditions of Theorem1.1are valid, and let

$$W(x) = { \vert x \vert ^{\frac{{n - \alpha}}{2}}}u(x),\qquad {W_{1}}(x) = { \vert x \vert ^{\frac{{n - \beta}}{2}}}v(x), $$

then$W(x)$and${W_{1}}(x)$are monotonically decreasing along the ray$\{ t\xi \vert {t \ge0} \}$ ($\vert \xi \vert = 1$).

For the bounded star-shaped domain about origin, we need that

• (iv) $f(v(x))$and$g(u(x))$are locally bounded positive functions inx, and for$r \ge0$, $f(r)$and$g(r)$are non-increasing inr;

• (v) $\frac{{f(r)}}{{{r^{q}}}}$and$\frac{{g(r)}}{{{r^{p}}}}$are non-decreasing inr;

• (vi) $k(x)$and$j(x)$are locally bounded positive functions and are non-increasing along each ray$\{ t\xi \vert {t \ge0} \}$for any unit vector$\xi \in R{^{n}}$.

Theorem 1.3

Let$\varOmega \subset R{^{n}}$ ($n \ge3$) be a bounded star-shaped domain about the origin and assume that (iv), (v), and (vi) are satisfied. If$u \in{L_{\alpha}} \cap C_{\mathrm{loc}}^{1,1}$and$v \in{L_{\beta}} \cap C_{\mathrm{loc}}^{1,1}$, then system (1.1) has no positive solution inΩ, unless$k(x)$and$j(x)$are constants.

The paper is organized as follows: In Sect. 2, we offer two maximum principles for narrow regions in the ball and out of the ball, respectively, which play important roles in the proofs of main results. It is worth stressing that the iterating technique in the proofs of maximum principles is employed since (1.1) involves two equations. Section 3 is devoted to proving Theorem 1.1 by the Kelvin transform, the maximum principle for the narrow region in the ball, and the direct method of moving spheres. The process in the proof of Theorem 1.1 leads immediately to Corollary 1.2. In Sect. 4, we prove Theorem 1.3 by the maximum principle for the narrow region out of the ball and the direct method of moving spheres.

Throughout the paper, we denote by c a positive constant depending on n, α, and β, which can be different from line to line.

2 Two narrow region principles

Denote by ${B_{\lambda}}(0)$ ($\lambda > 0$) a ball centered at the origin and radius λ, and for $u(x)$ and $v(x)$ satisfying (1.1), let

$$\begin{gathered} {x^{\lambda}} = \frac{{{\lambda^{2}}x}}{{{{ \vert x \vert }^{2}}}}, \\ {u_{\lambda}}(x) = { \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha }}u \bigl({x^{\lambda}} \bigr), \\ {v_{\lambda}}(x) = { \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \beta }}v \bigl({x^{\lambda}} \bigr).\end{gathered} $$

Denote

$$\begin{gathered} U(x) = u(x) - {u_{\lambda}}(x), \\ V(x) = v(x) - {v_{\lambda}}(x),\end{gathered} $$

and

$$\begin{gathered} {U_{\lambda}}(x) = {u_{\lambda}}(x) - u(x), \\ {V_{\lambda}}(x) = {v_{\lambda}}(x) - v(x).\end{gathered} $$

Then

$$\begin{aligned} {u_{\lambda}} \bigl({x^{\lambda}} \bigr) &= { \biggl( \frac{\lambda}{{ \vert {{x^{\lambda}}} \vert }} \biggr)^{n - \alpha}}u \bigl[{ \bigl({x^{\lambda}} \bigr)^{\lambda}} \bigr] \\ &= { \biggl(\frac{\lambda}{{ \vert {\frac{{{\lambda^{2}}x}}{{{{ \vert x \vert }^{2}}}}} \vert }} \biggr)^{n - \alpha}}u \biggl[ \frac{{{\lambda^{2}}(\frac {{{\lambda^{2}}x}}{{{{ \vert x \vert }^{2}}}})}}{{{{ \vert {\frac {{{\lambda^{2}}x}}{{{{ \vert x \vert }^{2}}}}} \vert }^{2}}}} \biggr] \\ &= { \biggl(\frac{{ \vert x \vert }}{\lambda} \biggr)^{n - \alpha}}u(x), \end{aligned} $$

and hence,

$$\begin{aligned} {U_{\lambda}}(x) &= {u_{\lambda}}(x) - u(x) \\ &= { \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha}}u \bigl({x^{\lambda}} \bigr) - u(x) \\ &= { \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha}}u \bigl({x^{\lambda}} \bigr) - { \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha}} {u_{\lambda}} \bigl({x^{\lambda}} \bigr) \\ &= { \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha }} \bigl[u \bigl({x^{\lambda}} \bigr) - {u_{\lambda}} \bigl({x^{\lambda}} \bigr) \bigr] \\ &= { \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha}}U \bigl({x^{\lambda}} \bigr). \end{aligned} $$

Similarly,

$${v_{\lambda}} \bigl({x^{\lambda}} \bigr)={ \biggl(\frac{{ \vert x \vert }}{\beta} \biggr)^{n - \alpha}}v(x),\qquad{V_{\lambda}}(x) = { \biggl( \frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \beta}}V \bigl({x^{\lambda}} \bigr). $$

Lemma 2.1

([25, Theorem 2.2])

Let$w \in{L_{\alpha}} \cap C_{\mathrm{loc}}^{1,1}(\varOmega)$be lower semi-continuous onΩ̅. If$c(x) < 0$is bounded from below inΩand

$$\left \{ \textstyle\begin{array}{l@{\quad}l} {( - \Delta)^{\alpha/2}}w(x) + c(x)w(x) \ge0, &x \in\varOmega \subset {B_{\lambda}}(0),\\ w(x) \ge0, & x \in{B_{\lambda}}(0)\setminus\varOmega,\\ w(x) = - {w_{\lambda}}(x), & x \in{B_{\lambda}}(0), \end{array}\displaystyle \right . $$

then there exists some sufficiently small$\delta > 0$such that if

$$\varOmega \subset{A_{\lambda - \delta,\lambda}}(0) \equiv \bigl\{ x \in R{^{n}}| { \lambda - \delta < \vert x \vert < \lambda} \bigr\} , $$

then

$$\inf_{x \in\varOmega} w(x) \ge0 . $$

Furthermore, if$w(x) = 0$for some$x \in\varOmega$, then$w(x) = 0$for almost every$x \in R{^{n}}$.

Lemma 2.2

Fix$\lambda > 0$small, we have, for$0 < \delta \ll\lambda$,

$${U_{\lambda}}(x) > 1,\qquad {V_{\lambda}}(x) > 1,\quad x \in{B_{\delta}}(0) \setminus \{ 0\}. $$

Proof

For $\vert x \vert$ large, there exists a positive constant c such that

$$\begin{aligned} u(x) &= \int_{{R^{n}}} {\frac{{k(x){v^{q}}(x) + f(v(x))}}{{{{ \vert {x - y} \vert }^{n - \alpha}}}}} \,dy \\ &\ge \int_{{B_{1}}(0)} {\frac{{k(x){v^{q}}(x) + f(v(x))}}{{{{ \vert {x - y} \vert }^{n - \alpha}}}}} \,dy \\ &\ge \int_{{B_{1}}(0)} {\frac{{{c}}}{{{{ \vert {x - y} \vert }^{n - \alpha}}}}} \,dy \\ &\sim{\frac{c}{{{{ \vert x \vert }^{n - \alpha}}}}}. \end{aligned} $$

For $0 < \delta \ll\lambda$ and $x \in{B_{\delta}}(0)\setminus\{ 0\}$, it is easy to see that $\vert {{x^{\lambda}}} \vert$ takes large values and hence

$${u_{\lambda}}(x) = { \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha }}u \bigl({x^{\lambda}} \bigr) \ge{ \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha}} \frac{c}{{{{ \vert {{x^{\lambda}}} \vert }^{n - \alpha}}}} = \frac{1}{{{\lambda^{n - \alpha}}}}. $$

Choosing λ sufficiently small, it holds

$${U_{\lambda}}(x) = {u_{\lambda}}(x) - u(x) \ge\frac{1}{{{\lambda^{n - \alpha}}}} - c > 1. $$

Similarly, we can prove that ${V_{\lambda}}(x) > 1$ and the proof of Lemma 2.2 is completed. □

For a bounded narrow region Ω in ${B_{\lambda}}(0)$, we have the following.

Lemma 2.3

Let$\varOmega \subset{A_{\lambda - \delta,\lambda}}(0) \equiv\{ x \in R{^{n}} \vert {\lambda - \delta < \vert x \vert < \lambda} \}$for$0\delta>0$sufficiently small. Assume that$U \in {L_{\alpha}} \cap C_{\mathrm{loc}}^{1,1}(\varOmega)$and$V \in{L_{\beta}} \cap C_{\mathrm{loc}}^{1,1}(\varOmega)$are lower semi-continuous onΩ̅. If${c_{i}}(x) < 0$ ($i = 1, 2$) is bounded from below onΩ̅and

$$ \textstyle\begin{cases} {( - \Delta)^{\alpha/2}}U(x) + {c_{1}}(x)V(x) \ge0,&\\ {( - \Delta)^{\beta/2}}V(x) + {c_{2}}(x)U(x) \ge0,& x \in\varOmega \subset{B_{\lambda}}(0),\\ U(x),V(x) \ge0, & x \in{B_{\lambda}}(0)\setminus\varOmega,\\ U(x) = - {U_{\lambda}}(x),V(x) = - {V_{\lambda}}(x),& x \in{B_{\lambda}}(0), \end{cases} $$

(2.1)

then

$$ U(x),V(x) \ge0,\quad x \in\varOmega. $$

(2.2)

Furthermore, if$U(x)$or$V(x)$equals to 0 at some point inΩ, then

$$ U(x) \equiv V(x) \equiv0,\quad x \in R{^{n}}. $$

(2.3)

Proof

We use the contradiction. If $U(x) \ge0$ ($x \in\varOmega$) in (2.2) does not hold, then from the lower semi-continuity of $U(x)$ on Ω̅, there exists some $\widetilde{x} \in \overline{\varOmega}$ such that

$$U(\widetilde{x}) = \min_{x \in\overline{\varOmega}} U(x) < 0. $$

Due to $U(x) \ge0$ ($x \in{B_{\lambda}}(0)\setminus\varOmega$) in (2.1), it implies $U(x) \ge0$ ($x \in\partial\varOmega$), so x̃ is in the interior of Ω.

From the proof process in Lemma 2.2, we have

$$ {( - \Delta)^{\alpha/2}}U(\widetilde{x}) \le \frac{c}{{\delta _{1}^{\alpha}}}U( \widetilde{x}). $$

(2.4)

Combining it with the first inequality of (2.1), it follows that $\frac{{cU(\widetilde{x})}}{{\delta_{1}^{\alpha}}} + {c_{1}}(\widetilde{x})V(\widetilde{x}) \ge0$, and by ${c_{1}}(\widetilde{x}) < 0$, we have

$$ V(\widetilde{x}) \le - \frac{{cU(\widetilde{x})}}{{\delta_{1}^{\alpha}{c_{1}}(\widetilde{x})}} < 0. $$

(2.5)

Using (2.5) and the lower semi-continuity of $V(x)$, there exists some x̂ such that

$$V(\widehat{x}) = \min_{x \in\varOmega} V(x) < 0. $$

Similar to the process of obtaining (2.4), we have

$$ {( - \Delta)^{\beta/2}}V(\widehat{x}) \le\frac{c}{{\delta_{2}^{\beta}}}V( \widehat{x}) < 0. $$

(2.6)

For $\delta < \min\{ {\delta_{1}},{\delta_{2}}\}$ sufficiently small, from the second inequality of (2.1), (2.6), $U(\widehat{x}) \ge U(\tilde{x})$, (2.5), and $V(\widehat{x}) < 0$, we have

$$\begin{aligned} 0 \le{}& {( - \Delta)^{\beta/2}}V(\widehat{x}) + {c_{2}}(\widehat{x})U(\widehat{x}) \\ \le{}& \frac{c}{{\delta_{2}^{\beta}}}V(\widehat{x}) + {c_{2}}(\widehat{x})U{{(}} \widetilde{x}{{)}} \\ \le{}& \frac{{cV(\widehat{x})}}{{\delta_{2}^{\beta}}} - \frac{{\delta _{1}^{\alpha}{c_{1}}(\widetilde{x}){c_{2}}(\widehat{x})V{{(}}\widetilde{x}{{)}}}}{c} \\ \le{}& \frac{{cV(\widehat{x})}}{{\delta_{2}^{\beta}}} \bigl(1 - {c_{1}}(\widetilde{x}){c_{2}}(\widehat{x}){\delta^{\alpha + \beta}} \bigr) \\ < {}& 0. \end{aligned}$$

It is a contradiction, and hence $U(x) \ge0$ ($x \in\varOmega$). Similarly, $V(x) \ge0$ ($x \in\varOmega$), so (2.2) is proved.

To prove (2.3), we also apply the contradiction and assume that there exists some ${\eta^{\prime}} \in\varOmega$ such that

$$V \bigl({\eta^{\prime}} \bigr) = 0. $$

The proof process for Lemma 2.1 (see [25]) shows

$$ {( - \Delta)^{\beta/2}}V \bigl({\eta^{\prime}} \bigr) = \int_{{B_{\lambda}}(0)} { \biggl(\frac{1}{{{{ \vert {\frac{{ \vert z \vert {\eta^{\prime}}}}{\lambda } - \frac{{\lambda z}}{{ \vert z \vert }}} \vert }^{n + \beta }}}}} - \frac{1}{{{{ \vert {{\eta^{\prime}} - z} \vert }^{n + \beta}}}} \biggr)V(z)\,dz. $$

(2.7)

Because for $z \in{B_{\lambda}}(0)$ it leads to

$${ \biggl\vert {\frac{{ \vert z \vert {\eta^{\prime}}}}{\lambda} - \frac {{\lambda z}}{{ \vert z \vert }}} \biggr\vert ^{2}} - { \bigl\vert {{\eta ^{\prime}} - z} \bigr\vert ^{2}} = \frac{{({{ \vert {{\eta^{\prime}}} \vert }^{2}} - {\lambda^{2}})({{ \vert z \vert }^{2}} - {\lambda^{2}})}}{{{\lambda ^{2}}}} > 0, $$

so

$$\frac{1}{{{{ \vert {\frac{{ \vert z \vert {\eta^{\prime}}}}{\lambda} - \frac{{\lambda z}}{{ \vert z \vert }}} \vert }^{n + \beta}}}} - \frac{1}{{{{ \vert {{\eta^{\prime}} - z} \vert }^{n + \beta}}}} < 0. $$

Hence, if $V(z)$ is not identically 0 in ${B_{\lambda}}(0)$, then (2.7) implies

$${( - \Delta)^{\beta/2}}V \bigl({\eta^{\prime}} \bigr) < 0. $$

Combining it with the second inequality of (2.1), it follows ${c_{2}}({\eta^{\prime}})U({\eta^{\prime}}) > 0$ and

$$U \bigl({\eta^{\prime}} \bigr) < 0, $$

since ${c_{2}}({\eta^{\prime}}) < 0$. This is a contradiction to (2.2) and then $V(x)$ is identically 0 in ${B_{\lambda}}(0)$. Thanks to

$$V(x) = - {V_{\lambda}}(x),\quad x \in{B_{\lambda}}(0), $$

it gives

$$ V(x) \equiv0, \quad x \in{R^{n}}. $$

(2.8)

From the second inequality of (2.1) and ${c_{2}}(x) < 0$, we know

$$U(x) \le0,\quad x \in\varOmega. $$

But noting

$$U(x) \ge0,\quad x \in\varOmega, $$

it implies

$$U(x) = 0,\quad x \in\varOmega. $$

Similar to the process of obtaining (2.8), we attain

$$U(x) = 0,\quad x \in R{^{n}}. $$

Now (2.3) is proved.

Similarly, if we first assume that there exists some ${\eta^{\prime}} \in \varOmega$ such that $U({\eta^{\prime}}) = 0$, then one can also check (2.3) following the previous proof. Now Lemma 2.3 is proved. □

For a bounded narrow region Ω out of ${B_{\lambda}}(0)$, we have the following.

Lemma 2.4

Let$\varOmega \subset{A_{\lambda - \delta,\lambda}}(0) \equiv\{ x \in{R^{n}} \vert {\lambda - \delta < \vert x \vert < \lambda} \}$for$0 < \delta < \lambda$sufficiently small. Assume that$U \in{L_{\alpha}} \cap C_{\mathrm{loc}}^{1,1}(\varOmega)$and$V \in {L_{\beta}} \cap C_{\mathrm{loc}}^{1,1}(\varOmega)$are lower semi-continuous onΩ̅. If${c_{i}}(x) < 0$ ($i = 1, 2$) is bounded from below onΩ̅and

$$ \textstyle\begin{cases} {( - \Delta)^{\alpha/2}}U(x) + {c_{3}}(x)V(x) \ge0,&\\ {( - \Delta)^{\beta/2}}V(x) + {c_{4}}(x)U(x) \ge0,&x \in\varOmega \subset R{^{n}}\setminus{B_{\lambda}}(0),\\ U(x),V(x) \ge0,&x \in R{^{n}}\setminus({B_{\lambda}}(0) \cup\varOmega),\\ U(x) = - {U_{\lambda}}(x),V(x) = - {V_{\lambda}}(x),&x \in R{^{n}}\setminus {B_{\lambda}}(0), \end{cases} $$

(2.9)

then

$$ U(x),V(x) \ge0, \quad x \in\varOmega. $$

(2.10)

Furthermore, if$U(x)$or$V(x)$equals to 0 at some point inΩ, then

$$ U(x) = V(x) = 0,\quad x \in R{^{n}}. $$

(2.11)

Proof: We note that $U(x) = - {U_{\lambda}}(x)$ and $V(x) = - {V_{\lambda}}(x)$, since $U(x)$ and $V(x)$ satisfy the anti-symmetry. If $U(x) \ge 0$ ($x \in\varOmega$) in (2.10) does not hold, then from the lower semi-continuity of $U(x)$ on Ω̅, there exists some $\bar{x} \in\overline{\varOmega}$ such that

$$ U(\bar{x}) = \min_{x \in\overline{\varOmega}} U(x) < 0. $$

(2.12)

By virtue of the third inequality of (2.9), we see further that x̄ is in the interior of Ω.

Let $\tilde{U}(x) = U(x) - U(\bar{x})$, then $\tilde{U}(\bar{x}) = 0$ and

$$ {( - \Delta)^{\alpha/2}}\tilde{U}(x) = {( - \Delta)^{\alpha/2}}U(x). $$

(2.13)

It follows from the anti-symmetry of $U(x)$ that

$${U_{\lambda}}(x) = { \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha }}U \bigl({x^{\lambda}} \bigr) = - U(x), $$

then

$$U \bigl({x^{\lambda}} \bigr) = - { \biggl(\frac{{ \vert x \vert }}{\lambda} \biggr)^{n - \alpha}}U(x) $$

and

$$\begin{aligned} {\biggl(\frac{\lambda}{{ \vert x \vert }}\biggr)^{n - \alpha}} \widetilde{U}\bigl({x^{\lambda}}\bigr)& = {\biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha }}\bigl(U\bigl({x^{\lambda}}\bigr) - U(\bar{x})\bigr) \\ &= {\biggl(\frac{\lambda}{{ \vert x \vert }}\biggr)^{n - \alpha}}\biggl[ - {\biggl( \frac {{ \vert x \vert }}{\lambda}\biggr)^{n - \alpha}}U(x) - U(\bar{x})\biggr] \\ &= - U(x) + U(\bar{x}) - U(\bar{x}) - {\biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha}}U(\bar{x}) \\ &= - \widetilde{U}(x) - \biggl[1 + {\biggl(\frac{\lambda}{{ \vert x \vert }}\biggr)^{n - \alpha}} \biggr]U(\bar{x}), \end{aligned} $$

i.e.,

$$ - \widetilde{U}(x) = { \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha}}\widetilde{U} \bigl({x^{\lambda}} \bigr) + \biggl[1 + { \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha}} \biggr]U(\bar{x}). $$

(2.14)

Together with $\widetilde{U}(\bar{x}) = 0$ and (2.14), we give

$$\begin{aligned} {( - \Delta)^{\alpha/2}}\widetilde{U}(\bar{x})={}& \int_{{R^{n}}} {\frac {{\widetilde{U}(\bar{x}) - \widetilde{U}(y)}}{{{{ \vert {\bar{x} - y} \vert }^{n + \alpha}}}}} \,dy \\ ={}& \int_{{B_{\lambda}}(0)} {\frac{{ - \widetilde{U}(y)}}{{{{ \vert {\bar{x} - y} \vert }^{n + \alpha}}}}} \,dy + \int_{{R^{n}}\setminus {B_{\lambda}}(0)} {\frac{{ - \widetilde{U}(y)}}{{{{ \vert {\bar{x} - y} \vert }^{n + \alpha}}}}} \,dy \\ ={}& \int_{{B_{\lambda}}(0)} {\frac{{{{(\frac{\lambda}{{ \vert y \vert }})}^{n - \alpha}}\widetilde{U}({y^{\lambda}})}}{{{{ \vert {\bar{x} - y} \vert }^{n + \alpha}}}}} \,dy + \int_{{B_{\lambda}}(0)} {\frac{{{{(1 + \frac{\lambda}{{ \vert y \vert }})}^{n - \alpha }}U(\bar{x})}}{{{{ \vert {\bar{x} - y} \vert }^{n + \alpha}}}}} \,dy \\ &+ \int_{{R^{n}}\setminus{B_{\lambda}}(0)} {\frac{{ - \widetilde{U}(y)}}{{{{ \vert {\bar{x} - y} \vert }^{n + \alpha}}}}} \,dy \\ : ={}& {J_{1}} + {J_{2}} + {J_{3}}. \end{aligned}$$

In ${J_{1}}$, letting $y = \frac{{{\lambda^{2}}z}}{{{{ \vert z \vert }^{2}}}}$, so $\vert y \vert = \frac{{{\lambda^{2}}}}{{ \vert z \vert }}$, ${y^{\lambda}} = \frac{{{\lambda^{2}}y}}{{{{ \vert y \vert }^{2}}}} = z$, and $\frac{\lambda}{{ \vert y \vert }} = \frac{\lambda}{{\frac{{{\lambda^{2}}}}{{ \vert z \vert }}}} = \frac{{ \vert z \vert }}{\lambda}$, it shows from $y \in {B_{\lambda}}(0)$ that $\vert z \vert > \lambda$, i.e., $z \in {R^{n}}\setminus{B_{\lambda}}(0)$, hence

$$\begin{aligned} {J_{1}} &= \int_{{B_{\lambda}}(0)} {\frac{{{{(\frac{\lambda}{{ \vert y \vert }})}^{n - \alpha}}\widetilde{U}({y^{\lambda}})}}{{{{ \vert {\bar{x} - y} \vert }^{n + \alpha}}}}} \,dy \\ &= \int_{{R^{n}}\setminus{B_{\lambda}}(0)} {\frac{{{{(\frac{{ \vert z \vert }}{\lambda})}^{n - \alpha}}\widetilde{U}(z)}}{{{{ \vert {\bar{x} - \frac{{{\lambda^{2}}z}}{{{{ \vert z \vert }^{2}}}}} \vert }^{n + \alpha}}}}} { \biggl(\frac{{{\lambda^{2}} \vert z \vert }}{{{{ \vert z \vert }^{2}}}} \biggr)^{n}}\,dz \\ &= \int_{{R^{n}}\setminus{B_{\lambda}}(0)} {\frac{{{{(\frac{{ \vert z \vert }}{\lambda})}^{n - \alpha}}\widetilde{U}(z)}}{{{{ \vert {\bar{x} - \frac{{{\lambda^{2}}z}}{{{{ \vert z \vert }^{2}}}}} \vert }^{n + \alpha}}}}} { \biggl(\frac{\lambda}{{ \vert z \vert }} \biggr)^{n - \alpha }} { \biggl(\frac{\lambda}{{ \vert z \vert }} \biggr)^{n + \alpha}}\,dz \\ &= \int_{{R^{n}}\setminus{B_{\lambda}}(0)} {\frac{{{1}}}{{{{ \vert {\frac{{\bar{x} \vert z \vert }}{\lambda} - \frac{{\lambda z}}{{ \vert z \vert }}} \vert }^{n + \alpha}}}}} \widetilde{U}(z)\,dz, \end{aligned}$$

(2.15)

and so

$${J_{1}} + {J_{3}} = \int_{{R^{n}}\setminus{B_{\lambda}}(0)} { \biggl(\frac {1}{{{{ \vert {\frac{{ \vert z \vert \bar{x}}}{\lambda} - \frac {{\lambda z}}{{ \vert z \vert }}} \vert }^{n + \alpha}}}}} - \frac{1}{{{{ \vert {\bar{x} - z} \vert }^{n + \alpha}}}} \biggr)\widetilde{U}(z)\,dz. $$

For $z \in R{^{n}}\setminus{B_{\lambda}}(0)$, it observes

$${ \biggl\vert {\frac{{ \vert z \vert \bar{x}}}{\lambda} - \frac {{\lambda z}}{{ \vert z \vert }}} \biggr\vert ^{2}} - \vert {\bar{x} - z} \vert = \frac{{({{ \vert {\bar{x}} \vert }^{2}} - {\lambda ^{2}})({{ \vert z \vert }^{2}} - {\lambda^{2}})}}{{{\lambda^{2}}}} > 0, $$

and then

$$ \frac{1}{{{{ \vert {\frac{{ \vert z \vert \bar{x}}}{\lambda} - \frac{{\lambda z}}{{ \vert z \vert }}} \vert }^{n + \alpha}}}} - \frac{1}{{{{ \vert {\bar{x} - z} \vert }^{n + \alpha}}}} < 0. $$

(2.16)

In Ω, we know by (2.12) that $\widetilde{U}(z) = U(z) - U(\bar{x}) \ge0$. In $R{^{n}}\setminus({B_{\lambda}}(0) \cup \varOmega)$, it follows from (2.9) that $\widetilde{U}(z) = U(z) - U(\bar{x}) \ge0$. Noting $R{^{n}}\setminus({B_{\lambda}}(0)) = [R{^{n}}\setminus({B_{\lambda}}(0) \cup\varOmega)] \cup\varOmega$, we obtain $\widetilde{U}(z) \ge0$ for every $z \in R{^{n}}\setminus ({B_{\lambda}}(0))$, and so ${J_{1}} + {J_{3}} \le0$. Using $U(\bar{x}) < 0$ and (2.13), it infers

$$ {( - \Delta)^{\alpha/2}}U(\bar{x}) = {J_{1}} + {J_{2}} + {J_{3}} \le {J_{2}} \le U(\bar{x}) \int_{{B_{\lambda}}(0)} {\frac{1}{{{{ \vert {\bar{x} - y} \vert }^{n + \alpha}}}}} \,dy. $$

(2.17)

Now we will estimate $\int_{{B_{\lambda}}(0)} {\frac{1}{{{{ \vert {\bar{x} - y} \vert }^{n + \alpha}}}}} \,dy$ in (2.17). Obviously, it follows from $\varOmega \subset {A_{\lambda - \delta ,\lambda}}(0) \equiv\{ x \in R{^{n}} \vert {\lambda - \delta < \vert x \vert < \lambda} \}$ that $\operatorname{dist}(\bar{x},\partial {B_{\lambda}}(0)) < \delta$, and for $\delta \le\lambda$,

$$\begin{aligned} & \int_{{B_{\lambda}}(0)} {\frac{1}{{{{ \vert {\bar{x} - y} \vert }^{n + \alpha}}}}} \,dy \\ &\quad\ge \int_{{B_{\lambda}}(0) \cap[{B_{2\lambda + \delta}}(\bar{x})\setminus{B_{\delta}}(\bar{x})]} {\frac{1}{{{{ \vert {\bar{x} - y} \vert }^{n + \alpha}}}}} \,dy \\ &\quad\ge\frac{{V({B_{\lambda/2}}(\bar{x}))}}{{V({B_{3\lambda}}(\bar{x}))}} \int_{{B_{2\lambda + \delta}}(\bar{x})\setminus{B_{\delta}}(\bar{x})} {\frac{1}{{{{ \vert {\bar{x} - y} \vert }^{n + \alpha}}}}} \,dy \\ &\quad\ge c \int_{{B_{2\lambda + \delta}}(\bar{x})\setminus{B_{\delta}}(\bar{x})} {\frac{1}{{{{ \vert {\bar{x} - y} \vert }^{n + \alpha }}}}} \,dy \\ &\quad= c \int_{\delta}^{2\lambda + \delta} { \int_{\partial{B_{s}}(\bar{x})} {\frac{1}{{{{ \vert {\bar{x} - y} \vert }^{n + \alpha}}}}\,ds} } \\ &\quad= c \int_{\delta}^{2\lambda + \delta} { \int_{\partial{B_{s}}(\bar{x})} {\frac{1}{{{s^{n + \alpha}}}}\,ds} } \\ &\quad=c \int_{\delta}^{2\lambda + \delta} {\frac{{{s^{n - 1}}}}{{{s^{n + \alpha}}}}} \,ds \\ &\quad=c \int_{\delta}^{2\lambda + \delta} {{s^{ - \alpha - 1}}} \,ds \\ &\quad=- c{s^{ - \alpha}}| {_{\delta}^{2\lambda + \delta}} \\ &\quad= c \biggl(\frac{1}{{{\delta^{\alpha}}}} - \frac{1}{{{{(2\lambda + \delta )}^{\alpha}}}} \biggr) \\ &\quad\ge c \biggl(\frac{1}{{{\delta^{\alpha}}}} - \frac{1}{{{{(3\delta )}^{\alpha}}}} \biggr) \\ &\quad= \frac{c}{{{\delta^{\alpha}}}}. \end{aligned}$$

Using it into (2.17) and recalling (2.12), we have

$$ {( - \Delta)^{\alpha/2}}U(\bar{x}) \le U(\bar{x}) \int_{{B_{\lambda}}(0)} {\frac{1}{{{{ \vert {\bar{x} - y} \vert }^{n + \alpha}}}}} \, dy \le\frac{c}{{{\delta^{n + \alpha}}}}U( \bar{x}). $$

(2.18)

Combining the first inequality in (2.9), it means

$$\frac{{c U(\bar{x})}}{{{\delta^{n + \alpha}}}} + {c_{3}}(\bar {x})V(\bar{x}) \ge0, $$

and from ${c_{3}}(\bar{x}) < 0$,

$$ V(\bar{x}) \le - \frac{{cU(\bar{x})}}{{{\delta^{n + \alpha }}{c_{3}}(\bar{x})}} < 0. $$

(2.19)

In terms of (2.19) and the lower semi-continuity of v, there exists some point ${x^{0}}$ such that

$$V \bigl({x^{0}} \bigr) = \min_{x \in\varOmega} V(x) < 0. $$

Similar to the process of obtaining (2.18), we have

$$ {( - \Delta)^{\beta/2}}V \bigl({x^{0}} \bigr) \le \frac{c}{{{\delta^{n + \beta }}}}V \bigl({x^{0}} \bigr) < 0. $$

(2.20)

By the second inequality in (2.9), (2.18), $U({x^{0}}) \ge U(\bar{x})$, (2.19), (2.20), and $V({x^{0}}) < 0$, it derives that, for δ sufficiently small,

$$\begin{aligned} 0 &\le{( - \Delta)^{\beta/2}}V \bigl({x^{0}}\bigr) + {c_{4}}\bigl({x^{0}}\bigr)U \bigl({x^{0}}\bigr) \\ &\le\frac{c}{{{\delta^{n + \beta}}}}V\bigl({x^{0}}\bigr) + {c_{4}} \bigl({x^{0}}\bigr)U{{(}}\bar{x} {{)}} \\ &\le\frac{{cV({x^{0}})}}{{{\delta^{n + \beta}}}} - \frac{{{\delta^{n + \alpha}}{c_{3}}(\bar{x}){c_{4}}({x^{0}})V{{(}}\bar{x}{{)}}}}{c} \\ &\le\frac{{cV({x^{0}})}}{{{\delta^{n + \beta}}}}\bigl(1 - {c_{3}}(\bar {x}){c_{4}} \bigl({x^{0}}\bigr){\delta^{\alpha + \beta + 2n}}\bigr) \\ &< 0. \end{aligned} $$

This is a contradiction and then $U(x) \ge0$ ($x \in\varOmega$). Similarly, we can verify $V(x) \ge0$ ($x \in\varOmega$), and (2.10) is proved.

To prove (2.11), assume that there exists some ${\xi^{\prime}} \in \varOmega$ such that

$$V \bigl({\xi^{\prime}} \bigr) = 0. $$

Letting $\xi = \frac{{{\lambda^{2}}z}}{{{{ \vert z \vert }^{2}}}}$, it gives by a similar way to (2.15) that

$$\begin{aligned} {( - \Delta)^{\beta/2}}V \bigl({\xi^{\prime}} \bigr) &= \int_{{R^{n}}} {\frac{{V({\xi ^{\prime}}) - V(\xi)}}{{{{ \vert {{\xi^{\prime}} - \xi} \vert }^{n + \beta }}}}} \,d\xi \\ &= \int_{{B_{\lambda}}(0)} {\frac{{ - V(\xi)}}{{{{ \vert {{\xi^{\prime}} - \xi} \vert }^{n + \beta}}}}} \,d\xi + \int_{{R^{n}}\setminus {B_{\lambda}}(0)} {\frac{{ - V(\xi)}}{{{{ \vert {{\xi^{\prime}} - \xi} \vert }^{n + \beta}}}}} \,d\xi \\ &= \int_{{B_{\lambda}}(0)} {\frac{{{{(\frac{\lambda}{{ \vert y \vert }})}^{n - \beta}}V({\xi^{\lambda}})}}{{{{ \vert {{\xi^{\prime}} - \xi} \vert }^{n + \beta}}}}} \,d\xi + \int_{{B_{\lambda}}(0)} {\frac {{{{(1 + \frac{\lambda}{{ \vert y \vert }})}^{n + \beta}}V({\xi ^{\prime}})}}{{{{ \vert {{\xi^{\prime}} - \xi} \vert }^{n + \beta}}}}} \,d\xi \\ &\quad+ \int_{{R^{n}}\setminus{B_{\lambda}}(0)} {\frac{{ - V(\xi )}}{{{{ \vert {{\xi^{\prime}} - \xi} \vert }^{n + \beta}}}}} \,d\xi. \end{aligned}$$

(2.21)

If $V(x)$ is not identically 0 in $R{^{n}}\setminus{B_{\lambda}}(0)$, then using (2.16) to (2.21) implies

$${( - \Delta)^{\beta/2}}V \bigl({\xi^{\prime}} \bigr) < 0. $$

Using the second inequality of (2.9) together with ${c_{4}}(x) < 0$, we gain

$$U \bigl({\xi^{\prime}} \bigr) < 0, $$

but it contradicts (2.10). Hence

$$ V(x) = 0,\quad x \in{R^{n}}\setminus{B_{\lambda}}(0). $$

(2.22)

Since $V(x)$ is symmetric in $R{^{n}}\setminus{B_{\lambda}}(0)$, i.e.,

$$V(x) = - V(x),\quad x \in R{^{n}}\setminus{B_{\lambda}}(0), $$

it shows

$$ V(x) = 0,\quad x \in R{^{n}}. $$

(2.23)

From the second inequality of (2.9) and ${c_{4}}(x) < 0$, we know

$$U(x) \le0,\quad x \in\varOmega. $$

Noting

$$U(x) \ge0,\quad x \in\varOmega, $$

we see

$$U(x) = 0,\quad x \in\varOmega. $$

Similar to (2.23), we achieve

$$U(x) = 0,\quad x \in R{^{n}}. $$

At this moment, (2.11) is proved. Similarly, if one assumes that there exists some point $\tilde{\xi}\in\varOmega$ such that $U(\tilde{\xi}) = 0$, we can also get (2.11). Lemma 2.4 is proved.

Lemma 2.3 will be used to (1.1) in the whole space $R{^{n}}$, while Lemma 2.4 to (1.1) in the bounded star-shaped domain.

3 Proof of Theorem 1.1

The following result is needed in the proof of Theorem 1.1.

Lemma 3.1

Let

$${\lambda_{0}} \equiv\sup \bigl\{ \lambda > 0| {{U_{{\lambda _{{0}}}}}(x) \ge0,} {V_{{\lambda_{{0}}}}}(x) \ge0, x \in {B_{\mu}}(0)\setminus \{ 0\} , 0 < \mu \le\lambda \bigr\} . $$

If${U_{{\lambda_{{0}}}}}(x) \ne{{0}}$or${V_{{\lambda_{{0}}}}}(x) \ne{{0}}$, $x \in{B_{{\lambda_{0}}}}(0)\setminus\{ 0\}$, then for$\varepsilon > 0$sufficiently small, there is a constantcsuch that

$$ {U_{{\lambda_{{0}}}}}(x) \ge c > 0,\qquad {V_{{\lambda_{{0}}}}}(x) \ge c > 0,\quad x \in{B_{\varepsilon}}(0)\setminus\{ 0\}. $$

(3.1)

Proof

Assuming

$${U_{{\lambda_{{0}}}}}(x) \ne{{0}},\quad x \in{B_{{\lambda _{0}}}}(0)\setminus\{ 0\}, $$

there exists some point ${y^{\prime\prime}} \in{B_{{\lambda_{0}}}}(0)\setminus\{ 0\}$ such that

$$ {U_{{\lambda_{{0}}}}} \bigl({y^{\prime\prime}} \bigr) > {{0}} {{.}} $$

(3.2)

Because of ${U_{{\lambda_{0}}}} \in C_{\mathrm{loc}}^{1,1}$, there exists small $\delta > 0$ such that

$${U_{{\lambda_{{0}}}}}(y) > {{0}},\quad x \in{B_{\delta}} \bigl({y^{\prime\prime}} \bigr). $$

The integral solution to equation

$${{{(}} - \Delta{{)}}^{\beta/2}}v(x) = j(x){u^{p}}(x) + g \bigl(u(x) \bigr),\quad x \in R{^{n}} $$

is of the form

$$v(x) = \int_{{R^{n}}} {\frac{1}{{{{ \vert {x - y} \vert }^{n - \beta }}}}} \bigl[j(x){u^{p}}(x) + g \bigl(u(x) \bigr) \bigr]\,dy. $$

With the similar way to treating ${J_{1}}$ in the proof of Lemma 2.2, we get

$$\begin{aligned} v(x)={}& \int_{{B_{{\lambda_{0}}}}(0)} {\frac{1}{{{{ \vert {x - y} \vert }^{n - \beta}}}}} \bigl[j(x){u^{p}}(x) + g \bigl(u(x) \bigr) \bigr]\,dy \\ &+ \int_{{R^{n}}\setminus{B_{{\lambda_{0}}}}(0)} {\frac{1}{{{{ \vert {x - y} \vert }^{n - \beta}}}}} \bigl[j(x){u^{p}}(x) + g \bigl(u(x) \bigr) \bigr]\,dy \\ ={}& \int_{{B_{{\lambda_{0}}}}(0)} {\frac{1}{{{{ \vert {x - y} \vert }^{n - \beta}}}}} \bigl[j(x){u^{p}}(x) + g \bigl(u(x) \bigr) \bigr]\,dy \\ & + \int_{{B_{{\lambda_{0}}}}(0)} {\frac{1}{{{{ \vert {x - \frac {{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}}} \vert }^{n - \beta }}}}} \biggl[j \biggl( \frac{{\lambda_{0}^{2}x}}{{{{ \vert x \vert }^{2}}}} \biggr){u^{p}} \biggl(\frac{{\lambda_{0}^{2}x}}{{{{ \vert x \vert }^{2}}}} \biggr) + g \biggl(u \biggl(\frac{{\lambda_{0}^{2}x}}{{{{ \vert x \vert }^{2}}}} \biggr) \biggr) \biggr]{ \biggl( \frac {{{\lambda_{0}}}}{{ \vert y \vert }} \biggr)^{2n}}\,dy \\ ={}& \int_{{B_{{\lambda_{0}}}}(0)} {\frac{1}{{{{ \vert {x - y} \vert }^{n - \beta}}}}} \bigl[j(x){u^{p}}(x) + g \bigl(u(x) \bigr) \bigr]\,dy \\ &+ \int_{{B_{{\lambda_{0}}}}(0)} {\frac{1}{{{{ \vert {\frac{{x \vert y \vert }}{{{\lambda_{0}}}} - \frac{{{\lambda_{0}}y}}{{ \vert y \vert }}} \vert }^{n - \beta}}}}} \biggl[j \biggl( \frac{{\lambda _{0}^{2}x}}{{{{ \vert x \vert }^{2}}}} \biggr){u^{p}} \biggl(\frac{{\lambda _{0}^{2}x}}{{{{ \vert x \vert }^{2}}}} \biggr) + g \biggl(u \biggl(\frac{{\lambda _{0}^{2}x}}{{{{ \vert x \vert }^{2}}}} \biggr) \biggr) \biggr]{ \biggl( \frac{{{\lambda_{0}}}}{{ \vert y \vert }} \biggr)^{n + \beta}}\,dy. \end{aligned}$$

It follows

$$\begin{aligned} {( - \Delta)^{\beta/2}} {v_{{\lambda_{0}}}}(x) &= { \biggl( \frac{{{\lambda _{0}}}}{{ \vert x \vert }} \biggr)^{n + \beta}} \bigl({( - \Delta)^{\beta /2}}v \bigr) \biggl( \frac{{\lambda_{0}^{2}x}}{{{{ \vert x \vert }^{2}}}} \biggr) \\ &= { \biggl(\frac{{{\lambda_{0}}}}{{ \vert x \vert }} \biggr)^{n + \beta }} \biggl[j \biggl( \frac{{\lambda_{0}^{2}x}}{{{{ \vert x \vert }^{2}}}} \biggr){u^{p}} \biggl(\frac {{\lambda_{0}^{2}x}}{{{{ \vert x \vert }^{2}}}} \biggr) + g \biggl(u \biggl(\frac{{\lambda _{0}^{2}x}}{{{{ \vert x \vert }^{2}}}} \biggr) \biggr) \biggr], \end{aligned}$$

and

$$\begin{aligned} {v_{{\lambda_{0}}}}(x)={}& \int_{{R^{n}}} {\frac{1}{{{{ \vert {x - y} \vert }^{n - \beta}}}}} { \biggl(\frac{{{\lambda_{0}}}}{{ \vert y \vert }} \biggr)^{n + \beta}} \biggl[j \biggl(\frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr){u^{p}} \biggl(\frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr) + g \biggl(u \biggl(\frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr) \biggr) \biggr]\,dy \\ = {}& \int_{{B_{{\lambda_{0}}}}(0)} {\frac{1}{{{{ \vert {x - y} \vert }^{n - \beta}}}}} { \biggl(\frac{{{\lambda_{0}}}}{{ \vert y \vert }} \biggr)^{n + \beta}} \biggl[j \biggl(\frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr){u^{p}} \biggl(\frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr) + g \biggl(u \biggl(\frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr) \biggr) \biggr]\,dy \\ &+ \int_{{R^{n}}\setminus{B_{{\lambda_{0}}}}(0)} {\frac {1}{{{{ \vert {x - y} \vert }^{n - \beta}}}}} { \biggl(\frac{{{\lambda _{0}}}}{{ \vert y \vert }} \biggr)^{n + \beta}} \biggl[j \biggl(\frac{{\lambda _{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr){u^{p}} \biggl(\frac{{\lambda _{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr) + g \biggl(u \biggl(\frac{{\lambda _{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr) \biggr) \biggr]\,dy \\ ={}& \int_{{B_{{\lambda_{0}}}}(0)} {\frac{1}{{{{ \vert {x - y} \vert }^{n - \beta}}}}} { \biggl(\frac{{{\lambda_{0}}}}{{ \vert y \vert }} \biggr)^{n + \beta}} \biggl[j \biggl(\frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr){u^{p}} \biggl(\frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr) + g \biggl(u \biggl(\frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr) \biggr) \biggr]\,dy \\ & + \int_{{B_{{\lambda_{0}}}}(0)} {\frac{1}{{{{ \vert {x - \frac {{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}}} \vert }^{n - \beta }}}}} { \biggl(\frac{{ \vert y \vert }}{{{\lambda_{0}}}} \biggr)^{n + \beta }} \bigl[j(y){u^{p}}(y) + g \bigl(u(y) \bigr) \bigr]{ \biggl(\frac{{{\lambda_{0}}}}{{ \vert y \vert }} \biggr)^{2n}}\,dy \\ = {}& \int_{{B_{{\lambda_{0}}}}(0)} {\frac{1}{{{{ \vert {x - y} \vert }^{n - \beta}}}}{{ \biggl(\frac{{{\lambda_{0}}}}{{ \vert y \vert }} \biggr)}^{n + \beta}}} \biggl[j \biggl(\frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr){u^{p}} \biggl(\frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr) + g \biggl(u \biggl( \frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr) \biggr) \biggr]\,dy \\ & + \int_{{B_{{\lambda_{0}}}}(0)} {\frac{1}{{{{ \vert {\frac{{x \vert y \vert }}{{{\lambda_{0}}}} - \frac{{{\lambda_{0}}y}}{{ \vert y \vert }}} \vert }^{n - \beta}}}}} \bigl[j(y){u^{p}}(y) + g \bigl(u(y) \bigr) \bigr]\,dy. \end{aligned}$$

Using ${u_{{\lambda_{0}}}}(y) = {(\frac{\lambda}{{ \vert y \vert }})^{n - \alpha}}u({y^{{\lambda_{0}}}})$, we know

$$\begin{gathered} u \biggl(\frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr) = u \bigl({y^{{\lambda _{0}}}} \bigr) = { \biggl( \frac{{ \vert y \vert }}{{{\lambda_{0}}}} \biggr)^{n - \alpha}} {u_{{\lambda_{0}}}}(y), \\ { \biggl(\frac{{{\lambda_{0}}}}{{ \vert y \vert }} \biggr)^{n + \beta}} {u^{p}} \biggl( \frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr) = { \biggl(\frac {{{\lambda_{0}}}}{{ \vert y \vert }} \biggr)^{n + \beta}} { \biggl[ { \biggl(\frac{{ \vert y \vert }}{{{\lambda_{0}}}} \biggr)^{n - \alpha}} \biggr]^{p}}u_{{\lambda_{0}}}^{p}(y) = u_{{\lambda_{0}}}^{p}(y)\end{gathered} $$

and

$$\begin{aligned} { \biggl(\frac{{{\lambda_{0}}}}{{ \vert y \vert }} \biggr)^{n + \beta}} g \biggl(u \biggl( \frac {{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr) \biggr) &= { \biggl(\frac{{{\lambda _{0}}}}{{ \vert y \vert }} \biggr)^{n + \beta}}g \biggl({ \biggl(\frac{{ \vert y \vert }}{{{\lambda_{0}}}} \biggr)^{n - \alpha}} {u_{{\lambda_{0}}}}(y) \biggr), \\ & = \frac{{g({{(\frac{{ \vert y \vert }}{{{\lambda_{0}}}})}^{n - \alpha}}{u_{{\lambda_{0}}}}(y))}}{{{{({{(\frac{{ \vert y \vert }}{{{\lambda_{0}}}})}^{n - \alpha}}u_{{\lambda _{0}}}^{p}(y))}^{p}}}}u_{{\lambda_{0}}}^{p}(y). \end{aligned}$$

It gives by $y \in{B_{{\lambda_{0}}}}(0)$ that $\frac{{{\lambda _{0}}}}{{ \vert y \vert }} \ge1$, $\frac{{ \vert y \vert }}{{{\lambda_{0}}}} \le1$, and $j(\frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}}) \ge j(y)$ from the monotonicity of $j(x)$ along each ray $t\xi \vert {t \ge0}$ for any unit vector $\xi \in R{^{n}}$. If $x \in{B_{\varepsilon}}(0)\setminus\{ 0\}$, $y \in{B_{\delta}}({y^{\prime\prime}})$, and $({B_{\varepsilon}}(0)\setminus\{ 0\} ) \cap ({B_{\delta}}({y^{\prime\prime}})) = \emptyset$, then

$${c_{11}} \le \vert {x - y} \vert < \biggl\vert {x - \frac{{\lambda _{0}^{2}y}}{{{{ \vert y \vert }^{2}}}}} \biggr\vert \le{c_{12}}, $$

and so

$$\frac{1}{{{{ \vert {x - y} \vert }^{n - \beta}}}} - \frac {1}{{{{ \vert {x - \frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}}} \vert }^{n - \beta}}}} \ge{c_{5}} > 0, $$

where $c_{11}$, $c_{12}$, and $c_{5}$ are positive constants. Using assumptions (i) and (ii), we have by (3.2) that

$$\begin{aligned} {V_{{\lambda_{0}}}}(x) ={}& {v_{{\lambda_{0}}}}(x) - v(x) \\ ={}& \int_{{B_{{\lambda_{0}}}}(0)} { \biggl(\frac{1}{{{{ \vert {x - y} \vert }^{n - \beta}}}} - } \frac{1}{{{{ \vert {x - \frac{{\lambda _{0}^{2}y}}{{{{ \vert y \vert }^{2}}}}} \vert }^{n - \beta}}}} \biggr) \\ &\cdot{{ \biggl\{ }} { \biggl(\frac{{{\lambda_{0}}}}{{ \vert y \vert }} \biggr)^{n + \beta}} \biggl[j \biggl(\frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr){u^{p}} \biggl( \frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr) + g \biggl(u \biggl(\frac{{\lambda_{0}^{2}y}}{{{{ \vert y \vert }^{2}}}} \biggr) \biggr) \biggr] - \bigl[j(y){u^{p}}(y) + g \bigl(u(y) \bigr) \bigr]{{ \biggr\} }}\,dy \\ \ge{}& \int_{{B_{{\lambda_{0}}}}(0)} { \biggl(\frac{1}{{{{ \vert {x - y} \vert }^{n - \beta}}}} - } \frac{1}{{{{ \vert {x - \frac{{\lambda _{0}^{2}y}}{{{{ \vert y \vert }^{2}}}}} \vert }^{n - \beta}}}} \biggr) \\ &\cdot \biggl[j(y)u_{{\lambda_{0}}}^{p}(y) - j(y){u^{p}}(y) + \frac{{g({{(\frac {{ \vert y \vert }}{{{\lambda_{0}}}})}^{n - \alpha}}{u_{{\lambda _{0}}}}(y))}}{{{{({{(\frac{{ \vert y \vert }}{{{\lambda_{0}}}})}^{n - \alpha}}{u_{{\lambda_{0}}}}(y))}^{p}}}}u_{{\lambda_{0}}}^{p}(y) - g \bigl(u(y) \bigr) \biggr]\, dy \\ \ge{}& \int_{{B_{{\lambda_{0}}}}(0)} { \biggl(\frac{1}{{{{ \vert {x - y} \vert }^{n - \beta}}}} - } \frac{1}{{{{ \vert {x - \frac{{\lambda _{0}^{2}y}}{{{{ \vert y \vert }^{2}}}}} \vert }^{n - \beta}}}} \biggr) \\ &\cdot \biggl[j(y)u_{{\lambda_{0}}}^{p}(y) - j(y){u^{p}}(y) + \frac {{g({u_{{\lambda_{0}}}}(y))}}{{{{({u_{{\lambda _{0}}}}(y))}^{p}}}}u_{{\lambda_{0}}}^{p}(y) - g \bigl(u(y) \bigr) \biggr] \,dy \\ \ge{}& \int_{{B_{{\lambda_{0}}}}(0)} { \biggl(\frac{1}{{{{ \vert {x - y} \vert }^{n - \beta}}}} - } \frac{1}{{{{ \vert {x - \frac{{\lambda _{0}^{2}y}}{{{{ \vert y \vert }^{2}}}}} \vert }^{n - \beta }}}} \biggr)j(y){u^{p - 1}}(y){U_{{\lambda_{0}}}}(y)\,dy \\ \ge{}& \int_{{B_{\delta}}({y^{\prime\prime}})} {{c_{{5}}} {c_{{6}}}} \,dy > c > 0. \end{aligned}$$

Then

$${V_{{\lambda_{0}}}}(x) > c > 0,\quad x \in{B_{\varepsilon}}(0)\setminus\{ 0\} $$

and

$${V_{{\lambda_{0}}}}(x) \ne0,\quad x \in{B_{{\lambda_{0}}}}(0)\setminus\{ 0\} . $$

Repeating a similar process from ${V_{{\lambda_{0}}}}(x) \ne0$, $x \in {B_{{\lambda_{0}}}}(0)\setminus\{ 0\}$, we have

$${U_{{\lambda_{0}}}}(x) > c > 0,\quad x \in{B_{\varepsilon}}(0)\setminus\{ 0\} , $$

and (3.1) is checked. □

Next, let us prove Theorem 1.1.

Proof of Theorem 1.1

Step 1. For $\lambda > 0$, let

$$B_{\lambda}^{-} (0) = \bigl\{ x \in{B_{\lambda}}(0)\setminus\{ 0\} | {{U_{\lambda}}(x) < 0\text{ or }{V_{\lambda}}(x) < 0} \bigr\} . $$

We will show that, for $\lambda > 0$ sufficiently small,

$$ {U_{\lambda}}(x),{V_{\lambda}}(x) \ge0,\quad x \in{B_{\lambda}}(0)\setminus \{ 0\}, $$

(3.3)

that is to say, for $\lambda > 0$ sufficiently small,

$$ B_{\lambda}^{-} (0) = \varPhi,\quad x \in{B_{\lambda}}(0) \setminus\{ 0\} . $$

(3.4)

To prove (3.4), we first know from Lemma 2.2 that, for $\lambda > 0$ sufficiently small, there exists $0 < \delta \ll \lambda$ such that

$${U_{\lambda}}(x),{V_{\lambda}}(x) \ge c > 0,\quad x \in{B_{\delta}}(0)\setminus\{ 0\}, $$

which shows

$$ B_{\lambda}^{-} (0) = \varPhi,\quad x \in{B_{\delta}}(0) \setminus\{ 0\}. $$

(3.5)

In the sequel, we will prove $B_{\lambda}^{-} (0) = \varPhi$, $x \in{B_{\delta}}(0)\setminus\{ 0\}$. A straightforward computation gives

$$\begin{aligned} {( - \Delta)^{\alpha/2}} {u_{\lambda}}(x) &= { \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n + \alpha}} \bigl({( - \Delta)^{\alpha/2}}u \bigr) \bigl({x^{\lambda}} \bigr) \\ &= { \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n + \alpha}} \biggl[k \biggl( \frac {{{\lambda^{2}}x}}{{{{ \vert x \vert }^{2}}}} \biggr){v^{q}} \bigl({x^{\lambda}} \bigr) + f \bigl(v \bigl({x^{\lambda}} \bigr) \bigr) \biggr]. \end{aligned}$$

From ${v_{\lambda}}(x) = {(\frac{\lambda}{{ \vert x \vert }})^{n - \beta}}v({x^{\lambda}})$, we have $v({x^{\lambda}}) = {(\frac{{ \vert x \vert }}{\lambda})^{n - \beta}}{v_{\lambda}}(x)$ and so

$$\begin{aligned} {( - \Delta)^{\alpha/2}} {U_{\lambda}}(x) ={}& {( - \Delta)^{\alpha /2}} {u_{\lambda}}(x) - {( - \Delta)^{\alpha/2}}u(x) \\ ={}&{ \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n + \alpha}} \biggl[k \biggl( \frac {{{\lambda^{2}}x}}{{{{ \vert x \vert }^{2}}}} \biggr){v^{q}} \bigl({x^{\lambda}} \bigr) + f \bigl(v \bigl({x^{\lambda}} \bigr) \bigr) \biggr] \\ & - k(x){v^{q}}(x) - f \bigl(v(x) \bigr) \\ ={}& { \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n + \alpha}}k \biggl( \frac {{{\lambda^{2}}x}}{{{{ \vert x \vert }^{2}}}} \biggr){ \biggl[{ \biggl(\frac{{ \vert x \vert }}{\lambda} \biggr)^{n - \beta}} {v_{\lambda}}(x) \biggr]^{q}} + { \biggl( \frac {\lambda}{{ \vert x \vert }} \biggr)^{n + \alpha}}f \bigl(v \bigl({x^{\lambda}} \bigr) \bigr) \\ &- k(x){v^{q}}(x) - f \bigl(v(x) \bigr) \\ ={}& k \biggl(\frac{{{\lambda^{2}}x}}{{{{ \vert x \vert }^{2}}}} \biggr)v_{\lambda}^{q}(x) + { \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n + \alpha }}f(v \bigl({x^{\lambda}} \bigr) \\ & - k(x){v^{q}}(x) - f \bigl(v(x) \bigr) \\ ={}& k \biggl(\frac{{{\lambda^{2}}x}}{{{{ \vert x \vert }^{2}}}} \biggr)v_{\lambda}^{q}(x) - k(x){v^{q}}(x) \\ &+ \frac{{f({{(\frac{{ \vert x \vert }}{\lambda})}^{n - \beta }}{v_{\lambda}}(x))}}{{{{({{(\frac{{ \vert x \vert }}{\lambda})}^{n - \beta}}{v_{\lambda}}(x))}^{q}}}}v_{\lambda}^{q}(x) - f \bigl(v(x) \bigr). \end{aligned}$$

(3.6)

Similarly, it implies

$$ {( - \Delta)^{\alpha/2}} {V_{\lambda}}(x) = j \biggl( \frac{{{\lambda ^{2}}x}}{{{{ \vert x \vert }^{2}}}} \biggr)u_{\lambda}^{p}(x) - j(x){u^{p}}(x) + \frac{{g({{(\frac{{ \vert x \vert }}{\lambda})}^{n - \alpha }}{u_{\lambda}}(x))}}{{{{({{(\frac{{ \vert x \vert }}{\lambda})}^{n - \alpha}}{u_{\lambda}}(x))}^{p}}}}u_{\lambda}^{p}(x) - g \bigl(u(x) \bigr). $$

(3.7)

By assumptions (ii) and (iii), we see that $\frac{{g(r)}}{{{r^{p}}}}$ and $\frac{{f(r)}}{{{r^{q}}}}$ are non-increasing, $k(x)$ and $j(x)$ are non-decreasing along each ray $t\xi \vert {t \ge0}$ for any unit vector $\xi \in R{^{n}}$, so

$$\begin{aligned}& k \biggl(\frac{{{\lambda^{2}}x}}{{{{ \vert x \vert }^{2}}}} \biggr) \ge k(x), \end{aligned}$$

(3.8)

$$\begin{aligned}& j \biggl(\frac{{{\lambda^{2}}x}}{{{{ \vert x \vert }^{2}}}} \biggr) \ge j(x), \end{aligned}$$

(3.9)

$$\begin{aligned}& \frac{{g({{(\frac{{ \vert x \vert }}{\lambda})}^{n - \alpha }}{u_{\lambda}}(x))}}{{{{({{(\frac{{ \vert x \vert }}{\lambda})}^{n - \alpha}}{u_{\lambda}}(x))}^{p}}}}u_{\lambda}^{p}(x) - \frac{{g({u_{\lambda}}(x))}}{{u_{\lambda}^{p}(x)}}u_{\lambda}^{p}(x) \ge0, \end{aligned}$$

(3.10)

$$\begin{aligned}& \frac{{f({{(\frac{{ \vert x \vert }}{\lambda})}^{n - \beta }}{v_{\lambda}}(x))}}{{{{({{(\frac{{ \vert x \vert }}{\lambda})}^{n - \beta}}{v_{\lambda}}(x))}^{q}}}}v_{\lambda}^{q}(x) - \frac{{f({v_{\lambda}}(x))}}{{v_{\lambda}^{q}(x)}}v_{\lambda}^{q}(x) \ge0. \end{aligned}$$

(3.11)

If ${U_{\lambda}}(x) < 0$, then ${u_{\lambda}}(x) < u(x)$. From $\frac {{ \vert x \vert }}{\lambda} \le1$, it follows

$${ \biggl(\frac{{ \vert x \vert }}{\lambda} \biggr)^{n - \alpha}} {u_{\lambda}}(x) < u(x), $$

and therefore,

$$ \frac{{g({{(\frac{{ \vert x \vert }}{\lambda})}^{n - \alpha }}{u_{\lambda}}(x))}}{{{{({{(\frac{{ \vert x \vert }}{\lambda})}^{n - \beta}}{u_{\lambda}}(x))}^{p}}}} - \frac{{g(u(x))}}{{{u^{p}}(x)}} \ge0. $$

(3.12)

If ${V_{\lambda}}(x) < 0$, similar to (3.12), it yields

$$ \frac{{f({{(\frac{{ \vert x \vert }}{\lambda})}^{n - \beta }}{v_{\lambda}}(x))}}{{{{({{(\frac{{ \vert x \vert }}{\lambda})}^{n - \beta}}{v_{\lambda}}(x))}^{q}}}} - \frac{{f(v(x))}}{{{v^{q}}(x)}} \ge0. $$

(3.13)

If ${U_{\lambda}}(x) \ge0$, by assumption (i), $g(r)$ is non-decreasing, and so

$$ g \bigl({u_{\lambda}}(x) \bigr) \ge g \bigl(u(x) \bigr). $$

(3.14)

If ${U_{\lambda}}(x) \ge0$, similar to (3.14), we achieve

$$ f \bigl({v_{\lambda}}(x) \bigr) \ge f \bigl(v(x) \bigr). $$

(3.15)

After these preparations, let us consider cases for $x \in B_{\lambda}^{-} (0)$: (a) ${U_{\lambda}}(x) \ge0$, ${V_{\lambda}}(x) < 0$; (b) ${U_{\lambda}}(x) < 0$, ${V_{\lambda}}(x) \ge0$; (c) ${U_{\lambda}}(x) < 0$, ${V_{\lambda}}(x) < 0$.

(a) If ${U_{\lambda}}(x) \ge0$, ${V_{\lambda}}(x) < 0$, then from (3.6)–(3.10), (3.13), and (3.14), we obtain

$$\begin{aligned} \begin{aligned}[b]{( - \Delta)^{\alpha/2}} {V_{\lambda}}(x)\ge{}& j(x) \bigl[u_{\lambda}^{p}(x) - {u^{p}}(x) \bigr] + \frac{{g({u_{\lambda}}(x))}}{{u_{\lambda}^{p}(x)}}u_{\lambda}^{p}(x) - g \bigl(u(x) \bigr) \\ \ge{}& j(x) \bigl[u_{\lambda}^{p}(x) - {u^{p}}(x) \bigr] \\ ={}& pj(x)\eta_{1}^{p - 1}(x){U_{\lambda}}(x),\end{aligned} \end{aligned}$$

(3.16)

where $u(x) < {\eta_{1}}(x) < {u_{\lambda}}(x)$, and

$$\begin{aligned} {( - \Delta)^{\alpha/2}} {U_{\lambda}}(x) &\ge k(x) \bigl[v_{\lambda}^{q}(x) - {v^{q}}(x) \bigr] + \frac{{f(v(x))}}{{{v^{q}}(x)}}v_{\lambda}^{q}(x) - f \bigl(v(x) \bigr) \\ &\ge \biggl[k(x) + \frac{{f(v(x))}}{{{v^{q}}(x)}} \biggr] \bigl[v_{\lambda}^{q}(x) - {v^{q}}(x) \bigr] \\ &= q \biggl[k(x) + \frac{{f(v(x))}}{{{v^{q}}(x)}} \biggr]\xi_{1}^{q - 1}(x){V_{\lambda}}(x), \end{aligned}$$

(3.17)

where ${v_{\lambda}}(x) < {\xi_{1}}(x) < v(x)$.

(b) If ${U_{\lambda}}(x) < 0$, ${V_{\lambda}}(x) \ge0$, then from (3.6)–(3.9), (3.11), (3.12), and (3.15), it follows

$$\begin{aligned} {( - \Delta)^{\alpha/2}} {V_{\lambda}}(x) &\ge \biggl[j(x) + \frac {{g(u(x))}}{{{u^{p}}(x)}} \biggr] \bigl[u_{\lambda}^{p}(x) - {u^{p}}(x) \bigr] \\ & = p \biggl[j(x) + \frac{{g(u(x))}}{{{u^{p}}(x)}} \biggr]\xi_{2}^{p - 1}(x){U_{\lambda}}(x), \end{aligned}$$

(3.18)

where ${u_{\lambda}}(x) < {\eta_{2}}(x) < u(x)$, and

$$\begin{aligned} {( - \Delta)^{\alpha/2}} {U_{\lambda}}(x) &\ge k(x) \bigl[v_{\lambda}^{q}(x) - {v^{q}}(x) \bigr] + \frac{{f({v_{\lambda}}(x))}}{{v_{\lambda}^{q}(x)}}v_{\lambda}^{q}(x) - f \bigl(v(x) \bigr) \\ & \ge k(x) \bigl[v_{\lambda}^{q}(x) - {v^{q}}(x) \bigr] \\ &= qk(x)\xi_{2}^{q - 1}(x){V_{\lambda}}(x), \end{aligned}$$

(3.19)

where $v(x) < {\xi_{2}}(x) < {v_{\lambda}}(x)$.

(c) If ${U_{\lambda}}(x) < 0$, ${V_{\lambda}}(x) < 0$, then from (3.6)–(3.9), (3.12), and (3.13), it gives (3.18) and (3.17).

Summing up the above analysis, we have

$$ \textstyle\begin{cases} {( - \Delta)^{\alpha/2}}{U_{\lambda}}(x) + {c_{1}}(x){V_{\lambda}}(x) \ge 0,&\\ {( - \Delta)^{\beta/2}}{V_{\lambda}}(x) + {c_{2}}(x){U_{\lambda}}(x) \ge 0,&x \in B_{\lambda}^{-} (0)\setminus\{ 0\},\\ {U_{\lambda}}(x),{V_{\lambda}}(x) \ge0,&x \in{B_{\lambda}}(0)\setminus (B_{\lambda}^{-} (0) \cup\{ 0\} ),\\ U(x) = - {U_{\lambda}}(x),V(x) = - {V_{\lambda}}(x),&x \in{B_{\lambda}}(0)\setminus\{ 0\}, \end{cases} $$

(3.20)

where ${c_{i}}(x) < 0$ ($i = 1,2$). Using Lemma 2.3 and the continuity of ${U_{\lambda}}$ and ${V_{\lambda}}$, we get that, for λ sufficiently small,

$$B_{\lambda}^{-} (0) = \phi,\quad x \in{B_{\lambda}}(0)\setminus{B_{\delta}}(0). $$

Combining it with (3.5), we derive (3.4). This completes the proof of Step 1.

In fact, Step 1 provides a starting point for moving spheres.

Step 2. Keep moving the sphere ${B_{\lambda}}$ until the limiting scale

$${\lambda_{0}} = \sup \bigl\{ \lambda \ge0| {{U_{\mu}}(x) \ge0,{V_{\mu}}(x) \ge0,x \in{B_{\mu}}(0)\setminus\{ 0\} } , \forall\mu < \lambda \bigr\} . $$

We point out that

$$ {\lambda_{0}} = \infty. $$

(3.21)

First, let us prove the following.

Claim 1

If${\lambda_{0}} < \infty$, then for every$x \in{B_{{\lambda _{0}}}}(0)\setminus\{ 0\}$,

$${u_{{\lambda_{0}}}}(x) = u(x),\qquad {v_{{\lambda_{0}}}}(x) = v(x). $$

Proof

We use the contradiction. In Lemma 3.1, we have proved if ${U_{{\lambda_{0}}}}(x) \ne0$ or ${V_{{\lambda_{0}}}}(x) \ne0$, then for $\varepsilon > 0$ sufficiently small,

$$ {U_{{\lambda_{0}}}}(x) \ge c > 0,\qquad {V_{{\lambda_{0}}}}(x) \ge c > 0,\quad x \in {B_{\varepsilon}}(0)\setminus(0). $$

(3.22)

By the definition of ${\lambda_{0}}$,

$$ {U_{{\lambda_{0}}}}(x) \ge0,\qquad {V_{{\lambda_{0}}}}(x) \ge0, \quad x \in {B_{{\lambda_{0}}}}(0)\setminus\{ 0\}. $$

(3.23)

It follows from (3.22) that there exist points ${x^{0}} \in {B_{{\lambda_{0}}}}(0)\setminus\{ 0\}$ and ${y^{0}} \in{B_{{\lambda _{0}}}}(0)\setminus\{ 0\}$ such that

$$ {U_{{\lambda_{0}}}} \bigl({x^{0}} \bigr) > 0,\qquad {V_{{\lambda_{0}}}} \bigl({y^{0}} \bigr) > 0, $$

(3.24)

hence,

$$ {U_{{\lambda_{0}}}}(x) > 0,\qquad {V_{{\lambda_{0}}}}(x) > 0,\quad x \in {B_{{\lambda_{0}}}}(0)\setminus\{ 0\}. $$

(3.25)

 □

To prove (3.25), suppose that (3.25) does not hold, then there exists some point ${x^{1}} \in{B_{{\lambda_{0}}}}(0)\setminus \{ 0\}$ such that

$$0 = {U_{{\lambda_{0}}}} \bigl({x^{1}} \bigr) = \min _{x \in{B_{{\lambda _{0}}}}(0)\setminus\{ 0\} } {U_{{\lambda_{0}}}}(x), $$

that is to say, ${u_{{\lambda_{0}}}}({x^{1}}) = u({x^{1}})$. By the proof process of Lemma 2.1, we know

$$ {( - \Delta)^{\alpha/2}} {U_{{\lambda_{0}}}} \bigl({x^{1}} \bigr) = \int _{{B_{{\lambda_{0}}}}(0)} { \biggl(\frac{1}{{{{ \vert {\frac{{ \vert z \vert {x^{1}}}}{{{\lambda_{0}}}} - \frac{{{\lambda_{0}}z}}{{ \vert z \vert }}} \vert }^{n + \alpha}}}}} - \frac{1}{{{{ \vert {{x^{1}} - z} \vert }^{n + \alpha}}}} \biggr){U_{{\lambda_{0}}}}(z)\,dz. $$

(3.26)

Gathering (2.16) and (3.23), it implies

$$ {( - \Delta)^{\alpha/2}} {U_{{\lambda_{0}}}} \bigl({x^{1}} \bigr) \le0. $$

(3.27)

From (3.16), we get

$$ {( - \Delta)^{\alpha/2}} {U_{{\lambda_{0}}}} \bigl({x^{1}} \bigr) \ge0. $$

(3.28)

Combining (3.27) and (3.28) gives

$${( - \Delta)^{\alpha/2}} {U_{{\lambda_{0}}}} \bigl({x^{1}} \bigr) = 0. $$

Therefore, we have from (3.26) that

$${U_{{\lambda_{0}}}}(x) \equiv0,\quad x \in{B_{{\lambda_{0}}}}(0)\setminus\{ 0\} , $$

which is a contradiction to (3.24). Hence,

$${U_{{\lambda_{0}}}}(x) > 0,\quad x \in{B_{{\lambda_{0}}}}(0)\setminus\{ 0\} . $$

Similarly, we can prove

$${V_{{\lambda_{0}}}}(x) > 0,\quad x \in{B_{{\lambda_{0}}}}(0)\setminus\{ 0\} . $$

Thus (3.25) is proved.

Using (3.25), we can show that, for ${\delta^{0}} > 0$ sufficiently small, it holds that, for any $\lambda \in[{\lambda _{0}},{\lambda_{0}} + {\delta^{0}})$,

$$ {U_{\lambda}}(x) \ge0,\qquad {V_{\lambda}}(x) \ge0,\quad x \in{B_{\lambda}}(0)\setminus\{ 0\}. $$

(3.29)

It will contradict the definition of ${\lambda_{0}}$ and Claim 1 will be proved.

To verify (3.29), we see from (3.25) that, for $\varepsilon > 0$ sufficiently small,

$$\min_{x \in{B_{{\lambda_{0}} - \varepsilon}}(0)\setminus\{ 0\} } {U_{{\lambda_{0}}}}(x): = {m_{0}} > 0,\qquad \min_{x \in{B_{{\lambda_{0}} - \varepsilon}}(0)\setminus\{ 0\} } {V_{{\lambda_{0}}}}(x): = {n_{0}} > 0. $$

By the continuity of ${U_{\lambda}}$ and ${V_{\lambda}}$ in λ, it implies

$$ {U_{\lambda}}(x) \ge\frac{{{m_{0}}}}{2} > 0,\qquad {V_{\lambda}}(x) \ge\frac {{{n_{0}}}}{2} > 0,\quad x \in{B_{{\lambda_{0}} - \varepsilon}}(0) \setminus \{ 0\} . $$

(3.30)

Noting computations similar to those in Step 1, Lemma 2.3, and the local boundedness of g and f, we arrive at

$${U_{\lambda}}(x) \ge0,\qquad{V_{\lambda}}(x) \ge0,\quad x \in{B_{\lambda}}(0)\setminus{B_{{\lambda_{0}} - \varepsilon}}(0). $$

Combining it with (3.30), we derive (3.29). Hence Claim 1 is true.

We continue to prove (3.21). Actually, if ${\lambda_{0}}$ is finite, then from Claim 1 we get

$$ {u_{{\lambda_{0}}}}(x) = u(x),\qquad {v_{{\lambda_{0}}}}(x) = v(x). $$

(3.31)

Owing to (3.6) and (3.7), we have

$$\begin{aligned} 0 &= {( - \Delta)^{\alpha/2}} {U_{{\lambda_{0}}}}(x) \\ &= \biggl(k \biggl(\frac{{\lambda_{0}^{2}x}}{{{{ \vert x \vert }^{2}}}} \biggr) - k(x) \biggr){v^{q}}(x) + \biggl[\frac{{f({{(\frac{{ \vert x \vert }}{{{\lambda _{0}}}})}^{n - \beta}}{v_{{\lambda_{0}}}}(x))}}{{{{({{(\frac{{ \vert x \vert }}{{{\lambda_{0}}}})}^{n - \beta}}{v_{{\lambda_{0}}}}(x))}^{q}}}} - \frac{{f(v(x))}}{{{v^{q}}(x)}} \biggr]{v^{q}}(x) \end{aligned}$$

(3.32)

and

$$\begin{aligned} 0 &= {( - \Delta)^{\alpha/2}} {V_{{\lambda_{0}}}}(x) \\ &= \biggl(j \biggl(\frac{{\lambda_{0}^{2}x}}{{{{ \vert x \vert }^{2}}}} \biggr) - j(x) \biggr){u^{p}}(x) + \biggl[\frac{{g({{(\frac{{ \vert x \vert }}{{{\lambda _{0}}}})}^{n - \alpha}}{u_{{\lambda_{0}}}}(x))}}{{{{({{(\frac{{ \vert x \vert }}{{{\lambda_{0}}}})}^{n - \alpha}}{u_{{\lambda_{0}}}}(x))}^{p}}}} - \frac{{g(u(x))}}{{{u^{p}}(x)}} \biggr]{u^{p}}(x). \end{aligned}$$

(3.33)

By assumptions (ii), (iii), it gives that $\frac{{f(r)}}{{{r^{q}}}}$ and $\frac{{g(r)}}{{{r^{p}}}}$ are non-increasing in r, $k(x)$ and $j(x)$ are non-decreasing. Together with (3.32) and (3.33), it follows

$$k \biggl(\frac{{\lambda_{0}^{2}x}}{{{{ \vert x \vert }^{2}}}} \biggr) = k(x),\quad \vert x \vert < { \lambda_{0}} $$

and

$$j \biggl(\frac{{\lambda_{0}^{2}x}}{{{{ \vert x \vert }^{2}}}} \biggr) = j(x),\quad \vert x \vert < { \lambda_{0}}, $$

which imply that $k(x)$ and $j(x)$ are constants. This is a contradiction, and we arrive at (3.21).

Since $\lambda_{0}= \infty$, from Lemma 5.7 in [41] (Proposition 3.3 in [25]) (also see [26] and [42]), we know $u = c$ and $v = c$ are constants. Taking this results into system (1.1), we know “$u = v = 0$”.

Thus we get a contradiction to the positivity of solutions. The proof of Theorem 1.1 is finished. □

Remark 3.1

When $k(x)$ and $j(x)$ are constants, we find from (3.32) and (3.33) that if $f(v) = {C_{1}}{v^{q}}$ and $g(u) = {C_{2}}{u^{p}}$, then $u(x)$ and $v(x)$ are nonnegative solutions of (1.1).

Proof of Corollary 1.2

The proof of Theorem 1.1 implies, for all $\lambda \in(0,\infty)$,

$${ \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha}}u \biggl(\frac {{{\lambda^{{2}}}x}}{{{{ \vert x \vert }^{2}}}} \biggr) \ge u(x), $$

so

$${ \biggl\vert {\frac{{{\lambda^{{2}}}x}}{{{{ \vert x \vert }^{2}}}}} \biggr\vert ^{\frac{{n - \alpha}}{2}}}u \bigl({x^{\lambda}} \bigr) = \frac{{{\lambda ^{n - \alpha}}}}{{{{ \vert x \vert }^{\frac{{n - \alpha }}{2}}}}}u \biggl(\frac{{{\lambda^{{2}}}x}}{{{{ \vert x \vert }^{2}}}} \biggr) \ge { \vert x \vert ^{\frac{{n - \alpha}}{2}}}u(x), $$

i.e.,

$${ \bigl\vert {{x^{\lambda}}} \bigr\vert ^{\frac{{n - \alpha }}{2}}}u \bigl({x^{\lambda}} \bigr) \ge{ \vert x \vert ^{\frac{{n - \alpha}}{2}}}u(x). $$

It shows that $W(x)$ is monotone decreasing along $\{ t\xi \vert {t \ge0} \}$ ($\vert \xi \vert = 1$).

The proof to ${W_{1}}(x)$ is similar. The statement is proved. □

4 Proof of Theorem 1.3

Proof of Theorem 1.3

Let ${B_{R}}({{0}})$ be the smallest ball centered at the origin containing Ω. For any $\lambda \in (0,R)$, we denote the set

$$\sum_{\lambda}{ = \varOmega\setminus \overline{{B_{\lambda}}} } (O). $$

Because of the star-shapedness of Ω, we know that ${u_{\lambda}}(x)$ and ${v_{\lambda}}(x)$ obtained by the Kelvin-transforms of $u(x)$ and $v(x)$ are well defined on $\sum_{\lambda}$.

Step 1. We will show that for $\varepsilon > 0$ ($0 < \varepsilon < R$) sufficiently small, if $\lambda \in[R - \varepsilon,R)$, then

$$ {U_{\lambda}}(x) \ge0,\qquad {V_{\lambda}}(x) \ge0,\quad x \in\sum _{\lambda}. $$

(4.1)

To view (4.1), denote

$$\sum_{\lambda}^{-} = \biggl\{ x \in\sum _{\lambda}{ \Big\vert {{U_{\lambda}}(x)} } < 0\text{ or }{V_{\lambda}}(x) < 0 \biggr\} . $$

Similar to the process of proving (3.13), we have

$$\textstyle\begin{cases} {( - \Delta)^{\alpha/2}}{U_{\lambda}}(x) + {c_{3}}(x){V_{\lambda}}(x) \ge 0,&\\ {( - \Delta)^{\beta/2}}{V_{\lambda}}(x) + {c_{4}}(x){U_{\lambda}}(x) \ge 0,& x \in\sum_{\lambda}^{-} ,\\ {U_{\lambda}}(x),{V_{\lambda}}(x) \ge0, & x \in R{^{n}}\setminus ({B_{\lambda}}(0) \cup\sum_{\lambda}^{-} ),\\ U(x) = - {U_{\lambda}}(x),V(x) = - {V_{\lambda}}(x),& x \in R{^{n}}\setminus {B_{\lambda}}(0), \end{cases} $$

where ${c_{i}}(x) < 0$ ($i = 3,4$). For $\varepsilon > 0$ sufficiently small, $\sum_{\lambda}^{-}$ is a narrow region for $\lambda \in[R - \varepsilon,R)$. Lemma 2.4 ensures

$${U_{\lambda}}(x) \ge0, \qquad{V_{\lambda}}(x) \ge0,\quad x \in\sum _{\lambda}^{-} . $$

Now (4.1) is proved.

This provides a starting point for moving spheres. Next we continue to move the spheres.

Step 2. Keep shrinking the sphere ${B_{\lambda}}$ until the limiting scale

$${\lambda_{0}} = \inf \biggl\{ \lambda \ge0\Big| {{U_{\mu}}(x) \ge0,{V_{\mu}}(x) \ge0,x \in\sum_{\mu}} ,\forall\lambda \le\mu < R \biggr\} . $$

We will show

$$ {\lambda_{0}} = {{0}}. $$

(4.2)

Before proving (4.2), let us note by the definition of $\lambda _{0}$ that

$${U_{{\lambda_{{0}}}}}(x) \ge0,\qquad{V_{{\lambda_{{0}}}}}(x) \ge0,\quad x \in \sum _{{\lambda_{{0}}}} . $$

It follows from (2.10) that either

$$ {U_{{\lambda_{{0}}}}}(x) > 0,\qquad {V_{{\lambda_{{0}}}}}(x) > 0,\quad x \in \sum _{{\lambda_{0}}} , $$

(4.3)

or

$$ {U_{{\lambda_{{0}}}}}(x) \equiv0,\qquad {V_{{\lambda_{{0}}}}}(x) \equiv0, \quad x \in \sum_{{\lambda_{0}}} . $$

(4.4)

Because Ω is a star-shaped domain, there exists some point ${y^{{0}}} \in\partial\varOmega$ such that

$${U_{{\lambda_{{0}}}}} \bigl({y^{0}} \bigr) > 0, \qquad{V_{{\lambda_{{0}}}}} \bigl({y^{0}} \bigr) > 0, $$

and the continuity of ${U_{{\lambda_{{0}}}}}(x)$, ${V_{{\lambda _{{0}}}}}(x)$ implies (4.3).

To prove (4.2), we use the contradiction and assume ${\lambda _{0}} > 0$. It allows us to move continuously the sphere $\partial {B_{{\lambda_{0}}}}$ and conclude that, for $\tau > 0$ sufficiently small,

$$ {U_{{\lambda_{0}} - \tau}}(x) \ge0,\qquad {V_{{\lambda_{0}} - \tau}}(x) \ge 0,\quad x \in \sum_{{\lambda_{0}} - \tau} . $$

(4.5)

But this contradicts the definition of ${\lambda_{0}}$ and it thus proves (4.2).

The remaining is to check (4.5). In fact, it is evident that ${U_{\lambda}}(x),{V_{\lambda}}(x) > 0$ in $R{^{n}}\setminus\bar{\varOmega}$ for all $\lambda > 0$. Note that the positivity of $u(x)$ and $v(x)$ does not necessarily imply the positivity of ${U_{\lambda}}$ and ${V_{\lambda}}$ on $\partial\sum_{\lambda}\setminus\partial {B_{\lambda}}(0)$, since Ω is not assumed to be strictly star-shaped. But, for each connected component ${Z_{\lambda}}$ of $\sum_{\lambda}$, we can choose $\delta > 0$ small as in Lemma 2.4 and cut a closed set ${K_{\lambda}}$ in ${Z_{\lambda}}$ such that meas$({Z_{\lambda}}\setminus{K_{\lambda}}) = \vert {{Z_{\lambda}}\setminus{K_{\lambda}}} \vert < \delta/2$ and $\operatorname{dist}(\partial {Z_{\lambda}},\partial{K_{\lambda}}) < \delta$. It follows by (4.3) that, for $\delta > 0$ sufficiently small,

$$ {U_{{\lambda_{0}}}} \ge c > 0,\qquad {V_{{\lambda_{0}}}} \ge c > 0,\quad x \in {K_{{\lambda_{0}}}}. $$

(4.6)

Since ${U_{\lambda}}$ and $V_{\lambda}$ depend continuously on λ, there exists $\varepsilon > 0$ sufficiently small, $\delta{{/2}} > \varepsilon > 0$, such that for all $\lambda \in({\lambda_{0}} - \varepsilon,{\lambda_{0}})$,

$$ {U_{\lambda}}(x) \ge0,\qquad {V_{\lambda}}(x) \ge0,\quad x \in{K_{{\lambda_{0}}}} $$

(4.7)

and

$$\vert {{Z_{\lambda}}\setminus{K_{{\lambda_{0}}}}} \vert < \delta. $$

Noting $\partial({Z_{\lambda}}\setminus{K_{{\lambda_{0}}}}) \subset \partial{Z_{\lambda}} \cup{K_{{\lambda_{0}}}}$, we see that

$${U_{\lambda}}(x) \ge0,\qquad {V_{\lambda}}(x) \ge0,\quad x \in\partial ({Z_{\lambda}}\setminus{K_{{\lambda_{0}}}}), $$

and so ${U_{\lambda}}(x)$ and ${V_{\lambda}}(x)$ satisfy

$$\textstyle\begin{cases} {( - \Delta)^{\alpha/2}}{U_{\lambda}}(x) + {c_{5}}(x){V_{\lambda}}(x) \ge 0,&\\ {( - \Delta)^{\beta/2}}{V_{\lambda}}(x) + {c_{6}}(x){U_{\lambda}}(x) \ge 0,& x \in{Z_{\lambda}}\setminus{K_{{\lambda_{0}}}},\\ {U_{\lambda}}(x),{V_{\lambda}}(x) \ge0, & x \in R{^{n}}\setminus (({Z_{\lambda}}\setminus{K_{{\lambda_{0}}}}) \cup{B_{\lambda}}(0)),\\ U(x) = - {U_{\lambda}}(x),V(x) = - {V_{\lambda}}(x),& x \in R{^{n}}\setminus {B_{\lambda}}(0), \end{cases} $$

where ${c_{i}}(x) < 0$ ($i = 5,6$). It follows from Lemma 2.4 that

$$ {U_{\lambda}} \ge0,\quad\quad {V_{\lambda}} \ge0,\quad x \in{Z_{\lambda}}\setminus {K_{{\lambda_{0}}}}. $$

(4.8)

By (4.7) and (4.8), we have that, for all $\lambda \in ({\lambda_{0}} - \varepsilon,{\lambda_{0}})$,

$${U_{\lambda}}(x) \ge0,\qquad {V_{\lambda}}(x) \ge0,\quad x \in\sum _{\lambda}, $$

which yields (4.5).

Step 3. Using (4.2) means that, for any fixed $x \in\varOmega$,

$${ \biggl(\frac{\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha}}u \biggl(\frac {{{\lambda^{2}}x}}{{{{ \vert x \vert }^{2}}}} \biggr) \ge u(x),\qquad { \biggl(\frac {\lambda}{{ \vert x \vert }} \biggr)^{n - \alpha}}v \biggl( \frac{{{\lambda ^{2}}x}}{{{{ \vert x \vert }^{2}}}} \biggr) \ge v(x) \quad\text{for all }\lambda \in \bigl(0, \vert x \vert \bigr). $$

Letting $\lambda \to0$, we obtain $u(x) = 0$ and $v(x) = 0$. It contradicts the positivity of $u(x)$ and $v(x)$, and the conclusion of Theorem 1.3 is derived. □