Existence of positive solutions to negative power nonlinear integral equations with weights

Abstract

This paper is devoted to the existence and non-existence of positive solutions to the following negative power nonlinear integral equation related to the sharp reversed Hardy–Littlewood–Sobolev inequality:

$$ f^{q-1}(x)= \int _{\varOmega }\frac{K(x)f(y)K(y)}{ \vert x-y \vert ^{n-\alpha }}\,dy+ \lambda \int _{\varOmega }\frac{G(x)f(y)G(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\,dy, \quad f\geq 0, x\in \overline{ \varOmega }, $$

where $0< q<1$, $\alpha >n$, $0<\beta <\alpha -n$, $\lambda \in \mathbb{R}$, Ω is a smooth bounded domain, $K(x)$, $G(x)$ are positive continuous functions in Ω̅. For $K\equiv G\equiv 1$, the existence and non-existence of positive solutions to the equation have been studied by Dou–Guo–Zhu (2019). In this paper we consider the existence and non-existence of positive solutions to the above integral equation with the general weight functions $K(x)$, $G(x)$.

1 Introduction

In this paper we consider the existence and non-existence of positive solutions to the following negative power nonlinear integral equation:

$$ f^{q-1}(x)= \int _{\varOmega }\frac{K(x)f(y)K(y)}{ \vert x-y \vert ^{n-\alpha }}\,dy+ \lambda \int _{\varOmega }\frac{G(x)f(y)G(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\,dy, \quad f \geq 0,x\in \overline{ \varOmega }, $$

(1.1)

where $0< q<1$, $\alpha >n$, $0<\beta <\alpha -n$, $\lambda \in \mathbb{R}$, Ω is a smooth bounded domain, $K(x)$, $G(x)$ are positive continuous functions in Ω̅.

For $0<\alpha <n$, $G(x)\equiv 1$, the existence and non-existence of positive solutions to (1.1) were studied by Dou–Zhu [2] and Guo–Wang [3] recently. Notice that when $0<\alpha <n$ this nonlinear integral equation is closely related to the sharp Hardy–Littlewood–Sobolev (HLS for short) inequality [4–7].

For $\alpha >n$, the existence and non-existence of positive solutions to (1.1) are also studied by Dou–Guo–Zhu [1] when $K(x)\equiv G(x)\equiv 1$. In this case the nonlinear integral equation is related to the sharp reversed HLS inequality obtained by Beckner [8] and Dou–Zhu [9], respectively. In fact, Eq. (1.1) (when $K(x)\equiv 1$, $\lambda =0$) can be seen as the Euler–Lagrange equation of the following minimizing problem related to the reversed HLS inequality:

$$ \xi _{\alpha }(\varOmega )=\inf_{f\in L^{\frac{2n}{n+\alpha }}(\varOmega ),f \ge 0, f \ne 0} \frac{\int _{\varOmega } \int _{\varOmega } f(x) \vert x-y \vert ^{-(n-\alpha )} f(y) \,dx \,dy}{ \Vert f \Vert ^{2}_{L^{\frac{2n}{n+\alpha }}(\varOmega )}}. $$

On the other hand, for Eq. (1.1) with $K(x)\equiv 1$ and $\lambda =0$, the blowup behavior of energy maximizing positive solutions as $q\to (\frac{2n}{n+\alpha })^{+}$ when $1<\alpha <n$, and the blowup behavior of energy minimizing positive solution as $q\to (\frac{2n}{n+\alpha })^{-}$ when $\alpha >n$ are also analyzed by Guo [10].

In this paper we consider the integral equation (1.1) for general weight functions $K(x)$, $G(x)$ and $\alpha >n$.

The following condition is needed.

$(\mathcal{T})$.:

$K(x_{*})-K(x)=o( \vert x-x_{*} \vert ^{\gamma })$ as $x\to x_{*}$, where $K(x_{*})=\min_{x\in \overline{\varOmega }}K(x)$, $\gamma >0$.

Denote $G(\tilde{x}_{*})=\max_{x\in \overline{\varOmega }}G(x)$.

The main results are stated as follows.

Theorem 1.1

Assume$\alpha >n$, $\beta \in (0,\alpha -n)$, Ωis a smooth bounded domain of diameter$d(\varOmega )$.

• (i) For$0< q<\frac{2n}{n+\alpha }$ (subcritical case), $-\frac{K^{2}(x_{*})}{d^{\beta }(\varOmega )G^{2}(\tilde{x}_{*})}< \lambda$, the positive functions$K(x),G(x)\in C^{1}(\overline{\varOmega })$, then there is a positive solution$f\in C^{1}(\overline{\varOmega })$to Eq. (1.1).

• (ii) For$q=\frac{2n}{n+\alpha }$ (critical case), $- \frac{K^{2}(x_{*})}{d^{\beta }(\varOmega )G^{2}(\tilde{x}_{*})}<\lambda <0$, the positive functions$K(x),G(x)\in C^{1}(\overline{\varOmega })$, assume further that$\beta < n$and$(\mathcal{T})$holds, then there is a positive solution$f\in C^{1}(\overline{\varOmega })$to Eq. (1.1).

• (iii) For$\frac{2n}{n+\alpha }\leq q<1$ (critical case and supercritical case), $\lambda \geq 0$, the nonnegative functions$K(x),G(x)\in C^{1}(\overline{\varOmega })$, ifΩis a star-shaped domain with respect tox̃, $(x-\tilde{x},\nabla K(x))\geq 0$and$(x-\tilde{x},\nabla G(x))\geq 0$, then there is not any positive$C^{1}(\overline{\varOmega })$solution to Eq. (1.1).

We use c, C throughout the paper to represent positive constants, which may vary from line to line.

2 Preliminaries

For simplicity, we denote $p_{\alpha }:=\frac{2n}{n-\alpha }$, $q_{\alpha }:=\frac{2n}{n+\alpha }$ throughout the paper. For $0< q<1$, we also denote $L^{q}(\varOmega ):=\{f\mid \int _{\varOmega } \vert f \vert ^{q}(x)\,dx<\infty \}$ for any domain $\varOmega \subset \mathbb{R}^{n}$, $L_{+}^{q}(\varOmega ):=\{f\in L^{q}(\varOmega )\setminus \{0\}:f\ge 0\}$ and define $\Vert f \Vert _{L^{q}(\varOmega )}:=(\int _{\varOmega } \vert f \vert ^{q}(x)\,dx)^{\frac{1}{q}}$ for $f\in L^{q}(\varOmega )$. Notice that $\Vert f \Vert _{L^{q}(\varOmega )}$ is not a norm if $0< q<1$.

We first recall the sharp reversed HLS inequality on $\mathbb{R}^{n}$.

Theorem A

(see [8, 9])

Let$\alpha >n$. Then

$$ \biggl\vert \int _{\mathbb{R}^{n}} \int _{\mathbb{R}^{n}}f(x) \vert x-y \vert ^{-(n- \alpha )}g(y)\,dx\,dy \biggr\vert \geq N_{\alpha } \Vert f \Vert _{L^{q_{\alpha }}( \mathbb{R}^{n})} \Vert g \Vert _{L^{q_{\alpha }}(\mathbb{R}^{n})} $$

(2.1)

for all$f,g\in L^{q_{\alpha }}(\mathbb{R}^{n})$, where$N_{\alpha }:=\pi ^{\frac{n-\alpha }{2}} \frac{\varGamma (\frac{\alpha }{2})}{\varGamma (\frac{n}{2}+\frac{\alpha }{2})} (\frac{\varGamma (\frac{n}{2})}{\varGamma (n)} )^{-\frac{\alpha }{n}}$. Moreover, the equality holds if and only if$f(x)=c_{1}g(x)=c_{2}(\frac{1}{c_{3}+ \vert x-x_{0} \vert ^{2}})^{ \frac{n+\alpha }{2}}$, where$c_{1}$, $c_{2}$, $c_{3}$are any constants, $x_{0}\in \mathbb{R}^{n}$.

3 Proofs of the main results

Here and hereafter we always assume $\alpha >n$.

3.1 Existence—subcritical case

We first prove the existence of positive solution to Eq. (1.1) in the subcritical case $0< q< q_{\alpha }$. The following lemma from [2] is needed.

Lemma 3.1

(see [2])

Let$q\in (0,q_{\alpha })$. There exists a positive constant$C(n,q, \alpha ,\varOmega )>0$such that

$$ \int _{\varOmega } \int _{\varOmega }f(x) \vert x-y \vert ^{-(n-\alpha )}f(y)\,dx\,dy \geq C(n,q, \alpha ,\varOmega ) \Vert f \Vert ^{2}_{L^{q}(\varOmega )} $$

(3.1)

for any nonnegative function$f\in L^{q}(\varOmega )$.

Now we prove the following lemma, which implies the existence result of part (i) in Theorem 1.1.

Lemma 3.2

Assume the positive functions$K(x), G(x)\in C^{1}(\overline{\varOmega })$. Then, for$0< q< q_{\alpha }$, $\lambda >- \frac{K^{2}(x_{*})}{d^{\beta }(\varOmega )G^{2}(\tilde{x}_{*})}$, the infimum

$$\begin{aligned}& Q_{\lambda ,q}(\varOmega ) \\& \quad :=\inf_{f\in L_{+}^{q}(\varOmega )} \frac{\int _{\varOmega }\int _{\varOmega }f(x)(K(x) \vert x-y \vert ^{-(n-\alpha )}K(y)+\lambda G(x) \vert x-y \vert ^{-(n-\alpha -\beta )}G(y))f(y)\,dx\,dy}{ \Vert f \Vert _{L^{q}(\varOmega )}^{2}} \end{aligned}$$

is attained by some nonnegative function in$L_{+}^{q}(\varOmega )$.

Proof

Notice that $\lambda >- \frac{K^{2}(x_{*})}{d^{\beta }(\varOmega )G^{2}(\tilde{x}_{*})}$ and

$$ K(x)K(y)+\lambda G(x) \vert x-y \vert ^{\beta }G(y)\geq K^{2}(x_{*})+\lambda d^{ \beta }(\varOmega )G^{2}(\tilde{x}_{*})>0,\quad x,y\in \varOmega . $$

Then by Lemma 3.1, $Q_{\lambda ,q}(\varOmega )>0$.

Now we can choose the minimizing positive sequence $\{f_{j}\}_{j=1}^{\infty }$ in $L_{+}^{q}(\varOmega )$ and argue as Lemma 3.2 in [1]. We sketch it for the reader’s convenience. Assume $f_{j}\in L^{q_{\alpha }}(\varOmega )$ and $\Vert f_{j} \Vert _{L^{q_{\alpha }}(\varOmega )}=1$. Then, up to a subsequence,

$$ f_{j}^{q}\rightharpoonup f_{*}^{q} \quad \text{in } L^{\frac{q_{\alpha }}{q}}( \varOmega ), \text{as } j\rightarrow \infty , $$

and

$$ \int _{\varOmega }f_{j}^{q}\rightarrow \int _{\varOmega }f_{*}^{q}, \quad \text{as } j \rightarrow \infty . $$

As in [1], we have $\Vert f_{j} \Vert _{L^{1}(\varOmega )}\leq C$. Thus $\int _{\varOmega }f_{*}^{q}>C>0$ via an interpolation inequality and $f_{j}^{q}\rightharpoonup f_{*}^{q}$ weakly in $L^{\frac{1}{q}}(\varOmega )$. For any fixed $x\in \overline{\varOmega }$, $f_{*}^{1-q}(y) \vert x-y \vert ^{\alpha -n}(K(x)K(y)+\lambda G(x) \vert x-y \vert ^{\beta }G(y)) \in L^{\frac{1}{1-q}}(\varOmega )$. Therefore

$$\begin{aligned}& \int _{\varOmega }f_{j}^{q}(y)f_{*}^{1-q}(y) \vert x-y \vert ^{\alpha -n}\bigl(K(x)K(y)+ \lambda G(x) \vert x-y \vert ^{\beta }G(y)\bigr)\,dy \\& \quad \rightarrow \int _{\varOmega }f_{*}(y) \vert x-y \vert ^{\alpha -n}\bigl(K(x)K(y)+ \lambda G(x) \vert x-y \vert ^{\beta }G(y)\bigr)\,dy, \quad\text{as } j\rightarrow \infty . \end{aligned}$$

Now we prove that the convergence is uniform for all $x\in \overline{\varOmega }$. Firstly, as Lemma 3.2 in [1], we have $\int _{\varOmega }f_{j}^{q}(y)f_{*}^{1-q}(y) \vert x-y \vert ^{\alpha -n}(K(x)K(y)+ \lambda G(x) \vert x-y \vert ^{\beta }G(y))\,dy$ is uniformly bounded for $x\in \overline{\varOmega }$. Now it is left to prove that $\int _{\varOmega }f_{j}^{q}(y)f_{*}^{1-q}(y) \vert x-y \vert ^{\alpha -n}(K(x)K(y)+ \lambda G(x) \vert x-y \vert ^{\beta }G(y))\,dy$ is equicontinuous in Ω̅. Notice that $K(x),G(x)\in C^{1}(\overline{\varOmega })$ and for any $x_{1},x_{2},y\in \overline{\varOmega }$,

$$ \bigl\vert \vert x_{1}-y \vert ^{\alpha -n}- \vert x_{2}-y \vert ^{\alpha -n} \bigr\vert \leq \textstyle\begin{cases} C \vert x_{1}-x_{2} \vert ^{\alpha -n}, &0< \alpha -n\leq 1, \\ C \vert x_{1}-x_{2} \vert , &\alpha -n>1. \end{cases} $$

Then since $\int _{\varOmega }f_{j}^{q}(y)f_{*}^{1-q}(y)K(y)\,dy$ is bounded, for any $x_{1},x_{2}\in \overline{\varOmega }$,

$$\begin{aligned} & \biggl\vert \int _{\varOmega }f_{j}^{q}(y)f_{*}^{1-q}(y) \vert x_{1}-y \vert ^{\alpha -n}K(y)\,dy- \int _{\varOmega }f_{j}^{q}(y)f_{*}^{1-q}(y) \vert x_{2}-y \vert ^{\alpha -n}K(y)\,dy \biggr\vert \\ &\quad \leq \int _{\varOmega }f_{j}^{q}(y)f_{*}^{1-q}(y)K(y) \bigl\vert \vert x_{1}-y \vert ^{ \alpha -n}- \vert x_{2}-y \vert ^{\alpha -n} \bigr\vert \,dy \\ &\quad \leq C\max \bigl( \vert x_{1}-x_{2} \vert ^{\alpha -n}, \vert x_{1}-x_{2} \vert \bigr). \end{aligned}$$

So $\int _{\varOmega }f_{j}^{q}(y)f_{*}^{1-q}(y) \vert x-y \vert ^{\alpha -n}K(y)\,dy$ and, by a similar argument, $\lambda \int _{\varOmega }f_{j}^{q}(y)f_{*}^{1-q}(y) \vert x- y \vert ^{\alpha + \beta -n}G(y)\,dy$ are equicontinuous in $x\in \overline{\varOmega }$. Thus we see that $\int _{\varOmega }f_{j}^{q}(y)f_{*}^{1-q}(y) \vert x-y \vert ^{\alpha -n} (K(x)K(y)+ \lambda G(x) \vert x-y \vert ^{\beta }G(y))\,dy$ is equicontinuous in Ω̅.

Now similar to Lemma 3.2 in [1], we can prove

$$\begin{aligned} & \liminf_{j\rightarrow \infty } \int _{\varOmega } \int _{ \varOmega }f_{j}(x) \vert x-y \vert ^{\alpha -n}\bigl(K(x)K(y)+\lambda G(x) \vert x-y \vert ^{\beta }G(y)\bigr)f_{j}(y)\,dx\,dy \\ &\quad \geq \int _{\varOmega } \int _{\varOmega }f_{*}(x) \vert x-y \vert ^{\alpha -n}\bigl(K(x)K(y)+ \lambda G(x) \vert x-y \vert ^{\beta }G(y)\bigr)f_{*}(y)\,dx\,dy. \end{aligned}$$

By $\Vert f_{j} \Vert _{L^{q}(\varOmega )}\rightarrow \Vert f_{*} \Vert _{L^{q}(\varOmega )}>0$ and the above,

$$\begin{aligned} & \liminf_{j\rightarrow \infty } \frac{\int _{\varOmega }\int _{\varOmega }f_{j}(x) \vert x-y \vert ^{\alpha -n}(K(x)K(y)+\lambda G(x) \vert x-y \vert ^{\beta }G(y))f_{j}(y)\,dx\,dy}{ \Vert f_{j} \Vert ^{2}_{L^{q}(\varOmega )}} \\ &\quad \geq \frac{\int _{\varOmega }\int _{\varOmega }f_{*}(x) \vert x-y \vert ^{\alpha -n}(K(x)K(y)+\lambda G(x) \vert x-y \vert ^{\beta }G(y))f_{*}(y)\,dx\,dy}{ \Vert f_{*} \Vert ^{2}_{L^{q}( \varOmega )}}. \end{aligned}$$

That is, $f_{*}$ is the minimizer. □

Again as that in [1], we obtain $u\in C^{1}(\overline{\varOmega })$. Thus we complete the proof of Theorem 1.1 (i).

Remark 3.3

We assume $\lambda >- \frac{K^{2}(x_{*})}{d^{\beta }(\varOmega )G^{2}(\tilde{x}_{*})}$ here to make sure that $Q_{\lambda ,q}(\varOmega )$ is positive.

3.2 Existence—critical case

Now we establish the existence and the regularity results for the weak solution to (1.1) with critical exponent for $\lambda <0$. Consider

$$\begin{aligned}& {Q_{\lambda ,q_{\alpha }}(\varOmega )}\\& \quad =\inf_{f\in L_{+}^{q_{ \alpha }}(\varOmega )} \frac{\int _{\varOmega }\int _{\varOmega }f(x)(K(x) \vert x-y \vert ^{-(n-\alpha )}K(y)+\lambda G(x) \vert x-y \vert ^{-(n-\alpha -\beta )}G(y))f(y)\,dy\,dx}{ \Vert f \Vert _{L^{q_{\alpha }}(\varOmega )}^{ 2}}. \end{aligned}$$

Notice that the corresponding Euler–Lagrange equation for extremal functions, up to a constant multiplier, is the integral equation (1.1) with $q=q_{\alpha }$.

We first show the following lemma.

Lemma 3.4

Assume that the positive functions$K(x),G(x)\in C^{1}(\overline{\varOmega })$and$(\mathcal{T})$holds. Then${Q_{\lambda ,q_{\alpha }}(\varOmega )}< K^{2}(x_{*})N_{\alpha }$for all$\lambda <0$. Further, $0<{Q_{\lambda ,q_{\alpha }}(\varOmega )}<K^{2}(x_{*})N_{ \alpha }$for any$\lambda \in (- \frac{K^{2}(x_{*})}{d^{\beta }(\varOmega )G^{2}(\tilde{x}_{*})},0)$, where$\beta >0$.

Proof

We distinguish two cases: (I) $x_{*}\in \varOmega$; (II) $x_{*} \in \partial \varOmega$.

(I) Let $x_{*}\in \varOmega$. By $(\mathcal{T})$, there exists small $R>0$ such that $K(x)-K(x_{*})\leq c \vert x-x_{*} \vert ^{\gamma }$ when $x\in B_{R}(x_{*})\subset \varOmega$. For small $\epsilon >0$, we define

$$ \widetilde{f}_{\epsilon }(x)= \textstyle\begin{cases} f_{\epsilon }(x),&x\in B_{R}(x_{*})\subset \varOmega , \\ 0,&x\in \mathbb{R}^{n}\setminus B_{R}(x_{*}), \end{cases} $$

where $f_{\epsilon }(x)=\epsilon ^{-\frac{n+\alpha }{2}}f_{1}( \frac{x-x_{*}}{\epsilon })=( \frac{\epsilon }{\epsilon ^{2}+ \vert x-x_{*} \vert ^{2}})^{\frac{n+\alpha }{2}}$, ${f_{1}(x)=(\frac{1}{1+ \vert x \vert ^{2}})^{\frac{n+\alpha }{2}}}$. Notice that $f_{1}$ and its conformal equivalent class $f_{\epsilon }$ are the extremal functions to the sharp reversed HLS inequality (2.1). Obviously, $\widetilde{f}_{\epsilon }\in L^{q_{\alpha }}(\mathbb{R}^{n})$. By $(\mathcal{T})$, we have

$$\begin{aligned}& \int _{\varOmega } \int _{\varOmega }\biggl(\frac{K(x)K(y)}{ \vert x-y \vert ^{n-\alpha }}+ \frac{\lambda G(x)G(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\biggr) \widetilde{f}_{ \epsilon }(x)\widetilde{f}_{\epsilon }(y)\,dx\,dy \\& \quad = \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})}\biggl( \frac{K(x)K(y)}{ \vert x-y \vert ^{n-\alpha }}+ \frac{\lambda G(x)G(y)}{ \vert x-y \vert ^{n-\alpha -\beta }} \biggr)f_{\epsilon }(x)f_{ \epsilon }(y)\,dx\,dy \\& \quad \leq \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})} \frac{(K(x_{*})+c \vert x-x_{*} \vert ^{\gamma })(K(x_{*})+c \vert y-x_{*} \vert ^{\gamma })}{ \vert x-y \vert ^{n-\alpha }}f_{ \epsilon }(x)f_{\epsilon }(y) \,dx\,dy \\& \quad \quad{} +\lambda \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})} \frac{G(x)G(y) f_{\epsilon }(x)f_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\,dx\,dy \\& \quad = \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})} \frac{K^{2}(x_{*})+K(x_{*})(c \vert x-x_{*} \vert ^{\gamma }+c \vert y-x_{*} \vert ^{\gamma })+c^{2} \vert x-x_{*} \vert ^{\gamma } \vert y-x_{*} \vert ^{\gamma }}{ \vert x-y \vert ^{n-\alpha }} \\& \quad\quad{}\times f_{ \epsilon }(x)f_{\epsilon }(y) \,dx\,dy \\& \quad\quad{} +\lambda \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})} \frac{G(x)G(y) f_{\epsilon }(x)f_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\,dx\,dy \\& \quad \leq \int _{\mathbb{R}^{n}} \int _{\mathbb{R}^{n}} \frac{K^{2}(x_{*})f_{\epsilon }(x)f_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy +\lambda \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})} \frac{G(x)G(y) f_{\epsilon }(x)f_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\,dx\,dy \\& \quad\quad{} +c^{2} \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})} \frac{ \vert x-x_{*} \vert ^{\gamma } \vert y-x_{*} \vert ^{\gamma }f_{\epsilon }(x)f_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy \\& \quad \quad{} +2c \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})} \frac{K(x_{*}) \vert x-x_{*} \vert ^{\gamma }f_{\epsilon }(x)f_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy \\& \quad = K^{2}(x_{*})N_{\alpha } \Vert f_{\epsilon } \Vert ^{2}_{L^{q_{\alpha }}( \mathbb{R}^{n})}+I_{1}+I_{2}+I_{3}, \end{aligned}$$

(3.2)

where

$$\begin{aligned} I_{1}:&=\lambda \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})} \frac{G(x)G(y) f_{\epsilon }(x)f_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\,dx\,dy, \\ I_{2}:&=c^{2} \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})} \frac{ \vert x-x_{*} \vert ^{\gamma } \vert y-x_{*} \vert ^{\gamma }f_{\epsilon }(x)f_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy, \\ I_{3}:&=2c \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})} \frac{K(x_{*}) \vert x-x_{*} \vert ^{\gamma }f_{\epsilon }(x)f_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy. \end{aligned}$$

For $I_{1}$, we have

$$\begin{aligned} I_{1}&= \lambda \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})} \frac{G(x)G(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\biggl( \frac{\epsilon }{\epsilon ^{2}+ \vert x-x_{*} \vert ^{2}} \biggr)^{\frac{n+\alpha }{2}}\biggl( \frac{\epsilon }{\epsilon ^{2}+ \vert y-x_{*} \vert ^{2}}\biggr)^{\frac{n+\alpha }{2}}\,dx\,dy \\ &\le C\lambda \epsilon ^{-(n-\alpha -\beta )-(n+\alpha )} \int _{B_{R}(0)} \int _{B_{R}(0)} \biggl\vert \frac{x-y}{\epsilon } \biggr\vert ^{-(n-\alpha -\beta )}\biggl(1+ \biggl\vert \frac{x}{\epsilon } \biggr\vert ^{2}\biggr)^{-\frac{n+\alpha }{2}}\biggl(1+ \biggl\vert \frac{y}{\epsilon } \biggr\vert ^{2}\biggr)^{-\frac{n+\alpha }{2}}\,dx\,dy \\ &=C\lambda \epsilon ^{\beta } \int _{B_{\frac{R}{\epsilon }}(0)} \int _{B_{ \frac{R}{\epsilon }}(0)} \vert \xi -\eta \vert ^{-(n-\alpha -\beta )}\bigl(1+ \vert \xi \vert ^{2}\bigr)^{- \frac{n+\alpha }{2}}\bigl(1+ \vert \eta \vert ^{2}\bigr)^{-\frac{n+\alpha }{2}}\,d\xi \,d\eta \\ &\leq C_{1}\lambda \epsilon ^{\beta }. \end{aligned}$$

For $I_{2}$, we have

$$\begin{aligned} I_{2}&:=c^{2} \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})} \frac{ \vert x-x_{*} \vert ^{\gamma } \vert y-x_{*} \vert ^{\gamma }f_{\epsilon }(x)f_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy \\ &\leq c^{2}R^{2\gamma } \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})} \frac{f_{\epsilon }(x)f_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy \\ &\leq c^{2}R^{2\gamma } \int _{\mathbb{R}^{n}} \int _{\mathbb{R}^{n}} \frac{f_{\epsilon }(x)f_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy \\ &=C_{2} R^{2\gamma }, \end{aligned}$$

where $C_{2}:=c^{2}N_{\alpha } \Vert f_{\epsilon } \Vert ^{2}_{L^{q_{\alpha }}( \mathbb{R}^{n})}$. For $I_{3}$, we have

$$\begin{aligned} I_{3}&:=2c \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})} \frac{K(x_{*}) \vert x-x_{*} \vert ^{\gamma }f_{\epsilon }(x)f_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy \\ &\leq 2cR^{\gamma } \int _{B_{R}(x_{*})} \int _{B_{R}(x_{*})} \frac{K(x_{*})f_{\epsilon }(x)f_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy \\ &\leq 2cR^{\gamma } \int _{\mathbb{R}^{n}} \int _{\mathbb{R}^{n}} \frac{K(x_{*})f_{\epsilon }(x)f_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy \\ & = C_{3} R^{\gamma }, \end{aligned}$$

where $C_{3}:=2c K(x_{*}) N_{\alpha } \Vert f_{\epsilon } \Vert ^{2}_{L^{q_{\alpha }}( \mathbb{R}^{n})}$. Therefore, for $\lambda <0$, we can take s satisfying $\frac{\beta }{\gamma }< s$, and $R=\epsilon ^{s}>0$ for some $\epsilon >0$ small enough, such that

$$\begin{aligned} I_{1}+I_{2}+I_{3}&\leq C_{1}\lambda \epsilon ^{\beta }+C_{2}R^{2 \gamma }+C_{3}R^{\gamma } \\ &=C_{1}\lambda \epsilon ^{\beta }+C_{2}\epsilon ^{2s\gamma }+C_{3} \epsilon ^{s\gamma }< 0. \end{aligned}$$

Combining this with (3.2), for $\lambda <0$, $\epsilon >0$ small enough, we have

$$ \int _{\varOmega } \int _{\varOmega }\biggl(\frac{K(x)K(y)}{ \vert x-y \vert ^{n-\alpha }}+ \frac{\lambda G(x)G(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\biggr) \widetilde{f}_{ \epsilon }(x)\widetilde{f}_{\epsilon }(y)\,dx\,dy < K^{2}(x_{*})N_{ \alpha } \Vert f_{\epsilon } \Vert ^{2}_{L^{q_{\alpha }}(\mathbb{R}^{n})}. $$

That is, for any $\lambda <0$, ${Q_{\lambda ,q_{\alpha }}(\varOmega )}< K^{2}(x_{*})N_{\alpha }$.

(II) Let $x_{*}\in \partial \varOmega$. By $(\mathcal{T})$, there exists $\rho _{1}>0$ such that $K(x)-K(x_{*})\leq c \vert x-x_{*} \vert ^{\gamma }$ when $x\in V:=\overline{\varOmega }\cap \overline{B(x_{*},\rho _{1})}$.

Let $0<\rho _{0}<\rho _{1}$, $x_{0}\in V$ satisfying $B_{\rho _{0}}(x_{0})\subset V-\partial V$, $\vert x_{0}-x_{*} \vert =2\rho _{0}$. Then, for any $x\in B_{\rho _{0}}(x_{0})$, we have

$$ K(x)-K(x_{*})\leq C\bigl( \vert x-x_{0} \vert ^{\gamma }+ \vert x_{0}-x_{*} \vert ^{\gamma }\bigr). $$

We define

$$ \widetilde{f}_{\epsilon }(x)= \textstyle\begin{cases} \overline{f}_{\epsilon }(x),&x\in B_{\rho _{0}}(x_{0})\subset \varOmega , \\ 0,&x\in \mathbb{R}^{n}\setminus B_{\rho _{0}}(x_{0}), \end{cases} $$

where $\overline{f}_{\epsilon }(x)=\epsilon ^{-\frac{n+\alpha }{2}}f_{1}( \frac{ \vert x-x_{0} \vert }{\epsilon })=( \frac{\epsilon }{\epsilon ^{2}+ \vert x-x_{0} \vert ^{2}})^{\frac{n+\alpha }{2}}$.

Similar to (I),

$$\begin{aligned}& \int _{\varOmega } \int _{\varOmega }\biggl(\frac{K(x)K(y)}{ \vert x-y \vert ^{n-\alpha }}+ \frac{\lambda G(x)G(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\biggr) \widetilde{f}_{ \epsilon }(x)\widetilde{f}_{\epsilon }(y)\,dx\,dy \\& \quad = \int _{B_{\rho _{0}}(x_{0})} \int _{B_{\rho _{0}}(x_{0})}\biggl( \frac{K(x)K(y)}{ \vert x-y \vert ^{n-\alpha }}+ \frac{\lambda G(x)G(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\biggr) \overline{f}_{ \epsilon }(x)\overline{f}_{\epsilon }(y)\,dx\,dy \\& \quad \leq \int _{B_{\rho _{0}}(x_{0})} \int _{B_{\rho _{0}}(x_{0})} \frac{(K(x_{*})+C( \vert x-x_{0} \vert ^{\gamma }+ \vert 2\rho _{0} \vert ^{\gamma }))(K(x_{*})+C( \vert y-x_{0} \vert ^{\gamma }+ \vert 2\rho _{0} \vert ^{\gamma }))}{ \vert x-y \vert ^{n-\alpha }} \\& \quad \quad{} \times\overline{f}_{\epsilon }(x) \overline{f}_{\epsilon }(y)\,dx\,dy \\& \quad \quad{} +\lambda \int _{B_{\rho _{0}}(x_{0})} \int _{B_{\rho _{0}}(x_{0})} \frac{G(x)G(y)\overline{f}_{\epsilon }(x)\overline{f}_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\,dx\,dy \\& \quad \leq \int _{\mathbb{R}^{n}} \int _{\mathbb{R}^{n}} \frac{K^{2}(x_{*})\overline{f}_{\epsilon }(x)\overline{f}_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy+ \lambda \int _{B_{\rho _{0}}(x_{0})} \int _{B_{\rho _{0}}(x_{0})} \frac{G(x)G(y) \overline{f}_{\epsilon }(x)\overline{f}_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\,dx\,dy \\& \quad \quad{} +2C \int _{B_{\rho _{0}}(x_{0})} \int _{B_{\rho _{0}}(x_{0})} \frac{K(x_{*})( \vert x-x_{0} \vert ^{\gamma }+ \vert 2\rho _{0} \vert ^{\gamma })\overline{f}_{\epsilon }(x)\overline{f}_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy \\& \quad \quad{} +C^{2} \int _{B_{\rho _{0}}(x_{0})} \int _{B_{\rho _{0}}(x_{0})} \frac{( \vert x-x_{0} \vert ^{\gamma }+ \vert 2\rho _{0} \vert ^{\gamma })( \vert y-x_{0} \vert ^{\gamma }+ \vert 2\rho _{0} \vert ^{\gamma })\overline{f}_{\epsilon }(x)\overline{f}_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy \\& \quad = K^{2}(x_{*})N_{\alpha } \Vert \overline{f}_{\epsilon } \Vert ^{2}_{L^{q_{ \alpha }}(\mathbb{R}^{n})}+J_{1}+J_{2}+J_{3}, \end{aligned}$$

(3.3)

where

$$\begin{aligned}& J_{1}:= \lambda \int _{B_{\rho _{0}}(x_{0})} \int _{B_{\rho _{0}}(x_{0})} \frac{G(x)G(y) \overline{f}_{\epsilon }(x)\overline{f}_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\,dx\,dy, \\& J_{2}:= 2C \int _{B_{\rho _{0}}(x_{0})} \int _{B_{\rho _{0}}(x_{0})} \frac{K(x_{*})( \vert x-x_{0} \vert ^{\gamma }+ \vert 2\rho _{0} \vert ^{\gamma })\overline{f}_{\epsilon }(x)\overline{f}_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy, \\& J_{3}:= C^{2} \int _{B_{\rho _{0}}(x_{0})} \int _{B_{\rho _{0}}(x_{0})} \frac{( \vert x-x_{0} \vert ^{\gamma }+ \vert 2\rho _{0} \vert ^{\gamma })( \vert y-x_{0} \vert ^{\gamma }+ \vert 2\rho _{0} \vert ^{\gamma })\overline{f}_{\epsilon }(x)\overline{f}_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\, dy. \end{aligned}$$

As in case (I), we know $J_{1}\leq C_{4}\lambda \epsilon ^{\beta }$. For $J_{2}$, we have

$$\begin{aligned} J_{2}&=2C \int _{B_{\rho _{0}}(x_{0})} \int _{B_{\rho _{0}}(x_{0})} \frac{K(x_{*})( \vert x-x_{0} \vert ^{\gamma }+ \vert 2\rho _{0} \vert ^{\gamma })\overline{f}_{\epsilon }(x)\overline{f}_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy \\ &\leq CK(x_{*})\rho _{0}^{\gamma } \int _{B_{\rho _{0}}(x_{0})} \int _{B_{ \rho _{0}}(x_{0})} \frac{\overline{f}_{\epsilon }(x)\overline{f}_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy \\ &\leq C_{5}\rho _{0}^{\gamma }. \end{aligned}$$

For $J_{3}$, we have

$$\begin{aligned} J_{3}&=C^{2} \int _{B_{\rho _{0}}(x_{0})} \int _{B_{\rho _{0}}(x_{0})} \frac{( \vert x-x_{0} \vert ^{\gamma }+ \vert 2\rho _{0} \vert ^{\gamma })( \vert y-x_{0} \vert ^{\gamma }+ \vert 2\rho _{0} \vert ^{\gamma })\overline{f}_{\epsilon }(x)\overline{f}_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy \\ &\leq C^{2}\rho _{0}^{2\gamma } \int _{B_{\rho _{0}}(x_{0})} \int _{B_{ \rho _{0}}(x_{0})} \frac{\overline{f}_{\epsilon }(x)\overline{f}_{\epsilon }(y)}{ \vert x-y \vert ^{n-\alpha }}\,dx\,dy \\ &\leq C_{6}\rho _{0}^{2\gamma }. \end{aligned}$$

Taking s with $\frac{\beta }{\gamma }< s$ and $\rho _{0}=\epsilon ^{s}>0$, then

$$\begin{aligned} J_{1}+J_{2}+J_{3}&\leq C_{4}\lambda \epsilon ^{\beta }+C_{5}\rho _{0}^{ \gamma }+C_{6} \rho _{0}^{2\gamma } \\ &=C_{4}\lambda \epsilon ^{\beta }+C_{5}\epsilon ^{s\gamma }+C_{6} \epsilon ^{2s\gamma }< 0 \end{aligned}$$

for $\epsilon >0$ small enough. Thus, combining this with (3.3), for $\lambda <0$, $\epsilon >0$ small enough, we have

$$ \int _{\varOmega } \int _{\varOmega }\biggl(\frac{K(x)K(y)}{ \vert x-y \vert ^{n-\alpha }}+ \frac{\lambda G(x)G(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\biggr) \widetilde{f}_{ \epsilon }(x)\widetilde{f}_{\epsilon }(y)\,dx\,dy < K^{2}(x_{*})N_{\alpha } \Vert \overline{f}_{\epsilon } \Vert ^{2}_{L^{q_{\alpha }}(\mathbb{R}^{n})}. $$

That is, for any $\lambda <0$, we have ${Q_{\lambda ,q_{\alpha }}(\varOmega )}< K^{2}(x_{*})N_{\alpha }$.

On the other hand, for any $\lambda \in (- \frac{K^{2}(x_{*})}{d^{\beta }(\varOmega ) G^{2}(\tilde{x}_{*})},0)$, we also have ${Q_{\lambda ,q_{\alpha }}(\varOmega )}>0$. So we complete the proof. □

In order to prove the existence of weak solutions, we need to prove that the minimal energy ${Q_{\lambda ,q_{\alpha }}(\varOmega )}$ is attained.

Proposition 3.5

For given$\lambda \in (- \frac{K^{2}(x_{*})}{d^{\beta }(\varOmega ) G^{2}(\tilde{x}_{*})},0)$, ${Q_{\lambda ,q_{\alpha }}(\varOmega )}$is attained by some positive function$f_{*}\in L^{q_{\alpha }}(\varOmega )$.

For $q< q_{\alpha }$, we consider

$$\begin{aligned}& Q_{\lambda ,q}(\varOmega )\\& \quad =\inf_{f\in L_{+}^{q}(\varOmega )} \frac{\int _{\varOmega }\int _{\varOmega }f(x)(K(x) \vert x-y \vert ^{-(n-\alpha )}K(y)+\lambda G(x) \vert x-y \vert ^{-(n-\alpha -\beta )}G(y))f(y)\,dx\,dy}{ \Vert f \Vert _{L^{q}(\varOmega )}^{ 2}}. \end{aligned}$$

By Lemma 3.2, the infimum is attained by the positive function $f_{q}$ which satisfies the integral equation with the subcritical exponent

$$ Q_{\lambda ,q}(\varOmega )f^{q-1}(x)= \int _{\varOmega } \frac{K(x)f(y)K(y)}{ \vert x-y \vert ^{n-\alpha }}\,dy+\lambda \int _{\varOmega } \frac{G(x)f(y)G(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\,dy, \quad x\in \overline{ \varOmega }. $$

(3.4)

That is, $f_{q}$ is the minimal energy solution to Eq. (3.4). It is easy to see that $\Vert f_{q} \Vert _{L^{q}(\varOmega )}=1$, $f_{q}\in C(\overline{\varOmega })$ and $Q_{\lambda ,q}\rightarrow Q_{\lambda }$ as $q\rightarrow (q_{\alpha })^{-}$.

Lemma 3.6

For$\lambda \in (- \frac{K^{2}(x_{*})}{d^{\beta }(\varOmega )G^{2}(\tilde{x}_{*})},0)$, $q\in (0,q_{ \alpha })$. Let$f_{q}>0$be a minimal energy solution to (3.4), where$\Vert f_{q} \Vert _{L^{q}(\varOmega )}=1$. If$0< Q_{\lambda ,q}(\varOmega )\leq K^{2}(x_{*})N_{\alpha }-\epsilon$for some$\epsilon >0$, then there exists$C>0$such that$\frac{1}{C}\leq f_{q}(x)\leq C$uniformly for all$x\in \overline{\varOmega }$, $q\in (0,q_{\alpha })$.

Proof

We prove it by modifying the argument of Lemma 4.3 in [1].

For any $x\in \overline{\varOmega }$, $q\in (0,q_{1})$, we see that $\max_{\overline{\varOmega }}f_{q}(x)=f_{q}(x_{q})\leq C<\infty$ holds uniformly, where $0< q_{1}< q_{\alpha }$.

We first prove that $\max_{\overline{\varOmega }}f_{q}(x)=f_{q}(x_{q})\leq C< \infty$ holds uniformly as $q\rightarrow (q_{\alpha })^{-}$. Otherwise, $f_{q}(x_{q})\rightarrow \infty$, where $x_{q}\to \tilde{x}$, up to a subsequence, as $q\rightarrow (q_{\alpha })^{-}$. Denote

$$ \mu _{q}:=f_{q}^{-\frac{2-q}{\alpha }}(x_{q}), \quad \quad \varOmega _{\mu }:= \frac{\varOmega -x_{q}}{\mu _{q}}=\biggl\{ z\Bigm| z= \frac{x-x_{q}}{\mu _{q}},x\in \varOmega \biggr\} . $$

We define

$$ g_{q}(z)=\mu _{q}^{\frac{\alpha }{2-q}}f_{q}(\mu _{q}z+x_{q}), \quad z\in \overline{\varOmega }_{\mu }. $$

Thus $g_{q}$ satisfies

$$\begin{aligned} Q_{\lambda ,q}(\varOmega )g_{q}^{q-1}(z)={}& \int _{\varOmega _{\mu }} \frac{K(\mu _{q}z+x_{q})g_{q}(y)K(\mu _{q}y +x_{q})}{ \vert z-y \vert ^{n-\alpha }}\,dy \\ &{} +\lambda \mu _{q}^{\beta } \int _{\varOmega _{\mu }} \frac{G(\mu _{q}z+x_{q})g_{q}(y)G(\mu _{q}y +x_{q})}{ \vert z-y \vert ^{n-\alpha -\beta }}\,dy, \quad z\in \overline{ \varOmega }_{\mu }, \end{aligned}$$

(3.5)

and $g_{q}(0)=1$, $g_{q}(z)\in (0,1]$.

For convenience, we define $h_{q}(z):=g_{q}^{q-1}(z)$. So (3.5) is equivalent to

$$\begin{aligned} Q_{\lambda ,q}(\varOmega )h_{q}(z)={}& \int _{\varOmega _{\mu }} \frac{K(\mu _{q}z+x_{q})h_{q}^{p-1}(y)K(\mu _{q}y+x_{q})}{ \vert z-y \vert ^{n-\alpha }}\,dy \\ &{}+\lambda \mu _{q}^{\beta } \int _{\varOmega _{\mu }} \frac{G(\mu _{q}z+x_{q})h_{q}^{p-1}(y)G(\mu _{q}y +x_{q})}{ \vert z-y \vert ^{n-\alpha -\beta }}\,dy, \quad z\in \overline{ \varOmega }_{\mu }, \end{aligned}$$

where $\frac{1}{p}+\frac{1}{q}=1$, $h_{q}(0)=1$, $h_{q}(z)\geq 1$.

Claim: There exist $C_{1},C_{2}>0$ such that

$$ 0< C_{1}\bigl(1+ \vert z \vert ^{\alpha -n} \bigr)\leq h_{q}(z)\leq C_{2}\bigl(1+ \vert z \vert ^{\alpha -n}\bigr),z \in \tilde{\varOmega }, $$

(3.6)

holds uniformly for any domain $\tilde{\varOmega }\subset \varOmega _{\mu }$ as $q\rightarrow (q_{\alpha })^{-}$.

The claim can be verified by a similar argument to that in [1], we omit it here. Thus $h_{q}(z)$ is equicontinuous in any bounded domain $\widehat{\varOmega }\subset \varOmega _{\mu }$ as $q\rightarrow (q_{\alpha })^{-}$. In fact, for $R>0$,

$$\begin{aligned}& Q_{\lambda ,q}(\varOmega )h_{q}(z) \\& \quad = \int _{\varOmega _{\mu }\setminus B(0,R)} \frac{K(\mu _{q}z+x_{q})h_{q}^{p-1}(y)K(\mu _{q}y+x_{q})}{ \vert z-y \vert ^{n-\alpha }}\,dy \\& \quad \quad {} + \int _{\varOmega _{\mu }\cap B(0,R)} \frac{K(\mu _{q}z+x_{q})h_{q}^{p-1}(y)K(\mu _{q}y+x_{q})}{ \vert z-y \vert ^{n-\alpha }}\,dy \\& \quad \quad {} +\lambda \mu _{q}^{\beta } \int _{\varOmega _{\mu }\setminus B(0,R)} \frac{G(\mu _{q}z+x_{q})h_{q}^{p-1}(y)G(\mu _{q}y +x_{q})}{ \vert z-y \vert ^{n-\alpha -\beta }}\,dy \\& \quad \quad {} +\lambda \mu _{q}^{\beta } \int _{\varOmega _{\mu }\cap B(0,R)} \frac{G(\mu _{q}z+x_{q})h_{q}^{p-1}(y)G(\mu _{q}y +x_{q})}{ \vert z-y \vert ^{n-\alpha -\beta }}\,dy. \end{aligned}$$

Notice that

$$\begin{aligned} & \int _{\varOmega _{\mu }\setminus B(0,R)} \frac{h_{q}^{p-1}(y)}{ \vert z-y \vert ^{n-\alpha }}\bigl(K(\mu _{q}z+x_{q})K( \mu _{q}y+x_{q})+ \lambda \mu _{q}^{\beta }G( \mu _{q}z+x_{q})G(\mu _{q}y +x_{q}) \vert z-y \vert ^{ \beta }\bigr)\,dy \\ &\quad \geq \bigl(K^{2}(x_{*})- \vert \lambda \vert d^{\beta }(\varOmega )G^{2}(\tilde{x}_{*})\bigr) \int _{\varOmega _{\mu }\setminus B(0,R)} \frac{h_{q}^{p-1}(y)}{ \vert z-y \vert ^{n-\alpha }}\,dy\geq 0. \end{aligned}$$

For any $\epsilon >0$ small enough, by (3.6), we have

$$\begin{aligned} 0 &\leq \int _{\varOmega _{\mu }\setminus B(0,R)} \frac{h_{q}^{p-1}(y)}{ \vert z-y \vert ^{n-\alpha }}\bigl(K(\mu _{q}z+x_{q})K( \mu _{q}y+x_{q}) \\ &\quad{} +\lambda \mu _{q}^{\beta }G(\mu _{q}z+x_{q}) \vert z-y \vert ^{\beta }G( \mu _{q}y+x_{q})\bigr)\,dy \\ &\leq C \int _{\varOmega _{\mu }\setminus B(0,R)} \frac{h_{q}^{p-1}(y)}{ \vert y \vert ^{n-\alpha }}\,dy \\ &\leq C \int _{R}^{\infty }r^{(\alpha -n)(p-1)+\alpha -1}\,dr \\ &=CR^{(\alpha -n)(p-1)+\alpha }< \epsilon , \end{aligned}$$

(3.7)

where $R>0$ is large enough, $q\rightarrow (q_{\alpha })^{-}$, $z\in \widehat{\varOmega }$. Since $\beta < n$, by using the same argument as above, we also have

$$ \biggl\vert \lambda \mu _{q}^{\beta } \int _{\varOmega _{\mu }\cap B(0,R)} \frac{G(\mu _{q}z+x_{q})h_{q}^{p-1}(y)G(\mu _{q}y +x_{q})}{ \vert z-y \vert ^{n-\alpha -\beta }}\,dy \biggr\vert < \epsilon , $$

(3.8)

when $R>0$ large enough, $q\rightarrow (q_{\alpha })^{-}$.

On the other hand, it is easy to see that, for $z\in \widehat{\varOmega }$, $\int _{\varOmega _{\mu }\cap B(0,R)} \frac{K(\mu _{q}z+x_{q})h_{q}^{p-1}(y)K(\mu _{q}y+x_{q})}{ \vert z-y \vert ^{n-\alpha }}\,dy \in C^{1}(\widehat{\varOmega })$. Combining this with (3.7) and (3.8), we conclude that $h_{q}(z)$ is equicontinuous in any bounded domain $\widehat{\varOmega }\subset \mathbb{R}^{n}$ as $q\rightarrow (q_{\alpha })^{-}$.

When $q\rightarrow (q_{\alpha })^{-}$, we distinguish two cases:

Case 1.$\varOmega _{\mu }\rightarrow \mathbb{R}^{n}_{T}:=\{(z_{1},z_{2},\ldots,z_{n})\mid z_{n}>-T\}$ for some $T\ge 0$, and $h_{q}(z)\rightarrow h(z)\in C(\mathbb{R}^{n}_{T})$ holds uniformly on any compact sets of $\mathbb{R}^{n}_{T}$, where $h(z)$ satisfying

$$ Q_{\lambda }h(z)=K^{2}(\widetilde{x}) \int _{\mathbb{R}^{n}_{T}} \frac{h^{p_{\alpha }-1}(y)}{ \vert z-y \vert ^{n-\alpha }}\,dy, \quad h(0)=1. $$

Then, similar to Lemma 4.3 in [1], we obtain a contradiction.

Case 2.$\varOmega _{\mu }\rightarrow \mathbb{R}^{n}$, $h_{q}(z)\rightarrow h(z)\in C(\mathbb{R}^{n})$ holds uniformly on any compact sets of $\mathbb{R}^{n}$, where $h(z)$ satisfying

$$ Q_{\lambda }h(z)=K^{2}(\widetilde{x}) \int _{\mathbb{R}^{n}} \frac{h^{p_{\alpha }-1}(y)}{ \vert z-y \vert ^{n-\alpha }}\,dy, \quad h(0)=1. $$

Again similar to Lemma 4.3 in [1], we obtain a contradiction.

Thus we conclude that there exists $C>0$ such that $f_{q}(y)\leq C$ uniformly for $y\in \overline{\varOmega }$, $q\in (0,q_{\alpha })$.

On the other hand, if $\min_{\overline{\varOmega }}f_{q}(x):=f_{q}(\widetilde{x}_{q}) \rightarrow 0$ as $q\rightarrow (q_{\alpha })^{-}$. Then

$$ \infty \leftarrow f_{q}^{q-1}(\widetilde{x}_{q})= \int _{\varOmega } \frac{K(\widetilde{x}_{q})f_{q}(y)K(y)}{ \vert \widetilde{x}_{q}-y \vert ^{n-\alpha }}\,dy+ \lambda \int _{\varOmega } \frac{G(\widetilde{x}_{q})f_{q}(y)G(y)}{ \vert \widetilde{x}_{q}-y \vert ^{n-\alpha -\beta }}\,dy \leq C< \infty , $$

as $q\rightarrow (q_{\alpha })^{-}$, which implies a contradiction. □

Proof of Proposition 3.5

By Lemma 3.6, $\{f_{q}\}$ are uniformly bounded above and bounded below by a positive constant. Thus the $\{f_{q}\}$ are equicontinuous due to Eq. (3.4). It follows that $f_{q}\rightarrow f_{*}$ as $q\rightarrow (q_{\alpha })^{-}$ in $C(\overline{\varOmega })$, and $f_{*}$ is the energy minimizer for $Q_{\lambda }$. □

Proof of Theorem 1.1 (ii)

Lemma 3.4 and Proposition 3.5 imply the existence of a positive solution $f\in L^{q_{\alpha }}(\varOmega )\cap C(\overline{\varOmega })$ to Eq. (1.1) for $q=\frac{2n}{n+\alpha }$, $\lambda \in (- \frac{K^{2}(x_{*})}{d^{\beta }(\varOmega )G^{2}(\tilde{x}_{*})},0)$. It is also easy to see that $f\in C^{1}(\overline{\varOmega })$. □

3.3 Nonexistence—critical and supercritical case

We first state a Pohozaev type identity.

Lemma 3.7

Assume that the origin is inΩand the domain is star-shaped with respect to the origin. If$u\in C^{1}(\overline{\varOmega })$is a nonnegative solution to

$$ u(x)= \int _{\varOmega }\frac{K(x)u^{p-1}(y)K(y)}{ \vert x-y \vert ^{n-\alpha }}\,dy+ \lambda \int _{\varOmega } \frac{G(x)u^{p-1}(y)G(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\,dy, \quad x\in\overline{ \varOmega }, $$

where$p\neq 0$, $\lambda \in \mathbb{R}$, $K(x),G(x)\in C^{1}( \overline{\varOmega })$, then

$$\begin{aligned} & \biggl(\frac{n}{p}+\frac{\alpha -n}{2}\biggr) \int _{\varOmega }u^{p}(x)\,dx \\ &\quad =-\frac{\lambda \beta }{2} \int _{\varOmega } \int _{\varOmega } \frac{G(x)u^{p-1}(x)u^{p-1}(y)G(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\,dy\,dx+ \frac{1}{p} \int _{\partial \varOmega }(x\cdot \nu )u^{p}(x)\,d\sigma \\ & \quad\quad{} - \int _{\varOmega } \int _{\varOmega } \frac{(x,\nabla K(x))u^{p-1}(x)u^{p-1}(y)K(y)}{ \vert x-y \vert ^{n-\alpha }}\,dy\,dx \\ & \quad\quad{} - \lambda \int _{\varOmega } \int _{\varOmega } \frac{(x,\nabla G(x))u^{p-1}(x)u^{p-1}(y)G(y)}{ \vert x-y \vert ^{n-\alpha -\beta }}\,dy\,dx, \end{aligned}$$

whereνis the outward unit normal vector to∂Ω.

Proof

The argument is standard. We omit it here. □

Proof of Theorem 1.1(iii)

We can prove by using Lemma 3.7 and a similar argument to that used in [1]. We omit it here. □