Effect of the protection zone in a diffusive ratio-dependent predator–prey model with fear and Allee effect

Abstract

In this paper, we study the influence of a protection zone for the prey on a diffusive predator–prey model with fear factor and Allee effect. The prior estimate, global existence, nonexistence of nonconstant positive solutions and bifurcation from semitrivial solutions are well discussed. We show the existence of a critical patch value $\lambda ^{D}_{1}(\Omega _{0})$ of the protection zone, described by the principal eigenvalue of the Laplacian operator over $\Omega _{0}$ with Neumann boundary conditions. When the mortality rate of the predator $\mu \geq d_{2}\lambda ^{D}_{1}(\Omega _{0})$, we show that the semitrivial solutions $(1,0)$ and $(\theta,0)$ are unstable and there is no bifurcation occurring along respective semitrivial branches.

1 Introduction

The predator–prey model is one of the most basic models to study the interspecific relationship, which is still being investigated widely [1, 2]. Because of the movement of the prey and predator, the predator–prey model can be modified in the presence of a spatial diffusion model. The interaction between predator and prey is described more accurately by the diffusive system [3–5].

In 1931, Allee offered a new population growth rate which was called by his name the Allee effect [6]. Allee effect is applied to populations with a too sparse species density, hence the growth function of endangered species usually is recognized as exhibiting Allee effect pattern [7]. In 2014, Cui and Shi [8] studied a diffusive predator–prey system with a strong Allee effect. They analyzed the dynamics and steady state solutions of the system. Their results show that the overexploitation phenomenon can be avoided if the Allee effect threshold is low. The works [9, 10] also confirm that the Allee effect can better explain the correlation between population size or density and the average individual fitness.

For surviving under the risk of predation, some species will form protection zones using herd behavior. In this case the predator hunt only prey herd at the boundary [11–15]. In addition, we can establish natural reserves in habitats to save endangered species. Du et al. [16] first proposed a diffusive predator–prey model with protection zone in 2006. They confirmed that building a protection zone is an effective method to stabilize interspecific relations. The main assumption is to assume that prey species are free in habitat and predator lives outside the protection zone. Hence prey can freely enter or leave the protection zone but the predator species are kept out. Later on, some experts [17–19] also studied the predator–prey model with a protection zone model.

In ecology, the fear effect reflects the fact that prey populations avoid predators for survival. Wang et al. [20] first proposed a predator–prey model incorporating the cost of the fear effect into prey reproduction in 2016. Their mathematical results show that high levels of fear can stabilize the predator–prey model by excluding the existence of periodic solutions. Moreover, some researchers also confirmed that the fear of prey will reduce the size of the prey population, and the influence level of fear even exceeds the direct killing of predators in some circumstances. See, for example, the works [21–23].

Inspired by the above description, for the purpose of protecting endangered species, we adopt a more appropriate Allee effect growth function with the fear effect. Meanwhile, a nature reserve is built in the habitat of predators and prey. When the reserve is established, the predator spends more time looking for prey, then predator growth rate is a function of the ratio of prey to predator abundance. This case can be modeled as a ratio-dependent function, which is more reasonable in the predator–prey model due to the absence of biological control paradox [18, 24]. To get close to reality, a homogeneous no-flux boundary condition is adopted, when the predator and prey species live in a closed environment. Therefore, a diffusive predator–prey model considering fear, Allee effect, and prey protection zone is as follows:

$$\begin{aligned} \textstyle\begin{cases} u_{t}=d_{1} \Delta {u} + u(1-u)(\frac{u}{\theta }-1) \frac{1}{1+\eta v}-\frac{b(x)uv}{mu+v} \quad\text{for } x \in {\Omega }, t>0, \\ v_{t}=d_{2} \Delta {v} + \frac{c(x)uv}{mu+v}-\mu v \quad\text{for } x\in {\Omega _{1}}, t>0, \\ \partial _{\nu } u= 0 \quad \text{for } x\in {\partial \Omega }, t>0,\qquad \partial _{\nu } v= 0 \quad\text{for } x\in {\partial \Omega _{1}}, t>0, \\ u(x,0)=u_{0}(x)\geq 0 \quad\text{for } x\in {\Omega },\qquad v(x,0)=v_{0}(x) \geq 0 \quad\text{for } x\in {\Omega _{1}}, \end{cases}\displaystyle \end{aligned}$$

(1.1)

where the model has been obtained with a nondimensionalization process as in [25]. In this model, Ω is bounded in $\mathbf{R}^{N}$ with a smooth boundary ∂Ω; $\Omega _{0}$ is a subdomain of Ω and $\partial \Omega _{0}$ is also smooth; the region $\Omega _{0}$ is a prey protection zone. It should be pointed out that the effective living space for prey species is Ω and that for predators is $\Omega _{1}:={\Omega }\backslash \overline{\Omega }_{0}$; $u(x,t)$ and $v(x,t)$ are the densities of prey and predators, and their diffusive coefficients are $d_{1}$ and $d_{2}$, respectively; the function $u(1-u)(\frac{u}{\theta }-1)$ refers to the Allee effect growth rate of prey; θ is called the Allee threshold value for the strong Allee effect so that $0<\theta <1$; weak Allee effect means $-1<\theta <0$; $\frac{1}{1+\eta v}$ stands for the fear of prey; η is the level of fear; $\frac{uv}{mu+v}$ is a ratio-dependent response function; $c(x)$ is the conversion rate of the prey captured by predators; μ is the mortality rate of predators. The function $b(x)$ stands for the loss of prey because of the predation, $b(x)=0$ for $x\in {\Omega _{0}}$, and $b(x)\geq c(x)$ for $x\in {\Omega _{1}}$.

The steady-state system corresponding to (1.1) is as follows:

$$\begin{aligned} \textstyle\begin{cases} -d_{1} \Delta {u} = u(1-u)(\frac{u}{\theta }-1)\frac{1}{1+\eta v}- \frac{b(x)uv}{mu+v} \quad\text{for } x\in {\Omega }, \\ -d_{2} \Delta {v} = \frac{c(x)uv}{mu+v}-\mu v \quad\text{for } x\in {\Omega _{1}}, \\ \partial _{\nu } u= 0 \quad\text{for } x\in {\partial \Omega }, \qquad \partial _{\nu } v= 0 \quad \text{for } x\in {\partial \Omega _{1}}. \end{cases}\displaystyle \end{aligned}$$

(1.2)

The organization of this paper is as follows. In Sect. 2, some useful dynamics are analyzed, including the global existence, prior estimates, and the nonexistence of nonconstant positive solutions. In Sect. 3, the bifurcations from semitrivial solutions $(1,0)$ and $(\theta,0)$ are proved, and the existence of a critical patch size $\lambda ^{D}_{1}(\Omega _{0})$ is given.

2 Dynamical analysis

In this section, the global existence of solutions to (1.1) is proved, and a priori estimates for (1.2) are established. Finally, the nonexistence of nonconstant positive steady state solutions of (1.2) is proved.

To facilitate the discussion, we have the following conditions:

(H₁) $\eta,m,\mu,d_{1},d_{2}>0$;

(H₂) $b(x)\geq c(x)>0$, for $b(x),c(x)\in C(\overline{\Omega }_{1}), \forall x\in { \overline{\Omega }_{1}}$;

(H₃) $b_{*}=\min_{x\in \overline{\Omega }_{1}}b(x), b^{*}=\max_{x\in \overline{\Omega }_{1}}b(x), c_{*}=\min_{x \in \overline{\Omega }_{1}}c(x), c^{*}=\max_{x\in \overline{\Omega }_{1}}c(x)$.

Theorem 2.1

Assume that $(H_{1})$ holds and $\theta \in (0,1)$.

$\mathrm{(i)}$ If $u_{0}(x)\geq 0$ for $x\in \Omega$, and $v_{0}(x)\geq 0$ for $x\in \Omega _{1}$, then the unique solution $(u(x,t),v(x,t))$ of (1.1) satisfies $u(x,t)>0$ for $(x,t)\in \overline{\Omega }\times (0,+\infty )$, and $v(x,t)>0$ for $(x,t)\in \overline{\Omega }_{1}\times (0,+\infty )$;

$\mathrm{(ii)}$ If $u_{0}(x)\leq \theta$ and $(u_{0}(x),v_{0}(x)) \not \equiv (\theta,0)$, then $\lim_{t\rightarrow \infty } u(x,t)=0$ for $x\in \overline{\Omega }$, $\lim_{t\rightarrow \infty }v(x,t)=0$ for $x\in \overline{\Omega }_{1}$.

Proof

Denote

$$\begin{aligned} P(u,v)=u(1-u) \biggl(\frac{u}{\theta }-1\biggr)\frac{1}{1+\eta v}- \frac{b(x)uv}{mu+v},\qquad Q(u,v)=\frac{c(x)uv}{mu+v}-\mu v. \end{aligned}$$

Let $(\underline{u}(x,t),\underline{v}(x,t))=(0,0)$, $(\overline{u}(x,t), \overline{v}(x,t))=(u^{*}(t),v^{*}(t))$, where $(u^{*}(t),v^{*}(t))$ is the unique spatially homogeneous solution of

$$\begin{aligned} \textstyle\begin{cases} \frac{\mathrm{d}u}{\mathrm{d}t}=u(1-u)(\frac{u}{\theta }-1) \frac{1}{1+\eta v} , \\ \frac{\mathrm{d}v}{\mathrm{d}t}=\frac{c(x)uv}{mu+v}-\mu v, \\ u(0)=u^{*}>0, \qquad v(0)=v^{*}>0, \end{cases}\displaystyle \end{aligned}$$

(2.1)

where $u^{*}=\sup_{x\in {\overline{\Omega }}}u_{0}(x)$, $v^{*}=\sup_{ x\in {\overline{\Omega }_{1}}}v_{0}(x)$. From the comparison principle, it is easy to get that $(\underline{u}(x,t),\underline{v}(x,t))$ and $(\overline{u}(x,t),\overline{v}(x,t))$ are the lower and upper solutions to (1.1), respectively. Since

$$\begin{aligned} \overline{u}_{t}-\triangle \overline{u}(x,t)\geq P(\overline{u}, \underline{v}),\qquad \underline{u}_{t}-\triangle \underline{u}(x,t) \leq P( \underline{u},\overline{v}), \end{aligned}$$

and

$$\begin{aligned} \overline{v}_{t}-\triangle \overline{v}(x,t)\geq Q(\overline{u}, \overline{v}),\qquad \underline{v}_{t}-\triangle \underline{v}(x,t) \leq Q( \underline{u},\underline{v}), \end{aligned}$$

the boundary conditions are satisfied. Therefore, the results for lower/upper-solutions in Theorem 8.3.3 in [26] show that (1.1) has a unique globally defined solution which satisfies

$$\begin{aligned} 0\leq u(x,t)\leq u^{*}(t),\qquad 0\leq v(x,t)\leq v^{*}(t). \end{aligned}$$

Meanwhile, by the strong maximum principle, we can get $u(x,t)>0$ for $(x,t)\in \overline{\Omega }\times (0,+\infty )$, and $v(x,t)>0$ for $(x,t)\in \overline{\Omega }_{1}\times (0,+\infty )$. This proves part (i).

Now we prove part (ii). We know $u_{0}(x)\leq u^{*}$ from part (i), therefore $u_{0}(x)\leq u^{*}<\theta$. If $(u_{0}(x),v_{0}(x))\neq (\theta,0)$ and $\theta <1$, then $u^{*}(t)\rightarrow 0,v^{*}(t)\rightarrow 0$ when $t\rightarrow \infty$, obviously $u(x,t)\rightarrow 0$ for $x\in \overline{\Omega }$, $v(x,t)\rightarrow 0$ for $x\in \overline{\Omega }_{1}$ as $t\rightarrow \infty$. □

It follows from above that the existence, uniqueness, and asymptotic properties of solutions have no direct connection with fear and prey protection zone. The asymptotic properties of solutions only depend on the strong Allee effect.

Lemma 2.2

([27])

Suppose $g(x,w)\in C(\overline{\Omega } \times \mathbb{R}^{1})$. If $w(x)\in C^{2}(\Omega ) \cap C^{1}(\overline{\Omega })$ satisfy

$$\begin{aligned} \Delta w(x)+g\bigl(x,w(x)\bigr)\geq 0(\leq 0),\quad x\in \Omega,\qquad \frac{\partial w}{\partial n} \leq 0(\geq 0),\quad x\in \partial \Omega, \end{aligned}$$

and $w(x_{0})=\max_{\bar{\Omega }} w(\min_{\bar{\Omega }} w)$, then $g(x_{0},w(x_{0}))\geq 0(\leq 0)$.

Theorem 2.3

Assume that $(H_{1})$–$(H_{2})$ hold, and $(u(x),v(x))$ is a nonnegative and nontrivial solution of (1.2). Then

$$\begin{aligned} 0< u(x)\leq 1 \quad\textit{for } x\in \Omega\quad \textit{and}\quad 0< v(x)\leq \frac{(1-\theta )^{2}}{4\mu \theta }+\frac{d_{1}}{d_{2}} \quad\textit{for } x\in \Omega _{1}. \end{aligned}$$

(2.2)

Proof

From Theorem 2.1, we know that $u>0,v>0$. Then, it is easy to see that $0< u(x)\leq 1$, by Lemma 2.2. Adding the two functions of system (1.1), we have

$$\begin{aligned} -(d_{1}\Delta u+d_{2}\Delta v)&=u(1-u) \biggl( \frac{u}{\theta }-1\biggr) \frac{1}{1+\eta v}+\bigl(c(x)-b(x)\bigr) \frac{uv}{mu+v}-\mu v \\ &\leq u(1-u) \biggl(\frac{u}{\theta }-1\biggr)\frac{1}{1+\eta v}-\mu v \\ &\leq u\frac{(1-\theta )^{2}}{4\theta }-\mu v \\ &\leq \biggl(\frac{(1-\theta )^{2}}{4\theta }+ \frac{\mu d_{1}}{d_{2}} \biggr)- \frac{\mu }{d_{2}}(d_{1}u + d_{2}v), \end{aligned}$$

which leads to

$$\begin{aligned} \Delta (d_{1} u+d_{2} v)+ \biggl(\frac{(1-\theta )^{2}}{4\theta }+ \frac{\mu d_{1}}{d_{2}} \biggr)-\frac{\mu }{d_{2}}(d_{1}u + d_{2}v) \geq 0. \end{aligned}$$

By Lemma 2.2, we obtain

$$\begin{aligned} d_{1} u+d_{2} v\leq \frac{d_{2}(1-\theta )^{2}}{4\mu \theta }+d_{1}, \end{aligned}$$

which implies

$$\begin{aligned} v(x)\leq \frac{(1-\theta )^{2}}{4\mu \theta }+\frac{d_{1}}{d_{2}}. \end{aligned}$$

 □

Theorem 2.4

Assume that $(H_{1})$–$(H_{3})$ hold, and $\theta \in (0,1)$. There exists a $D^{*}=D^{*}(m,\mu, b_{*},c^{*},\Omega )$ such that if $\min \{d_{1},d_{2}\}>D^{*}$, then (1.2) has no nonconstant positive solution.

Proof

Let $(u,v)$ be a nonconstant positive steady state solution of (1.2), and denote

$$\begin{aligned} \overline{u}=\frac{1}{ \vert \Omega \vert } \int _{\Omega } u\,\mathrm{d}x, \qquad\overline{v}=\frac{1}{ \vert \Omega \vert } \int _{\Omega } v\,\mathrm{d}x,\qquad F(u,v)=u(1-u) \biggl( \frac{u}{\theta }-1\biggr)\frac{1}{1+\eta v}. \end{aligned}$$

From Theorem 2.3, we know that $\overline{u} \leq 1$. Adding the two equations of (1.2) and integrating on Ω, we get

$$\begin{aligned} - \int _{\Omega }(d_{1} \Delta u+d_{2}\Delta v) \, \mathrm{d}x= \int _{ \Omega } \biggl(F(u,v)+\frac{(c(x)-b(x))uv}{mu+v}-\mu v \biggr) \, \mathrm{d}x. \end{aligned}$$

Since $b(x)\geq c(x)$ for $x\in \overline{\Omega }_{1}$, it is easy to obtain

$$\begin{aligned} \mu \int _{\Omega }v\,\mathrm{d}x= \int _{\Omega } \biggl(F(u,v)+ \frac{(c(x)-b(x))uv}{mu+v} \biggr)\, \mathrm{d}x\leq \int _{\Omega _{1}}F(u,v) \,\mathrm{d}x \leq \frac{(1-\theta )^{2}}{4\theta }. \end{aligned}$$

Thus

$$\begin{aligned} \overline{v}=\frac{1}{ \vert \Omega \vert } \int _{\Omega }v\,\mathrm{d}x\leq \frac{(1-\theta )^{2}}{4\mu \theta }. \end{aligned}$$

Multiplying the first equation of (1.2) by $(u-\overline{u})$, and integrating on Ω, it follows from the Green formula and Young inequality that

$$\begin{aligned} &d_{1}{ \int _{\Omega } \bigl\vert \nabla (u-\overline{u}) \bigr\vert ^{2}\,\mathrm{d}x} \\ &\quad= \int _{ \Omega } F(u,v) (u-\overline{u})\,\mathrm{d}x- \int _{\Omega } \frac{b(x)uv}{mu+v}(u-\overline{u})\,\mathrm{d}x \\ &\quad= \int _{\Omega } \bigl(F(u,v)-F(\overline{u},\overline{v})\bigr) (u- \overline{u}) \,\mathrm{d}x- \int _{\Omega } \frac{b(x)(u-\overline{u})(v-\overline{v})}{mu+v}(u-\overline{u}) \,\mathrm{d}x \\ &\quad\leq \frac{1+\theta }{\theta } \int _{\Omega } \frac{(u+\overline{u})(u-\overline{u})^{2}}{1+\eta v}\,\mathrm{d}x+ \eta \int _{\Omega } \frac{\overline{u}(1-\overline{u})(\frac{\overline{u}}{\theta }-1)(u-\overline{u})(v-\overline{v})}{(1+\eta v)(1+\eta \overline{v})} \,\mathrm{d}x \\ &\qquad{}+ \int _{\Omega } \frac{mb_{*}\overline{u}(u-\overline{u})[\overline{v}(u-\overline{u})+\overline{u}(\overline{v}-v)]}{(mu+v)(m\overline{u}+\overline{v})} \,\mathrm{d}x \\ &\quad\leq \biggl(\frac{2(\theta +1)}{\theta }+ \frac{b_{*}(1-\theta )^{2}}{4\mu \theta m}\biggr) \int _{\Omega }(u- \overline{u})^{2}\,\mathrm{d}x+\biggl( \frac{\eta (1-\theta )^{2}}{4\theta }+ \frac{b_{*}}{m}\biggr) \int _{\Omega } (u-\overline{u}) (v-\overline{v}) \,\mathrm{d}x \\ &\quad\leq \biggl(\frac{2(\theta +1)}{\theta }+ \frac{\eta (1-\theta )^{2}}{8\theta }+ \frac{b_{*}(1-\theta )^{2}}{4\mu \theta m}+ \frac{b_{*}}{2m} \biggr) \int _{\Omega }(u-\overline{u})^{2}\,\mathrm{d}x \\ &\qquad{}+ \biggl( \frac{\eta (1-\theta )^{2}}{8\theta }+\frac{b_{*}}{2m} \biggr) \int _{ \Omega } (v-\overline{v})^{2}\,\mathrm{d}x. \end{aligned}$$

(2.3)

Similarly, multiplying the second equation of (1.2) by $(v-\overline{v})$, and integrating on Ω, we get

$$\begin{aligned} &d_{2} \int _{\Omega } \bigl\vert \nabla (v-\overline{v}) \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad= \int _{ \Omega } \frac{c(x)uv}{mu+v}(v-\overline{v})\,\mathrm{d}x - \int _{ \Omega } \mu v(v-\overline{v})\,\mathrm{d}x \\ &\quad\leq \int _{\Omega }\frac{c^{*}u}{mu+v}(v-\overline{v})^{2}\, \mathrm{d}x+ \int _{\Omega } \frac{c^{*}u\overline{v}}{mu+v}(v-\overline{v}) \,\mathrm{d}x- \int _{\Omega }\mu (v-\overline{v}+\overline{v}) (v- \overline{v})\, \mathrm{d}x \\ &\quad\leq \frac{c^{*}}{m} \int _{\Omega } (v-\overline{v})^{2}\,\mathrm{d}x+ \int _{\Omega } \frac{c^{*}\overline{v}(v-\overline{v})}{(mu+v)(m\overline{u}+\overline{v})}\bigl[ \overline{v}(u- \overline{u})+\overline{u}(\overline{v}-v)\bigr]\,\mathrm{d}x- \mu \int _{\Omega } (v-\overline{v})^{2}\,\mathrm{d}x \\ &\quad\leq \biggl(\frac{c^{*}(1-\theta )^{4}}{32\mu ^{2}\theta ^{2}m^{2}}+ \frac{c^{*}(1-\theta )^{2}}{4\mu \theta m^{2}}+\frac{c^{*}}{m}-\mu \biggr) \int _{\Omega } (v-\overline{v})^{2}\,\mathrm{d}x+ \frac{c^{*}(1-\theta )^{4}}{32\mu ^{2}\theta ^{2}m^{2}} \int _{\Omega }(u- \overline{u})^{2}. \end{aligned}$$

(2.4)

Therefore, by (2.3), (2.4), and Poincáre inequality, we obtain

$$\begin{aligned} &d_{1} \int _{\Omega } \bigl\vert \nabla (u-\overline{u}) \bigr\vert ^{2}\,\mathrm{d}x+d_{2} \int _{\Omega } \bigl\vert \nabla (v-\overline{v}) \bigr\vert ^{2}\,\mathrm{d}x\\ &\quad\leq \frac{1}{\lambda _{1}} \biggl(\Lambda _{1} \int _{\Omega } \bigl\vert \nabla (u- \overline{u}) \bigr\vert ^{2}\,\mathrm{d}x +\Lambda _{2} \int _{\Omega } \bigl\vert \nabla (v- \overline{v}) \bigr\vert ^{2}\,\mathrm{d}x \biggr), \end{aligned}$$

where

$$\begin{aligned} &\Lambda _{1}=\frac{2(\theta +1)}{\theta }+ \frac{\eta (1-\theta )^{2}}{8\theta }+ \frac{b_{*}(1-\theta )^{2}}{4\mu \theta m}+\frac{b_{*}}{2m}+ \frac{c^{*}(1-\theta )^{4}}{32\mu ^{2}\theta ^{2}m^{2}}, \\ &\Lambda _{2}=\frac{\eta (1-\theta )^{2}}{8\theta }+\frac{b_{*}}{2m}+ \frac{c^{*}(1-\theta )^{4}}{32\mu ^{2}\theta ^{2}m^{2}}+ \frac{c^{*}(1-\theta )^{2}}{4\mu \theta m}+\frac{c^{*}}{m}-\mu, \end{aligned}$$

hence if

$$\begin{aligned} \min \{d_{1},d_{2}\}>\frac{1}{\lambda _{1}}\max \{\Lambda _{1}, \Lambda _{2}\}:=D^{*}\bigl(m, \mu,b_{*},c^{*},\Omega \bigr), \end{aligned}$$

we have

$$\begin{aligned} \nabla (u-\overline{u})=\nabla (v-\overline{v})=0, \end{aligned}$$

therefore $(u,v)$ must be a constant solution. □

From the analysis above, we find that the fear effect, protection zone, and the Allee effect on prey can make the prey–predator system (1.2) tend to a fixed stable state. Even if the scale of the protection zone is small, it will promote stability among the two species.

3 Bifurcation from semitrivial solutions

The steady state system (1.2) has two nonnegative constant semitrivial solutions $(1,0)$ and $(\theta,0)$. We take μ as the bifurcation parameter and start analyzing bifurcation along following semitrivial branches:

$$\begin{aligned} \Gamma _{u_{1}}=\bigl\{ (\mu;1,0):-\infty < \mu < +\infty \bigr\} , \qquad \Gamma _{u_{2}}= \bigl\{ (\mu;\theta,0):-\infty < \mu < +\infty \bigr\} , \end{aligned}$$

for $p>1$, we denote

$$\begin{aligned} &X_{1}=\bigl\{ u\in {W^{2,p}(\Omega )}:\partial _{\nu }u=0 \text{ on } \partial \Omega \bigr\} ,\qquad Y_{1}=L^{p}( \Omega ), \\ &X_{2}=\bigl\{ v\in {W^{2,p}(\Omega _{1})}:\partial _{\nu }v=0 \text{ on } \partial \Omega _{1}\bigr\} ,\qquad Y_{2}=L^{p}(\Omega _{1}). \end{aligned}$$

Now we discuss the bifurcations from the curve of semitrivial solutions $\Gamma _{u_{1}}$ and $\Gamma _{u_{2}}$.

Lemma 3.1

([28])

Let $\lambda ^{D}_{1}(\phi,O)$ and $\lambda ^{N}_{1}(\phi,O)$ represent the first eigenvalue of $-\Delta +\phi$ with Direchlet and Neumann boundary condition in the region O, respectively. The following properties are satisfied:

$\mathrm{(a)}$ $\lambda ^{D}_{1}(\phi,O)>\lambda ^{N}_{1}(\phi,O)$;

$\mathrm{(b)}$ $\lambda ^{B}_{1}(\phi _{1},O)>\lambda ^{B}_{1}(\phi _{2},O)$ for $\phi _{1}>\phi _{2}$ and $\phi _{1}\neq \phi _{2}$, where B= N or D;

$\mathrm{(c)}$ $\lambda ^{D}_{1}(\phi,O_{1})>\lambda ^{D}_{1}(\phi,O_{2})$ for $O_{1} \subset O_{2}$.

Theorem 3.2

Suppose $\eta,m,\mu,d_{1},d_{2}>0$. Then

$\mathrm{(i)}$ $\mu _{1}=-d_{2}\lambda ^{N}_{1}(-\frac{c(x)}{md_{2}},\Omega _{1})$ is a bifurcation point for the positive solutions of (1.2) from semitrivial branch $\Gamma _{u_{1}}$ and $\Gamma _{u_{2}}$, and the bifurcation at $\mu _{1}$ is subcritical;

$\mathrm{(ii)}$ when $\mu <\mu _{1}$, (1.2) has at least one positive solution, and (1.2) has no positive solution if and only if $\mu \geq \mu _{1}$;

$\mathrm{(iii)}$ if $\mu \geq d_{2}\lambda ^{D}_{1}(\Omega _{0})$, then $(1,0)$ and $(\theta,0)$ are unstable, and there is no bifurcation occurring along $\Gamma _{u_{1}}$ and $\Gamma _{u_{2}}$.

Proof

(i) Let $w=1-u$, and define $H:\mathbf{R}\times X_{1}\times X_{2}\rightarrow Y_{1}\times Y_{2}$ by

$$\begin{aligned} H(\mu;w,v)= \begin{pmatrix} d_{1}\Delta {w} - w(1-w)(\frac{1}{\theta }-\frac{w}{\theta }-1) \frac{1}{1+\eta v}+\frac{b(x)(1-w)v}{m(1-w)+v} \\ d_{2}\Delta {v} +\frac{c(x)(1-w)v}{m(1-w)+v} -\mu v \end{pmatrix} ^{T}. \end{aligned}$$

(3.1)

Some valuable calculations involving (3.1) are as follows:

$$\begin{aligned} &H_{(w,v)}(\mu;w,v)[\phi,\psi ]= \begin{pmatrix} d_{1}\Delta {\phi } - BE\phi -b(x)v^{2}A^{2}\phi +w\eta CDB^{2}\psi +mb(x)A^{2}C^{2} \psi \\ d_{2}\Delta {\psi } -c(x)v^{2}A^{2}\phi + mc(x)A^{2}C^{2}\psi -\mu \psi \end{pmatrix} ^{T}, \\ &H_{\mu }(\mu;w,v)=(0,-v),\qquad H_{\mu (w,v)}(\mu;w,v)[\phi,\psi ]=(0,- \psi ), \\ &H_{(w,v)(w,v)}(\mu;w,v)[\phi,\psi ]^{2}= \begin{pmatrix} P(\phi,\psi ) \\ Q(\phi,\psi ) \end{pmatrix} ^{T}, \end{aligned}$$

where

$$\begin{aligned} P(\phi,\psi )={}&{-}2 \biggl(\frac{B}{\theta }(3w-2+\theta )+mb(x)v^{2}A^{3} \biggr)\phi ^{2}-2C \bigl(wD\eta ^{2}B^{3} +mb(x)CA^{3} \bigr)\psi ^{2} \\ &{} +2 \bigl(\eta EB^{2}-2mb(x)vCA^{3} \bigr)\phi \psi, \\ Q(\phi,\psi )={}&{-}2mc(x)A^{3}(v^{2}\phi ^{2}+C^{2} \psi ^{2}+2vC\phi \psi \end{aligned}$$

and

$$\begin{aligned} &A=\frac{1}{m(1-w)+v},\qquad B=\frac{1}{1+\eta v},\qquad C=1-w,\\ & D=\frac{1}{\theta }- \frac{w}{\theta }-1,\qquad E=\frac{3}{\theta }w^{2}+2\biggl(1- \frac{2}{\theta }\biggr)w+\frac{1}{\theta }-1. \end{aligned}$$

By calculating $H_{(w,v)}(\mu;0,0)[\phi,\psi ]=0$, we can get

$$\begin{aligned} \textstyle\begin{cases} -d_{1}\Delta {\phi } = -(\frac{1}{\theta }-1)\phi +\frac{b(x)}{m}\psi & \text{in } {\Omega }, \\ -d_{2}\Delta {\psi } = \frac{c(x)}{m}\psi -\mu \psi & \text{in } {\Omega _{1}}, \\ \partial _{\nu } \phi = 0 \text{on} {\partial \Omega }, \partial _{\nu } \psi = 0 &\text{on } {\partial \Omega _{1}}, \end{cases}\displaystyle \end{aligned}$$

(3.2)

which has a solution with $\psi >0$ only when $\mu =\mu _{1}=-d_{2}\lambda ^{N}_{1}(-\frac{c(x)}{md_{2}},\Omega _{1})$. Thus $\mu _{1}$ is the only bifurcation point along $\Gamma _{u_{1}}$. We set kernel $\mathcal{N}(H_{(w,v)}(\mu _{1};0,0))=\operatorname{span}{(\varphi _{1}, \varphi _{2})}$, where $(\varphi _{1},\varphi _{2})\neq (0,0)$ is a solution of (3.2). Since $\mu _{1}=-d_{2}\lambda ^{1}_{N}(-\frac{c(x)}{md_{2}},\Omega _{1})$, we can choose $\varphi _{2}=1$, then

$$\begin{aligned} \varphi _{1}= \biggl(-\Delta +\frac{1}{d_{1}}\biggl( \frac{1}{\theta }-1\biggr) \biggr)^{-1} \frac{b(x)}{md_{1}}>0. \end{aligned}$$

The range of the operator is given by

$$\begin{aligned} \mathcal{R}\bigl(H_{(w,v)}(\mu _{1};0,0)\bigr)= \biggl\{ (f,g) \in Y_{1}\times Y_{2}: \int _{\Omega _{1}}g(x)\,\mathrm{d}x=0 \biggr\} . \end{aligned}$$

It is clear that $\mathrm{codim} \mathcal{R}(H_{(w,v)}(\mu _{1};0,0))=1$. We know $\int _{\Omega _{1}}1\,\mathrm{d}x>0$, thus

$$\begin{aligned} H_{\mu (w,v)}(\mu;w,v)[\varphi _{1},\varphi _{2}]=(0,-1) \notin \mathcal{R}\bigl(H_{(w,v)}(\mu _{1};0,0)\bigr). \end{aligned}$$

Now, by applying local bifurcation theorem, we can obtain the following smooth curve for the set of positive solution to (1.2) near $(\mu;1,0)$:

$$\begin{aligned} \Gamma _{u_{1}}=\bigl\{ \bigl(\mu _{1};1-u_{1}(s),v_{1}(s) \bigr):s\in [0,\delta )\bigr\} , \end{aligned}$$

such that $\mu _{1}(0)=\mu _{1}$, $u_{1}(s)=s\varphi _{1}(x)+o(|s|)$, $v_{1}(s)=s+o(|s|)$. Moreover, $\mu _{1}'(0)$ can be calculated

$$\begin{aligned} \mu _{1}'(0)=- \frac{\langle H_{(w,v)(w,v)}(\mu;0,0)[\phi,\psi ]^{2},l_{1}\rangle }{2\langle H_{\mu (w,v)}(\mu;0,0)[\phi,\psi ],l_{1}\rangle }=- \frac{c(x)}{m^{2}}< 0, \end{aligned}$$

where $l_{1}$ is the linear function on $Y_{1}\times Y_{2}$ defined by $\langle [f,g],l_{1} \rangle =\int _{\Omega _{1}}g(x)\,\mathrm{d}x$. Therefore the bifurcation at $(\mu _{1};1,0)$ is always subcritical.

Similarly, let $p=\theta -u$, the linearized equation of system (1.2) at $(\mu;\theta,0)$ is

$$\begin{aligned} G_{(p,v)}(\mu;0,0)[\phi,\psi ]= \begin{pmatrix} d_{1}\Delta {\phi } +(1-\theta )\phi +\frac{b(x)}{m}\psi \\ d_{2}\Delta {\psi } + \frac{c(x)}{m}\psi -\mu \psi \end{pmatrix} ^{T}, \end{aligned}$$

(3.3)

apparently the solution of second equation for (3.3) is the same as in (3.2) when $G_{(p,v)}(\mu; 0,0)[\phi,\psi ]=0$, which implies that the bifurcation proof from $(\theta,0)$ is similar to that from $(1,0)$.

(ii) Assume $(u,v)$ is a positive solution of (1.2). From (2.2), we obtain $0< u\leq 1$ in Ω. It is clear that

$$\begin{aligned} \frac{c(x)u}{mu+v}\leq \frac{c(x)}{m+v}\leq \frac{c(x)}{m}. \end{aligned}$$

By Lemma 3.1, we also have

$$\begin{aligned} \mu _{1}=-d_{2}\lambda ^{N}_{1} \biggl(- \frac{c(x)}{md_{2}},\Omega _{1} \biggr)>-d_{2}\lambda ^{N}_{1} \biggl(-\frac{c(x)u}{d_{2}(mu+v)}, \Omega _{1} \biggr)=\mu. \end{aligned}$$

Therefore (1.2) has no positive solution if and only if $\mu \geq \mu _{1}$.

(iii) Define $h(m)=-\lambda _{1}^{N}(-\frac{c(x)}{md_{2}},\Omega )$, then function $h(m)$ is monotonically decreasing from the properties of (3.1). Now we prove $h(m)\rightarrow \lambda _{1}^{D}(\Omega _{0})(m\rightarrow 0^{+})$. The variational characterization of the eigenvalue shows that

$$\begin{aligned} -\lambda _{1}^{N}\biggl(-\frac{c(x)}{md_{2}}, \Omega \biggr)=\inf_{\psi \in H^{1}(\Omega )} \frac{-\int _{\Omega } \vert \nabla \psi \vert ^{2}\,\mathrm{d}x+\int _{\Omega }\frac{c(x)}{md_{2}}\psi ^{2}\,\mathrm{d}x}{\int _{\Omega }\psi ^{2}\,\mathrm{d}x} \leq \lambda _{1}^{D}(\Omega _{0}). \end{aligned}$$

(3.4)

Here ψ is the eigenfunction associated with $\lambda _{1}^{D}(\Omega _{0})$ when $\psi >0$ in $x\in \Omega _{0}$ and $\psi =0$ when $x\in \Omega _{1}$. Let $\psi _{n}>0$ satisfy

$$\begin{aligned} \Delta \psi _{n}+\frac{c(x)}{m_{n}d_{2}}\psi _{n}=-\lambda _{1}^{N}\biggl(- \frac{c(x)}{m_{n}d_{2}}, \Omega \biggr)\psi _{n} \quad\text{in } \Omega,\qquad \partial _{\nu }\psi _{n}=0 \quad\text{on } \partial \Omega, \end{aligned}$$

(3.5)

where $\{m_{n}\}$ is a natural sequence satisfying $m_{n}\rightarrow 0^{+}$, and $\max_{x\in \overline{\Omega }}\psi _{n}(x)=1$. From (3.4) and (3.5), we have $\Delta \psi _{n}\leq \lambda _{1}^{D}(\Omega _{0})\psi _{n}$, then

$$\begin{aligned} - \int _{\Omega } \vert \nabla \psi _{n} \vert ^{2}\,\mathrm{d}x+ \int _{\Omega }\psi _{n}^{2} \,\mathrm{d}x\leq \bigl[\lambda _{1}^{D}(\Omega _{0})+1\bigr] \int _{\Omega }\psi _{n}^{2} \,\mathrm{d}x\leq \bigl[\lambda _{1}^{D}(\Omega _{0})+1\bigr] \vert \Omega \vert , \end{aligned}$$

which implies that $\{\psi _{n}\}$ converges to some ψ weakly in $H^{1}(\Omega )$ and strongly in $L^{p}(\Omega )$ for any $p>1$. Multiply (3.5) by $\psi _{n}$ and integrate over Ω to get

$$\begin{aligned} - \int _{\Omega } \vert \nabla \psi _{n} \vert ^{2}\,\mathrm{d}x-\lambda _{1}^{N}\biggl(- \frac{c(x)}{m_{n}d_{2}},\Omega \biggr) \int _{\Omega }\psi _{n}^{2}\,\mathrm{d}x=- \frac{1}{m_{n}d_{2}} \int _{\Omega _{1}} c(x)\psi _{n}^{2}\,\mathrm{d}x. \end{aligned}$$

Since $\frac{1}{m_{n}d_{2}}\rightarrow \infty$ when $m_{n}\rightarrow 0^{+}$, we must have $\psi \equiv 0$ in $\Omega _{1}$. Obviously, $\psi \neq 0$ in $\Omega _{0}$, otherwise $\psi =0$ in Ω, which contradicts with $\max_{x\in \overline{\Omega }}\psi _{n}(x)=1$. Hence $\lim_{m\rightarrow 0^{+}}h(m)=\lambda _{1}^{D}(\Omega _{0})$.

Now if $\mu \geq d_{2}\lambda ^{D}_{1}(\Omega _{0})$, then $\mu >-d_{2}\lambda ^{N}_{1}(-\frac{c(x)}{md_{2}},\Omega )$ for any $\mu >0$. Therefore, there is no bifurcation of positive solutions occurring along $\Gamma _{u_{1}}$. This also implies that $(1,0)$ is unstable. Similarly, $(\theta,0)$ is an unstable steady state of (1.2), and there is no bifurcation occurring along $\Gamma _{u_{2}}$. □

From Theorem 3.2, the prey and predator can coexist when the mortality rate of the predator μ is less than the threshold value $\mu _{1}$. Otherwise, at least numerous among prey and predator is ultimately extinct. The introduced protection zone plays an essential role in maintaining the stability of the ecosystem.

4 Conclusions

The article discusses a diffusion predator–prey model (1.2) with the Allee effect, fear effect, and protection zone. We mainly show some dynamical behavior of the model (1.2). Firstly, we discuss the global existence and a priori estimates of solutions, and ensure that predators and prey can coexist. The results further illustrate that the asymptotic property of solutions only depends on the strong Allee effect. Furthermore, the nonexistence of nonconstant positive solution is proved. The results show that the fear effect, protection zone, and Allee effect on prey can make the prey–predator system (1.2) tend to a stable state. Moreover, we also analyze the bifurcation from semitrivial solutions $(1,0)$ and $(\theta,0)$. We find that the bifurcation point from $(1,0)$ and $(\theta,0)$ is the same, namely $\mu _{1}$. When the death rate of predator species is less than $\mu _{1}$, predator and prey can coexist stably. We also prove that there is a critical path size $d_{2}\lambda ^{D}_{1}(\Omega _{0})$. When $\mu \geq d_{2}\lambda ^{D}_{1}(\Omega _{0})$, the solutions $(1,0)$ and $(\theta,0)$ are unstable, and the bifurcation will not occur.