Qualitative analysis of a prey–predator model with prey refuge and intraspecific competition among predators

Abstract

In this study, we consider a prey–predator model with prey refuge and intraspecific competition between predators using the Crowley–Martin functional response and investigate the dynamic characteristics of spatial and nonspatial prey–predator systems via both analytical and numerical methods. The local stability of nontrivial interior equilibrium, the existence of a Hopf bifurcation, and the stability of bifurcating periodic solutions are obtained in the absence of diffusion. For the spatial system, the Turing and non-Turing patterns are evaluated for some set of parametric belief functions, and we obtain some interesting results in terms of prey and predator inhabitants. We present the results of numerical simulations that demonstrate that both prey and predator populations do not converge to a stationary equilibrium state at any foreseeable future time when the parametric values are processed in the Turing domain.

1 Introduction

Understanding the dynamics of interacting prey–predator models is of paramount importance for gaining insights into the long-term behavior of species in ecological systems. Over the years, researchers have extensively investigated various nonlinear coupled ordinary differential equations to model the complex interactions among prey and their potential predators. Among the classical models, the Lotka–Volterra model, introduced independently by Lotka and Volterra in the early twentieth century, has served as a fundamental basis for studying predator–prey interactions. Subsequent advancements by Holling [18] in the form of the Rosenzweig–MacArthur model [33] incorporated density-dependent prey growth and functional response.

While these classical models have provided valuable insights into the general characteristics of prey–predator populations, recent research has shown that incorporating additional factors can better reflect the intricacies of natural ecosystems. Consequently, researchers have introduced various functional responses to represent the feeding rates of predators [5, 49, 50]. Notably, the Lotka–Volterra and Holling types have been widely used in this context.

Additionally, the most crucial concept in prey–predator models is the ‘functional response’, which describes the rate at which a predator attacks a given number of prey. The Lotka–Volterra and Holling types are the most commonly used functional responses to represent the average feeding rate of a predator.

$$\begin{aligned}& \text{Lotka--Volterra type} :\quad f_{1}(u)=\delta u, \\& \text{Holling Type-II} :\quad f_{2}(u)=\delta u/(1+\alpha u), \\& \text{Holling Type-III} :\quad f_{3}(u)=\delta u^{2}/\bigl(1+ \alpha u^{2}\bigr), \\& \text{Holling Type-IV} :\quad f_{4}(u)=\delta u/\bigl(1+\alpha u^{2}\bigr), \end{aligned}$$

where u denotes the density of the prey population, and δ and α are positive constants that respectively represent the effects of capture rate and handling time. Notably, these four functional response features are prey-dependent and generally not influenced by predator inhabitants. However, a crucial aspect often overlooked in these models is the consideration of mutual interference among potential predators. To address this limitation, Beddington [2] and DeAngelis [11] proposed a functional response

$$ f(u,v)=\delta u/(1+\alpha u+\beta v) $$

that accounts for predator $(v)$ interference β, while Crowley and Martin [10] introduced a similar response, taking into account both searching and handling interference:

$$ f(u,v)=\delta u/\bigl((1+\alpha u) (1+\beta v)\bigr), $$

which is referred to as the Crowley–Martin functional response. These modifications have proven essential in understanding the dynamics of real-world predator–prey systems, and they offer a more accurate representation of their behavior [38].

In recent years, the study of prey–predator models has witnessed significant developments, with researchers incorporating various factors such as the Allee effect, prey refuge, interference among predators, and harvesting [6, 15, 25–27, 29, 36, 37, 47, 48]. These enhancements have allowed for a more detailed reflection of the behavior of natural ecosystems, leading to a growing interest in the dynamic characteristics of these models [3, 4, 7, 8, 14, 19, 20, 22–24, 30, 31, 35, 39–46].

In light of these considerations, the present study focuses on formulating a prey–predator model that accounts for prey refuge and intraspecific competition among predators using the Crowley–Martin functional response. The inclusion of prey refuge and competition among predators in the model is crucial as it addresses important ecological aspects that have significant implications for species coexistence and ecosystem stability.

The rest of this paper is organized as follows: Sect. 2 presents the mathematical formulation of the prey–predator model with prey refuge and intraspecific competition among predators using the Crowley–Martin functional response. In Sect. 3, we analyze the existence and local and global dynamics of equilibria. Subsequently, in Sect. 4, we discuss the presence of Hopf bifurcation and its periodic solution behaviors. Section 5 focuses on the occurrence of Turing instability in the interior equilibrium point under the influence of diffusion. Theoretical results are verified through numerical simulations in Sect. 6, where we also uncover additional meaningful phenomena. Finally, Sect. 7 offers concluding remarks and highlights the significance of our findings within the broader context of ecological research.

By comprehensively investigating the dynamic behavior of this extended prey–predator model, we aim to contribute to a deeper understanding of ecological interactions and enhance the applicability of predator–prey models in real-world scenarios.

2 Mathematical model and analysis

Bazykin [1] introduced a prey–predator model based on the Crowley–Martin (C-M) functional response, which considers the simultaneous involvement of potential predators in various activities, including searching and handling prey as well as interacting with other predators. The C-M functional response is described by the total instantaneous per capita feeding rate equation

$$ f(X,Y):= \frac{\tilde{a} X Y }{1+\tilde{b}X+\tilde{c}(Y-1)}. $$

Here, X denotes the prey population, Y denotes the predator population, and c̃ (units: 1/predator) represents the magnitude of interference among predators. The term $Y-1$ is used to account for the mechanistic nature of predator dependence, indicating that a predator does not interfere with itself in these models. When $Y=1$, the equation reduces to the traditional Holling Type II response.

Prey refuge, a significant concept in prey–predator models, plays a crucial role in supporting a constant proportion of prey through predation [16, 17]. It profoundly influences the coexistence of predators and prey. In theoretical ecology, understanding the impact of prey refuge on the dynamics of prey–predator interactions has been a topic of interest. Many authors [9, 12, 21, 34] have found that prey refuge stabilizes prey–predator dynamics, preserving prey biomass and preventing extinction due to predation. Therefore, the strength of prey refuges affects the foraging efficiency of predators.

To incorporate the effect of prey refuge, we integrate it into the C-M functional response term. As a result, we arrive at the nonautonomous C-M type prey–predator model, accounting for prey refuge and intraspecific competition among predators for prey:

$$ \textstyle\begin{cases} \frac {dX(t)}{dt}=X(r_{1}-\rho X)- \frac{\delta _{1}(1-m)XY}{\chi _{1}+\chi _{2}(1-m)X+\chi _{3}((1-m)X)(Y-1)}, \\ \frac {dY(t)}{dt}=r_{2}Y- \frac {\delta _{2} Y^{2}}{\chi _{4}+(1-m)X}-\delta _{3} Y, \quad \\ X(0)=X_{0}> 0,\qquad Y(0)=Y_{0}>0. \end{cases} $$

(2.1)

In this model, $X(t)$ represents the biomass of the prey population, and $Y(t)$ denotes the biomass of the predator population at time t. The refuge protects a population of prey denoted by mX, where $m \in [0, 1)$ is a constant, making $(1-m)X$ the only prey available to predators. The parameters $r_{1}$, $r_{2}$, ρ, $\delta _{1}$, $\delta _{2}$, $\delta _{3}$, $\chi _{1}$, $\chi _{2}$, $\chi _{3}$, and $\chi _{4}$ are all positive constants with ecological meaning as follows in Table 1.

Table 1 Ecological meaning

Assuming a biological perspective, we confirm that $\chi _{1}+\chi _{2}(1-m)X+\chi _{3}((1-m)X)(Y-1)\neq 0$, ensuring the well-posedness of our system (2.1).

For simplicity, we proceed to nondimensionalize (2.1) using the following scaling:

$$ u\mapsto \rho X, \qquad v\mapsto Y, \qquad t\mapsto t. $$

This leads us to the following form of the model:

$$\begin{aligned} \textstyle\begin{cases} \dot{u}(t)= u (r-u )- \frac {\delta (1-m)uv}{1+\alpha (1-m)u+\beta ((1-m)u)(v-1)}, \\ \dot{v}(t)=\xi (\zeta v- \frac {v^{2}}{\eta +(1-m)u}-\sigma v ), \\ u(0)=u_{0}> 0,\qquad v(0)=v_{0}>0. \end{cases}\displaystyle \end{aligned}$$

(2.2)

In this formulation, we define dimensionless constants as follows: $r=r_{1}$, $\delta =\frac{\delta _{1}}{\chi _{1}}$, $\alpha = \frac{\chi _{2}}{\rho \chi _{1}}$, $\beta = \frac{\chi _{3}}{\rho \chi _{1}}$, $\xi =\frac{\delta _{2}}{\rho}$, $\zeta =\frac{r_{2} \rho}{\delta _{2}}$, $\eta =\rho \chi _{4}$, and $\sigma =\frac{\delta _{3} \rho}{\delta _{2}}$.

Our physical world exists within a spatial universe, where environmental interactions play a vital role in shaping ecological communities. Various aspects, such as the atmosphere, geological materials, and biological frameworks, differ significantly across the planet. Consequently, population dynamics depend on both space and time, incorporating spatial movement. To analyze the spatial dynamics of prey–predator models, we consider the following system of partial differential equations with homogeneous Neumann boundary conditions:

$$\begin{aligned} \textstyle\begin{cases} \frac{\partial u(x,t)}{\partial t}=u (r-u )- \frac{\delta (1-m)uv}{1+\alpha (1-m)u+\beta ((1-m)u)(v-1)}+D_{1} \nabla ^{2}u,\quad x\in \Omega , t>0, \\ \frac{\partial v(x,t)}{\partial t}=\xi (\zeta v- \frac{v^{2}}{\eta +(1-m)u}-\sigma v )+D_{2}\nabla ^{2}v,\quad x\in \Omega , t>0, \\ \partial _{\nu} u=\partial _{\nu} v=0, \quad x\in \partial \Omega , t>0, \\ u(x,0)=u_{0}(x)\geq 0,\qquad v(x,0)=v_{0}(x)\geq 0, \quad x\in \Omega . \end{cases}\displaystyle \end{aligned}$$

(2.3)

In this spatial extension, $u(x,t)$ and $v(x,t)$ represent the densities of prey and predators, respectively, in the fixed open bounded domain $\Omega \subset \mathbb{R}^{N}$ at time t. The diffusion coefficients $D_{1}$ and $D_{2}$ account for the spatial movement of prey and predators, respectively. The Laplacian operator $\nabla ^{2}=\frac{\partial ^{2}}{\partial u^{2}}+ \frac{\partial ^{2}}{\partial v^{2}}$ describes spatial interactions in a two-dimensional space.

The boundary ∂Ω of the domain Ω is smooth, and ν represents the outward unit normal vector of the boundary. The homogeneous Neumann boundary conditions indicate that the prey–predator system is self-contained with no population flux across the boundary.

3 Equilibrium points and stability analysis

3.1 Equilibria

System (2.2) has trivial and axial equilibrium points where $E_{0}$, $E_{1}$, $E_{2}$ always exist, where the following occur:

• (i) Trivial equilibrium point: $E_{0}=(0,0)$, both prey and predator are extinct.

• (ii) Axial equilibrium points: $E_{1}=(r,0)$, only prey survives; $E_{2}=(0,\zeta \eta -\eta \sigma )$, if $\zeta >\sigma$, only predators survive.

The nontrivial interior equilibrium point $E^{*}(u^{*},v^{*})\in \mathbb{R}_{+}^{2}$ can be obtained by solving the following system of equations:

$$\begin{aligned}& u (r-u )- \frac{\delta (1-m)uv}{1+\alpha (1-m)u+\beta ((1-m)u)(v-1)}=0, \\& \xi \biggl(\zeta v-\frac {v^{2}}{\eta +(1-m)u}-\sigma v \biggr)=0. \end{aligned}$$

Solving these nullclines, we obtain a cubic equation in u:

$$ \varDelta _{1} u^{3}+\varDelta _{2} u^{2}+\varDelta _{3} u+\varDelta _{4}=0, $$

(3.1)

where (H): $\varDelta _{1}=-(m-1)^{2} \beta (\zeta -\sigma )<0$, $\varDelta _{2}=(m-1)(\alpha +\beta (-1+((m-1)r+\eta )(\zeta -\sigma )))<0$, $\varDelta _{3}=-1-(m-1)^{2}\delta (\zeta -\sigma )-(m-1)r(\alpha + \beta (-1+\eta (\zeta -\sigma )))>0$, $\varDelta _{4}=r+(m-1)\delta \eta (\zeta -\sigma )>0$, provided the cubic equation (3.1) has at most one positive solution $u^{*}$ (say). Using the value of $u^{*}$, we obtain the value of $v^{*}$ as follows:

$$ v^{*}=(\zeta -\sigma ) \bigl(\eta +(1-m)u^{*}\bigr), $$

$\zeta >\sigma$ provided $v^{*}$ is also positive.

Below we assume that all $E^{*}(u^{*},v^{*})$ ensure the above conditions (H) and $\zeta >\sigma$.

3.2 Local stability analysis

The dynamic behavior of the equilibrium points can be studied by computing the eigenvalues of the Jacobian matrix J of system (2.2), namely,

$$ J= \begin{pmatrix} r-2u- \frac{\delta (1-m)v}{((m-1)u(\alpha +(v-1)\beta )-1)^{2}} & \frac{\delta (1-m)(-1+(m-1)u(\alpha -\beta ))u}{((m-1)u(\alpha +(v-1)\beta )-1)^{2}} \\ \frac{-\xi (m-1)v^{2}}{(\eta +(1-m)u)^{2}} & \xi (\zeta -\frac{2v}{\eta +(1-m)u}-\sigma ) \end{pmatrix}. $$

(3.2)

The existence and local stability of the equilibrium solutions can be stated as follows.

Theorem 3.1

• (i) The trivial equilibrium point $E_{0}=(0,0)$ is always unstable.

• (ii) If $\zeta <\sigma$, then the axial equilibrium point $E_{1} =(r, 0)$ is locally asymptotically stable.

• (iii) The axial equilibrium point $E_{2}=(0,\eta (\zeta -\sigma ))$ is always unstable.

Proof

(i) The Jacobian matrix of system (2.2) evaluated at the equilibrium point $E_{0}=(0,0)$ is given by

$$\begin{aligned} J\vert _{E_{0}}= \begin{pmatrix} r&0 \\ 0&\xi (\zeta -\sigma ) \end{pmatrix}, \end{aligned}$$

$\operatorname{tr} J\vert _{E_{0}}=r+\xi (\zeta -\sigma )$ and $\det J\vert _{E_{0}}=r\xi (\zeta -\sigma )$. The eigenvalues corresponding to $J\vert _{E_{0}}$ are $\lambda _{1}=r$ (>0) and $\lambda _{2}=\xi (\zeta -\sigma )$. Thus $E_{0}$ is unstable when $\zeta >\sigma$ ($\lambda _{2}>0$), and $E_{0}$ is a saddle point when $\zeta <\sigma$ ($\lambda _{2}<0$). Therefore, $E_{0}=(0,0)$ is always unstable.

(ii) The Jacobian matrix of system (2.2) evaluated at the axial equilibrium point $E_{1}=(r,0)$ is given by

$$\begin{aligned} J\vert _{E_{1}}= \begin{pmatrix} -r&\frac{-(m-1)r\delta}{(m-1)(\alpha -\beta )r-1} \\ 0&\xi (\zeta -\sigma ) \end{pmatrix}, \end{aligned}$$

$\operatorname{tr} J\vert _{E_{1}}=-r+\xi (\zeta -\sigma )$ and $\det J\vert _{E_{1}}=-r\xi (\zeta -\sigma )$. When $\zeta <\sigma$, $\det J\vert _{E_{1}}>0$ and $\operatorname{tr} J\vert _{E_{1}}<0$. Therefore, $E_{1}$ is locally asymptotically stable when $\zeta <\sigma$.

(iii) The Jacobian matrix of system (2.2) evaluated at the axial equilibrium point $E_{2}=(0,\eta (\zeta -\sigma ))$ is given by

$$\begin{aligned} J\vert _{E_{2}}= \begin{pmatrix} r+\delta \eta (m-1)(\zeta -\sigma )&0 \\ -\xi (m-1)(\zeta -\sigma )^{2}&-\xi (\zeta -\sigma ) \end{pmatrix}. \end{aligned}$$

Then the eigenvalues of the above Jacobi matrix $J\vert _{E_{2}}$ are $\mu _{1}= r+\delta \eta (1-m)(\zeta -\sigma )$ and $\mu _{2}=-\xi (\zeta -\sigma )$. When $\zeta <\sigma$ ($\mu _{2}<0$) and $r<\delta \eta (1-m)(\zeta -\sigma )$ ($\mu _{1}<0$), both eigenvalues are negative. Hence, the axial equilibrium point $E_{2}$ is locally asymptotically stable when $\zeta <\sigma$ and $r<\delta \eta (1-m)(\zeta -\sigma )$. This is not possible because $\zeta <\sigma \implies \delta \eta (1-m)(\zeta -\sigma )<0$ and because of our assumption that $r>0$. This shows that the locally asymptotically stable situation never occurs. Therefore, $E_{2}$ is always unstable. Also note that $\zeta <\sigma \implies E_{2}<0$ (i.e., when $E_{2}\notin \{\{0\}\cup \mathbb{R}\}_{+}^{2}$, the predators also die out). □

3.3 Interior equilibrium qualitative behaviors

The Jacobian matrix evaluated at the coexistence equilibrium point $E^{*}(u^{*},v^{*})$ is

$$\begin{aligned} J\vert _{E^{*}}= \begin{pmatrix} a_{11} & a_{12} \\ a_{21}& a_{22} \end{pmatrix}, \end{aligned}$$

(3.3)

where $a_{11}= r-2u^{*}- \frac{\delta (1-m)v^{*}}{((m-1)u^{*}(\alpha +(v^{*}-1)\beta )-1)^{2}}$, $a_{12}= \frac{\delta (1-m)(-1+(m-1)u^{*}(\alpha -\beta ))u^{*}}{((m-1)u^{*}(\alpha +(v^{*}-1)\beta )-1)^{2}}$, $a_{21}= \frac{-\xi (m-1){v^{*}}^{2}}{(\eta +(1-m)u^{*})^{2}}$, $a_{22}= \xi (\zeta - \frac{2v^{*}}{\eta +(1-m)u^{*}}-\sigma )$.

Then the trace and determinant of the Jacobian matrix (3.3) is

$$\begin{aligned} T=\operatorname{tr} J\vert _{E^{*}}=a_{11}+a_{22} \quad \text{and}\quad D=\det J\vert _{E^{*}} =a_{11}a_{22}-a_{21}a_{12}. \end{aligned}$$

Therefore, the characteristic equation of the linearized system of (2.2) at $E^{*}=(u^{*},v^{*})$ is

$$\begin{aligned} \lambda ^{2}-T \lambda +D=0. \end{aligned}$$

(3.4)

The qualitative behaviors of the interior equilibrium point $E^{*}(u^{*},v^{*})$ may be stated as follows.

Case (i)::

If $T<0$ and $D>0$, then the characteristic roots of (3.4) are either both negative reals or complex conjugates with a negative real part. Therefore, $E^{*}$ is either a stable node ($T^{2}>4D$) or a stable spiral ($T^{2}<4D$).

Case (ii)::

If $T>0$ and $D>0$, then the characteristic roots of (3.4) are either both positive reals or complex conjugates with a positive real part. Hence, $E^{*}$ is either an unstable node ($T^{2}>4D$) or an unstable spiral ($T^{2}<4D$).

Case (iii)::

If $D<0$ and $T\neq 0$, then the characteristic roots of (3.4) are both are real with opposite signs. Therefore $E^{*}$ is a saddle point.

Case (iv)::

If $D<0$ and $T=0$, then the characteristic roots of (3.4) both are real numbers with same magnitude and opposite signs. Hence, $E^{*}$ is a saddle node.

Case (v)::

If $D>0$ and $T=0$, then the characteristic roots of (3.4) are purely complex conjugate. Therefore, $E^{*}$ is a center.

3.4 Global stability analysis

Here, we provide the result to attain global stability in the nontrivial interior equilibrium $E^{*}=(u^{*},v^{*})$ for both local (2.2) and spatial (2.3) systems.

Theorem 3.2

If $(\delta (1-m)v^{*}-u^{*})\alpha <\frac {1}{1-m}+\beta u^{*}(v^{*}-1)$ and $(1-m)u^{*}(\beta -\alpha )<1$ hold, then the interior equilibrium $E^{*}=(u^{*},v^{*})$ of the local system (2.2) is globally asymptotically stable.

Proof

The proof is usually developed by applying the Lyapunov function. We consider the subsequent positive definite Lyapunov function in $\mathbb{R}$ about the equilibrium $E^{*}$:

$$\begin{aligned}& V_{1}=u-u^{*}-u^{*}\ln \biggl(\frac{u}{u^{*}} \biggr) \quad \text{and} \\& V_{2}=v-v^{*}-v^{*}\ln \biggl( \frac{v}{v^{*}} \biggr). \end{aligned}$$

This kind of Lyapunov function was first utilized in [13], and later it was broadly exploited by several researchers. We can simply verify that $\frac{\partial V_{1}}{\partial u}>0$ for $u>u^{*}$ and $\frac{\partial V_{1}}{\partial u} <0$ for $0< u< u^{*}$, and $\frac{\partial V_{2}}{\partial v}>0$ for $v>v^{*}$ and $\frac{\partial V_{2}}{\partial v} <0$ for $0< v< v^{*}$. If we estimate $\frac{\partial V_{1}}{\partial t}$ via (2.2), we obtain

$$\begin{aligned} \frac {dV_{1}(u)}{dt} =&\bigl(u-u^{*}\bigr) \frac {\dot{u}}{u}= \biggl(1- \frac {u^{*}}{u} \biggr) \biggl\{ u (r-u )- \frac{\delta (1-m)uv}{\varTheta} \biggr\} , \quad \\ =&\bigl(u-u^{*}\bigr) \biggl\{ -\bigl(u-u^{*}\bigr)- \biggl(\frac{\delta (1-m)v}{\varTheta}- \frac{\delta (1-m)v^{*}}{\widehat{\varTheta}} \biggr) \biggr\} , \quad \\ =&\bigl(u-u^{*}\bigr)^{2} \biggl\{ \frac {\delta (1-m)(\alpha (1-m)v^{*}-\beta (1-m)v^{*}(1-v))}{\varTheta \widehat{\varTheta}}-1 \biggr\} \\ &{}- \frac {\delta (1-m)(1+\alpha (1-m)u^{*}-\beta (1-m)u^{*})(u-u^{*}) (v-v^{*})}{\varTheta \widehat{\varTheta}}, \end{aligned}$$

(3.5)

where $\varTheta =1+\alpha (1-m)u+\beta ((1-m)u)(v-1)$, $\widehat{\varTheta}=1+\alpha (1-m)u^{*}+\beta ((1-m)u^{*})(v^{*}-1)$.

Similarly,

$$\begin{aligned} \frac {dV_{2}(v)}{dt} =&\bigl(v-v^{*}\bigr) \frac {\dot{v}}{v}= \biggl(1- \frac{v^{*}}{v} \biggr) \biggl\{ \xi \biggl( \zeta v- \frac {v^{2}}{\varPhi}-\sigma v \biggr) \biggr\} , =\bigl(v-v^{*}\bigr) \biggl\{ \frac {-\xi v}{\varPhi}+ \frac {\xi v^{*}}{\widehat{\varPhi}} \biggr\} , \\ =&\frac {-\xi (v-v^{*})^{2}}{\varPhi}+ \frac {\xi (1-m)v^{*}(u-u^{*})(v-v^{*})}{\varPhi \widehat{\varPhi}}, \end{aligned}$$

(3.6)

where $\varPhi =\eta +(1-m)u$, $\widehat{\varPhi}=\eta +(1-m)u^{*}$.

We define the Lyapunov function $V=V_{1}(u)+\frac {1}{C}V_{2}(v)$, where $C=\frac {\xi v^{*}}{\delta (1+\alpha (1-m)u^{*}-\beta (1-m)u^{*})}$. Generating $\frac{dV}{dt}$ via (3.5) and (3.6) gives

$$\begin{aligned} \frac{dV}{dt} =&\bigl(u-u^{*} \bigr)^{2} \biggl\{ \frac {\delta (1-m)(1+\alpha (1-m)u^{*}-\beta (1-m)v^{*}(1-v))}{\varTheta \widehat{\varTheta}}-1 \biggr\} \\ &{}-\bigl(v-v^{*}\bigr)^{2} \biggl\{ \frac {\delta (1+\alpha (1-m)u^{*}-\beta (1-m)u^{*})}{v^{*}\varPhi} \biggr\} . \end{aligned}$$

(3.7)

The coefficient of $(u-u^{*})^{2}$ is

$$\begin{aligned}& \frac {\delta (1-m)(1+\alpha (1-m)u^{*}-\beta (1-m)v^{*}(1-v))}{\varTheta \widehat{\varTheta}}-1 \\& \quad \leq \frac {\delta (1-m)(1+\alpha (1-m)u^{*}-\beta (1-m)v^{*}(1-v))}{\widehat{\varTheta}}-1. \end{aligned}$$

The coefficient of $(v-v^{*})^{2}$ is

$$ -\frac {\delta (1+\alpha (1-m)u^{*}-\beta (1-m)u^{*})}{v^{*}\varPhi} \leq -\frac {\delta (1+\alpha (1-m)u^{*}-\beta (1-m)u^{*})}{v^{*}}. $$

Therefore, if $(\delta (1-m)v^{*}-u^{*})\alpha <\frac{1}{1-m}+\beta u^{*}(v^{*}-1)$ and $(1-m)u^{*}(\beta -\alpha )<1$ hold consequently from (3.7), then we obtain $\frac{dV}{dt}<0$. Hence, by Lyapunov’s asymptotic stability theorem, the interior equilibrium $E^{*}$ of system (2.2) is globally asymptotically stable. □

Now, we select the Lyapunov function for the diffusion system (2.3)

$$ E(t)= \iint _{\Omega }V(u,v) \,dA. $$

(3.8)

Hence, differentiating $E(t)$ with respect to t along the solutions of system (2.3), we obtain

$$\begin{aligned} \frac {dE(t)}{dt}= \iint _{\Omega} \biggl(\frac{\partial V}{\partial u}d_{1} \nabla ^{2} u+\frac{\partial V}{\partial v} d_{2} \nabla ^{2} v \biggr) \,dA+ \iint _{\Omega }\frac {dV}{dt} \,dA. \end{aligned}$$

Considering the zero-flux boundary conditions $\partial _{\nu} u=\partial _{\nu} v=0$, $x\in \partial \Omega$ and applying Green’s first identity in the plane, we obtain

$$\begin{aligned} \frac {dE(t)}{dt} =& \iint _{\Omega}\frac {dV}{dt}\,dA- \biggl\{ \frac{d_{1} u^{*}}{u^{2}} \iint _{\Omega } \biggl( \biggl( \frac {\partial u}{\partial x} \biggr)^{2}+ \biggl( \frac {\partial u}{\partial y} \biggr)^{2} \biggr)\,dA \\ &{}+ \frac{d_{2} v^{*}}{v^{2}} \iint _{\Omega} \biggl( \biggl( \frac {\partial v}{\partial x} \biggr)^{2}+ \biggl( \frac {\partial v}{\partial y} \biggr)^{2} \biggr)\,dA \biggr\} \\ \leq & \iint _{\Omega }\frac {dV}{dt}\,dA \leq 0. \end{aligned}$$

(3.9)

Therefore, the equilibrium $E^{*}$ of the spatial system (2.3) is globally asymptotically stable.

4 Existence of Hopf bifurcation around $E^{*}$

We analyze the Hopf bifurcation occurring at $E^{*}(u^{*},v^{*})$ by choosing ξ as the bifurcation parameter. Also, note that $E^{*}(u^{*},v^{*})$ is independent of ξ. The characteristic equation of system (2.2) at $E^{*}(u^{*},v^{*})$ is

$$\begin{aligned} \lambda ^{2}-T(\xi ) \lambda +D(\xi )=0. \end{aligned}$$

(4.1)

The two roots of (4.1) are given as $\lambda =\frac{T\pm \sqrt{T^{2}-4D}}{2}=p_{1}(\xi )\pm i p_{2}(\xi )$.

Let $\xi _{0}= \bigl \{ \frac{-r+2u^{*}+\frac{\delta (1-m)v^{*}}{((m-1)u^{*}(\alpha +(v^{*}-1)\beta )-1)^{2}}}{ (\zeta -\frac{2v^{*}}{\eta +(1-m)u^{*}}-\sigma )} \bigr\}$. To see the instability of system (2.2), let us state the following theorem.

Theorem 4.1

(Hopf bifurcation theorem [28]) Let $T(\xi )$ and $D(\xi )$ be smooth functions of ξ in an open interval about $\xi _{0} \in \mathbb{R}$ such that the characteristic equation has a pair of imaginary eigenvalues $\lambda =p_{1}(\xi )\pm p_{2}(\xi )$ with $p_{1}(\xi )$ and $p_{2}(\xi )\in \mathbb{R}$ such that λ become purely imaginary at $\xi =\xi _{0}$ and $\frac{dp_{1}}{d\xi}\vert _{\xi =\xi _{0}}\neq 0$. Then a Hopf bifurcation occurs around $E^{*}$ at $\xi =\xi _{0}$ (i.e., stability changes of $E^{*}(x^{*},y^{*})$ are accompanied by the formation of a limit cycle when $\xi =\xi _{0}$).

Theorem 4.2

When ξ passes through $\xi _{0}$, the local system (2.2) attains a Hopf bifurcation around $E^{*}(x^{*},y^{*})$ provided $T(\xi _{0})=0$ and $D(\xi _{0})>0$.

Proof

At $\xi =\xi _{0}$, the characteristic equation (4.1) becomes

$$\begin{aligned} \lambda ^{2}+D(\xi _{0})=0. \end{aligned}$$

(4.2)

The roots of equation (4.2) are $\lambda _{1}=i\sqrt{D(\xi _{0})}$ and $\lambda _{2}=-i\sqrt{D(\xi _{0})}$. Thus, at $E^{*}$, two purely imaginary eigenvalues are present. Likewise, $T(\xi )$ and $D(\xi )$ are smooth functions of ξ. Therefore, in the neighborhood of $\xi _{0}$, the roots of equation (4.2) are of the form $\lambda _{1}=p_{1}(\xi )+i p_{2}(\xi )$ and $\lambda _{2}=p_{1}(\xi )-ip_{2}(\xi )$, where $p_{i}(\xi )$ are real functions for $i=1,2$. Subsequently, let us validate the transversality condition $\frac {d}{d\xi}(\operatorname{Re} \lambda _{i}(\xi ))\vert _{\xi =\xi _{0}}\neq 0$, $i=1,2$.

By substituting $\lambda (\xi )=p_{1}(\xi )+ip_{2}(\xi )$ into the characteristic equation of the Jacobi matrix $J\vert _{E^{*}}$ (3.4), we obtain

$$\begin{aligned} \bigl(p_{1}(\xi )+i p_{2}(\xi )\bigr)^{2}+T \bigl(p_{1}(\xi )+i p_{2}(\xi )\bigr)+D=0. \end{aligned}$$

(4.3)

Differentiating both sides with respect to ξ, we obtain

$$\begin{aligned} 2\bigl(p_{1}(\xi )+i p_{2}(\xi )\bigr) \bigl( \dot{p_{1}}(\xi )+i\dot{ p_{2}}(\xi )\bigr)+T\bigl( \dot{p_{1}}(\xi )+i\dot{ p_{2}}(\xi )\bigr)+\dot{T} \bigl(p_{1}(\xi )+i p_{2}( \xi )\bigr)+\dot{D}=0. \end{aligned}$$

(4.4)

Collating the real and imaginary parts from both sides, we obtain

$$\begin{aligned}& 2p_{1}(\xi )\dot{ p_{1}}(\xi )-2p_{2}(\xi )\dot{ p_{2}}(\xi )+T\bigl( \dot{ p_{1}}(\xi )\bigr)+ \dot{T}\bigl(p_{1}(\xi )\bigr)+D=0, \end{aligned}$$

(4.5)

$$\begin{aligned}& 2p_{1}(\xi )\dot{ p_{2}}(\xi )+2p_{2}(\xi ) \dot{ p_{1}}(\xi )+T\bigl( \dot{ p_{2}}(\xi )\bigr)+ \dot{T}\bigl(p_{2}(\xi )\bigr)=0. \end{aligned}$$

(4.6)

Equation (4.5) becomes

$$\begin{aligned}& \dot{ p_{1}}(\xi ) \bigl(2p_{1}(\xi )+T\bigr)+\dot{ p_{2}}(\xi ) \bigl(-2p_{2}(\xi )\bigr)+p_{1}( \xi )\dot{T}+\dot{D}=0 \\& \quad \implies\quad \dot{ p_{1}} B_{1}-\dot{ p_{2}}B_{2}+B_{3}=0. \end{aligned}$$

(4.7)

Moreover, equation (4.6) becomes

$$\begin{aligned}& \dot{ p_{1}}(\xi ) \bigl(2p_{2}(\xi )\bigr)+\dot{ p_{2}}(\xi ) \bigl(2p_{1}(\xi )+T\bigr)+p_{2}( \xi )\dot{T}=0 \\& \quad \implies\quad \dot{p_{1}}B_{2}+\dot{ p_{2}}B_{1}+B_{4}=0, \\& \text{where } B_{1}=\bigl(2p_{1}(\xi )+T\bigr), \qquad B_{2}=\bigl(-2p_{2}(\xi )\bigr),\qquad B_{3}=p_{1}( \xi )\dot{T}+\dot{D}, \qquad B_{4}=p_{2}(\xi )\dot{T}. \end{aligned}$$

(4.8)

Multiplying (4.7) by $B_{1}$ and (4.8) by $B_{2}$ and then adding, we obtain

$$\begin{aligned}& \dot{p_{1}}\bigl(B_{1}^{2}+B_{2}^{2} \bigr)+B_{1}B_{3}+B_{2}B_{4}=0 \\& \quad \implies\quad \dot{p_{1}}= \frac {-(B_{1}B_{3}+B_{2}B_{4})}{B_{1}^{2}+B_{2}^{2}}. \end{aligned}$$

At $\xi =\xi _{0}$,

• Case (i): $p_{1}=0$, $p_{2}=\sqrt{D}$.

  $$\begin{aligned}& \text{Then } B_{1}=T\neq 0, \qquad B_{2}=2 \sqrt{D}, \qquad B_{3}=\dot{D}, \qquad B_{4}= \dot{T} \sqrt{D}. \\& \therefore B_{1}B_{3}+B_{2}B_{4} \neq 0\quad \implies\quad \dot{ p_{1}}(\xi )\vert _{ \xi =\xi _{0}} \neq 0. \end{aligned}$$

• Case (ii): $p_{1}=0$, $p_{2}=-\sqrt{D}$.

  $$\begin{aligned}& \text{Then } B_{1}=T\neq 0, \qquad B_{2}=-2 \sqrt{D}, \qquad B_{3}=\dot{D}, \qquad B_{4}=- \dot{T} \sqrt{D}. \\& \therefore B_{1}B_{3}+B_{2}B_{4} \neq 0\quad \implies\quad \dot{ p_{1}}(\xi )\vert _{ \xi =\xi _{0}} \neq 0. \end{aligned}$$

 □

Theorem 4.3

Let $D(\xi )>0$. Then

• (i) the interior equilibrium point $E^{*}=(u^{*},v^{*})$ of system (2.2) is locally asymptotically stable when $\xi >\xi _{0}$ and unstable when $\xi <\xi _{0}$; and

• (ii) the local system (2.2) passes through the Hopf bifurcation at an interior equilibrium $E^{*}=(u^{*},v^{*})$ when $\xi =\xi _{0}$.

4.1 Stability behavior of Hopf bifurcation

Here, we discuss the behavior of Hopf bifurcation. For this specific purpose, we propose the perturbation $u=u_{1}+u_{1}^{\xi _{0}}$ and $v=v_{1}+v_{1}^{\xi _{0}}$ in our local system (2.2). Then, broadening in Taylor series, we obtain

$$\begin{aligned}& \dot{u_{1}}=a_{10}u_{1}+a_{01}v_{1}+a_{20} u_{1}^{2}+a_{11}u_{1}v_{1}+a_{30}u_{1}^{3}+a_{21}u_{1}^{2}v_{1}+ \cdots , \\& \dot{v_{1}}=b_{10}u_{1}+b_{01}v_{1}+b_{20} u_{1}^{2}+b_{11}u_{1}v_{1}+b_{30}u_{1}^{3}+b_{21}u_{1}^{2}v_{1}+ \cdots , \end{aligned}$$

where $a_{10}=r-2u- \frac{\delta (1-m)v}{((m-1)u(\alpha +(v-1)\beta )-1)^{2}}$, $a_{01}= \frac{\delta (1-m)(-1+(m-1)u(\alpha -\beta ))u}{((m-1)u(\alpha +(v-1)\beta )-1)^{2}}$, $b_{10}=\frac{-\xi _{0}(m-1)v^{2}}{(\eta +(1-m)u)^{2}}$, $b_{01}=\xi _{0} (\zeta -\frac{2v}{\eta +(1-m)u}-\sigma )$. Therefore, $a_{10}+b_{01}=0$ and $a_{10}b_{01}-a_{01}b_{10}>0$.

Various other coefficients are to be determined as given below.

$$\begin{aligned}& a_{20}=1/2 \biggl(\frac{\partial ^{2}F}{\partial u^{2}} \biggr)\bigg\vert _{\xi =\xi _{0}}=-1- \frac{(m-1)^{2}v^{*}(\alpha +(v^{*}-1)\beta )\delta}{((m-1)u^{*}(\alpha +(v^{*}-1)\beta )-1)^{3}}, \\& a_{11}= \biggl(\frac{\partial ^{2} F}{\partial u \partial v} \biggr)\bigg\vert _{\xi =\xi _{0}}= \frac{(m-1)((m-1)u^{*}(\alpha -(v^{*}+1)\beta )-1)\delta}{((m-1)u^{*}(\alpha +(v^{*}-1)\beta )-1)^{3}}, \\& a_{30}=1/6 \biggl(\frac{\partial ^{3}F}{\partial u^{3}} \biggr) \bigg\vert _{\xi =\xi _{0}}= \frac{(m-1)^{3}v^{*}(\alpha +(v^{*}-1)\beta )^{2}\delta}{(-1+(m-1)u^{*}(\alpha +(v^{*}-1)\beta ))^{4}}, \\& \begin{aligned} a_{21}={}&1/2 \biggl( \frac{\partial ^{3}F}{\partial u^{2} \partial v} \biggr)\bigg\vert _{ \xi =\xi _{0}} \\ ={}& - \bigl( \bigl((m-1)^{2} ( \alpha \bigl(-1+(m-1)u^{*}\alpha \bigr)+\beta -2v^{*}\beta \\ &{}-2(m-1)u^{*}\alpha \beta -(m-1)u^{*}\bigl({v^{*}}^{2}-1\bigr)\beta ^{2} )\delta \bigr)\bigr) \\ &{}\big/\bigl(\bigl(-1+(m-1)u^{*}\bigl(\alpha +\bigl(v^{*}-1 \bigr)\beta \bigr)\bigr)^{4}\bigr), \end{aligned} \\& b_{20}=1/2 \biggl(\frac{\partial ^{2}G}{\partial u^{2}} \biggr) \bigg\vert _{\xi =\xi _{0}}= \frac{-2(m-1)^{2}v^{*}\xi}{((1-m)u^{*}+\eta )^{3}}, \\& b_{11}= \biggl(\frac{\partial ^{2}G}{\partial u \partial v} \biggr)\bigg\vert _{\xi =\xi _{0}}= \frac{-2(m-1)v^{*}\xi}{((1-m)u^{*}+\eta )^{2}}, \\& b_{30}=1/6 \biggl(\frac{\partial ^{3}G}{\partial u^{3}} \biggr) \bigg\vert _{\xi =\xi _{0}}= \frac{-(m-1)^{3}{v^{*}}^{2}\xi}{((1-m)u^{*}+\eta )^{4}}, \\& b_{21}=1/2 \biggl( \frac{\partial ^{3}G}{\partial u^{2}\partial v}\biggr)\bigg\vert _{\xi = \xi _{0}} =\frac{-2(1-m)^{2}v^{*}\xi}{((1-m)u^{*}+\eta )^{3}}, \end{aligned}$$

where

$$\begin{aligned}& F= u (r-u )- \frac{\delta (1-m)uv}{1+\alpha (1-m)u+\beta ((1-m)u)(v-1)}, \\& G= \xi \biggl(\zeta v-\frac {v^{2}}{\eta +(1-m)u}- \sigma v \biggr). \end{aligned}$$

By [32], the first Lyapunov number to conclude the dynamics (stable or unstable) of the limit cycle arising through Hopf bifurcation has

$$\begin{aligned} \Lambda =&\frac{-3\pi}{2 a_{01}(a_{10}b_{01}-a_{01}b_{10})^{3/2}}\bigl(\bigl[a_{10}b_{10}a_{11}^{2}+a_{10}a_{01} \bigl(b_{11}^{2}+a_{20}b_{11} \bigr)-2a_{10}a_{01}a_{20}^{2} \\ &{}-a_{01}^{2}(2a_{20}b_{20}+b_{11}b_{20})- \bigl(a_{01}b_{10}-2a_{10}^{2} \bigr)a_{11}a_{20}\bigr] \\ &{}-\bigl(a_{10}^{2}+a_{01}b_{10} \bigr)\bigl[-3a_{01}a_{30}+2a_{10}(a_{21}+b_{12})+(b_{10}a_{12}-a_{01}b_{21}) \bigr]\bigr). \end{aligned}$$

Theorem 4.4

When $\Lambda <0$, the direction of Hopf bifurcation is supercritical and the bifurcated periodic solutions are stable; when $\Lambda >0$, the direction of Hopf bifurcation is subcritical and the bifurcated periodic solutions are unstable.

5 Diffusion-driven instability

Through this section, we concentrate on the prey–predator system with self-diffusion and examine the occurrence of Turing instability in the equilibrium point under diffusion effects (diffusion-driven instability).

Theorem (4.3) states that whenever $\xi >\xi _{0}$, the nontrivial interior equilibrium $E^{*}$ is locally asymptotically stable for the nondiffusion system (2.2).

We consider the influences of diffusion on the stable nontrivial interior equilibrium $E^{*}$ of (2.3) under the supposition $\xi > \xi _{0}$. Subsequently, for the diffusion system (2.3), we should consider the one-dimensional space $\Omega =(0, \pi )$ with a smooth boundary ∂Ω:

$$\begin{aligned}& \begin{aligned} & u_{t}=u (r-u )- \frac{\delta (1-m)uv}{1+\alpha (1-m)u+\beta ((1-m)u)(v-1)}+d_{1} u_{xx},\quad x\in (0,\pi ) , t>0, \\ & v_{t}=\xi \biggl(\zeta v-\frac{v^{2}}{\eta +(1-m)u}- \sigma v \biggr)+d_{2}v_{xx},\quad x\in (0,\pi ) , t>0, \\ &u_{x}(0,t)=u_{x}(\pi ,t)=0,\qquad v_{x}(0,t)=v_{x}(\pi ,t)=0,\quad t>0. \end{aligned} \end{aligned}$$

(5.1)

This is actually the notable operator $u\rightarrow -u_{xx}$ with Neumann boundary conditions. The analogous eigenvalues and normalized eigenfunctions are

$$ \varrho _{0}=0,\qquad \varphi _{0}(x)=\sqrt{ \frac{1}{\pi}}, \qquad \varrho _{k}=k^{2}, \qquad \varphi _{k}(x)=\sqrt{\frac{2}{\pi}} \cos (kx),\quad \text{where }k=1,2,3,\ldots . $$

Linearizing the above diffusion system (5.1) at $E^{*}$, we obtain

$$\begin{aligned} \begin{bmatrix} u_{t} \\ v_{t} \end{bmatrix}=L \begin{bmatrix} u \\ v \end{bmatrix}+J \begin{bmatrix} u \\ v \end{bmatrix}+D \begin{bmatrix} u_{xx} \\ v_{xx} \end{bmatrix}, \end{aligned}$$

where $D= \operatorname{diag} (d_{1},d_{2})$ and J is the Jacobian matrix pointed out in Sect. 3.3. L indicates a linear operator whose domain is $D_{L}=X_{\mathbb{C}}:=X\oplus i X=\{u1+i u2 : u1,u2\in X\}$, where $X:=\{(u,v)\in H^{2}[(0,\pi )]\times H^{2}[(0,\pi )]:u_{x}(0,t)=u_{x}( \pi ,t)=v_{x}(0,t)=v_{x}(\pi ,t)=0\}$, which is a real-valued Sobolev space.

Consider the characteristic equation $L(\varphi ,\psi )^{T}=\varrho (\varphi ,\psi )^{T}$, and let

$$\begin{aligned} (\varphi ,\psi )^{T}=\sum_{k=0}^{\infty}(a_{k},b_{k})^{T} \cos (kx). \end{aligned}$$

(5.2)

Let $J_{k}=J-k^{2}D$. Then we obtain $\sum_{k=0}^{\infty}(J_{k}-\varrho I)(a_{k},b_{k})^{T} \cos (kx)=0$.

Thus, all eigenvalues of L are obtained by the eigenvalues of $J_{k}$ for $k=1,2,3,\ldots$ . The characteristic equation of $J_{k}$ is

$$\begin{aligned} \varrho ^{2}-T_{k}\varrho +D_{k}=0, \quad k=1,2,3,\ldots, \end{aligned}$$

(5.3)

where

$$\begin{aligned}& T_{k}=\operatorname{tr} J_{k}=T-(d_{1}+d_{2})k^{2}, \\& \text{and} \quad D_{k}=\det J_{k} =D+d_{1}d_{2}k^{4}+(\delta d_{1}- \delta _{0} d_{2})k^{2}. \end{aligned}$$

Simply by examining the distribution of characteristic roots of $J_{k}$, we obtain the imminent conclusion.

Theorem 5.1

Assume that $\xi >\xi _{0}$ and $D(\xi )>0$. Then

• (i) the equilibrium $E^{*}=(u^{*},v^{*})$ of the nondiffused system (2.2) is locally asymptotically stable;

• (ii) the equilibrium $E^{*}=(u^{*},v^{*})$ of the diffused system (5.1) is locally asymptotically stable if and only if the following conditions hold:

  $$\begin{aligned}& (C1) \quad d_{1}\geq \frac {d_{2}\xi _{0}}{\xi}, \\& (C2) \quad d_{1}< \frac {d_{2}\xi _{0}}{\xi} \quad \textit{and} \quad \frac {d_{1}}{d_{2}}> \frac {\xi _{0}\xi +2D-2\sqrt{D(\xi \xi _{0}+D)}}{\xi ^{2}}. \end{aligned}$$

  Furthermore, for the diffused system (5.1), the solution $E^{*}$ is unstable (that is, Turing instability occurs) if

  $$\begin{aligned}& (C3) \quad 0< \frac {d_{1}}{d_{2}}< \frac {\xi _{0}\xi +2D-2\sqrt{D(\xi \xi _{0}+D)}}{\xi ^{2}} \quad ( \textit{Turing instability domain}). \end{aligned}$$

Proof

Because $\xi >\xi _{0}$ ($T<0$) signifies that $T_{k}<0$ for all $k\geq 0$. Moreover, by the meaning of $T_{k}$, we have the relationship that for every $k\geq 0$, $T_{k+1}< T_{k}$. We calculate $D_{k}$ like a quadratic function in $k^{2}$, that is, $D_{k}=D(k^{2})=d_{1}d_{2}(k^{2})^{2}+(\xi d_{1}-\xi _{0} d_{2})k^{2}+D$. Consequently, the real part of the characteristic values signs of (5.3) are discovered by the sign of $D(k^{2})$, separately. The symmetric axis of the graph $(k^{2}, D(k^{2}))$ is $k_{\min}^{2}=-\frac{(\xi d_{1}-\xi _{0} d_{2})}{2d_{1}d_{2}}$. We know that when $D_{k}<0$ ($D(k^{2})<0$), the characteristic roots of $J_{k}$ (5.3) are two real roots with opposite signs. Notice that in $D(k^{2})$, $d_{1}d_{2}>0$ and $k^{2}>0$. Therefore, whenever $\xi d_{1}-\xi _{0} d_{2}<0$, $D(k^{2})$ attains the bare minimum at $k^{2}=k_{\min}^{2}$.

Therefore, when (C3) holds, $D(k_{\min}^{2})$ is negative. However, this means that any one of the characteristic roots of $J_{k}$ have a positive real part, that is, that $E^{*}$ is the unstable solution of (5.1). As a result, we consider that whenever (C3) holds, Turing instability occurs. (C1) implies that $D_{k}>0$ for all $k\geq 0$ (because $D_{0}=D>0$), and (C2) implies that $D(k_{\min}^{2})$ is positive, and therefore that all the characteristic roots of $J_{k}$ have negative real parts. In this manner, any of conditions (C1) and (C2) guarantee that the characteristic roots of $J_{k}$ have negative real parts. Consequently, if any one of the conditions of (C1) and (C2) holds, then $E^{*}$ is the stable equilibrium solution of (5.1). □

6 Numerical simulation

In this section, we describe a few numerical simulations performed to verify our analytical findings using MATLAB. We consider system (2.2) with fixed parameters $\{r=0.7, \delta =1.5, \alpha =0.8, \beta =0.4, \xi =1.8, \zeta =1.7, \eta =0.4, \sigma =1.4, 0\leq m=0.35<1\}$. For these parametric values, the corresponding nullclines are shown in Fig. 1. The pink and orange curves represent the prey and predator nullclines, respectively. Intersecting points of nullclines are given by equilibrium points $E_{0}$, $E_{1}$, $E_{2}$, and $E^{*}(u^{*},v^{*})=(0.51525,0.22047)$. Note that, for the above set of parameters, one condition $\varDelta _{3}>0$ is violated in (H) although there exists a unique $E^{*}$ in $\mathbb{R}_{+}^{2}$. Therefore, the set of conditions we mentioned in (H) is necessary but not sufficient for the existence and uniqueness of $E^{*}\in \mathbb{R}_{+}^{2}$.

Figure 1

Nullclines of a prey–predator system (2.2) with parameters $\{r=0.7, \delta =1.5, \alpha =0.8, \beta =0.4, \xi =1.8, \zeta =1.7, \eta =0.4, \sigma =1.4, m=0.35\}$

Through the vector fields (blue arrows) in Fig. 1, we may understand the stability of the equilibriums. For this set of parameters, $E_{0}=(0,0)$ is unstable. In particular, for this case, $E_{0}$ is a nodal source. Also, where $\zeta =1.7>\sigma =1.4$, therefore $E_{1}=(r,0)=(1.7,0)$ and $E_{2}=(0,\zeta \eta -\eta \sigma )=(0,0.12)$ are also unstable (saddle points).

By fixing the same set of above parameters other than ζ and σ with condition $\zeta =1.4<\sigma =1.7$, our resultant phase plane is shown in Fig. 2. Because $\zeta <\sigma$, the axial equilibrium $E_{2}=(0,\zeta \eta -\eta \sigma )=(0,-0.12)\notin \{\{0\}\cup \mathbb{R}\}_{+}^{2}$. We drew different phase trajectories starting from various initial points. From the phase plane, we obtain that $E_{1}=(r,0)=(0.7,0)$ is locally asymptotically stable (a nodal sink). Also, Fig. 2 shows that $E_{0}$ is unstable (a saddle point) and $E_{2}$ is unstable (a nodal source). Also note that for this set of parametric values, the conditions $\varDelta _{1}<0$, $\varDelta _{3}>0$ do not hold in (H). Therefore, no interior equilibrium $E^{*}(u^{*},v^{*})$ exists in $\mathbb{R}_{+}^{2}$. However, there were three nontrivial equilibriums (yellow dots) on quadrants II and III of the phase plane. From the biological perspective, these are meaningless. The following set possesses all conditions of (H) and $\zeta >\sigma$: $\{r=1, \delta =0.8, \alpha =10, \beta =0.1, \xi =0.15, \zeta =7.1, \eta =0.01, \sigma =6, 0\leq m=0.01<1\}$. Then there exists a unique nontrivial interior equilibrium $E^{*}(u^{*},v^{*})=(0.920642,1.01358)\in \mathbb{R}_{+}^{2}$, which is shown in Fig. 3. For these parameter values, the trivial equilibrium point $E_{0}=(0,0)$ is unstable and axial equilibrium $E_{1}=(1,0)$ and $E_{2}=(0,0.011)$ are saddle points. Further, the above set of parameters follows the conditions $(\delta (1-m)v^{*}-u^{*})\alpha <\frac{1}{1-m}+\beta u^{*}(v^{*}-1)$ and $(1-m)u^{*}(\beta -\alpha )<1$. As per Theorem 3.2, Fig. 3 shows that different phase trajectories start from different initial points, but ultimately all the trajectories converge to the same interior equilibrium point $E^{*}=(0.920642,1.01358)$. This shows that $E^{*}$ is globally asymptotically stable.

Figure 2

Phase plane of the prey–predator system (2.2) with parameters $\{r=0.7, \delta =1.5, \alpha =0.8, \beta =0.4, \xi =1.8, \zeta =1.4, \eta =0.4, \sigma =1.7, m=0.35\}$

Figure 3

Existence and uniqueness of $E^{*}(u^{*},v^{*})$. The parametric values are taken as $\{r=1, \delta =0.8, \alpha =10, \beta =0.1, \xi =0.15, \zeta =7.1, \eta =0.01, \sigma =6, m=0.01\}$. Moreover, the steady state $E^{*}=(0.920642,1.01358)$ is globally asymptotically stable. Because $(\delta (1-m)v^{*}-u^{*})\alpha <\frac{1}{1-m}+\beta u^{*}(v^{*}-1)$ and $(1-m)u^{*}(\beta -\alpha )<1$

Now, consider the prey–predator model (2.2) together with the following set of parameters: $\{r=1, \delta =8, \alpha =10, \beta =0.1, \zeta =7.1, \eta =0.01, \sigma =6, m=0.01\}$. Then our ODE model becomes

$$\begin{aligned} \textstyle\begin{cases} \frac {du}{dt}= u (1-u )- \frac {8(1-0.01)uv}{1+10(1-0.01)u+0.1((1-0.01)u)(v-1)}, \\ \frac {dv}{dt}= \xi (7.1 v-\frac {v^{2}}{0.01+(1-0.01)u}-6 v ), \\ u(0)= u_{0}>0, \qquad v(0)=v_{0}>0. \end{cases}\displaystyle \end{aligned}$$

(6.1)

Under this set of parameters, the above system (6.1) has a unique nontrivial positive interior equilibrium $E^{*}(u^{*},v^{*})=(0.315572,0.3554658)$. Pertaining to the preceding parametric values, the crucial point value is $\xi _{0}=0.183713$, which is independent of ξ. Further, we vary the bifurcation parameter ξ only.

By fixing $\xi =0.2$, we obtain $\xi >\xi _{0}=0.183713$, and it follows from Theorem (4.3) that $E^{*}(u^{*},v^{*})$ is locally asymptotically stable (see Fig. 4).

Figure 4

Stability behavior with respect to time t and phase portraits of system (2.2) with $\xi >\xi _{0}$ and initial data $(u_{0},v_{0})=(0.5,0.3)$

Fixing $\xi =\xi _{0}=0.183713$ implies that $T(\xi _{0})=0$. Subsequently, by Theorem (4.3), $E^{*}$ loses its stability and a Hopf bifurcation arises, that is, a family group of intermittent periodic solutions bifurcating from the interior equilibrium $E^{*}$ (see Fig. 5).

Figure 5

Periodic behavior with respect to time t and phase portraits of system (2.2) with $\xi =\xi _{0}$ and initial data $(u_{0},v_{0})=(0.32,0.36)$

Through $\xi =0.15$, we certainly have $\xi <\xi _{0}$, and it follows by Theorem (4.3) that $E^{*}(u^{*}, v^{*})$ is unstable (see Fig. 6).

Figure 6

Unstable behavior with respect to time t and phase portraits of system (2.2) with $\xi <\xi _{0}$ and initial data $(u_{0},v_{0})=(0.32,0.36)$

Moreover, when ξ goes through $\xi _{0}$ from the right-hand side of $\xi _{0}$, the equilibrium point $E^{*}$ loses its steadiness and a Hopf bifurcation occurs, as shown in Fig. 5 and Fig. 6. For the above set of parameters, we obtain $\Lambda =-9.01885<0$. Therefore, from Theorem 4.4, the direction of the Hopf bifurcation at $\xi =\xi _{0}$ is supercritical, and the bifurcating periodic solutions are asymptotically stable, as shown in phase diagrams of Fig. 5 and Fig. 6.

The formation of a limit cycle around the interior equilibrium point with initial data $(0.4, 0.3)$ and $(0.5,0.5)$ inside and outside of the limit cycle, respectively, is shown in Fig. 7. The diagram shows that the limit cycle is stable.

Figure 7

Existence of a stable limit cycle around the interior equilibrium $E^{*}=(0.315572, 0.354658)$

To verify the occurrence of Turing instability of the diffusive prey–predator system (5.1), we consider the following set with Neumann boundary condition on a one-dimensional spatial domain $\Omega =(0,60)$: $\{r=1, \delta =8, \alpha =10, \beta =0.1, \zeta =7.1, \eta =0.01, \sigma =6, m=0.01\}$. We modify the diffusion coefficients $d_{1}$ and $d_{2}$ only.

Within the parameters $d_{1}=1$, $d_{2}=2$ and $\xi =0.2$ ($0.2=\xi > \xi _{0}=0.183713$), also $1=d_{1}< \frac{d_{2}\xi _{0}}{\xi}=1.83713$ and $\frac{d_{1}}{d_{2}}=0.5> \frac{\xi _{0}\xi +2D-2\sqrt{D(\xi \xi _{0}+D)}}{\xi ^{2}}=0.071$, that is, (C2) holds. By Theorem 5.1, the homogeneous equilibrium solution $E^{*}$ of system (5.1) is stable. The stability behavior and the corresponding contour diagram is shown in Fig. 8.

Figure 8

Numerical simulations of the diffusion system (5.1) with parameter restriction $\xi =0.2$ ($0.2=\xi > \xi _{0}=0.183713$), $d_{1}=1$ and $d_{2}=2$, with the initial condition ($t=0$, $0.1+0.05(\cos (\frac{1}{2}u))$, $0.1+0.03\sin (\frac{1}{2}v)$)

Under the parameters $d_{1}=0.01$, $d_{2}=1$ and $\xi =0.2$ ($\xi > \xi _{0}$), $0.01=\frac{d_{1}}{d_{2}}< \frac{\xi _{0}\xi +2D-2\sqrt{D(\xi \xi _{0}+D)}}{\xi ^{2}}=0.071$, that is, (C3) holds. By Theorem 5.1, the homogeneous equilibrium solution $E^{*}$ of system (5.1) is unstable. That is, Turing instability occurs. The Turing instability and the corresponding contour diagram are shown in Fig. 9.

Figure 9

Numerical simulations of the diffusion system (5.1) with parameter restriction $\xi =0.2$ ($0.2=\xi > \xi _{0}=0.183713$), $d_{1}=0.01$ and $d_{2}=1$, with the initial condition ($t=0, 0.31, 0.35$)

7 Conclusion

Our study focused on investigating the dynamic behavior of a prey–predator model that incorporates prey refuge and interference among predators. The interaction between prey and predators was governed by the Crowley–Martin response function. Our key contribution lies in the thorough analysis of the refuge function during intraspecific competition among predators for prey.

Throughout our investigation, we successfully established the existence criteria for biologically meaningful axial and interior equilibrium points, and we assessed their stability. Particularly, we found that when the growth rate of predators, based on the conversion coefficients from individual prey to individual predators, exceeds the death rate of predators ($\zeta =r_{2} \rho /\delta _{2}>\sigma =\delta _{3} \rho /\delta _{2}$), only predators will survive ($v^{*}>0$).

Moreover, we identified the influential role of the parameter ξ, independent of $E^{*}$, in causing a Hopf bifurcation around $E^{*}$. Theorem 4.3 reveals that when $D>0$ and the proportion between the maximal per capita predator consumption rate and the intensity of competition among individuals of the prey species ($\xi =\delta _{2}/\rho$) exceeds the critical value $\xi _{0}$, the prey–predator inhabitants are stable for any initial interior population. This emphasizes the significance of prey refuge and interference among predators in maintaining stable populations. However, when the ratio between the maximal per capita predator consumption rate and the intensity of competition among individuals of the prey species falls below the critical value $\xi _{0}$, the population size becomes unstable and cannot be precisely determined. When this ratio precisely equals the critical value $\xi _{0}$, the prey–predator inhabitant dynamics exhibit periodic changes due to the occurrence of Hopf bifurcation, as presented in Theorem 4.4.

Furthermore, we delved into the dynamics of the prey–predator population, considering both spatial and temporal motion. To capture spatial movements, we analyzed a diffusion system (5.1) and explored the diffusion-driven instability of the spatial system in detail. We observed that the stability of the interior equilibria $E^{*}$ could vary from stable to unstable, even when $\xi >\xi _{0}$ was satisfied due to the occurrence of diffusion-driven effects. The corresponding results are presented in Theorem 5.1. To validate our analytical findings, we provided numerical examples in Sect. 6, ensuring the robustness of our results.

As we conclude our study, we recognize that an exciting avenue for future research would involve exploring the impact of stochastic noise on the model, particularly in terms of habitat-dependent parameters. Incorporating stochastic elements could offer valuable insights into the system’s resilience and adaptability under varying environmental conditions.

In summary, our work contributes to a deeper understanding of the complex dynamics within prey–predator ecosystems, shedding light on how refuge and interactions among predators play pivotal roles in governing population stability and behavior. Our findings open the door to further investigations and offer potential applications in ecological management and conservation efforts.