Steady states of a diffusive predator-prey model with prey-taxis and fear effect

Abstract

In this paper, a diffusive predator-prey system with a prey-taxis response subject to Neumann boundary conditions is considered. The stability, the Hopf bifurcation, the existence of nonconstant steady states, and the stability of the bifurcation solutions of the system are analyzed. It is proved that a high level of prey-taxis can stabilize the system, the stability of the positive equilibrium is changed when χ crosses $\chi _{0}$, and the Hopf bifurcation occurs for the small s. The system admits nonconstant positive solutions around $(\bar{u}, \bar{v}, \chi _{i} )$, the stability of bifurcating solutions are controlled by $\int _{\Omega} \Phi _{i}^{3} \,\mathrm{d}x$ and $\int _{\Omega} \Phi _{i}^{4} \,\mathrm{d}x$. Finally, numerical simulation results are carried out to verify the theoretical findings.

1 Introduction

In biomathematics, the interactions among species and the spatial distributions of populations have always been the hot topics of ecosystems, and are important for developing research on economic benefits, pest control, and environmental governance. The predator-prey system has been deeply studied in [1–6].

In ecosystems, the prey populations are often affected by the indirect effects of predators, in addition to the direct killing, which is called the fear effect. It shows that the effect of the fear effect on the birth rate of prey is even more significant than the direct effect. In 2016, Wang [7] first introduced the fear effect into a predator-prey model that showed that the fear effect does not work on the system with a linear functional response function. However, for the model with a Holling-II functional response function, a high level of fear makes the system stable, while a low level of fear makes the system oscillate periodically. Wang and Zou [8] proposed a reaction-diffusion convective predator-prey model, which incorporated the fear effect into the local response term of the model. The authors of [9–14] studied the effect of the fear effect on the models from different aspects.

Predators prefer the areas with high prey density, in addition to the free diffusion, which is called prey-taxis. According to the positions of the stimuli, motions can be divided into positive taxis and negative taxis. The experiment was first proposed in 1970 by Keller and Segel in [15], and the results have shown that the solutions may blow up in finite time. For predator-prey systems, Kareiva and Odell [16] observed the prey-taxis model in 1987, and described that predators will gather near higher densities of prey to more effectively find prey. The model showed that prey-taxis increased the stability of the predator-prey system. Lee investigated prey-taxis for $\rho (u,v)= \chi$ is a constant, the paper showed that prey-taxis stabilizes the system and for large prey-taxis sensitivity, no pattern formation was observed in [17]. Wang discussed the effects of prey-taxis on the existence and stability of nonconstant steady states, gave the bifurcation type and stability of the bifurcation curve, and provided a stable wave model selection mechanism for the reaction-diffusion system with prey-taxis in [18] and [19]. Wu considered a more general predator-prey model with prey-taxis in [20], obtained the global existence and boundedness of the equilibria. Moreover, the prey can also adjust the corresponding position to reduce the risk of being caught, which is called predator-taxis [21]. Also, for the systems with three species [22–24], the authors investigated the global existence of the system solutions and the local stability conditions of the equilibria. More details about chemotaxis can be seen in [25–31].

Inspired by the above works, considering that the predator turns to another species when the favorite food is lacking, we introduce the modified Leslie-Gower term to make the model more realistic and established the following system,

$$ \textstyle\begin{cases} \frac{\partial u(x,t)}{\partial t}=d_{1}\Delta u+ \frac{r_{0}u}{1+av}-du-cu^{2}- \frac{puv}{u+kv},& x\in \Omega ,t>0, \\ \frac{\partial v(x,t)}{\partial t}=d_{2}\Delta v-\chi \nabla (v\nabla u)+sv (1- \frac{qv}{u+m} ),& x \in \Omega ,t>0, \\ \frac{\partial u(x,t)}{\partial \nu}= \frac{\partial v(x,t)}{\partial \nu}=0,& x\in \partial \Omega ,t>0, \\ u(x,0)=u_{0}(x)\geq 0, v(x,0)=v_{0}(x)\geq 0,&x\in \Omega , \end{cases} $$

(1)

where Ω is a bounded domain in $\mathbb{R}^{N}$, $N\geq 1$, with a smooth boundary ∂Ω; ∂ν is the outer flux; Δ and ∇ are Laplace and gradient operators defined in Ω. u, v represent the density of the prey and predator; $d_{1}$ and $d_{2}$ are self-diffusion coefficients; $r_{0}$ and d are the birth and death rate of prey; a is the fear factor; $-\chi \nabla (v\nabla u)$ stands for the prey-taxis term, χ is called the prey-taxis coefficient, prey-taxis is repulsive for $\chi <0$ and attractive for $\chi >0$; s is the intrinsic growth rate of predators, $\frac{puv}{u+kv}$ is the functional response; $\frac{qv}{u+m}$ is the modified Leslie-Gower term, all the parameters are positive. Furthermore, we discuss the effect of constant efficiency of predation p and the growth rate of predator species s, in addition to the prey-taxis in other articles. We mainly proved the existence of the steady states of the system, the dynamic equilibrium of biological populations helps to prevent species extinction, provides measures to control the population quantity, and forecasts the change of the population quantity, gives theoretical guidance for reasonable control of the population quantity.

The ODE corresponding to the system (1) can be expressed as,

$$ \textstyle\begin{cases} \frac{d u}{d t}= \frac{r_{0}u}{1+av}-du-cu^{2}- \frac{puv}{u+kv}, \\ \frac{d v}{d t}=sv (1- \frac{qv}{u+m} ), \\ u(0)\geq 0, \qquad v(0)\geq 0. \end{cases} $$

(2)

We can see that system (2) has two types of equilibria:

(1) the prey-free equilibrium $e_{01} (0, \frac{m}{q} )$;

(2) the predator-free equilibrium $e_{10} ( \frac{r_{0}-d}{c},0 )$, if $r_{0}>d$;

(3) the unique positive equilibrium $e_{2}(\overline{u}, \overline{v})$ with $\overline{v}= \frac{\overline{u}+m}{q}$, and u̅ is the positive root of the equation

$$ \frac{r_{0}q}{a\overline{u}+am+q}-d-c\overline{u}- \frac{p(\overline{u}+m)}{(q+k)\overline{u}+km}=0, $$

the positive equilibrium $e_{2}(\overline{u}, \overline{v})$ exists if $r_{0}> \frac{(dk+p)(q+am)}{kq}$.

The remainder of this paper is structured as follows. In Sect. 2, the stability of the positive equilibrium with and without the prey-taxis χ is verified, and the conditions for the Hopf bifurcation of the system (3) are given. In Sect. 3, the existence of the nonconstant steady states is proved, and the stability of the nonconstant solution is analyzed. Finally, numerical simulations are used to substantiate the theoretical findings in Sect. 4, with some conclusions being drawn in Sect. 5.

2 Stability analysis of the positive equilibrium

In order to investigate the stability of $e_{2}(\overline{u}, \overline{v})$ further, we denote $(u,v)=(\bar{u},\bar{v})+(U,V)$ to be a small perturbation away from $(\bar{u},\bar{v})$, and linearly expand the system (1) at $(\bar{u},\bar{v})$,

$$ \textstyle\begin{cases} U_{t}\approx d_{1}\Delta U+ ( \frac{p\bar{u}\bar{v}}{ (\bar{u}+k\bar{v} )^{2}} -c\bar{u} )U- ( \frac{ar_{0}\bar{u}}{ (1+a\bar{v} )^{2}} + \frac{p\bar{u}^{2}}{ (\bar{u}+k\bar{v} )^{2}} )V, \\ V_{t}\approx d_{2}\Delta V-\chi \bar{v}\Delta U+ \frac{s}{q}U-sV, \\ \frac{\partial U}{\partial \nu}= \frac{\partial V}{\partial \nu}=0. \end{cases} $$

(3)

The matrix of $(\bar{u},\bar{v})$ of (3) is

$$L_i:=\left(\begin{array}{cc}-d_1{\lambda }_i+a_{11}& a_{12}\\ \chi \overline{v}{\lambda }_i+\frac{s}{q}& -d_2{\lambda }_i-s\end{array}\right),i\in \mathbb{N},$$

where

$$ a_{11}= \frac{p\bar{u}\bar{v}}{(\bar{u}+k\bar{v})^{2}}-c \bar{u}, \qquad a_{12}=- \frac{ar_{0}\bar{u}}{(1+a\bar{v})^{2}}- \frac{p\bar{u}^{2}}{(\bar{u}+k\bar{v})^{2}}, $$

and $\lambda _{i}$ is the ith eigenvalue of −Δ over Ω under the Neumann boundary conditions that satisfy $0=\lambda _{0}<\lambda _{1}<\cdots <\lambda _{i}<\cdots <\infty$, the meaning of $\lambda _{i}$ is the same as in [19], and more details can be seen in that reference. The characteristic equation for the eigenvalue λ is

$$ \lambda ^{2}- \text{T}\lambda +\text{D}=0, $$

where

$$\begin{aligned} \text{T}&=-(d_{1}+d_{2})\lambda _{i}-s+a_{11}, \\ \text{D}&=d_{1}d_{2}{\lambda _{i}}^{2}+ (sd_{1}-a_{11}d_{2} -\chi \bar{v}a_{12} )\lambda _{i}- \biggl(sa_{11}+ \frac{s}{q}a_{12} \biggr). \end{aligned}$$

Theorem 2.1

Suppose $ck^{2}m \geq pq$ holds. Denote $\chi _{0}=\max_{i\in \mathbb{N}^{+}} \chi _{i}$, then the positive equilibrium $(\bar{u},\bar{v})$ is locally asymptotically stable (unstable) for $\chi >\chi _{0}$ ($\chi <\chi _{0}$); and the positive equilibrium $(\bar{u},\bar{v})$ is always stable for $i=0$.

Proof

The real part of λ is negative if and only if $\text{T}<0$ and $\text{D}>0$. First, we discuss the sign of $a_{11}$,

$$\begin{aligned} a_{11}&= \frac{p\bar{u}\bar{v}}{(\bar{u}+k\bar{v})^{2}}-c \bar{u} \\ &= \frac{\bar{u}}{(q\bar{u}+k\bar{u}+km)^{2}}\cdot \bigl[-c(q+k)^{2}\bar{u}^{2}- \bigl(2c (q+k )km-pq \bigr) \bar{u}-\bigl(ck^{2}m^{2}-pqm \bigr) \bigr]. \end{aligned}$$

When $ck^{2}m \geq pq$, $a_{11}<0$ for all $\bar{u}>0$, then $\text{T}=-(d_{1}+d_{2})\lambda _{i}-s+a_{11}<0$.

D is a linear function of χ, denoting

$$ \chi _{i}= \frac{q (d_{1}\lambda _{i}-A(\bar{u},\bar{v}) )(d_{2}\lambda _{i}+s)-sB(\bar{u},\bar{v})}{q\bar{v}B(\bar{u},\bar{v})\lambda _{i}} ,\quad i\in \mathbb{N}^{+}, $$

(4)

we rewrite D as

$$ \text{D}=\bar{v}B(\bar{u},\bar{v})\lambda _{i}(\chi _{i}- \chi ), $$

where $\text{D}>0$ is equivalent as $\chi _{i}<\chi$ for $B(\bar{u},\bar{v})<0$ and $\bar{v}>0$. Thus, the positive equilibrium $(\bar{u},\bar{v})$ of model (1) is locally asymptotically stable for $ck^{2}m\geq pq$ and $\chi >\chi _{i}$ according to standard linearized stability analysis. Moreover, $\text{T}=-s-a_{11}<0$, $\text{D}=- (sa_{11}+ \frac{s}{q}a_{12} )>0$ for $i=0$, the positive equilibrium $(\bar{u},\bar{v})$ is always stable. □

Next, we let $0< ck^{2}m< pq$ in order to further analyze the instability of the system. We give a Lemma about the $a_{11}$ first.

Lemma 2.2

For the $a_{11}(\bar{u})$, there exists $0< u_{1}< u_{2}$, such that $a_{11}(0)=a_{11}(u_{2})=0$, $a_{11}(\bar{u})$ is increasing for $\bar{u}\in (0,u_{1})$ and decreasing for $\bar{u}\in (u_{1},u_{2})$, obtains the local maximum value at $u_{1}$.

Proof

View $a_{11}$ as a function of ū,

$$\begin{aligned} a_{11}(\bar{u})&= \frac{-c(q+k)^{2}\bar{u} ^{3}+ (pq-2ckm(q+k) )\bar{u}^{2}-(ck^{2}m^{2}-pqm)\bar{u}}{(k\bar{u}+q\bar{u}+km)^{2}} \\ &= \frac{\bar{u}}{(q\bar{u}+k\bar{u}+km)^{2}}\cdot \bigl[-c(q+k)^{2}\bar{u}^{2}- \bigl(2c (q+k )km-pq \bigr) \bar{u}-\bigl(ck^{2}m^{2}-pqm \bigr) \bigr]. \end{aligned}$$

Now, $a_{11}(0)=0$, and there exists one and only one positive solution $u_{2}>0$ such that $a_{11}(u_{2})=0$ for $ck^{2}m< pq$.

For the derivation of $a_{11}(\bar{u})$,

$$ a_{11}^{\prime}(\bar{u})= \frac{-c(q+k)^{3}\bar{u} ^{3}-3ckm(q+k)^{2}\bar{u}^{2}- (pqm(q-k)+3ck^{2}m^{2}(q+k) )\bar{u} -km^{2}(ck^{2}m-pq)}{(k\bar{u}+q\bar{u}+km)^{3}}, $$

$pqm(q-k)+3ck^{2}m^{2}(q+k)>2ck^{2}m^{2}(2q+k)>0$ for $ck^{2}m< pq$. Then, there exists only one positive value $u_{1}$ that satisfies $0< u_{1}< u_{2}$, such that $a_{11}^{\prime}(u_{1})=0$, $a_{11}^{\prime}(\bar{u})>0$ for $\bar{u}\in (0,u_{1})$ and $a_{11}^{\prime}(\bar{u})<0$ for $\bar{u}>u_{1}$. Hence, for the $a_{11}(\bar{u})$, $a_{11}(0)=a_{11}(u_{2})=0$, $a_{11}(\bar{u})$ is increasing for $\bar{u}\in (0,u_{1})$, decreasing for $\bar{u}\in (u_{1},u_{2})$, $0< u_{1}< u_{2}$, and obtains the local maximum value $\max a_{11}(\bar{u})= a_{11}(u_{1})$. □

From [32], the Hopf bifurcation value u should satisfy the conditions: there exists $n\in \mathbb{N}$, such that

$$ \text{T}_{n}(u)=0, \qquad \text{D}_{n}(u)>0 \quad \text{and} \quad \text{T}_{j}(u)\neq{0}, \qquad \text{D}_{j}(u) \neq{0} \quad \text{for } j\neq{n}, $$

(5)

and for the complex eigenvalue $\alpha (u)\pm{i\omega (u)}$,

$$ \alpha '(u)\neq 0.$$

Then, any potential Hopf bifurcation point u of model (1) must lie in the interval $(0,u_{1})\cup (u_{1},u_{2})$, and satisfy the condition (5).

When $\chi >\max_{i\in \mathbb{N}^{+}} \chi _{i}$, then $\text{D}>0$. Combined with Lemma. 2.2, we have the two cases,

(i) if $a(u_{1})\leq s$, the positive equilibrium $(\bar{u}, \bar{v})$ is locally asymptotically stable;

(ii) if $a(u_{1})>s$, there exists $0< u_{0,-}< u_{j,-}< u_{1} < u_{j,+}< u_{0,+}< u_{2}$, such that $a(u_{0,\pm})=s$, $T_{0}(u_{0,\pm})=0$, $T_{n}(u_{0,\pm})\neq 0$ for $n\geq 1$; and $a(u_{j,\pm})=s+(d_{1}+d_{2})\lambda _{j}$, $T_{j}(u_{j,\pm})=0$, $T_{n}(u_{j,\pm})\neq 0$ for any $j\neq n$.

Summarizing the above discussion, we have the following theorem.

Theorem 2.3

Suppose $pq>ck^{2}m$ and $\chi >\max_{i\in \mathbb{N}^{+}} \chi _{i}$ hold. The positive equilibrium is locally asymptotically stable for $a(u_{1})\leq s$; for $a(u_{1})>s$, the system (1) undergoes Hopf bifurcation at $\bar{u}=u_{0,\pm}$ and $\bar{u}=u_{j,\pm}$, and the bifurcating periodic is spatially homogeneous when $\bar{u}=u_{0,\pm}$, the bifurcating periodic is spatially nonhomogeneous when $\bar{u}=u_{j,\pm}$.

Remark 2.4

Compared with the system without the prey-taxis, we find that when the Hopf bifurcation occurs in the system, the number and value of the bifurcation points in the case of spatial homogeneous and spatial heterogeneity are no different. When $\chi >\chi _{0}=\max_{i\in \mathbb{N}^{+}} \chi _{i}$, the value of χ does not affect the number and value of the bifurcation points of the system (1).

3 Steady states

3.1 The existence of nonconstant steady-state solutions

We know that the stability of $(\bar{u},\bar{v})$ is changed when χ passed $\max_{i\in \mathbb{N}^{+}}\chi _{i}$. Therefore, we want to further analyze the existence of nonconstant solutions of the following equation:

$$ \textstyle\begin{cases} d_{1}\Delta u+ \frac{r_{0}u}{1+av}-du-cu^{2}- \frac{puv}{u+kv}=0,& x\in \Omega , \\ d_{2}\Delta v-\chi \nabla (v\nabla u)+sv (1- \frac{qv}{u+m} )=0,& x\in \Omega , \\ \frac{\partial u}{\partial \nu}= \frac{\partial v}{\partial \nu}=0,& x\in \partial \Omega . \end{cases} $$

(6)

Treat χ as the bifurcation parameter to investigate the effect of prey-taxis, and introduce

$$ \mathcal{X}=\bigl\{ w\in W^{2,p}(\Omega ) | \partial _{\nu}w=0,x \in \partial \Omega \bigr\} , \quad p>N. $$

Rewrite $A(\bar{u},\bar{v})=a_{11}$, $B(\bar{u},\bar{v})=a_{12}$, and (6) as

$$ \mathcal{F}(u,v,\chi )=0,\quad (u,v,\chi )\in \mathcal{X} \times \mathcal{X} \times \mathbb{R}, $$

where

$$\mathcal{F}(u,v,\chi )=\left(\begin{array}{c}{d}_1\mathrm{\Delta }u+\frac{r_{0}u}{1+av}-du-c{u}^2-\frac{puv}{u+kv}\\ d_2\mathrm{\Delta }v-\chi \mathrm{\nabla }(v\mathrm{\nabla }u)+sv(1-\frac{qv}{u+m})\end{array}\right).$$

(7)

Obviously, $\mathcal{F}$ is a continuously differentiable mapping from $\mathcal{X} \times \mathcal{X}\times \mathbb{R}$ to $L^{P}(\Omega )\times L^{P}(\Omega )$. $\mathcal{F}(\bar{u},\bar{v},\chi )=0$ for each $\chi \in \mathbb{R}$. For any fixed $(\hat{u},\hat{v})\in \mathcal{X} \times \mathcal{X}$, the Fréchet derivative of $\mathcal{F}$ is given by

$$D_{(u,v)}\mathcal{F}(\hat{u},\hat{v},\chi )(u,v)=\left(\begin{array}{c}{d}_1\mathrm{\Delta }u+A(\hat{u},\hat{v})u+B(\hat{u},\hat{v})v\\ d_2\mathrm{\Delta }v-\chi \mathrm{\nabla }(\hat{v}\mathrm{\nabla }u+\mathrm{\nabla }\hat{u}v)+\frac{qs{\hat{v}}^2}{{(\hat{u}+m)}^2}u+(s-\frac{2qs\hat{v}}{\hat{u}+m})v\end{array}\right),$$

(7)

in particular, for $(\hat{u},\hat{v})=(\bar{u},\bar{v})$,

$$D_{(u,v)}\mathcal{F}(\overline{u},\overline{v},\chi )(u,v)=\left(\begin{array}{c}{d}_1\mathrm{\Delta }u+A(\overline{u},\overline{v})u+B(\overline{u},\overline{v})v\\ d_2\mathrm{\Delta }v-\chi \overline{v}\mathrm{\Delta }u+\frac{s}{q}u-sv\end{array}\right).$$

In order to apply the bifurcation theory from Crandall and Rabinowitz[33], we need to verify the following four conditions of the operator $\mathcal{F}$:

1. $\mathcal{F}(\bar{u},\bar{v},\chi )=0$ for each $\chi \in \mathbb{R}$.

2. The partial derivative $\frac{\mathrm{d}}{\mathrm{d} \chi} D_{(u,v)} \mathcal{F}(\bar{u},\bar{v},\chi )(u,v)$ exists and is continuous in χ.

3. $D_{(u,v)}\mathcal{F}(\bar{u},\bar{v},\chi )$ is a Fredholm operator with index 0, and $\dim \mathcal{N} (D_{(u,v)}\mathcal{F} (\bar{u},\bar{v}, \chi ) )=1$.

4. The transversality condition $\frac{\mathrm{d}}{\mathrm{d} \chi} D_{(u, v)} \mathcal{F}(\bar{u}, \bar{v}, \chi )(\bar{u}_{i}, \bar{v}_{i})|_{ \chi =\chi _{i}} \notin \mathcal{R} (D_{(u, v)} \mathcal{F} (\bar{u}, \bar{v}, \chi _{i} ) )$.

The first two conditions have already been proved. Next, we examine that $D_{(u,v)}\mathcal{F}(\bar{u},\bar{v},\chi )$ is a Fredholm operator with index 0, and $\dim \mathcal{N} (D_{(u,v)}\mathcal{F} (\bar{u},\bar{v}, \chi ) )=1$. For the condition

$$ \mathcal{N}\bigl(D_{(u,v)}\mathcal{F}(\bar{u},\bar{v},\chi )\bigr)\neq \{0\}, $$

(8)

we suppose that there exists $(u,v)$ that is the solution of the following system

$$ \textstyle\begin{cases} d_{1}\Delta u+A(\bar{u},\bar{v})u+B(\bar{u},\bar{v})=0, \\ d_{2}\Delta v-\chi \bar{v}\Delta u+ \frac{s}{q}u-sv=0, \\ \frac{\partial u}{\partial \nu}= \frac{\partial v}{\partial \nu}=0. \end{cases} $$

(9)

Expand $(u,v)$ into their eigenexpansions as

$$ u(x)=\sum_{i=0}^{\infty}T_{i}\Phi _{i}, \qquad v(x)=\sum_{i=0}^{\infty}S_{i} \Phi _{i}, $$

(9)

where $\text{T}_{i}$ and $\text{S}_{i}$ are constants, $\Phi _{i}$ is the eigenfunction corresponding to $\lambda _{i}$, $\lambda _{i}$ is the ith simple eigenvalue of (1) and $\Phi _{i}$ is normalized with $\int _{\Omega}\Phi _{i}^{2}\,\mathrm{d}x=1$, $i\in \mathbb{N}^{+}$. Substituting (9) into (7), we obtain

$$\left(\begin{array}{cc}-d_1{\lambda }_i+A(\overline{u},\overline{v})& B(\overline{u},\overline{v})\\ \chi \overline{v}{\lambda }_i+\frac{s}{q}& -d_2{\lambda }_i-s\end{array}\right)\left(\begin{array}{c}{\text{T}}_i\\ {\text{S}}_i\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right),i\in \mathbb{N}.$$

(10)

The case of $T_{0}$ and $S_{0}=0$ is ruled out since $\lambda _{0}=0$ for $i=0$. For $i\in \mathbb{N}^{+}$, we obtain condition (8) holds at $\chi =\chi _{i}$ through direct calculation

$$ \chi _{i}= \frac{q (d_{1}\lambda _{i}-A(\bar{u},\bar{v}) )(d_{2}\lambda _{i}+s) -sB(\bar{u},\bar{v})}{q\bar{v}B(\bar{u},\bar{v})\lambda _{i}}, $$

(10)

which is the same as the (4). Moreover, we can see that $\dim \mathcal{N}(D_{(u,v)}\mathcal{F}(\bar{u},\bar{v},\chi ))=1$ and

$$ \mathcal{N}\bigl(D_{(u,v)}\mathcal{F}(\bar{u},\bar{v},\chi )\bigr)= \operatorname{span}\bigl\{ ( \bar{u}_{i},\bar{v}_{i})\bigr\} , $$

where

$$ (\bar{u}_{i},\bar{v}_{i})=(1,Q_{i})\Phi _{i}, \qquad Q_{i}= \frac{d_{1}\lambda _{i}-A(\bar{u},\bar{v})}{B(\bar{u},\bar{v})}, \quad i\in \mathbb{N}^{+}. $$

(11)

Also, $D_{(u,v)}\mathcal{F}(\hat{u},\hat{v},\chi )$ is a Fredholm operator with index 0 according to Corollary 2.11 in [34].

Finally, we prove the transversality condition

$$ \frac{\mathrm{d}}{\mathrm{d} \chi} D_{(u, v)} \mathcal{F}( \bar{u}, \bar{v}, \chi ) (\bar{u}_{i}, \bar{v}_{i} ) |_{\chi =\chi _{i}} \notin \mathcal{R} \bigl(D_{(u, v)} \mathcal{F} (\bar{u}, \bar{v}, \chi _{i} ) \bigr), $$

(12)

where $\mathcal{R}$ denotes the range and

$$\frac{\mathrm{d}}{\mathrm{d}\chi }{D}_{(u,v)}\mathcal{F}(\overline{u},\overline{v},\chi )({\overline{u}}_i,{\overline{v}}_i){|}_{\chi ={\chi }_i}=\left(\begin{array}{c}0\\ -\overline{v}\mathrm{\Delta }{\overline{u}}_i\end{array}\right).$$

We argue by contradiction and suppose that there exists a nontrivial pair $(\widetilde{u}, \widetilde{v})$ such that (12) fails, thus the $(\widetilde{u}, \widetilde{v})$ satisfies

$$\left(\begin{array}{c}{d}_1\mathrm{\Delta }\tilde{u}+A(\overline{u},\overline{v})\tilde{u}+B(\overline{u},\overline{v})\tilde{v}\\ d_2\mathrm{\Delta }\tilde{v}-\chi \overline{v}\mathrm{\Delta }\tilde{u}+\frac{s}{q}\tilde{u}-s\tilde{v}\end{array}\right)=\left(\begin{array}{c}0\\ -\overline{v}\mathrm{\Delta }\widetilde{u_i}\end{array}\right),$$

(13)

multiplying both sides of (13) by $\Phi _{i}$ and integrating it over Ω, since $\int _{\Omega}\Phi _{i}^{2}\,\mathrm{d}x=1$,

$$\left(\begin{array}{cc}-d_1{\lambda }_i+A(\overline{u},\overline{v})& B(\overline{u},\overline{v})\\ \chi \overline{v}{\lambda }_i+\frac{s}{q}& -d_2{\lambda }_i-s\end{array}\right)\left(\begin{array}{c}{\int }_{\mathrm{\Omega }}\tilde{u}{\mathrm{\Phi }}_i^2\mathrm{d}x\\ {\int }_{\mathrm{\Omega }}\tilde{v}{\mathrm{\Phi }}_i^2\mathrm{d}x\end{array}\right)=\left(\begin{array}{c}0\\ \overline{v}{\lambda }_i\end{array}\right).$$

This is a contradiction in light of (10) and the transversality condition is verified. Moreover, to ensure $\chi _{i}\neq \chi _{j}$ we require $d_{1}d_{2}q\lambda _{i}\lambda _{j}\neq -sqA(\bar{u},\bar{v})+sB( \bar{u},\bar{v})$, for $i\neq j$, $i,j\in \mathbb{N}^{+}$.

All the above four conditions are satisfied then, according to Shi [34], we have the following result.

Theorem 3.1

Assume that

$$ d_{1}d_{2}q\lambda _{i}\lambda _{j} \neq -s\bigl(qA(\bar{u},\bar{v})+B( \bar{u},\bar{v})\bigr), \quad i\neq j, i,j\in \mathbb{N}^{+}, $$

then there exists a constant $\delta >0$, such that (6) admits nonconstant positive solutions around $(\bar{u}, \bar{v}, \chi _{i} )$, which consist of the bifurcation branch as $\Gamma _{i}(h)=\{ (u_{i}(h, x), v_{i}(h, x), \chi _{i}(h) ) \mid h \in (-\delta , \delta )\}$ with

$$ \textstyle\begin{cases} (u_{i}(h, x), v_{i}(h, x) )=(\bar{u}, \bar{v})+h (1, Q_{i} )\Phi _{i}+h (\xi _{i}(h, x), \zeta _{i}(h, x) ), \\ \chi _{i}(h)=\chi _{i}+h \mathcal{K}_{1}+h^{2} \mathcal{K}_{2}+O(h^{2}), \end{cases} $$

(14)

where $\mathcal{K}_{1}$, $\mathcal{K}_{2}$ are constants, $(\xi _{i}(h, x), \zeta _{i}(h, x) )$ in the closed complement $\mathcal{Z}$ of $\mathcal{N} (D_{(u, v)} \mathcal{F} (\bar{u}, \bar{v}, \chi _{i} ) )$ in $\mathcal{X} \times \mathcal{X}$ with $(\xi _{i}(0, x), \zeta _{i}(0, x) )=(0,0)$,

$$ \mathcal{Z}= \biggl\{ (u, v) \in \mathcal{X} \times \mathcal{X} \bigm| \int _{0}^{L} u \bar{u}_{i}+v \bar{v}_{i} \,d x=0 \biggr\} , $$

(15)

and $(\bar{u}_{i}, \bar{v}_{i} )$ is given in (11).

3.2 The stability of bifurcation solutions

Since $\mathcal{F}$ is $C^{4}$-smooth and $(u_{i}, v_{i}, \chi _{i} )$ are $C^{3}$-smooth functions of h, in order to further study the stability of the nonconstant solutions, we expand (14) as follows:

$$ \textstyle\begin{cases} u_{i}(h, x)=\bar{u}+h \Phi _{i}+h^{2} \varphi _{1}(x)+h^{3} \varphi _{2}(x)+o (h^{3} ), \\ v_{i}(h, x)=\bar{v}+h Q_{i} \Phi _{i}+h^{2} \psi _{1}(x)+h^{3} \psi _{2}(x)+o (h^{3} ), \\ \chi _{i}(h)=\chi _{i}+h \mathcal{K}_{1}+h^{2} \mathcal{K}_{2}+O (h^{2} ), \end{cases} $$

(16)

where $Q_{i}$ is given by (11) and $(\varphi (x), \psi (x) ) \in \mathcal{Z}$. It is easy to obtain through direct calculation that

$$ \textstyle\begin{cases} d_{1}\Delta u_{i}(h, x)=hd_{1}\Delta \Phi _{i} +h^{2} d_{1}\Delta \varphi _{1}(x)+h^{3} d_{1} \Delta \varphi _{2}(x)+o (h^{3} ), \\ d_{2}\Delta v_{i}(h, x)=hd_{2} Q_{k}\Delta \Phi _{i}+h^{2} d_{2} \Delta \psi _{1}(x)+h^{3} d_{2} \Delta \psi _{2}(x)+o (h^{3} ). \end{cases} $$

(17)

Moreover, from the Taylor’s expansion, we have

$$ \begin{aligned} &g\bigl(u_{i}(h, x),v_{i}(h, x) \bigr) \\ &\quad =h \biggl( \frac{s}{q}\Phi _{i}-sQ_{i}\Phi _{i} \biggr) \\ &\qquad {}+h^{2} \biggl( \frac{s}{q}\varphi _{1}(x) -s\psi _{1}(x)- \frac{s}{q(\bar{u}+m)}\Phi _{i}^{2} + \frac{2s}{\bar{u}+m}Q_{k}\Phi _{i}^{2} -sQ_{i}^{2}\Phi _{i}^{2} \biggr) \\ &\qquad {}+h^{3} \biggl( \frac{s}{q}\varphi _{2}(x)- s\psi _{2}(x)- \frac{2s}{q(\bar{u}+m)}\varphi _{1}(x)\Phi _{i} \\ &\qquad {}+ \frac{2s}{\bar{u}+m}\psi _{1}(x)\Phi _{i} + \frac{2s}{\bar{u}+m}Q_{i}\varphi _{1}(x)\Phi _{i} \\ &\qquad {}-2sQ_{i}\psi _{1}(x)\Phi _{i} - \frac{s}{q(\bar{u}+m)^{2}}\Phi _{i}^{3} - \frac{2s}{(\bar{u}+m)^{2}}Q_{i} \Phi _{i}^{3} + \frac{qs}{(\bar{u}+m)^{2}}Q_{i}^{2} \Phi _{i}^{3} \biggr) \\ &\qquad {}+o\bigl(h^{3}\bigr), \end{aligned} $$

(18)

and

$$\begin{aligned}& \nabla \bigl(v_{i}(h,x)\nabla u_{i}(h,x) \bigr) \\& \quad =h \bar{v}\Delta \Phi _{i} +h^{2} \bigl(\bar{v}\Delta \varphi _{1}(x) +Q_{i} \vert \nabla \Phi _{i} \vert ^{2} +Q_{i} \Phi _{i}\Delta \Phi _{i} \bigr) +h^{3} \bigl(\bar{v} \Delta \varphi _{2}(x) +Q_{i}\Delta \varphi _{1}(x)\Phi _{i} \\& \qquad {} +Q_{i}\nabla \varphi _{1}(x)\nabla \Phi _{i} +\nabla \psi _{1}(x) \nabla \Phi _{i} +\psi _{1}(x) \Delta \Phi _{i} \bigr)+o\bigl(h^{3} \bigr). \end{aligned}$$

(19)

From Wang [35], the stability of $(u_{i}(h,x),v_{i}(h,x),)$ is determined by the sign of $\mathcal{K}_{1}$ ($\mathcal{K}_{1}\neq 0$). For $\mathcal{K}_{1}=0$, we need to further discuss the sign of $\mathcal{K}_{2}$. Next, we discuss $\mathcal{K}_{1}$ first.

Theorem 3.2

Suppose all conditions in Theorem 3.1hold. Then, for each $i \in \mathbb{N}^{+}$, the sign of $\mathcal{K}_{1}$ is given by (21).

Proof

Substituting (16)–(19) into the second equation in (6) and collecting their $h^{2}$-terms, we have

$$ \begin{aligned} &d_{2} \Delta \psi _{1}(x) - \bigl(\bar{v}\Delta \varphi _{1}(x) +Q_{i}{ \vert \nabla \Phi _{i} \vert }^{2} +Q_{i}\Phi _{i} \triangle \Phi _{i} \bigr) \chi _{i} -\mathcal{K}_{1} \bar{v}\triangle \Phi _{i} \\ &\quad {} + \frac{s}{q}\varphi _{1}(x)-s\psi _{1}(x)- \frac{s}{q(\bar{u}+m)}\Phi _{i}^{2} + \frac{2s}{\bar{u}+m}Q_{i} \Phi _{i}^{2} -sQ_{i}^{2}\Phi _{i}^{2}=0, \end{aligned} $$

(20)

multiplying (20) by $\Phi _{i}$ and integrating it over Ω leads to

$$ \begin{aligned} \mathcal{K}_{1}\bar{v}\lambda _{i}=& (d_{2}\lambda _{i}+s ) \int _{\Omega} \psi _{1}(x)\Phi _{i}\, \mathrm{d}x - \biggl( \frac{s}{q}+\bar{v}\chi _{k}\lambda _{i} \biggr) \int _{ \Omega} \varphi _{1}(x)\Phi _{i}\, \mathrm{d}x \\ &{}+\chi _{i}Q_{i} \int _{ \Omega} \vert \nabla \Phi _{i} \vert ^{2}\Phi _{i}\,\mathrm{d}x \\ &{} + \biggl( \frac{s}{q(\bar{u}+m)}- \frac{2s}{\bar{u}+m}Q_{i}+sQ_{i}^{2} -\chi _{i}\lambda _{i}Q_{i} \biggr) \int _{\Omega} \Phi _{i}^{3} \,\mathrm{d}x \\ ={}& (d_{2}\lambda _{i}+s ) \int _{\Omega} \psi _{1}(x)\Phi _{i} \, \mathrm{d}x - \biggl( \frac{s}{q}+\bar{v}\chi _{k} \lambda _{i} \biggr) \int _{\Omega} \varphi _{1}(x)\Phi _{i} \, \mathrm{d}x \\ &{} + \biggl( \frac{s}{q(\bar{u}+m)}- \frac{2s}{\bar{u}+m}Q_{i} - \frac{1}{2}\chi _{i}\lambda _{i}Q_{i}+sQ_{i}^{2} \biggr) \int _{\Omega} \Phi _{i}^{3} \,\mathrm{d}x. \end{aligned} $$

(21)

Substituting (16)–(19) into the first equation in (6) and collecting the $h^{2}$-terms,

$$ d_{1}\Delta \varphi _{1}(x)+A(\bar{u},\bar{v})\varphi _{1}(x)+B( \bar{u},\bar{v})\psi _{1}(x) + \frac{1}{2} \bigl(\bar{f}_{uu}+2 \bar{f}_{uv}Q_{i}+ \bar{f}_{vv}Q_{i}^{2} \bigr)\Phi _{i}^{2}=0, $$

(22)

where

$$ \textstyle\begin{cases} \bar{f}_{uu}=-2c+ \frac{2pqk{\bar{u}+m}^{2}}{(q\bar{u}+k\bar{u}+km)^{3}}, \qquad \bar{f}_{uv}=- \frac{ar_{0}q}{(q+a\bar{u}+am)^{2}}- \frac{2pq^{2}k\bar{u}(\bar{u}+m)}{(q\bar{u}+k\bar{u}+km)^{3}}, \vspace*{3pt}\\ \bar{f}_{vv}= \frac{2a^{2}q^{3}r_{0}\bar{u}}{(q+a\bar{u}+am)^{3}}+ \frac{2pq^{2}k{\bar{u}}^{2}}{(q\bar{u}+k\bar{u}+km)^{3}}. \end{cases} $$

Multiplying (22) by $\Phi _{i}$ and integrating it over Ω,

$$ \bigl(-d_{1}\lambda _{i}+A(\bar{u},\bar{v}) \bigr) \int _{\Omega} \varphi _{1}(x)\Phi _{i} \, \mathrm{d}x + B(\bar{u},\bar{v}) \int _{ \Omega} \psi _{1}(x)\Phi _{i} \, \mathrm{d}x=\tilde{C_{0}}, $$

(23)

where

$$ \tilde{C_{0}}=- \frac{1}{2} \bigl(\bar{f}_{uu}+2 \bar{f}_{uv}Q_{i}+ \bar{f}_{vv}Q_{i}^{2} \bigr) \int _{\Omega} \Phi _{i}^{3} \,\mathrm{d}x. $$

For $(\varphi _{1}(x),\psi _{1}(x))\in \mathcal{Z}$ defined as (15), we obtain

$$ B(\bar{u},\bar{v}) \int _{\Omega} \varphi _{1}(x)\Phi _{i} \, \mathrm{d}x + \bigl(d_{1}\lambda _{i}-A(\bar{u},\bar{v}) \bigr) \int _{\Omega} \psi _{1}(x) \Phi _{i} \, \mathrm{d}x=0. $$

(24)

Combining (23) and (24),

$$\left(\begin{array}{cc}-d_1{\lambda }_i+A(\overline{u},\overline{v})& B(\overline{u},\overline{v})\\ B(\overline{u},\overline{v})& d_1{\lambda }_i-A(\overline{u},\overline{v})\end{array}\right)\left(\begin{array}{c}{\int }_{\mathrm{\Omega }}{\phi }_1(x){\mathrm{\Phi }}_i\mathrm{d}x\\ {\int }_{\mathrm{\Omega }}{\psi }_1(x){\mathrm{\Phi }}_i\mathrm{d}x\end{array}\right)=\left(\begin{array}{c}\widetilde{C_0}\\ 0\end{array}\right),$$

(25)

then we have two cases:

Case 1. for $\int _{\Omega} \Phi _{i}^{3} \,\mathrm{d}x\neq 0$, we have

$$ \begin{aligned} \int _{\Omega} \varphi _{1}(x)\Phi _{i} \, \mathrm{d}x= \frac{-B(\bar{u},\bar{v})\tilde{C_{0}}}{2 ( (-d_{1}\lambda _{i}+A(\bar{u},\bar{v}) )^{2}+B^{2}(\bar{u},\bar{v}) )}, \\ \int _{\Omega} \psi _{1}(x)\Phi _{i} \, \mathrm{d}x= \frac{ (d_{1}\lambda _{i}-A(\bar{u},\bar{v}) ) \tilde{C_{0}}}{2 ( (-d_{1}\lambda _{i}+A(\bar{u},\bar{v}) )^{2}+B^{2}(\bar{u},\bar{v}) )} . \end{aligned} $$

Then,

$$ \begin{aligned} \mathcal{K}_{1}\bar{v}\lambda _{i} ={}& \biggl\{ \frac{ (d_{2}\lambda _{i}+s ) (d_{1}\lambda _{i}-A(\bar{u},\bar{v}) )\tilde{C_{0}}}{ 2 ( (-d_{1}\lambda _{i}+A(\bar{u},\bar{v}) )^{2}+B^{2}(\bar{u},\bar{v}) )} + \frac{s}{q(\bar{u}+m)} - \frac{2s}{\bar{u}+m}Q_{i}+sQ_{i}^{2} \\ &{} + \frac{ ( \frac{s}{q}+\bar{v}\chi _{k}\lambda _{i} )B(\bar{u},\bar{v})\tilde{C_{0}}}{ 2 ( (-d_{1}\lambda _{i}+A(\bar{u},\bar{v}) )^{2}+B^{2}(\bar{u},\bar{v}) )} - \frac{1}{2}\chi _{i}\lambda _{i}Q_{i} \biggr\} \int _{ \Omega} \Phi _{i}^{3} \,\mathrm{d}x, \end{aligned} $$

the sign of $\mathcal{K}_{1}$ is decided by $\int _{\Omega} \Phi _{i}^{3} \,\mathrm{d}x$.

Case 2. for $\int _{\Omega} \Phi _{i}^{3} \,\mathrm{d}x=0$, we have

$$ \int _{\Omega} \varphi _{1}(x)\Phi _{i} \, \mathrm{d}x= \int _{\Omega} \psi _{1}(x)\Phi _{i} \, \mathrm{d}x=0. $$

(25)

 □

Remark 3.3

For the one-dimensional $\Omega =(0,L)$, $\int _{\Omega}\Phi _{i}^{3} \,\mathrm{d}x= \int _{0}^{L}\cos ^{3} \frac{i \pi x}{L} \,\mathrm{d}x=0$, then $\mathcal{K}_{1}=0$. As χ increases, $(\bar{u}, \bar{v})$ loses its stability, a family of nonconstant solutions $(\bar{u}, \bar{v}, \chi _{i} )$ emerge, thus system (6) undergoes pitchfork bifurcation.

Next, we continue to analyze the stability of bifurcating solutions.

Theorem 3.4

Suppose (25) is satisfied, we have that the sign of $\mathcal{K}_{2}$ is decided by (27).

Proof

Substituting (16)–(19) into the second equation in (6) and collecting their $h^{3}$-terms,

$$\begin{aligned}& d_{2}\Delta \psi _{2}(x) + \frac{s}{q}\varphi _{2}(x)-s \varphi _{2}(x) - \frac{2s}{q(\bar{u}+m)}\varphi _{1}(x) \Phi _{i} \\& \qquad {} + \frac{2s}{\bar{u}+m}Q_{i}\varphi _{1}(x) \Phi _{i} + \frac{2s}{\bar{u}+m}\psi _{1}(x)\Phi _{i} \\& \qquad {} -2sQ_{i}\psi _{1}(x)\Phi _{i} - \frac{s}{q(\bar{u}+m)^{2}}\Phi _{i}^{3} - \frac{2s}{(\bar{u}+m)^{2}}Q_{i} \Phi _{i}^{3} + \frac{qs}{(\bar{u}+m)^{2}}Q_{i}^{2} \Phi _{i}^{3} \\& \quad =\chi _{i} \bigl(\bar{v}\Delta \varphi _{2}(x) +Q_{i}\Delta \varphi _{1}(x) \Phi _{i} +Q_{i}\nabla \varphi _{1}(x) \vert \nabla \Phi _{i} \vert +\nabla \psi _{1}(x) \vert \nabla \Phi _{i} \vert +\psi _{1}(x) \vert \Delta \Phi _{i} \vert \bigr) \\& \qquad {}+ \mathcal{K}_{2}\bar{v} \vert \Delta \Phi _{i} \vert , \end{aligned}$$

(26)

multiplying (26) by $\Phi _{i}$ and integrating it over Ω,

$$ \begin{aligned} \mathcal{K}_{2}\bar{v}\lambda _{i}={}& d_{2}\lambda _{i} \int _{\Omega} \psi _{2}(x)\Phi _{i} \, \mathrm{d} x \\ &{}+ \biggl( \frac{sq-s}{q}-\chi _{i}\bar{v}\lambda _{i} \biggr) \int _{\Omega} \varphi _{2}(x)\Phi _{i} \, \mathrm{d} x -\chi _{i}\lambda _{i}Q_{i} \int _{\Omega} \varphi _{1}(x)\Phi _{i} \, \mathrm{d} x \\ & {}+ \biggl( \frac{2s}{q(\bar{u}+m)}- \frac{2s}{\bar{u}+m}Q_{i}+\chi _{i}\lambda _{i}Q_{i} \biggr) \int _{ \Omega} \varphi _{1}(x)\Phi _{i}^{2} \,\mathrm{d} x \\ &{}+ \biggl(2sQ_{i}- \frac{2s}{(\bar{u}+m)} \biggr) \int _{\Omega} \psi _{1}(x) \Phi _{i}^{2} \,\mathrm{d} x \\ & {}-\chi _{i}Q_{i} \int _{\Omega} \varphi _{1}(x) \vert \nabla \Phi _{i} \vert ^{2} \,\mathrm{d} x -\chi _{i} \int _{\Omega} \psi _{1}(x) \vert \nabla \Phi _{i} \vert ^{2} \,\mathrm{d} x \\ &{} + \biggl( \frac{s}{q(\bar{u}+m)^{2}}+ \frac{2s}{(\bar{u}+m)^{2}}Q_{i} - \frac{qs}{(\bar{u}+m)^{2}}Q_{i}^{2} \biggr) \int _{\Omega} \Phi _{i}^{4} \,\mathrm{d} x. \end{aligned} $$

(27)

From (25), we have $\int _{\Omega} \varphi _{1}(x)\Phi _{i} \,\mathrm{d}x=0$. Then, we need to calculate $\int _{\Omega} \varphi _{1}(x)\Phi _{i}^{2} \,\mathrm{d} x$, $\int _{\Omega} \varphi _{1}(x)\Phi _{i}^{2} \,\mathrm{d} x$, $\int _{\Omega} \varphi _{1}(x)|\nabla \Phi _{i}|^{2} \,\mathrm{d} x$ and $\int _{\Omega} \psi _{1}(x)|\nabla \Phi _{i}|^{2} \,\mathrm{d} x$.

Multiplying (20) by $\Phi _{i}^{2}$ and integrating over Ω,

$$ \begin{aligned} & \biggl(2\bar{v}\chi _{i}\lambda _{i}+ \frac{s}{q} \biggr) \int _{\Omega} \varphi _{1}(x)\Phi _{i}^{2} \,\mathrm{d}x -2 \bar{v}\chi _{i} \int _{\Omega} \varphi _{1}(x) \vert \nabla \Phi _{i} \vert ^{2} \,\mathrm{d}x \\ &\qquad {}+ (-2d_{2}\lambda _{i}-s ) \int _{\Omega} \psi _{1}(x)\Phi _{i}^{2} \,\mathrm{d}x \\ &\qquad {}+2d_{2} \int _{\Omega} \psi _{1}(x) \vert \nabla \Phi _{i} \vert ^{2} \\ &\quad = \biggl(- \frac{2}{3}\chi _{i}Q_{i}\lambda _{i}+ \frac{s}{q(\bar{u}+m)} - \frac{2s}{\bar{u}+m}Q_{i}+sQ_{i}^{2} \biggr) \int _{\Omega} \Phi _{i}^{4} \,\mathrm{d}x, \end{aligned} $$

(28)

multiplying (20) by $|\nabla \Phi _{i}|^{2}$ and integrating over Ω,

$$ \begin{aligned} &-2\bar{v}\chi _{i}\lambda _{i}^{2} \int _{\Omega} \varphi _{1}(x) \Phi _{i}^{2} \,\mathrm{d}x + \biggl(2\bar{v}\chi _{i}\lambda _{i}+ \frac{s}{q} \biggr) \int _{\Omega} \varphi _{1}(x) \vert \nabla \Phi _{i} \vert ^{2} \,\mathrm{d}x \\ &\qquad {}+2d_{2}\lambda _{i}^{2} \int _{ \Omega} \psi _{1}(x)\Phi _{i}^{2} \,\mathrm{d}x \\ &\qquad {}+ (-2d_{2}\lambda _{i}-s ) \int _{\Omega} \psi _{1}(x) \vert \nabla \Phi _{i} \vert ^{2} \\ &\quad = \biggl( \frac{2}{3}\chi _{i}Q_{i} \lambda _{i}^{2}+ \biggl( \frac{s}{q(\bar{u}+m)} - \frac{2s}{\bar{u}+m}Q_{i}+sQ_{i}^{2} \biggr) \frac{\lambda _{i}}{3} \biggr) \int _{\Omega} \Phi _{i}^{4} \,\mathrm{d}x, \end{aligned} $$

(29)

multiplying (22) by $\Phi _{i}^{2}$ and integrating over Ω,

$$\begin{aligned}& \bigl(-2d_{1}\lambda _{i}+A( \bar{u},\bar{v}) \bigr) \int _{\Omega} \varphi _{1}(x)\Phi _{i}^{2} \,\mathrm{d}x +2d_{1} \int _{\Omega} \varphi _{1}(x) \vert \nabla \Phi _{i} \vert ^{2} \,\mathrm{d}x + B(\bar{u}, \bar{v}) \int _{\Omega} \psi _{1}(x)\Phi _{i}^{2} \,\mathrm{d}x \\& \quad =- \frac{1}{2} \bigl(\bar{f}_{uu}+2\bar{f}_{uv}Q_{i}+ \bar{f}_{vv}Q_{i}^{2} \bigr) \int _{\Omega} \Phi _{i}^{4} \,\mathrm{d}x, \end{aligned}$$

(30)

multiplying (22) by $|\nabla \Phi _{i}|^{2}$ and integrating over Ω,

$$ \begin{aligned} &2d_{1}\lambda _{i}^{2} \int _{\Omega} \varphi _{1}(x)\Phi _{i}^{2} \,\mathrm{d}x + \bigl(-2d_{1}\lambda _{i}+A(\bar{u}, \bar{v}) \bigr) \int _{ \Omega} \varphi _{1}(x) \vert \nabla \Phi _{i} \vert ^{2} \,\mathrm{d}x + B( \bar{u},\bar{v}) \\ &\qquad {}\times \int _{\Omega} \psi _{1}(x) \vert \nabla \Phi _{i} \vert ^{2} \,\mathrm{d}x \\ &\quad =- \frac{1}{6}\lambda _{i} \bigl(\bar{f}_{uu}+2 \bar{f}_{uv}Q_{i}+ \bar{f}_{vv}Q_{i}^{2} \bigr) \int _{\Omega} \Phi _{i}^{4} \,\mathrm{d}x. \end{aligned} $$

(31)

Combining (28)–(31), we have,

$$\begin{array}{rl}& \left(\begin{array}{cccc}2\overline{v}{\chi }_i{\lambda }_i+\frac{s}{q}& -2\overline{v}{\chi }_i& -2d_2{\lambda }_i-s& 2d_2\\ -2\overline{v}{\chi }_i{\lambda }_i^2& 2\overline{v}{\chi }_i{\lambda }_i+\frac{s}{q}& 2d_2{\lambda }_i^2& -2d_2{\lambda }_i-s\\ -2d_1{\lambda }_i+A(\overline{u},\overline{v})& 2d_1& B(\overline{u},\overline{v})& 0\\ 2d_1{\lambda }_i^2& -2d_1{\lambda }_i+A(\overline{u},\overline{v})& 0& B(\overline{u},\overline{v})\end{array}\right)\\ & \times \left(\begin{array}{c}{\int }_{\mathrm{\Omega }}{\phi }_1(x){\mathrm{\Phi }}_i^2\mathrm{d}x\\ {\int }_{\mathrm{\Omega }}{\phi }_1(x)|\mathrm{\nabla }{\mathrm{\Phi }}_i{|}^2\mathrm{d}x\\ {\int }_{\mathrm{\Omega }}{\psi }_1(x){\mathrm{\Phi }}_i^2\mathrm{d}x\\ {\int }_{\mathrm{\Omega }}{\psi }_1(x)|\mathrm{\nabla }{\mathrm{\Phi }}_i{|}^2\mathrm{d}x\end{array}\right)\\ & =\left(\begin{array}{c}-\frac{2}{3}{\chi }_i{Q}_i{\lambda }_i+\frac{s}{q(\overline{u}+m)}-\frac{2s}{\overline{u}+m}{Q}_i+s{Q}_i^2\\ \frac{2}{3}{\chi }_i{Q}_i{\lambda }_i^2+\frac{{\lambda }_i}{3}(\frac{s}{q(\overline{u}+m)}-\frac{2s}{\overline{u}+m}{Q}_i+s{Q}_i^2)\\ -\frac{1}{2}({\overline{f}}_{uu}+2{\overline{f}}_{uv}{Q}_i+{\overline{f}}_{vv}{Q}_i^2)\\ -\frac{1}{6}{\lambda }_i({\overline{f}}_{uu}+2{\overline{f}}_{uv}{Q}_i+{\overline{f}}_{vv}{Q}_i^2)\end{array}\right){\int }_{\mathrm{\Omega }}{\mathrm{\Phi }}_i^4\mathrm{d}x.\end{array}$$

(32)

Denote

$$\begin{aligned}& \tilde{D_{1}} =- \frac{2}{3}\chi _{i}Q_{i}\lambda _{i}+ \frac{s}{q(\bar{u}+m)} - \frac{2s}{\bar{u}+m}Q_{i}+sQ_{i}^{2}, \\& \tilde{D_{2}} = \frac{2}{3}\chi _{i}Q_{i} \lambda _{i}^{2}+ \frac{\lambda _{i}}{3} \biggl( \frac{s}{q(\bar{u}+m)}- \frac{2s}{\bar{u}+m}Q_{i}+sQ_{i}^{2} \biggr), \\& \tilde{D_{3}} =- \frac{1}{2} \bigl(\bar{f}_{uu}+2 \bar{f}_{uv}Q_{i}+ \bar{f}_{vv}Q_{i}^{2} \bigr), \\& \tilde{D_{4}} =- \frac{1}{6}\lambda _{i} \bigl( \bar{f}_{uu}+2 \bar{f}_{uv}Q_{i}+ \bar{f}_{vv}Q_{i}^{2} \bigr), \end{aligned}$$

and solving (32),

$$ \int _{\Omega} \varphi _{1}(x)\Phi _{i}^{2} \,\mathrm{d}x= \frac{\tilde{E_{1}}}{\tilde{E_{0}}}, \qquad \int _{\Omega} \psi _{1}(x)\Phi _{i}^{2} \,\mathrm{d}x= \frac{\tilde{E_{2}}}{\tilde{E_{0}}}, $$

where

$$\begin{aligned}& \tilde{E_{0}}= \biggl(2\bar{v}\chi _{i}\lambda _{i}+ \frac{s}{q} \biggr)^{2}B^{2}( \bar{u},\bar{v}) + \bigl(2d_{1}\lambda _{i}-A( \bar{u}, \bar{v}) \bigr)^{2} (2d_{2}\lambda _{i}+s )^{2} -16d_{1}^{2}d_{2}^{2} \lambda _{i}^{4} \\& \hphantom{\tilde{E_{0}}=}{}+ (2d_{2}\lambda _{i}+s ) \bigl(-2d_{1}\lambda _{i}+A( \bar{u},\bar{v}) \bigr) \bigl(1+B(\bar{u},\bar{v}) \bigr) \biggl(2\bar{v} \chi _{i}\lambda _{i}+ \frac{s}{q} \biggr), \\& \tilde{E_{1}}=B^{2}(\bar{u},\bar{v})\tilde{D_{1}} \biggl(2\bar{v} \chi _{i}\lambda _{i}+ \frac{s}{q} \biggr) + \tilde{D_{3}} (2d_{2}\lambda _{i}+s )^{2} \bigl(-2d_{1} \lambda _{i}+A(\bar{u}, \bar{v}) \bigr) -8d_{1}d_{2}^{2}\lambda _{i}^{2} \tilde{D_{4}} \\& \hphantom{\tilde{E_{1}}=}{}+B(\bar{u},\bar{v})\tilde{D_{3}} (2d_{2}\lambda _{i}+s ) \biggl(2\bar{v}\chi _{i}\lambda _{i}+ \frac{s}{q} \biggr) +B(\bar{u},\bar{v})\tilde{D_{1}} \bigl(-2d_{2}\lambda _{i}+A(\bar{u}, \bar{v}) \bigr) \biggl(2 \bar{v}\chi _{i}\lambda _{i}+ \frac{s}{q} \biggr), \\& \tilde{E_{2}}=B(\bar{u},\bar{v})\tilde{D_{3}} \biggl(2 \bar{v}\chi _{i} \lambda _{i}+ \frac{s}{q} \biggr)^{2} -\tilde{D_{1}} (2d_{2}\lambda _{i}+s ) \bigl(-2d_{2}\lambda _{i}+A(\bar{u}, \bar{v}) \bigr)^{2} \\& \hphantom{\tilde{E_{2}}=}{}-8d_{1}d_{2}\bar{v}\chi _{i}\lambda _{i}^{2}\tilde{D_{4}} -8d_{1}^{2}d_{2} \lambda _{i}^{2}\tilde{D_{2}} -B(\bar{u},\bar{v}) \tilde{D_{1}} \bigl(-2d_{1} \lambda _{i}+A( \bar{u},\bar{v}) \bigr) \biggl(2\bar{v}\chi _{i} \lambda _{i}+ \frac{s}{q} \biggr) \\& \hphantom{\tilde{E_{2}}=}{}+\tilde{D_{3}} (2d_{2}\lambda _{i}+s ) \bigl(-2d_{1} \lambda _{i}+A(\bar{u},\bar{v}) \bigr) \biggl(2\bar{v}\chi _{i}\lambda _{i}+ \frac{s}{q} \biggr), \end{aligned}$$

and

$$ \int _{\Omega} \varphi _{1}(x) \vert \nabla \Phi _{i} \vert ^{2} \,\mathrm{d}x= \frac{\tilde{E_{3}}}{\tilde{E_{0}}}, \qquad \int _{\Omega} \psi _{1}(x) \vert \nabla \Phi _{i} \vert ^{2} \,\mathrm{d}x= \frac{\tilde{E_{4}}}{\tilde{E_{0}}}, $$

where

$$ \begin{aligned} \tilde{E_{3}}=&B^{2}(\bar{u}, \bar{v})\tilde{D_{2}} \biggl(2\bar{v} \chi _{i}\lambda _{i}+ \frac{s}{q} \biggr) + \tilde{D_{4}} (2d_{2}\lambda _{i}+s )^{2} \bigl(-2d_{1} \lambda _{i}+A(\bar{u},\bar{v}) \bigr) -8d_{1}d_{2}^{2} \lambda _{i}^{4} \tilde{D_{3}} \\ &+B(\bar{u},\bar{v})\tilde{D_{2}} (2d_{2}\lambda _{i}+s ) \bigl(-2d_{2}\lambda _{i}+A(\bar{u}, \bar{v}) \bigr) +B(\bar{u},\bar{v}) \tilde{D_{4}} (2d_{2} \lambda _{i}+s ) \biggl(2\bar{v}\chi _{i} \lambda _{i}+ \frac{s}{q} \biggr), \\ \tilde{E_{4}}=&B(\bar{u},\bar{v})\tilde{D_{4}} \biggl(2 \bar{v}\chi _{i} \lambda _{i}+ \frac{s}{q} \biggr)^{2} -\tilde{D_{2}} (2d_{2}\lambda _{i}+s ) \bigl(-2d_{1}\lambda _{i}+A(\bar{u}, \bar{v}) \bigr) \\ &-8d_{1}d_{2}\bar{v}\chi _{i}\lambda _{i}^{4}\tilde{D_{3}} -8d_{1}^{2}d_{2} \lambda _{i}^{4}\tilde{D_{1}} -B(\bar{u},\bar{v}) \tilde{D_{2}} \bigl(-2d_{1} \lambda _{i}+A( \bar{u},\bar{v}) \bigr) \biggl(2\bar{v}\chi _{i} \lambda _{i}+ \frac{s}{q} \biggr) \\ &+\tilde{D_{4}} (2d_{2}\lambda _{i}+s ) \bigl(-2d_{1} \lambda _{i}+A(\bar{u},\bar{v}) \bigr) \biggl(2\bar{v}\chi _{i}\lambda _{i}+ \frac{s}{q} \biggr), \end{aligned} $$

we suppose that $\tilde{E_{0}}$ is always nonzero.

For $\int _{\Omega} \varphi _{2}(x)\Phi _{i} \,\mathrm{d} x$, $\int _{\Omega} \psi _{2}(x)\Phi _{i} \,\mathrm{d} x$, equating $h^{3}$-terms in the first equation of (6),

$$ \begin{aligned} &d_{1}\Delta \varphi _{2} (x)+A( \bar{u},\bar{v})\varphi _{2}(x)+B( \bar{u},\bar{v})\psi _{2}(x) + (\bar{f}_{uu}+\bar{f}_{uv}Q_{i} )\varphi _{1}(x)\Phi _{i} \\ &\quad {}+\bar{f}_{uv}Q_{i}\psi _{1}(x)\Phi _{i} +\frac{1}{6} \bigl(\bar{f}_{u u u}+3 \bar{f}_{u u v} Q_{i}+3 \bar{f}_{u v v} Q_{i}^{2} +\bar{f}_{v v v} Q_{i}^{3} \bigr)\Phi _{i}^{3}=0, \end{aligned} $$

(33)

where

$$ \textstyle\begin{cases} \bar{f}_{uuu}=- \frac{6pq^{2}k(\bar{u}+m)^{2}}{(q\bar{u}+k\bar{u}+km)^{4}}, \qquad \bar{f}_{uuv}= \frac{4pq^{3}k\bar{u}(\bar{u}+m)-2pq^{2}k(\bar{u}+m)^{2}}{(q\bar{u}+k\bar{u}+km)^{4}}, \vspace*{3pt}\\ \bar{f}_{uvv}= \frac{a^{2}q^{3}r_{0}}{ (q+a\bar{u}+am)^{3}} - \frac{2pq^{3}k\bar{u}(q\bar{u}-2k(\bar{u}a+m))}{(q\bar{u}+k\bar{u}+km)^{4}}, \vspace*{3pt}\\ \bar{f}_{vvv}=- \frac{6a^{3}q^{4}r_{0}\bar{u}}{(q+a\bar{u}+am)^{4}}- \frac{6pq^{4}k^{2}{\bar{u}}^{2}}{(q\bar{u}+k\bar{u}+km)^{4}}. \end{cases} $$

Multiplying (33) by $\Phi _{i}$ and integrating it over Ω

$$ \begin{aligned} \bigl(-d_{1}\lambda _{i}+A( \bar{u},\bar{v}) \bigr) \int _{\Omega} \varphi _{2}(x) \Phi _{i} \, \mathrm{d} x+B(\bar{u},\bar{v}) \int _{ \Omega} \psi _{2}(x)\Phi _{i} \, \mathrm{d} x=\tilde{C_{1}}, \end{aligned} $$

(34)

where

$$ \begin{aligned} \tilde{C_{1}}={}&- (\bar{f}_{uu}+ \bar{f}_{uv}Q_{i} ) \int _{\Omega} \varphi _{1}(x)\Phi _{i}^{2} \,\mathrm{d} x -\bar{f}_{uv}Q_{i} \int _{ \Omega} \psi _{1}(x)\Phi _{i}^{2} \,\mathrm{d} x \\ & {}-\frac{1}{6} \bigl(\bar{f}_{u u u}+3 \bar{f}_{u u v} Q_{i}+3 \bar{f}_{u v v} Q_{i}^{2} + \bar{f}_{v v v} Q_{i}^{3} \bigr) \int _{\Omega} \Phi _{i}^{4} \,\mathrm{d} x. \end{aligned} $$

Since $(\varphi _{2}(x), \psi _{2}(x)) \in \mathcal{Z}$ as defined in (15), we have

$$ B(\bar{u},\bar{v}) \int _{\Omega} \varphi _{2}(x) \Phi _{i} \, \mathrm{d} x + \bigl(-d_{1}\lambda _{i}+A(\bar{u},\bar{v}) \bigr) \int _{ \Omega} \psi _{2}(x)\Phi _{i} \, \mathrm{d} x=0. $$

(35)

Combining (34) and (35)

$$\left(\begin{array}{cc}-d_1{\lambda }_i+A(\overline{u},\overline{v})& B(\overline{u},\overline{v})\\ B(\overline{u},\overline{v})& d_1{\lambda }_i-A(\overline{u},\overline{v})\end{array}\right)\left(\begin{array}{c}{\int }_{\mathrm{\Omega }}{\phi }_2(x){\mathrm{\Phi }}_i\mathrm{d}x\\ {\int }_{\mathrm{\Omega }}{\psi }_2(x){\mathrm{\Phi }}_i\mathrm{d}x\end{array}\right)=\left(\begin{array}{c}\widetilde{C_1}\\ 0\end{array}\right),$$

(36)

solving equation (36), we have

$$ \begin{aligned} & \int _{\Omega} \varphi _{2}(x) \Phi _{i} \, \mathrm{d} x = \frac{ (-d_{1}\lambda _{i}+A(\bar{u},\bar{v}) )\tilde{C_{1}}}{ (-d_{1}\lambda _{i}+A(\bar{u},\bar{v}) )^{2}+B^{2}(\bar{u},\bar{v})}, \\ & \int _{\Omega} \psi _{2}(x)\Phi _{i} \, \mathrm{d} x = \frac{B(\bar{u},\bar{v})\tilde{C_{1}}}{ (-d_{1}\lambda _{i}+A(\bar{u},\bar{v}) )^{2}+B^{2}(\bar{u},\bar{v})}. \end{aligned} $$

The sign of $\mathcal{K}_{2}$ is determined by (27). □

Remark 3.5

By Theorem 3.2 of Wang [35], we know that $\operatorname{sgn}(\lambda )=\operatorname{sgn}(\mathcal{K}_{2})$ for $h\in (-\delta , \delta )$, $h\neq 0$, then $(u_{i_{0}}(h,x), v_{i_{0}}(h,x))$ is stable if $\mathcal{K}_{2}<0$ and it is unstable if $\mathcal{K}_{2}>0$ for $h\in (-\delta , \delta )$, $h\neq 0$, $\mathcal{K}_{1}=0$. Denoting

$$ D_{(u,v)}\mathcal{F}\bigl(\bigl(u_{i}(h,x), v_{i}(h,x)\bigr),\chi _{i}(h)\bigr) (u,v)= \lambda (u,v), \quad u,v\in \mathcal{X}\times \mathcal{X}, $$

(37)

we know that 0 is a simple eigenvalue of $D_{(u, v)}\mathcal{F}(\bar{u}, \bar{v}, \chi _{i})$ with an eigenspace $\operatorname{span}\{(1, Q_{i})\cos \frac{i \pi x}{L}\}$. For each $i \neq i_{0}$, (37) has the eigenvalue with a positive real part when $h=0$, then from the standard eigenvalue perturbation theory in [36], it also holds for small h. This shows that $(u_{i}(h,x), v_{i}(h,x), \chi _{i})$ is unstable for each $i \in \mathbb{N}^{+} \backslash \{i_{0}\}$.

4 Numerical simulations

In this section, we will give some numerical simulations to verify our findings. The parameters are chosen as $d_{1}=0.02$, $d_{2}=1$, $a=2$, $d=2$, $c=3$, $k=1$, $s=1$, $q=1$, $m=1$. Then equation (1) is

$$ \textstyle\begin{cases} \frac{\partial u(x,t)}{\partial t}=0.02\Delta u+ \frac{r_{0}u}{1+2v}-2u-3u^{2}- \frac{puv}{u+v}, \\ \frac{\partial v(x,t)}{\partial t}=\Delta v-\chi \nabla (v\nabla u)+v (1- \frac{v}{u+1} ). \end{cases} $$

In Fig. 1 we show that prey-taxis can stabilize the positive equilibrium of system (1) for $\chi >0$, and destabilize the positive equilibrium for $\chi <0$, it is easy to see that the high level of prey-taxis can accelerate the stability of the system.

Figure 1

The positive equilibrium $(\bar{u},\bar{v})\approx (0.6467, 1.6467)$ for $r_{0}=80$, $p=1$. The first line represents the number of predators, the second line represents the number of prey, the positive equilibrium $(\bar{u},\bar{v})$ becomes stable faster with increasing χ

Summarizing the results of Theorems 2.1 and 2.3 we have Table 1.

Table 1 The stability of positive equilibrium $(\overline{u},\overline{v})$ for different χ and other parameters

Figure 2 describes the phenomenon of the positive equilibrium $(\overline{u},\overline{v})$ that is stable when $\chi >\chi _{0}$, and loses stability when χ crosses $\chi _{0}$ for $pq\leq ck^{2}m$, which confirms the theoretical results of Theorem 2.1. Let $r_{0}=80$, $p=3=ck^{2}m$, $\ell =1$, the positive equilibrium $(\bar{u},\bar{v})\approx (2.2751, 3.2751)$, and $\chi _{0}\approx -0.3164$ for $i=5$.

Figure 2

$(\overline{u},\overline{v})$ loses stability when $\chi =-0.327<\chi _{0}$ and becomes stable when $\chi =-0.2$, $\chi =0.5>\chi _{0}$

Figure 3 shows how the intrinsic growth rate of predator s and prey-taxis χ affect the spatial-temporal patterns over the domain (0,1), the parameters are chosen to be $r_{0}=80$, $p=18>ck^{2}m$, the positive equilibrium $(\bar{u},\bar{v})\approx (0.8678, 1.8678)$, the maximum value of $\chi _{0}\approx 0.0359$ for $i=4$, and the maximum value $a_{11}(u_{1})\approx 1.911$ at $u_{1}\approx 0.3999$. For $s\geq 1.911$, there is no Hopf bifurcation for the system (1), for $s<1.911$, the distribution of stable regions is shown in Fig. 4.

Figure 3

The first line shows the stability of system (1) changes with the values of s when $\chi >\chi _{0}$, Hopf bifurcation occurs as s decreases through the point 1.911. The second line shows that the prey populations are unstable for $\chi <\chi _{0}$, and Hopf bifurcation occurs for $\chi >\chi _{0}$

Figure 4

The distribution of stable regions for $s=1$, there is no Hopf bifurcation for $\bar{u}\in (0,0.0902)\cup (1,+\infty )$, and the Hopf bifurcation may occur for $\bar{u}\in [0.0902,1]$

Next, we discuss the steady states of system (1) for $\Omega =(0,L)$, thus the eigenexpansions (9) with eigenvalue $\lambda _{i}= ( \frac{i\pi}{L} )^{2}$ are expressed as

$$ u(x)=\sum_{i=0}^{\infty}T_{i}\cos \frac{i\pi x}{L}, \qquad v(x)=\sum_{i=0}^{\infty}S_{i} \cos \frac{i\pi x}{L}. $$

By Theorem 3.1, model (6) admits a bifurcation branch as $\Gamma _{i}(h)=\{ (u_{i}(h, x), v_{i}(h, x)\}$. Let $d_{1}=0.01$, $d_{2}=2$, $r_{0}=80$, $\Omega =(0,1)$, the other parameters are the same as in Fig. 1,

$$ \textstyle\begin{cases} \frac{\partial u(x,t)}{\partial t}=0.01\Delta u+ \frac{80u}{1+2v}-2u-3u^{2}- \frac{3uv}{u+v},& x\in (0,1),t>0, \\ \frac{\partial v(x,t)}{\partial t}=2\Delta v-\chi \nabla (v\nabla u)+v (1- \frac{v}{u+1} ),& x \in (0,1),t>0, \end{cases}\displaystyle \vspace{-0.1cm} $$

then $(\bar{u},\bar{v})\approx (0.8678,1.8678)$. When $|h|$ is small, $(u_{i}(h,x),v_{i}(h,x))$ is the approximate solution near $(\overline{u},\overline{v},\chi _{i})$ of model (6), see Fig. 5.

Figure 5

The main part of $(u_{i}(h,x),v_{i}(h,x))$ of model (6) in the interval of $(0,1)$, the first line is the image of the prey corresponding to $i=1,2,3$ with $h=\pm 0.02, \pm 0.01$ and $h=0$, respectively, and the second line is the predator

We further find that if $(\bar{u},\bar{v})$ is locally asymptotically stable when $\chi =0$, it will maintain stability for $\chi >\max \chi _{i}$, and change the stability for $\chi <\max \chi _{i}$, it also implied that the $(\overline{u},\overline{v})$ is stable for all $\chi >0$ when $\max_{i\in \mathbb{N}^{+}}\chi _{i}<0$. Let $d_{1}=0.02$, $d_{2}=1$, $r_{0}=20$, $a=2$, $d=2$, $c=3$, $p=3$, $k=1$, $s=1$, $q=1$, $m=1$, $L=\pi$,

$$ \textstyle\begin{cases} \frac{\partial u(x,t)}{\partial t}=0.02\Delta u+ \frac{20u}{1+2v}-2u-3u^{2}- \frac{3uv}{u+v},& x\in (0,\pi ),t>0, \\ \frac{\partial v(x,t)}{\partial t}=\Delta v-\chi \nabla (v\nabla u)+v (1- \frac{v}{u+1} ),& x \in (0,\pi ),t>0. \end{cases} $$

By calculation, $(\bar{u},\bar{v})\approx (0.3445, 1.3445)$, $\max \chi _{i}\approx -0.6104$ for $i=3$, see Fig. 6. Moreover, by Matlab, we have $K_{2}\approx -2.3938<0$, the bifurcation solution $\Gamma _{3}(h)$ is stable through Theorem 3.4, see Fig. 7.

Figure 6

The positive equilibrium $(\bar{u},\bar{v})\approx (0.3445, 1.3445)$ loses its stability when $\chi =-0.65<\max \chi _{i}$, and maintains stability for $\chi =-0.5$, $0.5>\max \chi _{i}$

Figure 7

The bifurcation solution $\Gamma _{3}(h)$ is stable for $\chi _{3}= -0.6104$, $K_{2}\approx -2.3938<0$

5 Conclusion

In this paper, we consider a diffusive predator–prey system with prey-taxis. We show that the positive equilibrium $(\bar{u},\bar{v})$ is stable for $\chi >\max \chi _{i}$ and $ck^{2}m\geq pq$, the stability changes for $\chi <\max \chi _{i}$ or $ck^{2}m< pq$. For $\chi >\max \chi _{i}$, $ck^{2}m< pq$, the stability is determined by s, the system undergoes a spatially homogeneous Hopf bifurcation at $\bar{u}=u_{0,\pm}$ and a spatially nonhomogeneous Hopf bifurcation at $\bar{u}=u_{j,\pm}$ for small s. When χ> passes $\chi _{0}$, the system loses stability, also we examined the existence of nonconstant steady states by the bifurcation theory from Crandall and Rabinowitz. The stability of the solution $(\bar{u}, \bar{v}, \chi _{i} )$ is investigated, the sharpnesses of the bifurcation branches are determined by the value of $\int _{\Omega} \Phi _{i}^{3} \,\mathrm{d}x\neq 0$ that are given by Theorem 3.2 or $\int _{\Omega} \Phi _{i}^{4} \,\mathrm{d}x\neq 0$ for $\int _{\Omega} \Phi _{i}^{3} \,\mathrm{d}x=0$ in Theorem 3.4. In particular, the bifurcation curve $\Gamma _{i}(h)$ around $(\bar{u}, \bar{v}, \chi _{i} )$ is pitchfork for a 1D space.