Introduction

In the real world, intra and inter-species interactions significantly impact the extinction and persistence of the constituent subpopulations1. To understand the effect of synergistic and antagonistic ecological interactions on the extinction and co-evolution of species, researchers have proposed and analyzed prey-predator systems1,2,3,4, where predators typically consume the prey for their survival. The availability of prey and natural food resources directly impact the reproduction rates of predator and prey, respectively. Moreover, the ability to exploit only ecologically accessible space can have a beneficial effect on the sub-populations5,6. Hence, consideration of free-space availability as an ecological variable in prey-predator competition dynamics can influence the extinction, survival, and coexistence of species7,8,9,10,11,12,13,14.

In a general resource-consumer model, consumers (i.e., predators) typically enhance their reproductive success by exploiting the available resources i.e., prey). However, when resources develop direct or indirect defense mechanisms—such as chemical deterrents, behavioral adaptations, or morphological traits— consumer consumption efficiency declines. While this defensive strategy lowers predation pressure, it comes at the cost of a slower resource accumulation rate, potentially altering the stability and persistence of both populations15,16. Aposematism, or warning coloration, is a strategic defense mechanism used by prey to reduce their predation-induced mortality rate17. Aposematic prey employ conspicuous coloration as a visual warning to predators, signaling the harmful consequences of consuming chemically defended, toxic individuals. Predators assess the trade-off between the nutritional gains from ingestion and the physiological costs associated with toxin intake, and adjusting their consumption rate of visually distinct toxic prey accordingly18,19,20. The degree of toxicity, which correlates with the intensity of noxiousness, serves as an evolutionary trait, as the co-evolution of toxin production and warning coloration functions as a defensive strategy that minimizes predation pressure21.

Aposematism is widespread across various animal taxa, including invertebrates, fishes, amphibians, snakes, and birds22,23,24. For instance, the dendrobatid frog Oophaga pumilio23, the venomous coral snake M. fulvius25, and the mealworm Tenebrio molitor18 frequently exhibit bright coloration and distinct patterns to signal their unpalatability or toxicity to potential predators. At La Selva Biological Station in Costa Rica, Saporito et al.23 experimentally demonstrated that red-colored plasticine models of O. pumilio were subjected to fewer attacks by avian predators compared to brown frog models. This finding suggests that the striking aposematic coloration in alkaloid-rich dendrobatid frogs functions as an effective chemical defense mechanism against predation23. Similarly, the tricolor-banded pattern of venomous coral snakes serves as an aposematic signal for free-ranging avian predators, as confirmed through experimental studies by Brodie25. While the experiments in refs. 23,25 utilized plasticine models to illustrate aposematic signaling in poison frogs and coral snakes, Rowe et al.18 conducted in-vitro experiments using live T. molitor larvae to assess the influence of warning coloration on their avian predators, European starlings Sturnus vulgaris). Evidence of toxin production to reduce predation-induced exploitation is also present in unicellular organisms, such as phenotypically variable microbial biofilms26. Cooperative producers release toxins (e.g., hydrogen cyanide) to restrict the uncontrolled exploitation of public goods, such as elastase, by toxin-sensitive, non-productive defective mutants. This policing mechanism, wherein cooperative producers secrete toxins to suppress defectors in the opportunistic human pathogen Pseudomonas aeruginosa, conceptually parallels the toxin production by aposematic prey (acting as cooperators) to deter potential predators (functioning as defectors). Both systems exemplify an evolutionary strategy where cooperative traits emerge as a defense against exploitation, reinforcing the stability of cooperative communities in both microbial and ecological contexts27,28.

Ecologists and evolutionary biologists around the globe typically assume that evolutionary processes operate on longer timescales than ecological processes. However, recent studies indicate that ecological shifts and species evolution can occur on similar time scales29, i.e., ecological and evolutionary dynamics are inherently coupled, influencing each other in complex and reciprocal ways30,31. We now recognize that the coupling between ecology and evolution contributes to the processes driving biodiversity32. The analysis of this eco-evolutionary feedback in the form of eco-evolutionary dynamics7,8,9,11,12 can provide a qualitative justification for the effect of environmental changes on the evolution of population and vice versa.

The effect of evolutionary genetic feedback on the ecological dynamics of natural populations is discussed by Pimentel33. The importance of incorporating ecological interactions, by considering changes in population size, in evolutionary dynamics was identified by Hauert et al.5,6,34. They proposed a natural extension of replicator dynamics35,36, where the per capita growth rate of constituent sub-populations is proportional to their evolutionary fitness and also the availability of surrounding free space (ecological variable), which regulates the rate of reproduction of the population. Empirical studies, supported by theoretical predictions, that investigate the interconnection between ecological and evolutionary dynamics are performed on algal–rotifer prey-predator system by Kasada et al.16,37. They demonstrated how various forms of fitness trade-offs between prey defense mechanisms against rotifer predation and their competitive ability to acquire the limiting resource influence the resulting eco-evolutionary dynamics. Specifically, their study showed how stronger defensive traits, while offering protection against predation, can come at the cost of reduced resource uptake efficiency, thereby altering species interactions, population stability, and long-term coexistence patterns. By examining these trade-offs, they provided insights into how ecological pressures and evolutionary constraints shape adaptive strategies, ultimately driving shifts in community composition and biodiversity. Hiltunen et al.38 proposed an eco-evolutionary model for a three-species marine planktonic food web, where flagellates and rotifers act as predators, with both consuming prey algae, while rotifers additionally prey on flagellates (intraguild predation). Their study demonstrated that rapid evolution of a defensive trait in algae significantly influences prey-predator dynamics, leading to substantial alterations in the system’s stability and interactions. Colombo et al.31,39 used an individual-based model to investigate the existence of eco-evolutionary feedback between the spatial distribution of species in an ecological community and the evolution of perceptual ranges of a predator.

The overarching goal of this spatio-temporal eco-evolutionary framework is to address key questions regarding the interplay between ecological constraints, evolutionary adaptations, and spatial self-organization in predator-prey systems40. Specifically, we seek to shed light on the following questions: How does the initial availability of free space (an ecological variable) shape population evolution? How do the combined effects of free-space availability and the toxicity levels of aposematic prey (an evolutionary parameter) influence population extinction, persistence, and coexistence? What is the route of different spatio-temporal patterns induced by self-diffusion with the variation of ecological and evolutionary factors?

Motivated by this, in this paper, we formalize and present the predator–prey interactions in the presence of aposematism in prey via modified replicator dynamics5,6,34,35, where the per-capita growth rate of the population is determined not only by its intrinsic reproductive fitness but also by the availability of neighboring free-space, an essential ecological variable. We assume that each species allocates its energy and time among defense against predation, foraging, and reproduction2,15. When a species invests more in self-defense, such as toxin production or delay-induced warning coloration, its reproductive rate declines; however, its survival probability increases as predation pressure decreases. Building on these principles, we develop a three-dimensional diffusive eco-evolutionary prey-predator model consisting of non-toxic, non-aposematic prey, toxic aposematic prey, and predators. The toxic prey’s reproductive rate and predation rate are assumed to vary inversely with its toxicity level, an evolutionary trait. Resources for both prey types are supplied at a constant rate, reflecting rapid environmental recovery, and prey mortality due to intra- and interspecific competition is assumed to be negligible. A simplified Beverton-Holt recruitment function41 is used to describe prey birth. For predators, both natural and density-dependent deaths are considered. We examine how spatial structure affects eco-evolutionary outcomes by incorporating self-diffusion mechanisms for both prey and predator species, and explore the role of dispersal in shaping the spatial organization and persistence of interacting subpopulations. Our analysis reveals that the ecological factor, initial availability of accessible free space, and the evolutionary factor, the toxicity level of aposematic prey, jointly determine whether species face extinction, persist individually, or coexist dynamically.

Methods

In our model, we consider a predator species that feeds on two types of prey: an undefended type and a defended type. The defended prey exhibits traits such as chemical toxicity or behavioral mechanisms that reduce its palatability or increase the energetic cost to the predator. The ecological consequences of consuming these prey types differ, with undefended prey offering a net energetic benefit to predators, and defended prey potentially imposing costs. To model this trade-off, we later introduce a toxicity-dependent conversion function that captures how predator energy gain varies continuously with prey defense levels.

To model the eco-evolutionary dynamics of the prey-predator system with aposematic prey, we consider U, V and W to be the fractions of undefended prey, defended prey, and predators in the population, respectively. To account for the effect of free space on growth of populations, we denote Z as the fraction of ecologically accessible empty space. If N represents the total number of prey and predators integrated over the entire space, then the numbers of undefended prey, defended prey, and predators are NU, NV, and NW, respectively. We assume that defended prey divide their time between two activities: aposematism and searching for food for reproduction. We denote by ε (0 ≤ ε ≤ 1) the proportion of time they devote to aposematism and 1 − ε the proportion of time they devote to searching for food and reproduction. Since undefended prey do not invest time in self-defense, they spend all of their time searching for food, and subsequently, for reproduction.

We consider the spatial distribution of prey and predator in a two-dimensional (2D) domain with spatial coordinates (X, Y) ∈ [0, R] × [0, R], where \(R\in {\mathbb{R}}\). We assume further that all species move randomly by self-diffusion. As stated previously, we suppose that the fraction of neighboring free space (Z = Z(X, Y, t)) induces positive feedback on the per-capita growth rate of the populations. Combining the above effects, we deduce that the spatio-temporal eco-evolutionary dynamics of the prey-predator replicator system can be written in the following form,

$$\frac{\partial U}{\partial \widehat{t}} = {D}_{U}{\nabla }^{2}U+U(Z{f}_{U}-{d}_{U}),\\ \frac{\partial V}{\partial \widehat{t}} = {D}_{V}{\nabla }^{2}V+V(Z{f}_{V}-{d}_{V}),\\ \frac{\partial W}{\partial \widehat{t}} = {D}_{W}{\nabla }^{2}W+W(Z{f}_{W}-{d}_{W}).$$
(1)

As the proposed system (1) comprises prey and predators integrated with ecologically-viable free-space, we assume \(U(X,Y,\widehat{t})+V(X,Y,\widehat{t})+W(X,Y,\widehat{t})+Z(X,Y,\widehat{t})=1\)42. It abstracts the constraint imposed by finite space and resource, taken to be unity, and does not imply that species exert identical spatial or competitive effects. Inter-species facilitation and competition are modeled explicitly through the dynamic interaction terms43. In Eq. (1), fU = fU(U, V, W), fV = fV(U, V, W) and fW = fW(U, V, W) denote the growth functions of undefended prey, defended prey and predator, respectively. Also, dU = dU(U, V, W), dV = dV(U, V, W) and dW = dW(U, V, W) are the corresponding death functions. These six functions depend on various ecological and evolutionary factors (their functional forms are defined below). The parameters DU, DV, and DW are the diffusion coefficients, assumed constant for undefended prey, defended prey, and predator, respectively.

We consider the spatial distribution of prey and predator populations within a two-dimensional ecological domain \(\Omega =[0,R]\times [0,R]\subset {{\mathbb{R}}}^{2}\). The boundary of the domain is denoted by ∂Ω. All species are assumed to disperse through self-diffusion within Ω, and the environment is considered spatially and temporally homogeneous, implying that all model parameters are constant in space and time. To ensure conservation of population within the domain, we impose zero-flux (Neumann) boundary conditions,

$$\frac{\partial U}{\partial \widehat{n}}=\frac{\partial V}{\partial \widehat{n}}=\frac{\partial W}{\partial \widehat{n}}=0,\,(X,Y)\in \partial \Omega ,$$
(2)

where \(\widehat{n}\) denotes the outward unit normal to the boundary ∂Ω. The system is initialized with strictly positive population densities: U(X, Y, 0) = U0(X, Y) > 0, V(X, Y, 0) = V0(X, Y) > 0, W(X, Y, 0) = W0(X, Y) > 0, and the initial fraction of free ecological space is defined by Z(X, Y, 0) = 1 − U0(X, Y) − V0(X, Y) − W0(X, Y) subject to the constraint 0 ≤ U0(X, Y) + V0(X, Y) + W0(X, Y) ≤ 1 for (X, Y) ∈ Ω, which reflects the assumption of a finite ecological carrying capacity. This constraint implies that the combined biomass or resource occupancy of all species is bounded, modeling the limited availability of nutrients, habitat, or space, rather than direct spatial conflict among species.

Let I be the input of resources such as natural food, habitat, and other opportunities provided by the ecological space as growth factors for both prey. Then, the available resource for each undefended and defended prey are \(\frac{I}{NU}\) and \(\frac{I}{NV}\), respectively. If s0 is the searching efficiency of resources for both prey, then the resource uptakes by each undefended and defended prey in one unit of time are \(\frac{I}{NU}{s}_{0}\) and \(\frac{I}{NV}{s}_{0}(1-\varepsilon )\), respectively. For simplicity, we use the Beverton-Holt recruitment function \(\frac{{K}_{1}m}{1+{K}_{2}m}\) to model the per capita birth rate of undefended and defended prey, incorporates the effect of saturation in converting the acquired food resource m into offspring. Hence, \(m=\frac{I}{NU}{s}_{0}\) and \(m=\frac{I}{NV}{s}_{0}(1-\varepsilon )\) for the undefended and defended prey44, respectively. After some simple calculations, the reproduction rate of undefended and defended prey can be written respectively as \(\frac{r}{\rho +U}\) and \(\frac{r(1-\varepsilon )}{\rho (1-\varepsilon )+V}\) where \(r=\frac{{K}_{1}{s}_{0}I}{N}\) and \(\rho =\frac{{K}_{2}{s}_{0}I}{N}\). Here r and ρ denote the maximum potential reproduction rate and the saturation parameter based on resources, respectively. We also assume that the rate at which predators consume prey can be described via a Holling-type II functional response. In particular, we define the growth functions fU, fV, fW and the death functions dU, dV, dW as follows,

$${f}_{U} = \, \frac{r}{\rho +U},\,\,{f}_{V}=\frac{r(1-\varepsilon )}{\rho (1-\varepsilon )+V},\,\,{f}_{W}=\frac{{e}_{1}{d}_{2}U+{e}_{2}{d}_{2}(1-\epsilon )V}{\kappa +U+V},\\ {d}_{U} = \;{d}_{1}+\frac{{d}_{2}W}{\kappa +U+V},{d}_{V}={d}_{1}+\frac{{d}_{2}(1-\varepsilon )W}{\kappa +U+V},{d}_{W}={d}_{3}+{d}_{4}W.$$
(3)

Here, for simplicity, we assume that natural death rate d1 is the same for undefended and defended prey. Parameters e1 and e2 are the conversion efficiencies from undefended and defended prey biomass to predator biomass, respectively. The parameters d3 and d4 are natural and density-dependent death rates of the predator population, respectively. The terms d2 and d2(1 − ε) represent the predation rates of undefended and defended prey, respectively, while κ is the competition intensity per individual for both prey types. The fitness and mortality functions (fU and dU in Eq. (3)) of undefended prey are unaffected by the toxicity-induced self-protection measure ε. However, from the expressions for fV and dV in Eq. (3), it follows that an increase in toxicity level ε reduces both the per-capita growth and predation-induced mortality of defended prey, reinforcing the trade-off between self-defense and reproduction in aposematic species.

All model parameters (r, ρ, d1, d2, κ, ε, e1, d3, d4, DU, DV, and DW) are non-negative constants. We assume that when 0 ≤ ε ≤ 0.5, consuming defended prey provides a net positive effect on predator reproduction. However, for 0.5 < ε ≤ 1, ingesting highly toxic prey negatively affects predator growth. The conversion efficiency from undefended prey biomass to predator biomass is given by e1 > 0, ensuring that predation on undefended prey always supports predator growth.

To incorporate the negative effect of consuming highly toxic aposematic prey, we define the conversion efficiency from defended prey biomass to predator biomass, e2, as a transition term of the form \(s(x;k)=\tanh (kx)\)45, where x is a centered driver variable and k > 0 controls the steepness of the transition. As k → ∞, S approaches a step function. Accordingly, we define

$${e}_{2}(\varepsilon )={e}_{1}\,\tanh \left(k\left(1-\frac{\varepsilon }{0.5}\right)\right),$$

which ensures e2(0) = e1, e2(1/2) = 0, and e2(1) = − e1, consistent with the biological constraints. To encode an arbitrarily steep but bounded transition, we re-parameterize the steepness as

$$k={\tanh }^{-1}\left(1-\frac{\epsilon }{{e}_{1}}\right),$$

so that k → ∞ as ϵ → 0. Substituting this expression for k into the definition of e2(ε) and taking the sharp-switch limit ϵ → 0+ yields

$${e}_{2}={lim}_{\epsilon \to 0}{e}_{1}\tanh \left[\left(1-\frac{\varepsilon }{0.5}\right) \, {\tanh }^{-1}\left(1-\frac{\epsilon }{{e}_{1}}\right)\right].$$
(4)

The functional form of Eq. (4) was deliberately chosen to provide a smooth and bounded transition between beneficial and harmful energetic outcomes for predators consuming defended prey, depending on the toxicity level. The use of a bounded hyperbolic tangent ensures analytical tractability, avoids discontinuities that could cause numerical instability, and captures a biologically realistic trade-off in eco-evolutionary predator-prey interactions. Equation (4) ensures that  − e1 < e2 < e1, also restricts the negative impact of consuming highly toxic prey to a finite level, reflecting empirical observations that ingestion of defended prey in many aposematic systems often results in sublethal effects and learned avoidance rather than immediate death46. This formulation assumes that as ε increases, the net benefit of consuming defended prey declines, and becomes deleterious for highly toxic individuals, aligning with empirical observations of predator-prey interactions involving chemically defended species.

For example, predators feeding on poison frogs24, ladybird beetles47, or toxic butterflies show reduced reproduction or foraging efficiency rather than direct lethality. Such sublethal costs are consistent with the sigmoidal form, which parallels approaches in functional ecology where smooth transition functions are used to model threshold-like biological responses without unrealistic discontinuities. While the present formulation aligns with observed predator behavior in both macro- & microbial systems, the model can be readily generalized to incorporate more extreme penalties, should empirical data indicate the necessity.

Although U, V and W are dimensionless population fractions, other model parameters (DU, DV, DW, r, d1, d2, d3, d4) and spatio-temporal variables \((\widehat{t},X,Y)\) are dimensional. For clarity, we non-dimensionalize the eco-evolutionary dynamics (1) as,

$$\frac{\partial u}{\partial t} = {D}_{u}{\nabla }^{2}u+\frac{zu}{\rho +\kappa u}-\xi u-\frac{\gamma uw}{1+u+v},\\ \frac{\partial v}{\partial t} = {D}_{v}{\nabla }^{2}v+\frac{zv}{\rho +\kappa \theta v}-\xi v-\frac{\gamma vw}{\theta (1+u+v)},\\ \frac{\partial w}{\partial t} = {\nabla }^{2}w+(\frac{{\beta }_{1}u+{\beta }_{2}v}{1+u+v})zw-\delta w-\mu {w}^{2},$$
(5)

where \(t=\widehat{t}r\kappa\), \(x=X\sqrt{\frac{r\kappa }{{D}_{W}}}\), \(y=Y\sqrt{\frac{r\kappa }{{D}_{W}}}\), \(u(x,y,t)=\frac{U(X,Y,\widehat{t})}{\kappa }\), \(v(x,y,t)=\frac{V(X,Y,\widehat{t})}{\kappa }\), \(w(x,y,t)=\frac{W(X,Y,\widehat{t})}{\kappa }\), \(z(x,y,t)=\frac{Z(X,Y,\widehat{t})}{\kappa }\), \(\xi =\frac{{d}_{1}}{r\kappa }\), \(\gamma =\frac{{d}_{2}}{r\kappa }\), η = εκ, β1 = e1γκ, \(\delta =\frac{{d}_{3}}{r\kappa }\), \(\mu =\frac{{d}_{4}}{r}\), \(\theta =\frac{\kappa }{\kappa -\eta }\), \({\beta }_{2}={lim}_{\epsilon \to 0}\frac{{\beta }_{1}}{\theta }\tanh [(1-\frac{2\eta }{\kappa }){\tanh }^{-1}(1-\frac{\epsilon \gamma \kappa }{{\beta }_{1}})]\), \({D}_{u}=\frac{{D}_{U}}{{D}_{W}}\), \({D}_{v}=\frac{{D}_{V}}{{D}_{W}}\), and \(u(x,y,t)+v(x,y,t)+w(x,y,t)+z(x,y,t)=\frac{1}{\kappa }\).

The dimensional model (1) has 12 parameters (DU, DV, DW, r, s, κ, ε, e1, d1, d2, d3, d4), while the dimensionless model (5) has 10 parameters (Du, Dv, ρ, κ, η, β1, ξ, γ, δ, μ).

Incorporation of free space induced reproduction means that the total population fraction u + v + w is bounded below and above by 0 and \(\frac{1}{\kappa }\). If the population reaches its maximum carrying capacity \(\frac{1}{\kappa }\), then there is no territorial free space left. By contrast, if the population becomes extinct (u + v + w = 0), then the entire ecological territory is empty space. For physically realistic solutions we require θ > 0, i.e., \(0 < \eta < \kappa\)). If η = 0, then there is no concept of aposematic prey, i.e., the system consists of only undefended prey and predator). Moreover, if η = κ (θ = ∞) then there is no predation of maximally toxic defended preys. In other words, when κ = μ, β2 = 0 in Eq. (5) (i.e., the predator’s reproduction depends only on undefended prey). However, to maintain the trade-off between the times for defense and reproduction, the accumulation of maximal toxicity induces a negligibly small reproduction rate; hence, in the long-run, the defended prey will diminish. A moderately high κ and a wide toxicity range η ∈ [0, κ] can significantly inhibit predation on aposematic prey and predator growth, initially boosting prey density. However, increased population density in limited ecological space reduces free-space-mediated reproduction, ultimately promoting extinction. In a summary, we conclude that high values of κ reduce the availability of free-space and in turn, reduce the carrying capacity of the population, which can also be justified mathematically as \(u+v+w=\frac{1}{\kappa }-z\). For reference, the parameters appearing in Eq. (5) are tabulated in Table 1. The parameter values in Table 1 are selected so that the system yields an ecologically meaningful regime where three species — undefended prey, defended prey, and predators — can coexist and stable in the absence of diffusion, and destabilize under the analytically derived diffusion conditions. Understanding the interplay between ecological (κ, μ) and evolutionary (η) parameters is one of the main objectives of this study.

Table 1 Summary of dimensionless model (5) parameters
Full size table

The functional choices made in the proposed model reflect widely observed ecological and evolutionary mechanisms. The trade-off in defended prey between toxin production and reproduction has strong empirical support across taxa. Prey reproduction is represented using the Beverton-Holt recruitment function, a classical form that accounts for resource saturation44. Predator consumption follows a Holling type II functional response50, while the energetic effect of defended prey is modeled via a smooth hyperbolic tangent function, which avoids unrealistic discontinuities and captures the sublethal costs of toxicity observed in predator-prey interactions53. These formulations ensure that the model is both mathematically tractable and biologically grounded.

Results

Linear stability analysis

In this section, our primary motivation is to understand how the initial abundance of free space (κ, ecological parameter) and the degree of defensiveness of aposematic prey (η, evolutionary parameter) affect the evolution of the population. We write the eco-evolutionary system (5) with boundary condition (2) more compactly as,

$$\frac{\partial u}{\partial t} = {D}_{u}{\nabla }^{2}u+F(u,v,w),\\ \frac{\partial v}{\partial t} = {D}_{v}{\nabla }^{2}v+G(u,v,w),\\ \frac{\partial w}{\partial t} = {\nabla }^{2}w+H(u,v,w),$$
(6)

where the reaction terms are as follows

$$F(u,v,w) = u\left[\frac{\frac{1}{\kappa }-u-v-w}{\rho +\kappa u}-\xi -\frac{\gamma w}{1+u+v}\right],\\ G(u,v,w) = v\left[\frac{\frac{1}{\kappa }-u-v-w}{\rho +\kappa \theta v}-\xi -\frac{\gamma w}{\theta (1+u+v)}\right], \\ H(u,v,w) = w\left[\left(\frac{{\beta }_{1}u+{\beta }_{2}v}{1+u+v}\right)\left(\frac{1}{\kappa }-u-v-w\right)-\delta -\mu w\right].$$
(7)

In the following sections, we undertake a linear stability analysis of the system (6), using the functional forms (7), when spatially effects are negligible, i.e., u(x, y, t) = u(t), etc. We also establish the fundamental principles of Turing instability for coexistence solutions. Finally, we examine how parameter variations shape the morphology of the emerging spatio-temporal patterns, highlighting the eco-evolutionary significance of dynamic shifts across different ecological and evolutionary states within the system (6).

Stability analysis of the eco-evolutionary dynamics without spatial effects

In this section, we neglect spatial effects and identify the equilibrium points of the ordinary differential equations which is discussed in supplementary Note 1. The spatially homogeneous and ecologically viable equilibrium points of the diffusion-free system of Eq. (5), conditions for their existence, and stability are summarized in Table S1.

To complement our analysis of the linear stability of spatially homogeneous equilibrium points, we perform numerical simulations of the isolated system of Eq. (5). We systematically vary κ, which controls the availability of ecologically viable free space for reproduction, and η, which regulates the defended prey’s toxicity level. These two parameters shape the ecological and evolutionary trajectories of the system, influencing the persistence and coexistence of the three subpopulations. Recall that the availability of free space decreases as κ increases as given by \(z=\frac{1}{\kappa }-(u+v+w)\).

By varying κ and η, we capture dynamic behaviors, ranging from extinction to non-oscillatory and oscillatory coexistence of the species. To illustrate the effect of κ and η on the system, we plot two bifurcation diagrams, varying η for two fixed values of κ, in Fig. 1a, e, respectively.

Fig. 1: Dynamical behavior by changing η.
Fig. 1: Dynamical behavior by changing η.The alternative text for this image may have been generated using AI.
Full size image

Bifurcation diagram concerning the toxicity level (η) of the defended prey in the model (6) for (a) κ = 0.15, and e κ = 0.2. Several stable and unstable scenarios are delineated out of this experiment, (I) and (V) denote oscillatory and non-oscillatory coexistence of defended prey and predator, respectively. (III) and (IV) denote the non-oscillatory and oscillatory coexistence of undefended prey, defended prey, and predator, respectively. (II) denotes the non-oscillatory coexistence of undefended prey and defended prey. Blue (light blue), red (mauve) and green (deep green) lines respectively denote \({u}_{\max }({u}_{\min })\), \({v}_{\max }({v}_{\min })\) and \({w}_{\max }({w}_{\min })\) in regions (I) and (IV) of (a, e) where oscillatory behaviors are observed. b, c, d, f, and g Represent the time series of u (blue), v (red) and w (green) portraits of regimes (I), (II), (III), (IV), and (V), respectively. The other parameter values are fixed at ρ = 1.76, ξ = 0.1, γ = 3.73, β1 = 0.5, δ = 0.1, and μ = 0.15, with (0.1, 0.1, 0.1) as the initial fractions of the three species.

In Fig. 1a for κ = 0.15, we observe three distinct coexistence states corresponding to regions (I), (II), and (III). For small values of η, the system stabilizes into oscillatory coexistence of defended prey and predators, with the undefended prey driven to extinction (region (I), time series in Fig. 1b. This behavior aligns with ecological theory, where predator-prey oscillations can emerge in the presence of chemically defended prey, which regulates predator populations by imposing energetic costs on predation54,55. Similar dynamics have been observed in amphibian communities where toxic species, such as dendrobatid frogs, influence predator populations by altering their predation rates and foraging behaviors56. With a slight increase in η, the predator population becomes extinct, and both prey populations stabilize in a non-oscillatory manner, corresponding to equilibrium point \({Q}_{{p}_{1}}\) (region (II), stable time series in Fig. 1c. This suggests that when toxicity levels exceed a threshold, the energetic trade-offs associated with predator foraging become unsustainable, leading to trophic downgrading57. Further increasing η results in stable coexistence of the three species, equilibrium point Q* (region (III), time series in Fig. 1d. This shift highlights the role of adaptive foraging and eco-evolutionary feedback in stabilizing multi-species interactions, as observed in experimental studies on bacterial communities where predator-prey coexistence is mediated by evolutionary adaptations58.

For κ = 0.2, the system exhibits oscillatory co-existence across all sub-populations within a narrow interval of η (region (IV), Fig. 1e, spanning approximately η = 0 to η = 0.003. Such transient oscillatory coexistence can emerge from delayed feedback mechanisms inherent in eco-evolutionary systems, where species interactions co-evolve over ecological and evolutionary timescales simultaneously59. This phenomenon parallels empirical studies on predator-prey cycles in aquatic ecosystems, where time-lagged responses to environmental fluctuations drive transient oscillations before stabilizing into alternative states. As η increases further, the undefended prey population goes extinct, and the system stabilizes at equilibrium point \({Q}_{{p}_{3}}\) (region (V), corresponding time series in Fig. 1g. This scenario is consistent with empirical observations of predator-prey systems, where increasing prey toxicity selects against generalist predators and leads to alternative stable states, as seen in marine food webs where prey chemical defenses drive predator specialization and trophic restructuring60.

Overall, the analysis reveals that η (governing the strength of prey chemical defense) plays a crucial role in determining species persistence and interaction stability. Further, changes in η and κ with κ (which dictates the initial spatial resource availability) lead to diverse ecological outcomes, ranging from predator-driven oscillations to stable coexistence. The results provide insight into real-world ecological scenarios, such as toxin-mediated trophic interactions in amphibian communities56 and evolutionary shifts in predator-prey systems under anthropogenic pressures60.

Regions of local asymptotic stability of the equilibrium points \({Q}_{{p}_{1}}\), \({Q}_{{p}_{2}}\), \({Q}_{{p}_{3}}\) and Q* are highlighted in Fig. 2a, c. We identify regions in the κ − η plane where the equilibrium points \({Q}_{{p}_{1}}\),\({Q}_{{p}_{2}}{Q}_{{p}_{3}}\) and Q* co-exist. We analyze their existence and stability for the κ > η regime. The stability conditions for the four equilibrium points, summarized in Table S1, serve as the foundation for these stability maps. As shown in Fig. 2a, the stability regions associated with \({Q}_{{p}_{1}}=(u^{\prime} ,v^{\prime} ,0)\) (the predator-free terrain) and \({Q}_{{p}_{2}}=(u^{\prime\prime} ,0,w^{\prime\prime} )\) (coexistence of undefended prey and predator) reveal distinct ecological regimes. The red-shaded region represents parameter combinations where \({Q}_{{p}_{1}}\) is locally stable, indicating that the two types of prey species can coexist in the absence of the predator. The black curve represents the stability boundary of \({Q}_{{p}_{2}}\), indicating scenarios where the undefended prey coexists exclusively with the predator. In the white region, neither of these equilibria is stable, implying potential oscillatory or unstable dynamics. In Fig. 2c, the stability domains of \({Q}_{{p}_{3}}=(0,{v}^{\prime \prime \prime} ,{w}^{\prime \prime \prime} )\) (coexistence of defended prey and predator) and Q* = (u*, v*, w*) (full coexistence of all three species) further emphasize the impact of the carrying capacity (κ) and the level of toxicity of the defended prey (η) in shaping ecological outcomes. In the cyan-shaded region, \({Q}_{{p}_{3}}\), where both the defended prey and predator species coexist in the absence of the undefended prey, is stable, while in the orange-shaded region Q*, where all three species can coexist is stable. The existence and stability of Q* aligns with key ecological principles such as competitive exclusion and predator-mediated persistence, emphasizing that species coexistence is highly sensitive to parameter variations. Here, bistability between \({Q}_{{p}_{1}}\) and \({Q}_{{p}_{3}}\) (cyan region) and between \({Q}_{{p}_{1}}\) and Q* (orange region). Note that, the presence of white regimes (η > κ) exist here in both diagrams 2a, c, underscores the fragility of equilibrium states under certain parameter choices, suggesting the possibility of oscillatory behavior, chaotic dynamics, or extinction events, whereas the yellow regime in Fig. 2c highlights the unstable terrain for \({Q}_{{p}_{3}}\) and Q* only. These findings offer theoretical insight into species persistence and extinction, contributing to a deeper understanding of biodiversity dynamics and informing conservation strategies. Figure 2b, d testifies the discontinuous stable persistence of \({Q}_{{p}_{2}}\) and Q* along κ = η, and κ = η + 0.009 line, as we observe in the two-dimensional parameter space  κ − η (Fig. 2a, c, respectively).

Fig. 2: Two-parameter bifurcation diagram.
Fig. 2: Two-parameter bifurcation diagram.The alternative text for this image may have been generated using AI.
Full size image

Different stability regions shaped by the equilibrium points of the system (6) in the two-dimensional parameter planes of inverse carrying capacity (κ) and toxicity level of defended prey (η). Stability regions of (a) \({Q}_{{p}_{1}}\) (red portion) and \({Q}_{{p}_{2}}\) (black dots, varying through κ = η only), and (c) \({Q}_{{p}_{3}}\) (cyan) and Q* (orange) in the κ-η parameter spaces while keeping the other parameter values fixed at ρ = 1.76, ξ = 0.1, γ = 3.73, β1 = 0.5, δ = 0.1 and μ = 0.15. To verify the inconsistency of the region  of these stable portions, we conduct two experiments (through (b, d). In (b), we fix η = κ, and plot all the existing \({Q}_{{p}_{2}}\), and in (d), we vary κ, through fixing η = κ − 0.009) to see the existence and stability of the interior Q*. The purple background showcases the non-existence or unstable region, that validates our parameter spaces (a, c). The white regions in (a, c) represent η > κ, whereas the yellow region in c denotes the unstable terrain for \({Q}_{{p}_{3}}\), and Q*.

Stability analysis of the spatial model

We consider the spatio-temporal model (6). We perform linear stability analysis to identify conditions for diffusion-driven instability and verify the results numerically. Now, we investigate the mathematical basis for Turing instability in the 3D spatio-temporal model (6), where the spatially homogeneous steady state Q*(u*, v*, w*) is stable in the absence of diffusion of the undefended prey, defended prey and predators, but unstable in the presence of diffusion. We linearize the model (6) about the spatially homogeneous equilibrium Q*(u*, v*, w*), and introduce linearized PDEs for \((\widetilde{u},\widetilde{v},\widetilde{w})\): \(\widetilde{u}=u-{u}_{* }\), \(\widetilde{v}=v-{v}_{* }\), \(\widetilde{w}=w-{w}_{* }\), where \((\widetilde{u},\widetilde{v},\widetilde{w})\) are small perturbations of (u, v, w) in a neighborhood of Q*(u*, v*, w*). At leading order, we obtain the following:

$$\frac{\partial \widetilde{u}}{\partial t} = \, {D}_{u}\left(\frac{{\partial }^{2}\widetilde{u}}{\partial {x}^{2}}+\frac{{\partial }^{2}\widetilde{u}}{\partial {y}^{2}}\right)+{F}_{u}({u}_{* },{v}_{* },{w}_{* })\widetilde{u}\\ +{F}_{v}\left({u}_{* },{v}_{* },{w}_{* }\right)\widetilde{v}+{F}_{w}({u}_{* },{v}_{* },{w}_{* })\widetilde{w},\\ \frac{\partial \widetilde{v}}{\partial t} = \, {D}_{v}\left(\frac{{\partial }^{2}\widetilde{v}}{\partial {x}^{2}}+\frac{{\partial }^{2}\widetilde{v}}{\partial {y}^{2}}\right)+{G}_{u}({u}_{* },{v}_{* },{w}_{* })\widetilde{u}\\ +{G}_{v}({u}_{* },{v}_{* },{w}_{* })\widetilde{v}+{G}_{w}({u}_{* },{v}_{* },{w}_{* })\widetilde{w},\\ \frac{\partial \widetilde{w}}{\partial t} = \left(\frac{{\partial }^{2}\widetilde{w}}{\partial {x}^{2}}+\frac{{\partial }^{2}\widetilde{w}}{\partial {y}^{2}}\right)+{H}_{u}({u}_{* },{v}_{* },{w}_{* })\widetilde{u}\\ +{H}_{v}({u}_{* },{v}_{* },{w}_{* })\widetilde{v}+{H}_{w}({u}_{* },{v}_{* },{w}_{* })\widetilde{w}.$$
(8)

We consider spatio-temporal perturbations of the form

$$\widetilde{u}(x,y,t) = {\aleph }_{u}{e}^{\sigma t}\sin ({k}_{x}x)\sin ({k}_{y}y),\\ \widetilde{v}(x,y,t) = {\aleph }_{v}{e}^{\sigma t}\sin ({k}_{x}x)\sin ({k}_{y}y),\\ \widetilde{w}(x,y,t) = {\aleph }_{w}{e}^{\sigma t}\sin ({k}_{x}x)\sin ({k}_{y}y),$$

where σ = σ(kx, ky) is the growth rate of spatial perturbations with wave numbers kx and ky. ℵu, ℵv and ℵw are the amplitudes of the perturbation. Using Eq. (8), it is straightforward to show that the Jacobian matrix is given by

$${J}_{D}=\left[\begin{array}{ccc}{F}_{u}-{D}_{u}{k}^{2} & {F}_{v} & {F}_{w}\\ {G}_{u} & {G}_{v}-{D}_{v}{k}^{2} & {G}_{w}\\ {H}_{u} & {H}_{v} & {H}_{w}-{k}^{2}\end{array}\right],$$
(9)

where \({k}^{2}={k}_{x}^{2}+{k}_{y}^{2}\). Now, we determine the conditions for the diffusion-driven instability of Q*.

We assume that Q* is locally asymptotically stable in the absence of diffusion, i.e., Ξ < 0, ϒ < 0 and \(\Omega > \frac{\Upsilon }{\Xi }\) (see Eq. S14). The eigen-value equation in a neighborhood of Q* can be written in terms of the Jacobian (9) as

$${\lambda }^{3}-{\alpha }_{2}{\lambda }^{2}+{\alpha }_{1}\lambda -{\alpha }_{0}=0,$$
(10)

where α2 = Ξ − (Du + Dv + 1)k2 < 0, \({\alpha }_{1}=({D}_{u}+{D}_{v}+{D}_{u}{D}_{v}){k}^{4}-\{{F}_{u}^{* }+{G}_{v}^{* }+({D}_{u}+{D}_{v}){H}_{w}^{* }+{D}_{v}{F}_{u}^{* }+{D}_{u}{G}_{v}^{* }\}{k}^{2}+\Omega\), \({\alpha }_{0}=-{D}_{u}{D}_{v}{k}^{6}+\{{D}_{v}{F}_{u}^{* }+{D}_{u}{G}_{v}^{* }+{D}_{u}{D}_{v}{H}_{w}^{* }\}{k}^{4}-\{{F}_{u}^{* }{G}_{v}^{* }+{D}_{v}{F}_{u}^{* }{H}_{w}^{* }+{D}_{u}{G}_{v}^{* }{H}_{w}^{* }-{D}_{u}{G}_{w}^{* }{H}_{v}^{* }-{F}_{v}^{* }{G}_{u}^{* }-{D}_{v}{F}_{w}^{* }{H}_{u}^{* }\}{k}^{2}+\Upsilon\), \({F}_{u}^{* }={F}_{u}({Q}_{* })\) and so on.

Applying the Routh-Hurwitz criterion to Eq. (10), we deduce that a sufficient condition for the instability of Q* in the presence of diffusion is α0 > 0. Now, α0 can be written as

$${\alpha }_{0}={\vartheta }_{3}{k}^{6}+{\vartheta }_{2}{k}^{4}+{\vartheta }_{1}{k}^{2}+{\vartheta }_{0},$$
(11)

where ϑ3 = − DuDv < 0, \({\vartheta }_{2}={D}_{v}{F}_{u}^{* }+{D}_{u}{G}_{v}^{* }+{D}_{u}{D}_{v}{H}_{w}^{* }\), \({\vartheta }_{1}=-[{F}_{u}^{* }{G}_{v}^{* }+{D}_{v}{F}_{u}^{* }{H}_{w}^{* }+{D}_{u}{G}_{v}^{* }{H}_{w}^{* }-{D}_{u}{G}_{w}^{* }{H}_{v}^{* }-{F}_{v}^{* }{G}_{u}^{* }-{D}_{v}{F}_{w}^{* }{H}_{u}^{* }]\), ϑ0 = ϒ < 0.

From Eq. (11), the sufficient conditions for α0 > 0 can be written as

$${\vartheta }_{2} > 0,$$
(12a)

and

$$\Delta =18{\vartheta }_{3}{\vartheta }_{2}{\vartheta }_{1}{\vartheta }_{0}-4{\vartheta }_{2}^{3}{\vartheta }_{0}+{\vartheta }_{2}^{2}{\vartheta }_{1}^{2}-4{\vartheta }_{3}{\vartheta }_{1}^{3}-27{\vartheta }_{3}^{2}{\vartheta }_{0}^{2} > 0.$$
(12b)

Expressions (12a) and (12b) are the necessary conditions for diffusion-driven instability of Q* in model (6).

We validate these analytical results in Fig. 3, where we use (11) to show how α0 changes as k2 varies for three fixed values of μ. As the strength of intra-species competition among predators (μ) increases, the range of values of k2 for which spatial patterning is predicted increases.

Fig. 3: Variation of α0 by changing χ = k2.
Fig. 3: Variation of α0 by changing χ = k2.The alternative text for this image may have been generated using AI.
Full size image

Plot of α0(χ) vs χ from Eq. (11) of the model (6) for different values of μ (μ = 0.6 brown (dashed line), μ = 0.72 olive green (dotted line) and μ = 0.89 pink (solid line)). The positiveness of α0 for positive values of the wave numbers χ = k2 justifies the sufficient condition for Turing instability analytically. It is observed that for increasing values of μ, the ranges of χ increases for which α0 > 0. Other parameter values are fixed at κ = 0.15, ξ = 0.55, γ = 3.73, η = 0.005, β1 = 0.5, δ = 0.1 and Du = Dv = 0.01.

The local stability of the equilibrium point Q* is governed by the eigenvalue Eq. (10), where the coefficients α2, α1, and α0 depend on the diffusion coefficients Du, Dv and the functional responses of the species. The critical coefficient α0 determines the onset of diffusion-driven instability; according to the Routh-Hurwitz criterion, instability arises when α0 > 0, leading to spatial pattern formation. As shown in Fig. 3, the stability region (α0 < 0) shrinks with increasing μ, and the maximum value of α0 rises accordingly. This indicates that stronger intra-predator competition promotes conditions favorable for diffusion-induced instability and the emergence of spatial structures.

To understand this behavior, we recall the role of μ in modifying the effective mortality rate of the predator population. As μ increases, competitive pressure among the predators intensifies, leading to different predation rates on the defended and undefended prey. This altered predation pressure impacts the spatial distribution of the two prey populations, driving the system unstable for certain parameter values.

Eco-evolutionary significance of the spatial patterns

The system defined by Eq. (6) exhibits a range of spatial patterns, which emerge from the intricate interplay between diffusive movement and eco-evolutionary interactions between species. In this section, we explore the mechanisms that drive these patterns by examining the effects of varying three key parameters κ, μ, and η, while holding all other parameters fixed as in Table 1, where the interior equilibrium Q* is stable. We adopt a controlled approach, holding two parameters fixed and varying the third, to reveal how it affects the spatial distribution of the undefended prey. In this way, we capture the critical transitions in the pattern dynamics. In order to generate numerical simulations, we discretize the 2D spatial domain into a grid of size 100 × 100, with Δx = Δy = 0.33. We use an explicit finite-difference method, employing a forward Euler scheme for time integration and a standard second-order central difference scheme for spatial discretization, to solve Eq. (6), by tuning a small perturbation of order 10−5 around the fixed points of the interior of the spirally stable (Q*), viewed at time snap 105 considering time step of Δt = 0.025.

By varying the intra-species conflict rate μ for predators while keeping the resource conversion efficiency fixed at κ = 0.15 (similar to the analysis seen in Fig. 3), distinct spatial structures emerge, reflecting different ecological rhythms for increasing μ values (0.6, 0.72, 0.8, 0.85 and 0.95, the black dots marked in Fig. 4a). Figure 4a highlights the regime where conditions (12a) and (12b) ensure the stable coexistence of all three species at Q*, while Fig. 4b−f depict the sequence of transformations in the spatial distribution of the undefended prey species in a 100 × 100 grid. Similar shifts in pattern are also observed for the defended prey and predator populations as μ varies. The color bar, which varies from blue (low abundance, 0) to red (high abundance, 1), captures the heterogeneity in the distribution of species, revealing a transition from spotted structures to striped and perforated structures as μ increases.

Fig. 4: Spatio-temporal plots by changing μ.
Fig. 4: Spatio-temporal plots by changing μ.The alternative text for this image may have been generated using AI.
Full size image

The diffusion-induced instability region around the interior equilibrium point Q*, satisfying the analytical conditions (12a) and (12b), is shown in the κ-μ parameter space (a) for the system (6). This region highlights the emergence of instability driven by diffusion mechanisms near the interior fixed point Q*. To investigate the corresponding spatio-temporal dynamics, a two-dimensional spatial domain of size 100 × 100 is considered with spatial steps Δx = Δy = 0.33, where the dynamics are governed by the spatial abundance of the undefended prey u. Fixing the inverse of the carrying capacity, controlling the strength of density-dependent limitations, κ = 0.15, five increasing values of the mortality rate due to the intra-species conflict rate among predators, i.e., the strength of density-dependent predator competition μ = 0.6, 0.72, 0.8, 0.89, and 0.95 are selected, each marked by black dots in (a). These values correspond to a range of distinct spatio-temporal patterns observed in the system: b a spot pattern at μ = 0.6, c a mixture of spots and stripes at μ = 0.72, d a stripe pattern at μ = 0.8, e a mixture of stripes and holes at μ = 0.89, and f a hole pattern at μ = 0.95. All pattern snapshots are taken at 5 × 105 time iterations with Δt = 0.025.

For small values of μ (μ = 0.6) with κ = 0.15 and η = 0.005, the system stabilizes in a spotted pattern, where low-density areas surround high-abundance regions of the undefended prey. Fig. 4b shows that this state reflects localized refuges where prey clusters can persist despite predation pressure. As μ increases from μ = 0.6 to μ = 0.72, the system transitions into a mixed state of spots and stripes, where prey aggregates form elongated structures interspersed with isolated patches benefited through a higher extinction trend of the predator kinds, indicating a shift towards partial spatial coherence (see Fig. 4c). Further, μ = 0.8leads to a fully developed striped pattern, where the undefended prey distribute in parallel bands, possibly reflecting the dominance of traveling waves of predator-prey interactions that arise due to spatially varying predation intensity (see Fig. 4d). This pattern likely shows that predator-prey interactions create moving waves, caused by differences in how strongly predators hunt in different areas. At higher values of μ, the prey distribution transitions into mixed stripes and holes (μ = 0.89, as shown in Fig. 4e, and depletion zones begin to emerge within the stripes, highlighting the increasing instability of prey refuges. Finally, when μ = 0.95, the system reaches a perforated pattern, where low-abundance prey regions dominate, leaving isolated pockets of high-density prey populations (see in Fig. 4f). The empty spaces in Fig. 4f indicate regions where prey extinction has occurred, suggesting that intense intra-species conflict among predators has led to a breakdown in prey persistence.

The gradual decline in the undefended prey population from Fig. 4b–f, reveals that as μ increases, intensified competition with predators indirectly leads to increased predation efficiency, leading to a progressive depletion in prey populations. In simpler terms, stronger predator competition makes them hunt more effectively, gradually reducing the number of prey. This transition highlights how eco-evolutionary pressures regulate species coexistence through spatially driven interactions. These spatial configurations are crucial in understanding the eco-evolutionary dynamics of predator-prey interactions. The spotted and striped phases suggest stable coexistence, where prey populations can persist by forming spatial refuges that limit over-exploitation. However, as mortality increases, the emergence of holes and fragmented structures indicates ecological stress, where prey abundance declines in specific regions, potentially altering species coexistence dynamics. Such transitions between spatial states play a fundamental role in regulating biodiversity, species persistence, and the stability of ecological networks in heterogeneous environments.

In Fig. 5, we show how varying κ impacts the spatio-temporal patterns of the undefended prey when μ = 0.8 and η = 0.0024. The parameter space in Fig. 5a highlights the transition from homogeneous to structured patterns, with black dots representing specific values of κ (0.13, 0.17, and 0.2) at η = 0.0024 and μ = 0.8, for which the spatial distributions are analyzed in Fig. 5b−d.

Fig. 5: Spatio-temporal plots by varying κ.
Fig. 5: Spatio-temporal plots by varying κ.The alternative text for this image may have been generated using AI.
Full size image

a Highlights the region in the 2D κ and η parameter space at μ = 0.8, satisfying the diffusion-induced instability conditions (12a) and (12b), while supporting the coexistence of all three species in system (6). For a fixed rate of toxicity, η = 0.0024, we select three values of the controller of density-dependent limitations and per-capita crowding effects κ (0.13, 0.17 and 0.2, marked by black dots in (a) and the resulting changes in the spatio-temporal patterns of the undefended prey u. Different spatio-temporal patterns at (b) κ = 0.13, c κ = 0.17 and d κ = 0.2. The snapshots are taken after 5 × 105 time steps.

When κ = 0.13, the system exhibits a labyrinthine pattern, where the undefended prey maintains a relatively high abundance (indicated by the dominance of the red and yellow regions) in Fig. 5b. As κ increases to 0.17, we observe a decline in the overall abundance of the undefended prey, with more extensive low-density zones (Fig. 5c). Finally, at κ = 0.2, the system evolves into a stripe-like structure, where the levels of undefended prey are further reduced (in Fig. 5d).

We describe the observed decline in prey abundance as κ increases due to reduced resource accessibility and heightened interspecific competition.

The functional response terms in Eq. (6) contain (ρ + κu) and (ρ + κθv) in their denominators, indicating that as κ increases, the availability of resources for the growth of prey decreases. Since κ is regulatory factor that modulates the carrying capacity, higher values of κ amplify density-dependent limitations on prey reproduction. This reduces prey population growth, making them more vulnerable to predation and competition for resources to evolve.

Additionally, the term \(\frac{1}{\kappa }-u-v-w\) appearing in F(u, v, w) and G(u, v, w) suggests that as κ increases, the effective habitat size for prey declines, exacerbating competition among individuals. This contraction in available resources drives a shift from a relatively stable coexisting state (as seen at lower κ) to a regime where the undefended prey struggles to maintain a viable population. The predator-prey dynamics also changes, producing more structured spatial distributions that manifest as stripe-like patterns at higher κ.

From an ecological perspective, the decline in prey abundance as κ increases signifies a transition to a more competitive and resource-limited environment, where species interactions become more spatially constrained. This highlights the importance of κ in regulating the eco-evolutionary dynamics of the system: κ determines whether prey populations persist in the face of increasing competition and predation.

Figure 6a maps the Turing instability regime shaped by the interplay between toxicity (η) and predator conflict (μ), and further shows how, by fixing μ, varying η transforms the spatial patterns formed by the undefended prey. In Fig. 6a, the stability diagram in η − μ parameter space marks the region where spatio-temporal patterns emerge (shaded area). The upper and lower boundaries of this domain indicate the threshold conditions for the pattern formation, beyond which homogeneous states dominate. The three black dots represent selected values of η = 0.002, 0.010 and η = 0.015 when μ = 0.84, and correspond to the distinct spatial structures presented in Fig. 6b–d. The predation efficiency is set at κ = 0.15, ensuring that prey-predator interactions remain consistent between different values of η.

Fig. 6: Spatio-temporal plots by changing η.
Fig. 6: Spatio-temporal plots by changing η.The alternative text for this image may have been generated using AI.
Full size image

a The parameter regime in η − μ plane for κ = 0.15 where system (6) satisfies the diffusion-induced instability conditions (12a) and (12b), ensuring stable coexistence of the three species. The transition in spatial patterns of the undefended prey (u) as the system parameter η increases (b) η = 0.002, c η = 0.01 and d η = 0.015 at μ = 0.84. The snapshots are taken at 5 × 105 time iterations.

The parameter η governs the intensity of interspecific prey competition, directly influencing the spatial distribution of the undefended prey. At low η = 0.002 (Fig. 6b), the spatial pattern is labyrinthine, characterized by winding, maze-like structures. This indicates that the undefended prey species can still maintain a relatively high abundance despite predation pressures. The prevalence of red and yellow regions in the colormap suggests that competition is weak, allowing undefended prey to establish extended spatial domains. The labyrinthine structure emerges due to an optimal balance between local prey reproduction and predation-driven regulation, leading to self-organized heterogeneity in the ecosystem.

As interspecific competition increases to η = 0.01 (Fig. 6c), the spatial organization undergoes a morphological transition. The previously connected labyrinthine structures begin to fragment into a mixture of patch-like domains and isolated filamentary structures. This change signifies that stronger inter-specific competition restricts the spatial expansion of the undefended prey, forcing it into smaller, localized clusters. The reduced dominance of high-abundance regions (red and yellow) in the colormap indicates that the species is experiencing greater spatial constraints. In particular, the emergence of thin, maze-like corridors suggests the formation of localized ecological niches where undefended prey persists in dynamically shifting regions. This transition marks the onset of spatial segregation, where different species struggle for dominance within defined territories.

At a higher induction of η = 0.015 (Fig. 6d), the spatial structure further disintegrates into a spotted pattern dominated by isolated prey clusters. The blue and cyan regions are now predominant, signifying a sharp reduction in overall prey abundance. The transition from labyrinthine to fragmented patches and ultimately to isolated spots follows a well-known Turing instability mechanism, where increased competition destabilizes large-scale spatial structures and favors smaller, disconnected domains. The spotted pattern suggests that undefended prey can only persist in localized refuges, as competitive pressures have significantly reduced its spatial reach. The low connectivity between clusters further weakens the species’ resilience against predation, increasing the likelihood of local extinctions.

The progressive transformation of spatial patterns with increasing η underscores its critical role in regulating species coexistence and spatial self-organization. When competition is weak (η ≈ 0), the ecosystem maintains a high degree of spatial heterogeneity, allowing undefended prey to thrive despite predation. As η increases, competition intensifies, restricting prey expansion and leading to territorial fragmentation. Beyond a critical threshold, the system exhibits self-organized spatial segregation, where species are confined to isolated patches, reducing their ability to recover from stochastic perturbations.

This transition is particularly important in eco-evolutionary dynamics, as it highlights the trade-off between competition and spatial persistence. While competition is necessary for structuring species interactions, excessive interspecific competition drives species toward spatial collapse. The results presented in Fig. 6 reveal a nonlinear response of spatial structures to competition intensity, demonstrating emergent phenomena such as Turing-like pattern transitions from labyrinthine to spotted states, spatial refugia formation where prey species can persist in isolated clusters despite high predation and competition, and network-like fragmentation, indicating a loss of connectivity among prey-dominated regions as η increases.

Overall, Fig. 6 underscores how competition and predation jointly shape biodiversity patterns, influencing species survival and spatial dominance in eco-evolutionary systems. We further emphasize a complementary numerical experiment in which the mobility of both prey species is set to zero, while only the predators are allowed to move. Specifically, we discuss the results for Du = Dv = 0, and Dw = 1 (see Supplementary Note 2), and the spatial distribution of the evolutionary species are shown in Fig. S1.

Discussion

This study presents a comprehensive analysis of the eco-evolutionary dynamics of a tri-trophic predator-prey system incorporating diffusive movement and inter-specific competition. By systematically varying the ecological parameter κ (which controls resource availability and reproductive opportunities), the evolutionary parameter η (which governs toxicity levels of defended prey), and the intra-species competition among predators μ, we have uncovered a diverse range of self-organized spatio-temporal patterns. The emergence of these patterns is directly linked to the stability conditions of different equilibrium states, as outlined in our analytical investigations. The formation and transitions of spatial patterns provide deep insights into the underlying mechanisms that govern species persistence, coexistence, and extinction in structured ecosystems.

The results demonstrate that κ plays a fundamental role in shaping the spatial organization of species. When κ is low, resource availability is high, allowing prey populations to expand and form labyrinthine or spotted spatial structures. As κ increases, competition intensifies, leading to a gradual decline in prey abundance and the fragmentation of their spatial distributions into stripe-like patterns. This finding aligns with real-world ecological scenarios where habitat fragmentation due to environmental stressors reduces population connectivity, thereby altering predator-prey interactions. The observed spatial transitions suggest that increasing resource limitation can force populations into dynamic equilibrium states, where dispersal and local competition dictate survival probabilities.

In contrast, η, which regulates the level of toxicity in defended prey, introduces an evolutionary aspect into the system. At low η, defended prey coexist with undefended prey and predators in a well-mixed environment, with oscillatory predator-prey cycles dominating population dynamics. However, as η increases, the system undergoes spatial segregation, with defended prey forming isolated patches that deter predators. This outcome is ecologically significant, as it resembles real-world predator avoidance mechanisms, such as aposematic (warning) coloration and toxin accumulation, which influence predator behavior and prey survival. When η exceeds a critical threshold, undefended prey face extinction, and defended prey become the dominant species, highlighting an evolutionarily driven shift in species dominance.

The intra-species conflict among predators (μ) introduces another layer of complexity, influencing the predator-prey interactions through top-down regulation. The simulations reveal that as μ increases, predator competition reduces their effective hunting efficiency, allowing prey to persist in structured spatial domains. At low μ, predation pressure is high, leading to prey suppression and the emergence of spatial refuges where prey populations are patchily distributed. As μ increases, prey abundance rises, but their spatial organization transitions from structured patterns to perforated states, where high predation mortality results in scattered prey clusters. This transition suggests that predator competition can indirectly stabilize prey populations by modulating predation intensity, an observation that aligns with empirical findings in multi-predator ecosystems where inter-predator aggression leads to prey release.

The qualitative outcomes of the proposed framework parallel a wide range of ecological observations and also provide some testable predictions. For instance, the oscillatory coexistence between defended prey and predators at low toxicity reflects predator–prey cycles mediated by toxin-based defenses in amphibians (e.g., dendrobatid frogs) and microbial systems (e.g., toxin-producing Pseudomonas biofilms)23. At high levels of toxicity, the model predicts predator extinction accompanied by stabilization of prey populations, a phenomenon consistent with trophic downgrading in marine food webs where chemical defenses restructure trophic interactions61. Furthermore, the diffusion-driven emergence of spotted, striped, and labyrinthine patterns in the simulations resemble the formation of spatial refugia observed in vegetation landscapes and coral reef ecosystems, where prey persist in clusters that reduce the risk of over-exploitation.

In addition to reproducing known phenomena, the model yields several other predictions. First, it identifies toxicity thresholds in the defended prey (η) that determine whether predators persist, collapse, or coexist, thereby suggesting measurable trait boundaries in empirical systems. Second, it shows that habitat fragmentation, encoded through the carrying-capacity parameter κ, can trigger sharp transitions in coexistence regimes, analogous to ecological shifts under resource limitation or anthropogenic disturbance. Finally, it reveals that intra-specific predator competition (μ) can indirectly stabilize prey by promoting the formation of self-organized refugia, highlighting a top-down regulatory effect that has not been emphasized in previous theory. These insights strengthen the biological relevance of the framework and suggest clear avenues for experimental validation in a microcosm systems (e.g., algal–rotifer assemblages) as well as field communities.

Conclusions

The emergence of distinct spatio-temporal patterns and perforated structures demonstrates that self-organization is a key factor in species co-existence. For example, the transition from spotted to striped patterns, as observed in the parameter sweeps, suggests that environmental constraints and species interactions work together to regulate spatial distributions. These findings parallel Turing-type pattern formation in natural ecosystems, where diffusion-driven instability leads to the spontaneous emergence of ordered structures, such as the patchiness observed in vegetation landscapes and predator-prey distributions in coral reef ecosystems.

Furthermore, the study highlights the existence of spatial refugia, where prey populations persist in clusters despite strong predation pressure. These refugia are particularly evident in high η and μ regimes, where prey establish stable pockets of abundance that act as buffers against overexploitation. Such mechanisms are well documented in ecological studies, where prey use spatial heterogeneity to avoid extinction, either by migrating to less accessible regions or by exploiting microhabitats that predators cannot efficiently navigate.

The results of this study have significant implications for understanding how interactions between species influence biodiversity in ecosystems with spatial structure. The interplay between diffusion, competition, and predation not only determines population persistence but also influences the stability of ecological communities. The results suggest that competition-driven spatial fragmentation may act as a precursor to species extinction, emphasizing the need to consider spatial structure in conservation strategies. We can also visualize this analogy in cancer, with prey being considered as cancer cells and predators as immune cells, and with cancer cells hiding their distinctive markers, (i.e., antigens) to evade predation. This analogy highlights how immune evasion drives tumor survival and progression.

Future work should explore how external environmental perturbations, such as climate change and habitat destruction, affect these eco-evolutionary dynamics. Moreover, the diffusion coefficients can significantly influence the spatial dynamics in reaction-diffusion systems, particularly in determining how the system’s behavior will change in slow versus high diffusion, can be further studied. In addition, the choice of the diffusion-reaction ratio, (i.e., the Damköhler number) plays a crucial role in shaping the emergent patterns and can serve as an important parameter for future investigations. Extending the model to include adaptive predator strategies or higher-order interactions could provide deeper insights into the mechanisms driving long-term ecological stability. Experimental validation of these spatio-temporal patterns, particularly in microbial predator-prey systems or controlled laboratory ecosystems, would further reinforce the theoretical predictions derived from the model.

In conclusion, this study highlights the intricate balance between ecological and evolutionary forces in shaping spatially structured populations. The observed transitions between different pattern states underscore the importance of considering spatial self-organization in ecological modeling, with direct implications for understanding species persistence, community stability, and ecosystem resilience in the face of environmental change.