Introduction

Complex physical, biological, and social systems often exhibit higher-order interdependencies that cannot be reduced to pairwise interactions between their components1,2. Recent studies suggest that higher-order organisation is not the exception but the norm, providing various mechanisms for its emergence3,4,5,6. Modelling studies have revealed that higher-order interactions (HOIs) underlie collective activities such as bistability, hysteresis, and ‘explosive’ phase transitions associated with abrupt discontinuities in order parameters4,7,8,9,10,11.

HOIs are particularly important for the functioning of biological and artificial neural systems. For instance, they shape the collective activity of biological neurons12,13, being directly responsible for their inherent sparsity5,13,14,15 and possibly underlying critical dynamics16,17. HOIs have also been shown to enhance the computational capacity of artificial recurrent neural networks18,19. More specifically, ‘dense associative memories’ with extended memory capacity20,21,22,23 are realised by specific non-linear activation functions, which effectively incorporate HOIs. These non-linear functions are related to attention mechanisms of transformer neural networks24 and the energy landscape of diffusion models25,26, leading to the conjecture that HOIs underlie the success of these state-of-the-art deep learning models.

Despite their importance, existent studies of HOIs face significant computational challenges. Analytically tractable models that incorporate HOIs typically limit interactions to a single order (e.g., \(p\)-spin models22,27,28). Otherwise, attempting to represent diverse HOIs exhaustively results in a combinatorial explosion29. This issue is pervasive, restricting investigations of high-order interaction models—such as contagion9, Ising19, or Kuramoto30 models—to highly homogeneous scenarios3,16 or to models of relatively low-order9,11,31. While attempts have been made to model all orders of HOIs and perform theoretical analyses20,21,22,23,32,33,34,35,36,37, it is currently unclear how to construct parsimonious models to address the diverse effects of HOIs in a principled manner.

To address this challenge, here we employ an extension of the maximum entropy principle to capture HOIs through the deformation of the space of statistical models. When applied to neural networks, our approach generalises classical neural network models to yield a family of curved neural networks that effectively incorporate HOIs of all orders. The resulting models have rich connections with the literature on the statistical physics of neural networks21,22,27,34. These features enable the exploration of various aspects of HOIs using techniques including mean-field approximations, quenched disorder analyses, and path integrals.

Our analyses reveal how relatively simple curved neural networks exhibit some of the hallmark characteristics of higher-order phenomena, such as explosive phase transitions, arising both in mean-field models and in more complex transitions to spin-glass states. These phenomena are driven by a self-regulated annealing process, which accelerates memory retrieval through positive feedback between energy and an ‘effective’ temperature—a perspective that can also explain memory-retrieval dynamics in other modern artificial networks. Furthermore, we show—both analytically and experimentally—that this mechanism can lead to an increase in the memory capacity or robustness of memory retrieval in these neural networks. Overall, the core contributions of this work are (i) the development of a parsimonious neural network model based on the maximum entropy principle that captures interactions of all orders, (ii) the discovery of a self-regulated annealing mechanism that can drive explosive phase transitions, and (iii) the demonstration of enhanced memory capacity resulting from this mechanism.

Results

High-order interactions in curved manifolds

The maximum entropy principle (MEP) is a general modelling framework based on the principle of adopting the model with maximal entropy compatible with a given set of observations, under the rationale that one should not assume any structure beyond what is specified by the assumptions or features selected from the data38,39. The traditional formulation of the MEP is based on Shannon’s entropy40, and the resulting models correspond to Boltzmann distributions of the form \(p({{\boldsymbol{x}}})=\exp \left({\sum }_{a}{\theta }_{a}{f}_{a}({{\boldsymbol{x}}})-\varphi \right)\), where x = (x1, …, xn), φ is a normalising potential, and θa are parameters constraining the average value of observables \(\langle \, {f}_{a}({{\boldsymbol{x}}}) \rangle\). While observables are often set to low orders (e.g. fi(x) = xi, fij(x) = xixj, corresponding to first and second order statistics), higher-order interdependencies can be included by considering observables of the type fI(x) = ∏i∈Ixi, where I is a set of indices of order k = ∣I∣. Unfortunately, an exhaustive description of interactions up to order k ≫ 1 becomes unfeasible in practice due to an exponential number of terms (for more details on the MEP, see Supplementary Note 1).

The MEP can be expanded to include other entropy functionals such as Tsallis’41 and Rényi’s42. Concretely, maximising the Rényi entropy (with the scaling parameter γ ≥ −1)43

$${H}_{\gamma }(p)=-\frac{1}{\gamma }\ln {\sum}_{{{\boldsymbol{x}}}}p{({{\boldsymbol{x}}})}^{1+\gamma }$$
(1)

while constraining \(\langle \, {f}_{a}({{\boldsymbol{x}}})\rangle\) (i.e., the expectation of features by p(x)) results in models of the form (see Supplementary Note 1):

$${p}_{\gamma }({{\boldsymbol{x}}})=\exp (-{\varphi }_{\gamma })\left[1+\gamma \beta {\sum}_{a}{\theta }_{a}{f}_{a}({{\boldsymbol{x}}})\right]_{+}^{1/\gamma },$$
(2)

where φγ is a normalising constant given by

$${\varphi }_{\gamma }=\ln {\sum}_{{{\boldsymbol{x}}}} \left[1+\gamma \beta {\sum}_{a}{\theta }_{a}{f}_{a}({{\boldsymbol{x}}})\right]_{\!{+}}^{\!{1/\gamma }}.$$
(3)

Above, the square bracket operator sets negative values to zero, \({\left[x\right]}_{+}=\max \{0,x\}\). We refer to distributions following (2) as the deformed exponential family, which maximises both Rényi and Tsallis entropies44,45. When γ → 0, Rényi’s entropy tends to Shannon’s and (2) to the standard exponential family42.

A fundamental insight explored in this study is that higher-order interdependencies can be efficiently captured by deformed exponential family distributions46,47. Starting from a standard Shannon’s MEP model with low-order interactions, it can be shown that varying γ in (2) results in a deformation of the statistical manifold which, in turn, enhances the capability of pγ(x) to account for higher-order interdependencies. In effect, the consequence of deformation can be investigated by rewriting (2) via Taylor expansion of the exponent

$${p}_{\gamma }({{\boldsymbol{x}}})=\exp \left({\sum }_{k=1}^{\infty }\frac{-1}{k\gamma }{\left(-\gamma \beta {\sum}_{a}{\theta }_{a}{f}_{a}({{\boldsymbol{x}}})\right)}^{k}-{\varphi }_{\gamma }\right),$$
(4)

which is valid for the case 1 + γ∑aθafa(x) > 0, and otherwise pγ(x) = 0. This shows that the deformed manifold contains interactions of all orders even if fa(x) is restricted to lower orders while establishing a specific dependency structure across the orders, thereby avoiding a combinatorial explosion of the number of required parameters. The deformation resulting from the maximisation of a non-Shannon entropy has been shown to reflect a curvature of the space of possible models in information geometry42,45,48,49. This leads to a particular foliation of the space of possible models50 (an ‘onion-like’ manifold structure, Fig. 1), which has properties that allow to re-derive the MEP from fundamental geometric properties—for technical details, see Supplementary Note 1.

Fig. 1: Higher-order decomposition resulting from the foliation of a statistical manifold.
figure 1

Illustration of a family of standard MEP models (right) and its deformed counterpart (bottom left). The space of MEP distributions with constraints of different orders constitute nested sub-manifolds29, giving rise to a hierarchy of sub-families of models of the form \({{{\mathcal{E}}}}_{k}^{\gamma }=\{{p}_{\gamma }^{(k)}({{\boldsymbol{x}}})={e}^{-{\varphi }_{\gamma }}{\left[1-\gamma \beta {E}_{k}({{\boldsymbol{x}}})\right]}_{+}^{1/\gamma }\}\) such that \({{{\mathcal{E}}}}_{1}^{\gamma }\subset {{{\mathcal{E}}}}_{2}^{\gamma }\subset \cdots \subset {{{\mathcal{E}}}}_{n}^{\gamma }\)42. The foliation depends on the curvature γ, and in general \({{{\mathcal{E}}}}_{k}^{\gamma }\ne {{{\mathcal{E}}}}_{k}^{0}\) but rather \({{{\mathcal{E}}}}_{k}^{\gamma }\cap {{{\mathcal{E}}}}_{r}^{0}\ne\) ∅ for k < r. For small values of ∣γ∣, it is possible to neglect higher-order terms in (4), and therefore certain subsets of \({{{\mathcal{E}}}}_{k}^{\gamma }\) effectively approximate \({{{\mathcal{E}}}}_{r}^{0}\).

Full size image

Curved neural networks

Several well-known neural network models adhere to the MEP, such as Ising-like models51 and Boltzmann machines52. Interestingly, these models can encode patterns in their weights in the form of ‘associative memories’ as in Nakano-Amari-Hopfield networks53,54,55, being amenable for investigations using tools from equilibrium and nonequilibrium statistical physics literature56,57,58,59. Following the principles laid down in the previous section, we now introduce a family of recurrent neural networks that we call curved neural networks.

For this purpose, let us consider N binary variables x1, …, xN taking values in {1, −1} following a joint probability distribution

$${p}_{\gamma }({{\boldsymbol{x}}})=\exp (-{\varphi }_{\gamma }){\left[1-\gamma \beta E({{\boldsymbol{x}}})\right]}_{+}^{1/\gamma },$$
(5)

where φγ is a normalising constant. Above, we call E(x) and β the (stochastic) energy function (i.e., Hamiltonian) and the inverse temperature, due to their similarity with the Gibbs distribution in statistical physics when γ → 0. Note that, unlike exponential families, these models do not exhibit energy invariance under constant shifts. However, as demonstrated in Ref. 41, deformed exponential models can be related to energy-invariant models by rescaling their temperature, which can be seen as maximising entropy with respect to escort statistics rather than the original natural statistics.

Neural network models are typically defined by considering pγ(x) as defined in (5) with an energy function of the form

$$E({{\boldsymbol{x}}})=-{\sum }_{i=1}^{N}{H}_{i}{x}_{i}-\frac{1}{N}{\sum}_{i < j}{J}_{ij}{x}_{i}{x}_{j},$$
(6)

where Jij is the coupling strength between neurons xi and xj, and Hi are bias terms. In the limit γ → 0, p0(x) recovers the Ising model. Emulating classical associative memories, the weights Jij can be made to encode a collection of M neural patterns \({{{\boldsymbol{\xi }}}}^{a}=\{{\xi }_{1}^{a},\ldots {\xi }_{N}^{a}\}\), \({\xi }_{1}^{a}=\pm \! 1\) and a = 1, …, M by using the well-known Hebbian rule55,56

$${J}_{ij}=J{\sum }_{a=1}^{M}{\xi }_{i}^{a}{\xi }_{j}^{a},$$
(7)

where J is a scaling parameter.

Before proceeding with our main analysis, one can gain insights into the effect of the curvature γ from the dynamics of a recurrent neural network that behaves as a sampler of the equilibrium distribution described by (5). For this, we adapt the classic Glauber dynamics to curved neural networks (see Supplementary Note 2) to obtain

$$p({x}_{i}| {{{\boldsymbol{x}}}}_{\backslash i})={\left(1+{\left[1-\gamma {\beta }^{{\prime} }({{\boldsymbol{x}}})\Delta E({{\boldsymbol{x}}})\right]}_{\!\!+}^{\!{1}/\gamma }\right)}^{-1},$$
(8)

where x\i denotes the state of all neurons except xi, \(\Delta E({{\boldsymbol{x}}})=2{x}_{i}({H}_{i}+\frac{1}{N}{\sum }_{j}\,{J}_{ij}{x}_{j})\) is the energy difference associated with detailed balance, and \({\beta }^{{\prime} }({{\boldsymbol{x}}})\) is an effective inverse temperature given by

$${\beta }^{{\prime} }({{\boldsymbol{x}}})=\frac{\beta }{{\left[1-\gamma \beta E({{\boldsymbol{x}}})\right]}_{+}}.$$
(9)

Again, γ → 0 recovers the classic Glauber dynamics and \({\beta }^{{\prime} }({{\boldsymbol{x}}})=\beta\). Thus, the curvature affects the dynamics through the deformed nonlinear activation function (8) and the state-dependent effective temperature \({\beta }^{{\prime} }({{\boldsymbol{x}}})\) (9), with higher \({\beta }^{{\prime} }({{\boldsymbol{x}}})\) inducing lower degrees of randomness in the transitions. The effect of E(x) on \({\beta }^{{\prime} }({{\boldsymbol{x}}})\) depends then on the sign of γ. A negative γ increases \({\beta }^{{\prime} }({{\boldsymbol{x}}})\) during relaxation, reducing the stochasticity of the dynamics and accelerating convergence to a low-energy state. This, in turn, raises \({\beta }^{{\prime} }\), creating a positive feedback loop between energy and effective temperature. The effect is similar to simulated annealing, but the coupling of the energy and effective inverse temperature lets the annealing scheduling self-regulate to accelerate convergence. In contrast, positive γ decelerates the dynamics through negative feedback. Such accelerating or decelerating dynamics underlie non-trivial complex collective behaviours of the curved neural networks, which will be examined in the subsequent sections.

Mean-field behaviour of curved associative-memory networks

As with regular associative memories58, one can solve the behaviour of curved associative-memory networks through mean-field methods in the thermodynamic limit N → ∞ (Supplementary Note 3). Here the energy is extensive, meaning that it scales with the system’s size N. To ensure the deformation parameter remains independent of system properties such as size or temperature, we scale it as follows:

$$\gamma=\frac{{\gamma }^{{\prime} }}{N\beta }.$$
(10)

Under this condition, we calculate the normalising potential φγ by introducing a delta integral and calculating a saddle-node solution, resulting in a set of order parameters m = {m1, …, mM}, \({m}_{a}=\frac{1}{N}{\sum }_{i}{\xi }_{i}^{a}\langle {x}_{i}\rangle\) in the limit of size N → ∞. This calculation assumes 1 − γβE(x) > 0 so that \({[]}_{+}\) operators can be omitted and φγ is differentiable. The solution results in (for Hi = 0):

$${\varphi }_{\gamma }= N\frac{\beta }{{\gamma }^{{\prime} }}\ln \frac{{\beta }^{{\prime} }}{\beta }-{\sum }_{a=1}^{M}{\beta }^{{\prime} }NJ{m}_{a}^{2}\\ +{\sum }_{i=1}^{N}\ln \left(2\cosh \left({\beta }^{{\prime} }J{\sum }_{a=1}^{M}{\xi }_{i}^{a}{m}_{a}\right)\right),$$
(11)

where \({\beta }^{{\prime} }\) is given by

$${\beta }^{{\prime} }=\frac{\beta }{1+{\gamma }^{{\prime} }\frac{1}{2}J{\sum }_{a}{m}_{a}^{2}},$$
(12)

and the values of the mean-field variables ma are found from the following self-consistent equations:

$${m}_{a}={\sum }_{i=1}^{N}\frac{{\xi }_{i}^{a}}{N}\tanh \left({\beta }^{{\prime} }J{\sum }_{b=1}^{M}{\xi }_{i}^{b}{m}_{b}\right).$$
(13)

Similarly, using a generating functional approach59, we use the Glauber rule in (8) to derive a dynamical mean-field given by path integral methods (see Supplementary Note 4). This yields

$${\dot{m}}_{a}=-{m}_{a}+{\sum }_{i=1}^{N}\frac{{\xi }_{i}^{a}}{N}\tanh \left({\beta }^{{\prime} }J{\sum }_{b=1}^{M}{\xi }_{i}^{b}{m}_{b}\right),$$
(14)

where \({\beta }^{{\prime} }\) is defined as in (12) for each m. Note that in large systems, we recover the classical nonlinear activation function, and the deformation affects the dynamics only through the effective temperature \({\beta }^{{\prime} }\).

Explosive phase transitions

To illustrate these findings, let us focus on a neural network with a single associative pattern (M = 1), which is similar to the Mattis model60 and equivalent to a homogeneous mean-field Ising model61 (with energy \(E({{\boldsymbol{x}}})=-\frac{1}{N}J{\sum }_{i < j}{x}_{i}{x}_{j}\)) by changing a variable xi ← ξixi. Rewriting (13), we find that a one-pattern curved neural network follows a mean-field model given by

$$m=\tanh \left({\beta }^{{\prime} }Jm\right),$$
(15)
$${\beta }^{{\prime} }=\frac{\beta }{1+{\gamma }^{{\prime} }\frac{1}{2}J{m}^{2}}.$$
(16)

This result generalises the well-known Ising mean-field solution \(m=\tanh \left(\beta Jm\right)\), which is recovered for γ = 0.

By evaluating these equations, one finds that the model exhibits the usual order-disorder phase transition for positive and small negative values of \({\gamma }^{{\prime} }\) (Fig. 2a top). However, for large negative values of \({\gamma }^{{\prime} }\), a different behaviour emerges: an explosive phase transition8 that displays hysteresis due to HOIs (Fig. 2a bottom). The resulting phase diagram (Fig. 2b) closely resembles phase transitions in higher-order contagion models9,11 and higher-order synchronisation observed in Kuramoto models30.

Fig. 2: Explosive phase transitions in curved neural networks.
figure 2

a Phase transitions of the curved neural network with one associative memory, for J = 1 and values of \({\gamma }^{{\prime} }=-\!0.5\) (top, displaying a second-order phase transition) and \({\gamma }^{{\prime} }=-\!1.5\) (bottom, displaying an explosive phase transition). Solid lines represent the stable fixed points, and dotted lines correspond to unstable fixed points. b Phase diagram of the system. The areas indicated by P and M refer to the usual paramagnetic (disordered) and magnetic (ordered) phases, respectively. The area indicated by Exp represents a phase where ordered and disordered states coexist in an explosive phase transition characterised by a hysteresis loop. (c) Solutions of (15)-(16) for \({\beta }^{{\prime} },m,\beta\) (black line) for \({\gamma }^{{\prime} }=-\!1.2\), and projections to the plane m = 0, β = 0 and \({\beta }^{{\prime} }=0\), obtaining respectively the relation between \(\beta,{\beta }^{{\prime} }\) and solutions of the flat and the deformed models respectively (grey lines). (d) Mean-field dynamics of the single-pattern neural network for β = 1.001 (near criticality from the ordered phase) for some values of \({\gamma }^{{\prime} }\) in [ −1.5, 0]. For large negative \({\gamma }^{{\prime} }\) the dynamics ‘explodes’, with m (top) and \({\beta }^{{\prime} }\) (bottom) converging abruptly.

Full size image

One can intuitively interpret the effect of the deformation parameter \({\gamma }^{{\prime} }\) by noticing that, for a fixed \({\beta }^{{\prime} }\), m is the solution of a function of \({\beta }^{{\prime} }\). For \({\gamma }^{{\prime} }=0\), this results in the mean-field behaviour of the regular exponential model, which assigns a value of m to each inverse temperature \(\beta={\beta }^{{\prime} }\). In the case of the deformed model, the possible pairs of solutions \((m,{\beta }^{{\prime} })\) are the same, but their mapping to the inverse temperatures β changes. Namely, this deformation can be interpreted as a stretching (or contraction) of the effective temperature, which maps each pair \((m,{\beta }^{{\prime} })\) to an inverse temperature \(\beta={\beta }^{{\prime} }(1+\frac{1}{2}{\gamma }^{{\prime} }J{m}^{2})\) according to (16). Thus, one can obtain the mean-field solutions of the deformed patterns as mappings of the solutions of the original model. This is illustrated in Fig. 2c, where the solution of \({\beta }^{{\prime} },m,\beta\) is projected to the planes β = 0 and \({\beta }^{{\prime} }=0\), obtaining the solutions for the flat (\({\gamma }^{{\prime} }=0\)) and the deformed (\({\gamma }^{{\prime} }=-1.2\)) models respectively.

In order to gain a deeper understanding of the explosive nature of this phase transition, we study the dynamics of the single-pattern neural network. By rewriting (14) for M = 1, and under the change of variables mentioned above to remove ξ, the dynamical mean-field equation of the system reduces to

$$\dot{m}=-m+\tanh \left({\beta }^{{\prime} }Jm\right),$$
(17)

where \({\beta }^{{\prime} }\) is calculated as in (16). Simulations of the dynamical mean-field equations for values of β just above the critical point are depicted in Fig. 2d. Trajectories with strongly negative \({\gamma }^{{\prime} }\) saturate earlier than smaller negative \({\gamma }^{{\prime} }\), confirming accelerated convergence. During this process, the effective inverse temperature \({\beta }^{{\prime} }\) rapidly increases until it saturates, creating a positive feedback loop between \({\beta }^{{\prime} }\) and m that gives rise to the explosive nature of the phase transition. This positive loop occurs only if \({\gamma }^{{\prime} }\) is negative; otherwise, negative feedback simply makes the convergence of m slower.

Overlaps between memory basins of attraction

A key property of associative-memory networks is their ability to retrieve patterns in different contexts. In the case of one-pattern associative-memory networks, the energy function \(E({{\boldsymbol{x}}})=-\frac{J}{N}{\sum }_{i < j}{x}_{i}{\xi }_{i}{\xi }_{j}{x}_{j}\) is a quadratic function with two minima at x = ± ξ, which configure global attractors. Instead, a two-pattern associative-memory network has an energy function with four minima (if sufficiently separated), but their attraction basins can overlap when the patterns are correlated.

To study the degree of the overlap between pairs of patterns, we analyse solutions of (13) for a network with two patterns with correlation \(\langle {\xi }_{i}^{1}{\xi }_{i}^{2}\rangle=C\) (see Supplementary Note 3.3 for details). In this scenario, the system is described by two mean-field patterns:

$${m}_{a}= \frac{1}{2}(1+C)\tanh \left({\beta }^{{\prime} }J({m}_{1}+{m}_{2})\right)\\ +w\frac{1}{2}(1-C)\tanh \left({\beta }^{{\prime} }J({m}_{1}-{m}_{2})\right)$$
(18)

with w = 3 − 2a = ± 1 for a = 1, 2, and

$${\beta }^{{\prime} }=\frac{\beta }{1+\gamma \frac{1}{2}J({m}_{1}^{2}+{m}_{2}^{2})}.$$
(19)

Figure 3 shows how the hysteresis effect and explosive phase transitions persist in the case of two patterns for C = 0.2 with negative \({\gamma }^{{\prime} }\). This example shows two consecutive, overlapping explosive bifurcations (going from 1 to 2, and then to 4 fixed points), creating a hysteresis involving 7 fixed points within a more compressed parameter range of β than the classical case. Consequently, the memory-retrieval region for the four embedded memories expands. These results illustrate complex hysteresis cycles as well as an increased memory capacity for finite temperatures by negative values of \({\gamma }^{{\prime} }\). This enhanced capability for memory retrieval is further investigated through the replica analyses in the next section.

Fig. 3: Interaction between two encoded memories.
figure 3

a Values of φγ for different mean-field values m1, m2, indicating the attractor structure of the network for different values of β with J = 1, C = 0.2 for \({\gamma }^{{\prime} }=0\) (top row) and \({\gamma }^{{\prime} }=-1.2\) (bottom row). b Bifurcations of the order parameters m1, m2. For \({\gamma }^{{\prime} }=0\) we observe an attractor bifurcating into two and then into four. For \({\gamma }^{{\prime} }=-1.2\), we observe the same sequence, but with a coexistence hysteresis regime in which 7 attractors are possible.

Full size image

Memory retrieval with an extensive number of patterns

Next, we investigate how the deformation related to γ impacts the memory-storage capacity of associative memories. In classical associative networks of N neurons, the energy function is defined as \(E({{\boldsymbol{x}}})=-\frac{J}{N}{\sum }_{a=1}^{M}{\sum }_{i < j}{x}_{i}{\xi }_{i}^{a}{\xi }_{j}^{a}{x}_{j}\) with M = αN. As the number of patterns learned by the network increases, the system transitions to a disordered spin-glass state in the thermodynamic limit. Furthermore, one can analytically solve this model62,63,64,65. For example, using the replica-trick method can determine the memory capacity of the system62, and theoretically identify the critical value of α at which memory retrieval becomes impossible—leading to a disordered spin-glass phase. Here, we apply a similar approach to reveal how deformed associative memory networks afford an enhanced memory capacity.

Applying the replica trick in conjunction with the methods outlined in previous sections allows us to solve the system (see Supplementary Note 5). This method entails computing a mean-field variable m corresponding to one of the patterns ξ a and averaging over the others. For simplicity, a pattern with all positive unity values ξ a = (1, 1, …, 1) is considered, which is equivalent to any other single pattern just by a series of sign flip variable changes. The degree of similarity or overlap of this pattern with other patterns in the system introduces a new order parameter q, which contributes to measuring disorder in the system. After introducing the relevant order parameters and solving under a replica-symmetry assumption, the normalising potential is derived as

$${\varphi }_{\gamma }= N\frac{\beta }{{\gamma }^{{\prime} }}\ln \frac{\beta }{{\beta }^{{\prime} }}-N{\beta }^{{\prime} }J{m}^{2}-N\frac{1}{2}\alpha {({\beta }^{{\prime} }J)}^{2}(r+R-2qr)\\ -N\frac{1}{2}\alpha \left(\ln \left(1-{\beta }^{{\prime} }J(1-q)\right)-{\beta }^{{\prime} }J\sqrt{rq}\right)\\ +N\int\,Dz\,\ln \left(2\cosh \left({\beta }^{{\prime} }Jm+{\beta }^{{\prime} }J\sqrt{\alpha r}z\right)\right),$$
(20)

where J is a scaling factor, and the order parameters are defined as

$$m=\int\,Dz\,\tanh \left({\beta }^{{\prime} }Jm+{\beta }^{{\prime} }J\sqrt{\alpha r}z\right),$$
(21)
$$q=\int\,Dz\,{\tanh }^{2}\left({\beta }^{{\prime} }Jm+{\beta }^{{\prime} }J\sqrt{\alpha r}z\right),$$
(22)

with

$$r=\frac{q}{{(1-{\beta }^{{\prime} }J(1-q))}^{2}},\quad R=\frac{{({\beta }^{{\prime} }J)}^{-1}-(1-2q)}{{(1-{\beta }^{{\prime} }J(1-q))}^{2}}.$$
(23)

As in previous cases, the model is governed by an effective temperature

$${\beta }^{{\prime} }=\frac{\beta }{1+{\gamma }^{{\prime} }\frac{1}{2}\left(J{m}^{2}+\alpha J({\beta }^{{\prime} }(R-qr)-1)\right)}.$$
(24)

This solution differs from the models in previous sections by the self-dependence of \({\beta }^{{\prime} }\).

To obtain a phase diagram, we solved (21)-(22) numerically for given \(\alpha,{\beta }^{{\prime} }\) at \({\gamma }^{{\prime} }=0\), and rescaled the inverse temperature as in the previous section to obtain the corresponding values of β for each \({\gamma }^{{\prime} }\). Using the resulting order parameters and calculating the free energy for each \(\alpha,\beta,{\gamma }^{{\prime} }\), we constructed the phase diagram of the system (similarly to regular associative memories58,62) characterised by the following distinct phases (Fig. 4):

  • A paramagnetic phase (P), corresponding to disordered solutions with m = q = 0, where memory-retrieval fails due to the dominance of fluctuations.

  • A ferromagnetic phase (F), corresponding to stable memory-retrieval solutions with m > 0 and q > 0.

  • A spin-glass phase (SG), exhibiting spurious-retrieval solutions with m = 0 and q > 0.

  • A mixed phase (M), where F and SG types of solutions coexist, being the spin-glass solutions a global minimum of the normalising potential φγ.

Fig. 4: Memory capacity is enhanced by geometric deformation.
figure 4

Phase diagram of a curved associative memory with an extensive number of encoded patterns M = αN and J = 1 for (a) different T = 1/β at \({\gamma }^{{\prime} }=0\) (black dashed lines), 0.8, − 0.8 (solid lines), and for (b) different \({\gamma }^{{\prime} }\) at β = 2. F indicates the ferromagnetic (i.e., memory retrieval) phase, SG the spin-glass phase (where saturation makes memory retrieval inviable), M a mixed phase, and P the paramagnetic region. Both in F and M, ferromagnetic and spin-glass solutions coexist, but we differentiate these by calculating respectively whether memory-retrieval or spin-glass solutions are the global minimum of the normalising potential φγ. The dotted lines in (a) near T = 0 indicate the AT lines, below which the replica-symmetric solution is not valid. Increasing \({\gamma }^{{\prime} }\) to larger negative values extends the retrieval phase into larger values of α, indicating an increased memory capacity, while larger positive values reduce the extension of the mixed phase, increasing robustness of memory retrieval.

Full size image

For \({\gamma }^{{\prime} }=0\) (black dashed lines), the phase transition reflects the behaviour of associative memories near saturation58,62. With negative \({\gamma }^{{\prime} }\) (red lines), we observe an expansion of the ferromagnetic and mixed phases, indicating an enhanced memory-storage capacity by the deformation. Conversely, a positive value of \({\gamma }^{{\prime} }\) (yellow lines) decreases the memory capacity but reduces the extent of the mixed phase. In the mixed phase, retrieved memories (m > 0) are represented at a local—but not global—minimum of the normalising potential φγ in (20), indicating a larger probability of observing spurious patterns. Thus, we expect positive values of \({\gamma }^{{\prime} }\) to result in more robust memory retrieval.

The stability of the replica symmetry solution is given by the condition

$${\left(1+{\beta }^{{\prime} }(1-q)\right)}^{2} > \alpha {\beta }^{{\prime} 2}\int\,Dz\,{\cosh }^{-4}{\beta }^{{\prime} }\left(Jm+J\sqrt{\alpha r}z\right),$$
(25)

which is captured by the dotted lines near zero temperature in Fig. 4a. Note that all solutions in Fig. 4b are stable under the replica symmetry assumption.

We complement the analysis from the previous section with an experimental study of a system encoding patterns from an image classification benchmark. The patterns are sourced from the CIFAR-100 dataset, which comprises 60,000 32 × 32 colour images66. To adapt the dataset to binary patterns suitable for storage in an associative memory, we processed each RGB channel by assigning a value of 1 to pixels with values greater than the channel’s median value and  −1 otherwise (Fig. 5a). The resulting array of N = 32 ⋅ 32 ⋅ 3 binary values for each image was assigned to patterns ξ a. Note that associative memories (as well as our theory above) usually assume that patterns are relatively uncorrelated, and specific methods are required to adapt them to correlated patterns67,68. To simplify the problem, we conducted experiments using a selection of 100 images with covariance values smaller than \(10/\sqrt{N}\) (the standard deviation of the covariance values for uncorrelated patterns is \(1/\sqrt{N}\)). We used a random search to select patterns with low correlations: we randomly picked an image and replaced it if its correlation exceeded the threshold, repeating until all correlations were below it.

Fig. 5: Simulation study for the effect of deformation on image encoding.
figure 5

a Examples of CIFAR-100 images (top) and their RGB binarised versions (bottom). Every 32 × 32 × 3 binary RGB pixel value for each image a is assigned to the value of one position of pattern \({\xi }_{i}^{a}\). b, c Mean and variance of pattern retrieval values obtained in experiments, measured by the overlap between the final state of the network and the encoded pattern.

Full size image

We evaluated the memory retrieval capacity of networks with various degrees of curvature γ by encoding different numbers of memories, as described in (7). As a measure of performance, we evaluated the stability of the network by assigning an initial state x = ξ a and calculating the overlap \(o={\sum }_{i}{x}_{i}{\xi }_{i}^{a}\) after T = 30N Glauber updates for β = 2, J = 1. The process was repeated R = 500 times from different initial conditions (different encoded patterns and different initial states) to estimate the value of m in (21). Experimental outcomes confirm our theoretical results, revealing that memory capacity increases with negative values of \({\gamma }^{{\prime} }\), while positive values reduce the memory capacity (Fig. 5b), but reduce the extent and magnitude of the high variability region in pattern retrieval (Fig. 5c), which is consistent with the reduction of the mixed phase. Note that the resulting memory capacity of the system observed in our experiments (i.e., the value of α at which the transition happens) is diminished due to the presence of correlations among some of the memorised patterns.

Finally, we investigated transitions near the spin-glass phase boundaries. First, we note that, for J → 0 and α = J−2, the model in (21)-(22) converges to (see Supplementary Note 5)

$$q=\int\,Dz\,{\tanh }^{2}\left({\beta }^{{\prime} }\sqrt{q}z\right),$$
(26)
$${\beta }^{{\prime} }=\frac{\beta }{1+\frac{1}{2}{\gamma }^{{\prime} }{\beta }^{{\prime} }(1-{q}^{2})},$$
(27)

which at γ = 0 recovers the well-known Sherrington-Kirkpatric model69 (see Supplementary Note 6). While in the classical case, a phase transition occurs from a paramagnetic to a spin-glass phase, the curvature effect of \({\gamma }^{{\prime} }\ne 0\) modifies the nature of this transition. For small values of \({\gamma }^{{\prime} }\), the system exhibits a continuous phase transition akin to the Sherrington–Kirkpatrick spin-glass, where \(\frac{dq}{d\beta }\) shows a cusp (Fig. 6a). However, for \({\gamma }^{{\prime} }=-\!1\) the phase transition becomes second-order, displaying a divergence of \(\frac{dq}{d\beta }\) at the critical point (Fig. 6b). Moreover, increasing the magnitude of negative \({\gamma }^{{\prime} }\) leads to a first-order phase transition with hysteresis (Fig. 6c), resembling the explosive phase transition observed in the single-pattern associative-memory network. This hybrid phase transition combines the typical critical divergence of a second-order phase transition with a genuine discontinuity, similar to ‘type V’ explosive phase transitions8.

Fig. 6: Explosive spin glasses.
figure 6

Phase transitions for order parameter q for replica-symmetric disordered spin models displaying (a) a cusp phase transition for \({\gamma }^{{\prime} }=-\!0.5\), (b) a second-order phase transition for \({\gamma }^{{\prime} }=-\!1.0\) and (c) an explosive phase transition for \({\gamma }^{{\prime} }=-\!1.2\). d Phase diagram of the explosive spin glass, displaying a paramagnetic (P), spin-glass (SG) and an explosive phase (Exp).

Full size image

We analytically calculated the properties of these phase transitions (see Supplementary Note 6). By computing the solution at \({\gamma }^{{\prime} }=0\) and rescaling \({\beta }^{{\prime} }\), we determined that the critical point is located at \({\beta }_{c}=1+\frac{1}{2}{\gamma }^{{\prime} }\) (consistent with Fig. 6a–c). The slope of the order parameter around the critical point is, for \({\gamma }^{{\prime} }\le -\!1\), equal to \({(1+{\gamma }^{{\prime} })}^{-1}\), indicating the onset of a second-order phase transition as depicted in Fig. 6b. The resulting phase diagram of the curved Sherrington-Kirkpatrick model is shown in Fig. 6d.

Comparison with other dense associative memory models

Although our primary objective is to develop a parsimonious model of HOIs to explain higher-order phenomena, our framework can also be used to explain the behaviour of modern networks with HOIs, including the recently proposed relativistic Hopfield model32,33,34 and dense associative memories20,21. For this, let us consider the energy \({{\mathcal{F}}}[E]\) of the exponential family distribution \(p({{\boldsymbol{x}}}) \sim {e}^{-\beta {{\mathcal{F}}}[E]}\) given by the nonlinear transformation (denoted by \({{\mathcal{F}}}\)) of the classical energy E(x). The deformed exponential models in this study correspond to \({{\mathcal{F}}}[E]=-\frac{N}{{\gamma }^{{\prime} }}\ln (1-{\gamma }^{{\prime} }E/N)\), while the relativistic model corresponds to \({{\mathcal{F}}}[E]=-\frac{N}{{\gamma }^{{\prime} }}\sqrt{1-{\gamma }^{{\prime} }E/N}\). For the deformed exponential, the term \({{\mathcal{F}}}[E]\) can be expanded as

$${{\mathcal{F}}}[E]=E+\frac{{\gamma }^{{\prime} }}{2N}{E}^{2}+\frac{{\gamma }^{{\prime} 2}}{3{N}^{2}}{E}^{3}+\ldots$$
(28)

When E depends on the quadratic Mattis magnetisation (i.e., \(E=-{\sum }_{a}\frac{1}{N}{\left({\sum }_{i}{\xi }_{i}^{a}{x}_{i}\right)}^{2}\)), then \({{\mathcal{F}}}[E]\) expands in terms of even-order HOIs of \({\sum }_{i}{\xi }_{i}^{a}{x}_{i}\). For \({\gamma }^{{\prime} } < 0\), all coefficients of \({\sum }_{i}{\xi }_{i}^{a}{x}_{i}\) in the expansion are negative, indicating that embedded memories have deeper energy minima than in the classical case. The same signs appear for each order in the relativistic energy with \({\gamma }^{{\prime} } < 0\). We also note that β in the free energy of both the deformed exponential and relativistic models in the limit of large N appears scaled according to an effective temperature given by \({\beta }^{{\prime} }=\beta {\partial }_{E}{{\mathcal{F}}}[E]\) (e.g., (11) and Eq. (6.2) in Ref. 34). Moreover, the input in the Glauber dynamics is approximated for large sizes as

$$\beta \Delta {{\mathcal{F}}}[E]\approx \beta {\partial }_{E}{{\mathcal{F}}}[E]\,\Delta E({{\boldsymbol{x}}})={\beta }^{{\prime} }\Delta E({{\boldsymbol{x}}}).$$
(29)

The effective inverse temperatures \({\beta }^{{\prime} }=\beta {(1-{\gamma }^{{\prime} }E/N)}^{-1}\) for the deformed exponential and \({\beta }^{{\prime} }={2}^{-1}{(1-{\gamma }^{{\prime} }E/N)}^{-1/2}\) for the relativistic models are decreasing functions of E when \({\gamma }^{{\prime} } < 0\), resulting in an acceleration of memory retrieval—with lower energy E resulting in higher \({\beta }^{{\prime} }\) (lower temperature). While the relativistic model has been studied for \({\gamma }^{{\prime} } > 0\)32,33,34, we conjecture it may exhibit explosive phase transitions if \({\gamma }^{{\prime} } < 0\). Conversely, a positive \({\gamma }^{{\prime} }\) introduces alternating signs in even-order terms of \({\sum }_{i}{\xi }_{i}^{a}{x}_{i}\), and a shallower energy landscape due to a reduction in \({\beta }^{{\prime} }\). This shallower energy landscape reduces the memory capacity of the deformed exponential networks by expanding the spin-glass phases (Fig. 4), but also enlarges the recall (ferromagnetic) region by mitigating the formation of spurious memories given by overlapping patterns in the mixed phase (in alignment with previous work32 on mitigation of spurious memories in the relativistic model).

This perspective on accelerated memory retrieval by nonlinearity extends to dense associative memories20,21, which achieve supralinear memory capacities through nonlinear pattern encoding. Specifically, their energy function is given by \({{\mathcal{F}}}=-\!{\sum }_{a}F({\sum }_{i}{\xi }_{i}^{a}{x}_{i})\) with F being e.g., a thresholded power function20, \(F(z)={\left[z\right]}_{\!+}^{p}\) or an exponential nonlinearity21 F(z) = ez at zero temperature. These nonlinearities narrow basins of attraction, reducing memory overlap and preventing transitions to the spin-glass phase. The jumps in the Glauber dynamics of such systems are weighed by an accelerating function. Namely, from our perspective, the dynamics of such systems can be described via positive feedback on weights linked to a specific memory, which increase during memory retrieval. This follows from the fact that, relating the linear difference in Mattis terms \(\Delta {\epsilon }_{k}^{a}\equiv 2{\xi }_{k}^{a}{x}_{k}\) with the nonlinear difference \(\Delta {F}_{k}^{a}\equiv F\left({\sum }_{i}{\xi }_{i}^{a}{x}_{i}\right)-F\left({\sum }_{i}{\xi }_{i}^{a}{x}_{i}-\Delta {\epsilon }_{k}^{a}\right)\), the update of the kth neuron is determined by the sign of

$$\Delta {{\mathcal{F}}}({{\boldsymbol{x}}})=\mathop{\sum}_{a}\frac{\Delta {F}_{k}^{a}}{\Delta {\epsilon }_{k}^{a}}\Delta {\epsilon }_{k}^{a}={\sum}_{a}{w}_{k}^{a}\Delta {\epsilon }_{k}^{a}.$$
(30)

Here, we show that the effective weight \({w}_{k}^{a}\equiv \frac{\Delta {F}_{k}^{a}}{\Delta {\epsilon }_{k}^{a}}\) becomes an increasing function of \({\sum }_{i}{\xi }_{i}^{a}{x}_{i}\) when F is the power, exponential, or more generally, a convex function (See Supplementary Note 7). Thus, increasing \({\sum }_{i}{\xi }_{i}^{a}{x}_{i}\) as pattern ξ a is retrieved strengthens its basin of attraction and ensures positive feedback. Meanwhile, retrieval of ξ a reduces \({\sum }_{i}{\xi }_{i}^{b}{x}_{i}\) for orthogonal patterns ξ b, lowering their weights, suppressing their recall to minimise interference. This competitive mechanism highlights the higher memory capacity of these models compared to curved neural networks with uniform temperature scaling. Unlike the effective inverse temperature in curved networks, which depends only on the system’s state or energy, the effective weight in updating the k-th neuron additionally depends on the neuron’s state xk, thus no longer representing a global modulation of the energy.

Discussion

HOIs play a critical role in enabling emergent collective phenomena in natural and artificial systems. Modelling HOIs is, however, highly non-trivial, often requiring advanced analytic tools (such as simplicial complexes or hypergraphs) that entail an exponential increase in parameters for large systems. In this paper, we addressed this issue by leveraging the maximum entropy principle to effectively capture HOIs in models via a deformation parameter γ, which is associated with the Rényi entropy. Given their close connection with statistical physics, this family of models provides a useful setup to investigate the effect of HOIs on spin systems, including explosive ferromagnetic and spin-glass phase transitions, extending studies on anomalous phase transitions found in other systems2,7,8,9,11, and the capability of networks to store memories.

The observed effects in curved neural networks can be explained via an effective temperature, inducing a positive or negative feedback effect in memory retrieval. As we discussed above, this effect is present in different forms across other dense associative memories20,21,34. A similar argument may apply to diffusion models framed within dense associative memories25,26, where the energy follows a log-sum-exp nonlinearity. Thus, the accelerated mechanism found in this study clarifies memory retrieval in advanced associative networks, providing an important step toward designing extended memory capacities and improved noise scheduling.

Curved neural networks also provide insights into biological neural systems, where evidence suggests the presence of alternating positive and negative HOIs for even and odd orders, respectively. This alternation leads to sparse neuronal activity, which has been shown to be instrumental for enabling extended periods of total silence5,13,14,15,35. Interestingly, such sparse activity patterns may coexist with the accelerated memory retrieval dynamics, as both involve positive even-order HOIs. The attainment of enhanced memory, combined with sparse activity, presents a promising direction for understanding energy-efficient biological neuronal networks35,36. Future work may investigate how curved neural networks might support both energy efficiency and high memory capacities, potentially by adopting a thresholded, supralinear neuronal activation function20,35. Additionally, developing statistical methods for fitting these models to experimental data (i.e., theories for learning) represents an important, yet largely unexplored, research avenue. Together, these research directions offer a compelling path to uncover the principles of efficient information coding in biological neural systems.

Overall, our results demonstrate the benefits of considering the maximum entropy principle, emergent HOIs, and nonlinear network dynamics as theoretically intertwined notions. As showcased here, such an integrated framework reveals how information encoding, retrieval dynamics, and memory capacity in neural networks are mediated by HOIs, providing principled, analytically tractable tools and insights from statistical mechanics and nonlinear dynamics. More generally, the framework presented in this work extends beyond neural networks and contributes to a general theory of HOIs, paving the road toward a principled study of higher-order phenomena in complex networks.