Statistical Mechanics of Neural Network Learning and Models

Statistical mechanics provides a principled framework for understanding how large collections of adjustable parameters in neural networks organise and generalise when trained on data. By mapping the space of synaptic weights to an energy landscape, one can employ tools such as partition functions, cumulants and renormalisation to derive macroscopic observables like generalisation error, capacity and learning curves. In the infinite-width limit, networks converge to Gaussian processes, while finite systems exhibit data-dependent fluctuations that can be treated as slow variables. Phase transitions akin to those in physical systems arise during training, marking abrupt changes in representation or performance. The interplay between network depth, width, data structure and optimisation dynamics can thus be cast in terms of thermodynamic variables, enabling quantitative predictions of feature learning, overparameterisation effects and the emergence of invariant internal representations.

Research from Nature Portfolio

Recent studies have demonstrated a separation of scales in deep convolutional and fully connected networks, showing that layers interact predominantly through the second cumulant of activations, which fluctuates nearly Gaussian in finite systems. This leads to a tractable thermodynamic theory in which data-aware kernels adapt during training, yielding accurate predictions for generalisation and feature learning in overparameterised networks. Complementarily, advances in the statistical mechanics of kernel regression have produced analytical expressions for generalisation error across arbitrary data distributions and kernel choices. This work reveals an inductive bias favouring simple functions, explains non-monotonic learning curves when data are noisy or poorly matched to the kernel, and characterises the conditions under which additional data can degrade performance.

Statistical Mechanics of Neural Network Learning and Models publication trend

The graph below shows the total number of articles in statistical mechanics of neural network learning and models across all publications each year (not limited to Nature Index journals).

Technical terms

Partition function: A sum over weight configurations encoding the statistical weight of each network state.

Cumulant: A measure of joint fluctuations of activations or pre-activations beyond mean and variance.

Gaussian process: A distribution over functions corresponding to infinitely wide neural networks.

Kernel regression: A non-parametric method that predicts outputs via inner products in feature space.

Inductive bias: The predisposition of a learning algorithm to favour certain functions or representations.

Generalisation error: The expected discrepancy between network predictions and true outputs on new data.

Teacher-student model: A theoretical setup where a ‘student’ network learns data generated by a fixed ‘teacher’ network.

Feature map: A transformation that projects raw inputs into a representation space, either predefined or learned.

References

1. Separation of scales and a thermodynamic description of feature learning in some CNNs. Nature Communications (2023).
2. How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model. Physical Review X (2024).
3. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature Communications (2021).
4. Learning curves of generic features maps for realistic datasets with a teacher-student model* *This article is an updated version of: Loureiro B, Gerbelot C, Cui H, Goldt S, Krzakala F, Mezard M and Zdeborová L 2021 Learning curves of generic features maps for realistic datasets with a teacher-student model Advances in Neural Information Processing Systems vol 34 ed M Ranzato, A Beygelzimer, Y Dauphin, P S Liang and J Wortman Vaughan (New York: Curran Associates) pp 18137–51.. Journal of Statistical Mechanics Theory and Experiment (2022).