Divergence Measures in Information Theory and Statistics

Divergence measures quantify the dissimilarity between probability distributions and lie at the heart of modern information theory and statistical inference. Originating with the Kullback–Leibler divergence as a means to compare statistical models, the field has broadened to embrace a rich family of f-divergences, α-divergences and Bregman divergences. These measures share key properties such as non-negativity, convexity and, in many cases, the data-processing inequality, which ensures that information cannot increase under stochastic transformations. Common examples include the symmetric and bounded Jensen–Shannon divergence, the Rényi divergence of adjustable order and the χ²-divergence used in goodness-of-fit testing. Applications span hypothesis testing, coding theory, clustering of complex data and the training of deep neural networks via cross-entropy loss. Recent work has focused on normalising classical divergences to lie in [0,1], on computationally efficient estimation in high dimensions and on robust formulations that withstand model misspecification. This confluence of theory and practice continues to shape disciplines as diverse as machine learning, statistical physics and bioinformatics, illustrating the enduring global significance of divergence measures.

Research from Nature Portfolio

No recent Nature Portfolio content available.

Divergence Measures in Information Theory and Statistics publication trend

The graph below shows the total number of articles in divergence measures in information theory and statistics across all publications each year (not limited to Nature Index journals).

Technical terms

Divergence measure: A function that quantifies how one probability distribution differs from another, typically satisfying non-negativity and convexity.

Kullback–Leibler divergence: An asymmetric measure of relative entropy that quantifies the inefficiency of assuming one distribution when the truth is another.

f-divergence: A broad class of divergences defined via a convex generating function f, encompassing many standard measures including total variation and χ²-divergence.

Jensen–Shannon divergence: A symmetric and bounded divergence derived from averaging two Kullback–Leibler divergences, often used for clustering and distribution comparison.

Rényi divergence: A one-parameter generalisation of KL divergence that interpolates between different sensitivities to probability tails, useful in hypothesis testing and information bounds.

Cross-entropy loss: A measure of discrepancy between true labels and predicted probability distributions, widely employed as an objective function in supervised learning.

References

1. A Maximum Value for the Kullback–Leibler Divergence between Quantized Distributions. Information (2024).
2. On a Generalization of the Jensen–Shannon Divergence and the Jensen–Shannon Centroid. Entropy (2020).
3. On f-Divergences: Integral Representations, Local Behavior, and Inequalities. Entropy (2018).
4. On Data-Processing and Majorization Inequalities for f-Divergences with Applications. Entropy (2019).
5. On Relations between the Relative Entropy and χ2-Divergence, Generalizations and Applications. Entropy (2020).
6. Correlations of Cross-Entropy Loss in Machine Learning. Entropy (2024).