Statistical Inference for Sparse Mixture Models

Inference for sparse mixture models addresses the challenge of detecting and estimating a small subpopulation of observations whose distribution differs subtly from a dominant background. Such problems arise in fields as diverse as genomics, astrophysics and anomaly detection, where signals may be faint and only manifest in a tiny fraction of data. Theoretical work has characterized sharp detection boundaries delineating regimes in which reliable identification is possible from those in which any test must fail. Classical likelihood‐ratio approaches remain optimal in certain dense regimes, but modern techniques such as higher criticism and scan statistics excel when the signal is both rare and weak. These methods interrogate the combined behaviour of extreme order statistics or local clusters to amplify subtle departures from the null. Recent advances have refined exact p‐value computations, extended asymptotic optimality results to non‐Gaussian and discrete mixtures, and developed computationally efficient algorithms that scale to large datasets. Overall, statistical inference for sparse mixtures balances rigorous asymptotic theory with practical algorithms to meet the demands of high‐dimensional and large‐scale applications.

Research from Nature Portfolio

No recent Nature Portfolio content available.

Research from all publishers

One line of work has rigorously compared the higher criticism statistic and the scan statistic in the canonical normal mixture model, establishing that both attain the minimax detection boundary in the rare‐weak regime and characterizing their relative finite‐sample performance. Another strand has addressed sparse Poisson mixtures, demonstrating that a specialised form of higher criticism remains optimal when mean counts exceed logarithmic order, whereas simple multiple‐testing with Bonferroni correction suffices in ultra‐sparse scenarios with smaller means. A further contribution has investigated detection of sparse positive dependence in a bivariate setting, showing that rank‐based higher criticism tests achieve the theoretical boundary under Gaussian copula models, though power may degrade in the most extreme sparsity. Complementary computational advances have introduced efficient algorithms for exact p‐value calculation of supremum‐based statistics, yielding practical tools that maintain asymptotic guarantees while delivering accurate inference in moderate sample sizes.

Statistical Inference for Sparse Mixture Models publication trend

The graph below shows the total number of articles in statistical inference for sparse mixture models across all publications each year (not limited to Nature Index journals).

Technical terms

Sparse mixture model: A statistical model in which a small proportion of observations follow an alternative distribution while the majority follow a baseline distribution.

Detection boundary: The threshold in parameter space separating regimes where a signal can or cannot be detected with high probability.

Higher criticism statistic: A test based on the maximal deviation of empirical p‐values from their expected uniform distribution, optimised for rare‐weak signals.

Scan statistic: A localised test that searches over subsets or windows for unusually high aggregate signal, sensitive to clustered alternatives.

Rare‐weak regime: An asymptotic setting where the signal proportion tends to zero and individual effect sizes decrease with sample size.

Likelihood ratio test: A classical hypothesis test comparing the likelihoods under the null and alternative models, often optimal in dense settings.

References

1. Detection of sparse mixtures: higher criticism and scan statistic. Electronic Journal of Statistics (2019).
2. The sparse Poisson means model. Electronic Journal of Statistics (2015).
3. Detection of sparse positive dependence. Electronic Journal of Statistics (2020).
4. On the exact Berk-Jones statistics and their $p$-value calculation. Electronic Journal of Statistics (2016).
5. Exact asymptotics for the scan statistic and fast alternatives. Electronic Journal of Statistics (2016).