Optimization Algorithms for Likelihood Estimation

Likelihood estimation lies at the heart of statistical inference, providing a principled framework to fit models to data by maximising the probability of observed outcomes. Classical approaches such as Newton–Raphson and Fisher scoring exploit gradient and curvature information to iteratively ascend the likelihood surface, offering rapid local convergence but sometimes suffering from instability or prohibitive computational cost in high dimensions. The Expectation–Maximization (EM) algorithm and its generalisations, including Majorization–Minimization (MM) schemes, construct surrogate functions that guarantee monotonic ascent and facilitate handling of latent variables or complex constraints. Hybrid strategies have emerged to combine global convergence properties with fast local rates, integrating damping techniques like Levenberg–Marquardt to balance stability and efficiency. Recently, profile-likelihood methods have gained traction for decomposing high-dimensional problems into more tractable subproblems and providing automatic starting values. These developments respond to the growing demand for scalable, robust, and convergent algorithms in applications ranging from large-scale genomics to geophysical inverse problems. The interplay of theoretical convergence guarantees, numerical stability and practical implementation considerations continues to drive innovation in optimisation algorithms for likelihood estimation.

Research from Nature Portfolio

No recent Nature Portfolio content available.

Research from all publishers

A novel profile‐likelihood‐based algorithm addresses the need for automatic and stable computation of maximum likelihood estimates in complex crash-frequency models. By splitting the parameter vector and applying successive one-dimensional profile-likelihood optimisations, the method yields guaranteed global convergence from an automatically generated starting guess. Simulation studies demonstrate superior numerical stability and faster convergence compared with classical Newton and quasi-Newton routines.

In geophysical applications, efficient estimation of covariance parameters via Fisher scoring has been enhanced through Levenberg–Marquardt damping. The combined approach ensures reliable convergence even when standard Fisher scoring iterations falter, and allows tuning of convergence speed by adjusting the damping parameter. This development offers a practical route to fast, stable maximum likelihood and restricted maximum likelihood estimation in large-scale collocation problems.

For multinomial logistic models with latent class structures, a Majorization–Minimization algorithm has been developed to minimise the negative log-likelihood without resorting to squared-distance approximations. This method constructs tight surrogate functions that guarantee monotonic decrease of the objective, simplifying model interpretation and selection. Empirical applications illustrate improved convergence behaviour and enhanced graphical diagnostics compared with earlier distance-based unfolding techniques.

Optimization Algorithms for Likelihood Estimation publication trend

The graph below shows the total number of articles in optimization algorithms for likelihood estimation across all publications each year (not limited to Nature Index journals).

Technical terms

Maximum Likelihood Estimation (MLE): A method for estimating model parameters by maximising the probability (likelihood) of observed data under the model.

Expectation–Maximization (EM) Algorithm: An iterative procedure that alternates between estimating expected sufficient statistics (E‐step) and maximising a surrogate likelihood (M‐step) in the presence of latent variables.

Newton–Raphson Method: A root-finding algorithm applied to the score equations of the likelihood, using second-order Taylor expansions for rapid local convergence.

Fisher Scoring: A variant of Newton–Raphson that replaces the observed Hessian with its expected value (the Fisher information), often improving numerical stability.

Majorization–Minimization (MM) Algorithm: A general framework that constructs a surrogate function at each iteration which is easier to optimise and majorises (or minorises) the original objective, ensuring monotonic improvement.

Profile Likelihood: A technique that reduces a multivariate optimisation problem by maximising the likelihood with respect to a subset of parameters for fixed values of the others, producing a lower-dimensional optimisation.

Levenberg–Marquardt Algorithm: A damped least-squares approach that interpolates between gradient descent and Gauss–Newton steps to enhance convergence stability, often applied in likelihood contexts.

References

1. An Automated Profile‐Likelihood‐Based Algorithm for Fast Computation of the Maximum Likelihood Estimate in a Statistical Model for Crash Data. Journal of Applied Mathematics (2022).
2. Fast Estimation of Covariance Parameters in Least-Squares Collocation by Fisher Scoring with Levenberg–Marquardt Optimization. Surveys in Geophysics (2017).
3. Multinomial Restricted Unfolding. Journal of Classification (2024).