Statistical Methods for Method Comparison Studies

Method comparison studies aim to evaluate whether two or more measurement techniques yield sufficiently similar results to be used interchangeably. Traditional approaches such as linear regression and correlation quantify association rather than agreement and can mask systematic biases or scale differences. The Bland-Altman method, based on plotting the differences between paired measurements against their means and calculating limits of agreement, has become a cornerstone for continuous outcomes. Yet its underlying assumptions—constant bias and homoscedasticity—are often violated in practice. Extensions using regression techniques allow for bias that varies with magnitude, while nonparametric quantile estimators and tolerance intervals offer more robust limits when normality fails. In parallel, variance-component and hierarchical models decompose sources of variability in repeated-measure designs, accommodating multiple observers, devices or time points. The concordance correlation coefficient and total deviation index provide summary statistics that blend accuracy and precision. Advances in graphical tools and statistical software have improved transparency and ease of use, promoting wider application across disciplines from clinical chemistry to environmental monitoring. These methods support evidence-based decisions on method interchangeability, assurance of measurement quality and regulatory compliance worldwide.

Research from Nature Portfolio

No recent Nature Portfolio content available.

Research from all publishers

Recent work has revisited the Bland-Altman framework in high-precision bioanalytical settings. Researchers re-evaluated the classic limits of agreement in mass spectrometry comparisons, proposing quantitative acceptance criteria based on calibration against gold-standard instruments and emphasising the relationship between difference and mean in diverse biological matrices. This study underlined the need for method-specific thresholds rather than generic limits.

Another investigation designed a formal comparison between a low-cost open-source pH logger and an industrial reference device. Using linear mixed-effects models, the authors derived indices of agreement and graphical diagnostics, revealing initial fixed bias and demonstrating how recalibration improved concordance. This approach illustrates how mixed-effects modelling can yield nuanced insights into device performance in field conditions.

Moreover, an examination of settings where one method has virtually no measurement error highlighted shortcomings of the Bland-Altman approach. It showed that when one technique is effectively error-free, limits of agreement become biased and recommended linear regression of differences on the precise method as an unbiased alternative. This work challenges the universality of Bland-Altman analysis and promotes context-adapted strategies.

Statistical Methods for Method Comparison Studies publication trend

The graph below shows the total number of articles in statistical methods for method comparison studies across all publications each year (not limited to Nature Index journals).

Technical terms

Bland-Altman plot: A scatter plot of paired differences versus means, used to visualise bias and limits of agreement between two measurement methods.

Limits of agreement: The interval within which a specified proportion (typically 95%) of differences between methods is expected to lie, calculated as bias ±1.96 × standard deviation of differences.

Bias: The average difference between measurements obtained by two methods, indicating systematic deviation.

Mixed-effects model: A statistical model incorporating both fixed effects (systematic factors) and random effects (sources of variability across subjects, observers or devices).

Tolerance interval: An interval expected to contain a given proportion of a population’s differences with a specified confidence level, offering exact coverage properties under distributional assumptions.

Concordance correlation coefficient: A statistic combining measures of precision and accuracy to quantify agreement between quantitative methods in a single metric.

References

1. Mass Spectrometry-Based Evaluation of the Bland–Altman Approach: Review, Discussion, and Proposal. Molecules (2023).
2. Formal Assessment of Agreement and Similarity between an Open-Source and a Reference Industrial Device with an Application to a Low-Cost pH Logger. Sensors (2024).
3. The Bland-Altman method should not be used when one of the two measurement methods has negligible measurement errors. PLOS ONE (2022).
4. Nonparametric Limits of Agreement in Method Comparison Studies: A Simulation Study on Extreme Quantile Estimation. International Journal of Environmental Research and Public Health (2020).
5. To tolerate or to agree: A tutorial on tolerance intervals in method comparison studies with BivRegBLS R Package. Statistics in Medicine (2020).
6. A multivariate hierarchical Bayesian approach to measuring agreement in repeated measurement method comparison studies. BMC Medical Research Methodology (2009).
7. BA.plot: An R function for Bland-Altman analysis. Clinical Epidemiology and Global Health (2021).