Multiple Testing Procedures in Statistical Analysis

When researchers test several hypotheses simultaneously, the risk of false positive findings increases unless adjustments are made to account for multiplicity. Classical approaches focus on controlling the family-wise error rate (FWER), the probability of making at least one type I error, through methods such as the Bonferroni correction and its sequential variants (Holm, Hochberg). In exploratory settings where some false discoveries may be tolerated, the false discovery rate (FDR) offers a less stringent criterion, with the Benjamini–Hochberg procedure and its extensions commonly employed. Closed testing procedures provide a general framework for strong control of the FWER by testing all intersection hypotheses in a hierarchical fashion. Graphical and gatekeeping strategies enable flexible allocation of error rates across correlated endpoints or ordered families of hypotheses, reflecting scientific priorities and logical relationships. Advances in computational methods, including resampling-based and permutation techniques, have made it possible to estimate adjusted p-values or critical constants accurately under complex dependency structures. Recent work also emphasises the impact of multiplicity adjustments on statistical power, motivating simulation-based tools for sample size and detectable effect size calculations in multi-level designs. Taken together, these developments enhance rigour and interpretability in fields ranging from clinical trials and genomics to complex observational studies.

Research from Nature Portfolio

No recent Nature Portfolio content available.

Research from all publishers

In the field of trial design, an R package has been introduced to estimate statistical power, minimum detectable effect size and required sample size in multi-level randomised experiments with multiple outcomes, explicitly accounting for a range of multiple testing procedures via simulation of joint test-statistic distributions. In omics research, a novel shortcut for exact closed testing has been developed to control the FWER when assessing global hypotheses across thousands of metabolomic feature sets, enabling post hoc selection of pathways without inflating error rates. In biomedical studies of binary endpoints, comparative evaluation of multiple marginal models versus nonparametric vector-based resampling approaches demonstrates that resampling-based adjustments can consistently yield higher power while still strongly controlling the FWER in small-sample settings, whereas marginal modelling offers versatility for complex effect measures and simultaneous confidence intervals.

Multiple Testing Procedures in Statistical Analysis publication trend

The graph below shows the total number of articles in multiple testing procedures in statistical analysis across all publications each year (not limited to Nature Index journals).

Technical terms

Family-wise error rate (FWER): Probability of making one or more type I errors when testing multiple hypotheses.

False discovery rate (FDR): Expected proportion of incorrectly rejected null hypotheses among all rejections.

Closed testing procedure: Hierarchical approach that tests all combinations of hypotheses to ensure strong control of the FWER.

Gatekeeping strategy: Sequential framework that organises hypotheses into families and allocates error rates to preserve overall multiplicity control.

Resampling-based procedure: Data-driven method using repeated sampling (e.g. bootstrap, permutation) to estimate adjusted p-values or error rates under dependency.

References

1. PUMP: Estimating Power, Minimum Detectable Effect Size, and Sample Size When Adjusting for Multiple Outcomes in Multi-Level Experiments. Journal of Statistical Software (2024).
2. Closed Testing with Globaltest, with Application in Metabolomics. Biometrics (2022).
3. Simultaneous Inference of Multiple Binary Endpoints in Biomedical Research: Small Sample Properties of Multiple Marginal Models and a Resampling Approach. Biometrical Journal (2024).