Statistical Methods for Multiple Hypothesis Testing

When researchers conduct many statistical tests simultaneously, the chance of obtaining false positives increases. Traditional methods for multiple comparisons, such as the Bonferroni correction, control the family-wise error rate by adjusting significance thresholds conservatively, but often at the expense of statistical power. In contrast, procedures that control the false discovery rate strike a balance between discovering true effects and limiting spurious findings. The Benjamini–Hochberg procedure ranks p-values and applies an adaptive threshold, offering greater sensitivity when testing hundreds or thousands of hypotheses. More recent advances incorporate auxiliary information—covariates or grouping structures—to weight or prioritise tests, thereby improving power without sacrificing error control. Hierarchical and group-wise methods exploit known dependencies among hypotheses, while empirical Bayes approaches estimate the proportion of true nulls to refine thresholds. Emerging two-dimensional procedures further harness complex data structures, such as those in genomics or neuroimaging, by accounting simultaneously for multiple sources of variation. These innovations have enhanced the reliability of high-throughput studies and underpin discoveries in fields ranging from molecular biology to social science.

Research from Nature Portfolio

No recent Nature Portfolio content available.

Research from all publishers

A new two-dimensional group Benjamini–Hochberg procedure extends classical false discovery control by exploiting dual grouping structures in high-throughput data. By iteratively adjusting thresholds along two axes—such as genes and cell types—this method enhances power in settings where tests exhibit block dependence, outperforming one-way adaptive procedures while maintaining rigorous error rates. Another study applies information-theoretic principles to modify false discovery rate procedures under arbitrary correlation structures. By deriving thresholds from the conditional Fisher information of ordered test statistics, three novel variants provide flexible control across low to high correlation regimes. These approaches bridge the gap between conservative and liberal corrections, optimising feature selection in high-dimensional genomic analyses. In parallel, the harmonic mean p-value framework offers a means to combine dependent tests while controlling the family-wise error rate. By aggregating p-values through a harmonic mean transformation, this method gains sensitivity to detect small effect sizes across correlated hypotheses. It has demonstrated superior power to the Benjamini–Hochberg procedure in genome-wide association studies and can reveal coherent signals among groups of marginally significant tests, thereby enhancing discovery potential in large-scale investigations.

Statistical Methods for Multiple Hypothesis Testing publication trend

The graph below shows the total number of articles in statistical methods for multiple hypothesis testing across all publications each year (not limited to Nature Index journals).

Technical terms

False Discovery Rate (FDR): Expected proportion of false positives among all rejected hypotheses.

Family-Wise Error Rate (FWER): Probability of making one or more false discoveries across multiple tests.

p-value: Probability of observing data at least as extreme as the sample result under the null hypothesis.

q-value: Adjusted p-value reflecting the minimum false discovery rate at which a test may be deemed significant.

Benjamini–Hochberg procedure: Step-up algorithm that orders p-values and sets a data-driven threshold to control the false discovery rate.

References

1. 2dGBH: Two-dimensional group Benjamini–Hochberg procedure for false discovery rate control in two-way multiple testing of genomic data. Bioinformatics (2024).
2. Modifying the false discovery rate procedure based on the information theory under arbitrary correlation structure and its performance in high-dimensional genomic data. BMC Bioinformatics (2024).
3. The harmonic mean p-value for combining dependent tests. Proceedings of the National Academy of Sciences of the United States of America (2019).