Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Commentary
  • Published:

The fickle P value generates irreproducible results

Nature Methods volume 12, pages 179–185 (2015)Cite this article

Subjects

The reliability and reproducibility of science are under scrutiny. However, a major cause of this lack of repeatability is not being considered: the wide sample-to-sample variability in the P value. We explain why P is fickle to discourage the ill-informed practice of interpreting analyses based predominantly on this statistic.

Access through your institution
Buy or subscribe

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Simulated data distributions of two populations.
Figure 2: Small samples show substantial variation.
Figure 3: A larger sample size estimates effect size more precisely.
Figure 4: Sample size affects the distribution of P values.
Figure 5: How sample size alters estimated effect size.
Figure 6: Characterizing the precision of effect size using the 95% CI of the difference between the means.

References

  1. Woolston, C. Nature 513, 283 (2014).

    Article  Google Scholar 

  2. Mobley, A., Linder, S.K., Braeuer, R., Ellis, L.M. & Zwelling, L. PLoS ONE 8, e63221 (2013).

    Article  CAS  Google Scholar 

  3. Anonymous. Economist 26–30 (19 October 2013).

  4. Russell, J.F. Nature 496, 7 (2013).

    Article  CAS  Google Scholar 

  5. Bohannon, J. Science 344, 788–789 (2014).

    Article  CAS  Google Scholar 

  6. Van Noorden, R. Nature News doi:10.1038/nature.2014.15509 (2014).

  7. Anonymous. Nature 515, 7 (2014).

  8. McNutt, M. Science 346, 679 (2014).

    Article  CAS  Google Scholar 

  9. Vaux, D.L. Nature 492, 180–181 (2012).

    Article  CAS  Google Scholar 

  10. Button, K.S. et al. Nat. Rev. Neurosci. 14, 365–376 (2013).

    Article  CAS  Google Scholar 

  11. Nuzzo, R. Nature 506, 150–152 (2014).

    Article  CAS  Google Scholar 

  12. Fidler, F., Burgman, M.A., Cumming, G., Buttrose, R. & Thomason, N. Conserv. Biol. 20, 1539–1544 (2006).

    Article  Google Scholar 

  13. Tressoldi, P.E., Giofré, D., Sella, F. & Cumming, G. PLoS ONE 8, e56180 (2013).

    Article  CAS  Google Scholar 

  14. Sharpe, D. Psychol. Methods 18, 572–582 (2013).

    Article  Google Scholar 

  15. Ellison, A.M., Gotelli, N.J., Inouye, B.D. & Strong, D.R. Ecology 95, 609–610 (2014).

    Article  Google Scholar 

  16. Murtaugh, P.A. Ecology 95, 611–617 (2014).

    Article  Google Scholar 

  17. Cohen, J. Am. Psychol. 49, 997–1003 (1994).

    Article  Google Scholar 

  18. Krzywinski, M. & Altman, N. Nat. Methods 10, 1139–1140 (2013).

    Article  CAS  Google Scholar 

  19. Fisher, R.A. Statistical Methods for Research Workers (Oliver and Boyd, 1925).

    Google Scholar 

  20. Fisher, R.A. Statistical Methods and Scientific Inference 2nd edn. (Hafner, 1959).

    Google Scholar 

  21. Krzywinski, M. & Altman, N. Nat. Methods 10, 809–810 (2013).

    Article  CAS  Google Scholar 

  22. McCormack, J., Vandermeer, B. & Allan, G.M. BMC Med. Res. Methodol. 13, 134 (2013).

    Article  Google Scholar 

  23. Boos, D.D. & Stefanski, L.A. Am. Stat. 65, 213–221 (2011).

    Article  Google Scholar 

  24. Cumming, G. Perspect. Psychol. Sci. 3, 286–300 (2008).

    Article  Google Scholar 

  25. Cumming, G. Psychol. Sci. 25, 7–29 (2014).

    Article  Google Scholar 

  26. Maxwell, S.E. Psychol. Methods 9, 147–163 (2004).

    Article  Google Scholar 

  27. Salsburg, D.S. Am. Stat. 39, 220–223 (1985).

    Google Scholar 

  28. Johnson, V.E. Proc. Natl. Acad. Sci. USA 110, 19313–19317 (2013).

    Article  CAS  Google Scholar 

  29. Johnson, D.H. J. Wildl. Mgmt. 63, 763–772 (1999).

    Article  Google Scholar 

  30. Krzywinski, M. & Altman, N. Nat. Methods 10, 921–922 (2013).

    Article  CAS  Google Scholar 

  31. Masson, M.E. & Loftus, G.R. Can. J. Exp. Psychol. 57, 203–220 (2003).

    Article  Google Scholar 

  32. Drummond, G.B. & Vowler, S.L. J. Physiol. (Lond.) 589, 1861–1863 (2011).

    Article  CAS  Google Scholar 

  33. Lavine, M. Ecology 95, 642–645 (2014).

    Article  Google Scholar 

  34. Loftus, G.R. Behav. Res. Methods Instrum. Comput. 25, 250–256 (1993).

    Article  Google Scholar 

  35. Martínez-Abraín, A. Acta Oecol. 34, 9–11 (2008).

    Article  Google Scholar 

  36. Nakagawa, S. & Cuthill, I.C. Biol. Rev. Camb. Philos. Soc. 82, 591–605 (2007).

    Article  Google Scholar 

  37. Curran-Everett, D. Adv. Physiol. Educ. 33, 87–90 (2009).

    Article  Google Scholar 

  38. Grissom, R.J. & Kim, J.J. Effect Sizes for Research: Univariate and Multivariate Applications 2nd edn. (Routledge, 2011).

    Google Scholar 

  39. Fearon, P. Psychologist 16, 632–635 (2003).

    Google Scholar 

  40. Maxwell, S.E., Kelley, K. & Rausch, J.R. Annu. Rev. Psychol. 59, 537–563 (2008).

    Article  Google Scholar 

  41. Rosnow, R.L. & Rosenthal, R. Am. Psychol. 44, 1276–1284 (1989).

    Article  Google Scholar 

  42. Lew, M.J. Br. J. Pharmacol. 166, 1559–1567 (2012).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We thank J.W. Huber, C.M. Bishop and P.A. Stephens for helpful comments on drafts of this manuscript.

Author information

Authors and Affiliations

  1. Lewis G. Halsey is in the Department of Life Sciences, University of Roehampton, London, UK,

    Lewis G Halsey

  2. Douglas Curran-Everett is in the Division of Biostatistics and Bioinformatics, and the Department of Biostatistics and Informatics, National Jewish Health, Colorado School of Public Health, University of Colorado Denver, Denver, Colorado, USA,

    Douglas Curran-Everett

  3. Sarah L. Vowler is at Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, UK,

    Sarah L Vowler

  4. Gordon B. Drummond is at the University of Edinburgh, Edinburgh, UK.,

    Gordon B Drummond

Authors
  1. Lewis G Halsey
    View author publications

    Search author on:PubMed Google Scholar

  2. Douglas Curran-Everett
    View author publications

    Search author on:PubMed Google Scholar

  3. Sarah L Vowler
    View author publications

    Search author on:PubMed Google Scholar

  4. Gordon B Drummond
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Lewis G Halsey.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Halsey, L., Curran-Everett, D., Vowler, S. et al. The fickle P value generates irreproducible results. Nat Methods 12, 179–185 (2015). https://doi.org/10.1038/nmeth.3288

Download citation

  • Published:

  • Issue date:

  • DOI: https://doi.org/10.1038/nmeth.3288

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing