Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

Remove, rather than redefine, statistical significance

Nature Human Behaviour volume 2, page 4 (2018)Cite this article

Subjects

To the Editor — Benjamin et al.1 propose to redefine statistical significance with a trichotomy: what was once ‘highly significant’ (P < 0.005) becomes ‘significant’, what was once significant (P < 0.05) becomes ‘suggestive’, and what was ‘nonsignificant’ (P > 0.05) remains nonsignificant. Trichotomization is better than dichotomization, and we agree that P values around 0.05 convey only limited evidence against the tested hypothesis (which is usually a ‘null’ hypothesis of no effect)2.

We also agree that P hacking, selective reporting and publication bias “are arguably the bigger problems”1 than false positives arising by chance. Nonetheless, imposing a more stringent significance threshold will aggravate those problems3. Benjamin et al. say that their proposal “should not be used to reject publications of novel findings with 0.005 < P < 0.05”. But rejections due to P > 0.05 will remain, and rejections due to P > 0.005 will now also occur, leading to more intense P hacking and selective reporting, with increased bias in reported effects (because estimates from studies that are selected for having P < 0.005 are usually more inflated than those selected for having P < 0.05)3,4.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to the full article PDF.

USD 39.95

Prices may be subject to local taxes which are calculated during checkout

References

  1. Benjamin, D.  J. et al. Nature Hum. Behav. 1, 0189 (2017).

    Google Scholar 

  2. Greenland, S. The need for cognitive science in methodology. Am. J. Epidemiol. (in the press).

  3. Amrhein, V., Korner-Nievergelt, F. & Roth, T. PeerJ 5, e3544 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  4. Button, K. S. et al. Nat. Rev. Neurosci. 14, 365–376 (2013).

    Article  CAS  PubMed  Google Scholar 

  5. Gelman, A. & Loken, E. Am. Sci. 102, 460–465 (2014).

    Article  Google Scholar 

  6. Fiedler, K., Kutzner, F. & Krueger, J. I. Persp. Psy. Sci. 7, 661–669 (2012).

    Article  Google Scholar 

  7. Greenland, S. Eur. J. Epidemiol. 32, 3–20 (2017).

    Article  PubMed  Google Scholar 

  8. Greenland, S. et al. Eur. J. Epidemiol. 31, 337–350 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  9. Oakes, M. Statistical Inference: A Commentary for the Social and Behavioural Sciences (Wiley, Chichester, 1986).

    Google Scholar 

  10. Wasserstein, R. L. & Lazar, N. A. Am. Stat. 70, 129–133 (2016).

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Zoological Institute, University of Basel, 4051 Basel, Switzerland

    Valentin Amrhein

  2. Swiss Ornithological Institute, 6204 Sempach, Switzerland

    Valentin Amrhein

  3. Department of Epidemiology and Department of Statistics, University of California, Los Angeles, CA, USA

    Sander Greenland

Authors
  1. Valentin Amrhein
    View author publications

    Search author on:PubMed Google Scholar

  2. Sander Greenland
    View author publications

    Search author on:PubMed Google Scholar

Corresponding authors

Correspondence to Valentin Amrhein or Sander Greenland.

Ethics declarations

Competing interests

The authors declare no competing interests.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Amrhein, V., Greenland, S. Remove, rather than redefine, statistical significance. Nat Hum Behav 2, 4 (2018). https://doi.org/10.1038/s41562-017-0224-0

Download citation

  • Published:

  • Issue date:

  • DOI: https://doi.org/10.1038/s41562-017-0224-0

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics