Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

Small sample size is not the real problem

Nature Reviews Neuroscience volume 14, page 585 (2013)Cite this article

Subjects

Access through your institution
Buy or subscribe

Widespread poor research practices raise difficult questions about how to bring about improvements. Unfortunately, I believe that the Analysis article by Button et al. (Power failure: why small sample size undermines the reliability of neuroscience. Nature Rev. Neurosci. 14, 365–376 (2013))1, along with previous similar discussion of sample size2, misidentifies small sample size as a fundamental cause of problems in research and at the same time uncritically accepts a very harmful overemphasis on whether p < 0.05.

Much of their argument is undercut by the fact that the positive predictive value of p < 0.05 (PPV) is an unacceptably poor measure of the evidence that a study provides. PPV ignores distinctions between different p values below 0.05, such as p = 0.049 versus p < 0.0001, and therefore wastes information. Estimated effects, confidence intervals and exact p values should be considered when interpreting a study's results, and these make power irrelevant for interpreting completed studies3,4,5. In addition, any specific result (for example, p = 0.040) is not weaker evidence because of small sample size per se than the same p value would be with a larger sample size6.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

References

  1. Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nature Rev. Neurosci. 14, 365–376 (2013).

    Article  CAS  Google Scholar 

  2. Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2, e124 (2005).

    Article  Google Scholar 

  3. Goodman, S. N. & Berlin, J. A. The use of predicted confidence intervals when planning experiments and the misuse of power when interpreting results. Ann. Intern. Med. 121, 200–206 (1994).

    Article  CAS  Google Scholar 

  4. Hoenig, J. M. & Heisey, D. M. The abuse of power: the pervasive fallacy of power calculations for data analysis. Am. Stat. 55, 19–24 (2001).

    Article  Google Scholar 

  5. Senn, S. J. Power is indeed irrelevant in interpreting completed studies. BMJ 325, 1304 (2002).

    Article  Google Scholar 

  6. Royall, R. M. The effect of sample-size on the meaning of significance tests. Am. Stat. 40, 313–315 (1986).

    Google Scholar 

  7. Bacchetti, P. Current sample size conventions: flaws, harms, and alternatives. BMC Med. 8, 17 (2010).

    Article  Google Scholar 

  8. Bacchetti, P., Wolf, L. E., Segal, M. R. & McCulloch, C. E. Ethics and sample size. Am. J. Epidemiol. 161, 105–110 (2005).

    Article  Google Scholar 

  9. Bacchetti, P., McCulloch, C. E. & Segal, M. R. Simple, defensible sample sizes based on cost efficiency. Biometrics 64, 577–594 (2008).

    Article  Google Scholar 

  10. Bacchetti, P., Deeks, S. G. & McCune, J. M. Breaking free of sample size dogma to perform innovative translational research. Sci. Transl. Med. 3, 87ps24 (2011).

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Peter Bacchetti is at the Department of Epidemiology & Biostatistics, University of California, San Francisco, California 94143–0560, USA.,

    Peter Bacchetti

Authors
  1. Peter Bacchetti
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Peter Bacchetti.

Ethics declarations

Competing interests

The author declares no competing financial interests.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bacchetti, P. Small sample size is not the real problem. Nat Rev Neurosci 14, 585 (2013). https://doi.org/10.1038/nrn3475-c3

Download citation

  • Published:

  • Issue date:

  • DOI: https://doi.org/10.1038/nrn3475-c3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing