Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Reply
  • Published:

Empirical evidence for low reproducibility indicates low pre-study odds

Nature Reviews Neuroscience volume 14, page 877 (2013)Cite this article

Subjects

Access through your institution
Buy or subscribe

In response to our Analysis article (Power failure: why small sample size undermines the reliability of neuroscience. Nature Rev. Neurosci. 14, 365–376 (2013))1, Hoppe (A test is not a test. Nature Rev. Neurosci.http://dx.doi.org/10.1038/nrn3475-c5 (2013))2 correctly points out that the figures in our article cover only pre-study odds of an effect (R) up to 1 (that is, up to 50% prior prevalence of a non-null effect). In principle, of course, R can be higher — the prior prevalence of a non-null effect can vary from 0 to 100%. However, in the large majority of research studies, it will be 50% or lower.

When the prior probability of an effect is very high, such as that required to justify a large, confirmatory clinical trial, it will still only approach 50% (R = 1). Empirically, Djulbegovic et al.3 have recently shown this to be the case: only slightly more than 50% of Phase 3 clinical trials show superiority of the intervention arm over the comparator arm. As large-scale clinical trials are arguably the end stage of a research pipeline that begins in the basic sciences, such trials should represent the case when R (on average) can be expected to achieve the highest values. As R increases above 1 (that is, prior prevalence >50%), the incremental value of further research decreases; when it is very high (for example, prior prevalence >90%), further research is probably not necessary, because there is already high confidence in the outcome. Most neuroscience research is far removed from this situation, as most neuroscience involves testing exploratory hypotheses and addressing measurements with high complexity and extreme multiplicity. In other words, there are many variables that can be explored for associations and effects, often with little prior insight regarding which of them may be important.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

References

  1. Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nature Rev. Neurosci. 14, 365–376 (2013).

    Article  CAS  Google Scholar 

  2. Hoppe, C. A test is not a test. Nature Rev. Neurosci. http://dx.doi.org/10.1038/nrn3475-c5 (2013).

  3. Djulbegovic, B., Kumar, A., Glasziou, P., Miladinovic, B. & Chalmers, I. Medical research: trial unpredictability yields predictable therapy gains. Nature 500, 395–396 (2013).

    Article  CAS  Google Scholar 

  4. Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature 483, 531–533 (2012).

    Article  CAS  Google Scholar 

  5. Mobley, A., Linder, S. K., Braeuer, R., Ellis, L. M. & Zwelling, L. A survey on data reproducibility in cancer research provides insights into our limited ability to translate findings from the laboratory to the clinic. PLoS ONE 8, e63221 (2013).

    Article  CAS  Google Scholar 

  6. Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nature Rev. Drug Discov. 10, 712 (2011).

    Article  CAS  Google Scholar 

  7. Gelman, A. & Tuerlinckx, F. Type S error rates for classical and Bayesian single and multiple comparison procedures. Comput. Stat. 15, 373–390 (2000).

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Experimental Psychology, University of Bristol, BS8 1TU, UK

    Katherine S. Button, Claire Mokrysz & Marcus R. Munafò

  2. School of Social and Community Medicine, University of Bristol, BS8 2BN, UK

    Katherine S. Button

  3. Stanford University School of Medicine, 94305, California, USA

    John P. A. Ioannidis

  4. Department of Psychology, University of Virginia, Charlottesville, 22903, Virginia, USA

    Brian A. Nosek

  5. Wellcome Trust Centre for Human Genetics, University of Oxford, OX 7BN, UK

    Jonathan Flint

  6. School of Physiology and Pharmacology, University of Bristol, BS8 1TD, UK

    Emma S. J. Robinson

Authors
  1. Katherine S. Button
    View author publications

    Search author on:PubMed Google Scholar

  2. John P. A. Ioannidis
    View author publications

    Search author on:PubMed Google Scholar

  3. Claire Mokrysz
    View author publications

    Search author on:PubMed Google Scholar

  4. Brian A. Nosek
    View author publications

    Search author on:PubMed Google Scholar

  5. Jonathan Flint
    View author publications

    Search author on:PubMed Google Scholar

  6. Emma S. J. Robinson
    View author publications

    Search author on:PubMed Google Scholar

  7. Marcus R. Munafò
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Marcus R. Munafò.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Button, K., Ioannidis, J., Mokrysz, C. et al. Empirical evidence for low reproducibility indicates low pre-study odds. Nat Rev Neurosci 14, 877 (2013). https://doi.org/10.1038/nrn3475-c6

Download citation

  • Published:

  • Issue date:

  • DOI: https://doi.org/10.1038/nrn3475-c6

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing