Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Matters Arising
  • Published:

Claims about scientific rigour require rigour

Nature Human Behaviour volume 8, pages 1890–1891 (2024)Cite this article

Subjects

The Original Article was published on 09 November 2023

Access through your institution
Buy or subscribe

arising from J. Protzko et al. Nature Human Behaviour https://doi.org/10.1038/s41562-023-01749-9 (2023)

Protzko et al.1 describe a project in which internal tests of pilot-tested hypotheses and independent replications embraced “rigour-enhancing practices” such as confirmatory tests, large sample sizes, preregistration and methodological transparency. The authors report a high estimate of replicability, which, in their appraisal, “justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries” (see Abstract). However, replicability was not the original outcome of interest in the project, and analyses associated with replicability were not preregistered as claimed. Instead of replicability, the originally planned study set out to examine whether the mere act of scientifically investigating a phenomenon (data collection or analysis) could cause effect sizes to decline on subsequent investigation (https://osf.io/ba8p7). This “decline effect” hypothesis, posited by one of the authors and not articulated in the published manuscript, invokes phenomena that, if found, could revise the “laws of reality”2. The project did not yield support for this preregistered hypothesis; the preregistered analyses on the decline effect and the resulting null findings were largely relegated to the supplement, and the published article instead focused on replicability, with a set of non-preregistered measures and analyses, despite claims to the contrary.

This is a preview of subscription content, access via your institution

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

References

  1. Protzko, J. et al. High replicability of newly discovered social-behavioural findings is achievable. Nat. Hum. Behav. https://doi.org/10.1038/s41562-023-01749-9 (2023).

  2. Schooler, J. Unpublished results hide the decline effect. Nature 470, 437 (2011).

    Article  CAS  PubMed  Google Scholar 

  3. Schauer, J. M. & Hedges, L. V. Reconsidering statistical methods for assessing replication. Psychol. Methods 26, 127–139 (2021).

    Article  CAS  PubMed  Google Scholar 

  4. Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).

    Article  Google Scholar 

  5. Cova, F. et al. Estimating the reproducibility of experimental philosophy. Rev. Phil. Psychol. 12, 9–44 (2021).

    Article  Google Scholar 

  6. Klein, R. A. et al. Investigating variation in replicability. Soc. Psychol. 45, 142–152 (2014).

    Article  Google Scholar 

  7. Soto, C. J. How replicable are links between personality traits and consequential life outcomes? The Life Outcomes of Personality Replication Project. Psychol. Sci. 30, 711–727 (2019).

    Article  PubMed  Google Scholar 

Download references

Author information

Author notes

  1. These authors contributed equally: Joseph Bak-Coleman, Berna Devezer.

Authors and Affiliations

  1. Department of Collective Behaviour, Max Planck Institute of Animal Behavior, Konstanz, Germany

    Joseph Bak-Coleman

  2. Centre for the Advanced Study of Collective Behaviour, University of Konstanz, Konstanz, Germany

    Joseph Bak-Coleman

  3. Department of Business, University of Idaho, Moscow, ID, USA

    Berna Devezer

  4. Department of Mathematics and Statistical Science, University of Idaho, Moscow, ID, USA

    Berna Devezer

Authors
  1. Joseph Bak-Coleman
    View author publications

    Search author on:PubMed Google Scholar

  2. Berna Devezer
    View author publications

    Search author on:PubMed Google Scholar

Contributions

Both authors contributed equally to the drafting and revising of the manuscript.

Corresponding author

Correspondence to Joseph Bak-Coleman.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Human Behaviour thanks Jessica Hullman and Daniel Lakens for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bak-Coleman, J., Devezer, B. Claims about scientific rigour require rigour. Nat Hum Behav 8, 1890–1891 (2024). https://doi.org/10.1038/s41562-024-01982-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue date:

  • DOI: https://doi.org/10.1038/s41562-024-01982-w

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing