Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Matters Arising
  • Published:

High variability in LLMs’ analogical reasoning

Nature Human Behaviour volume 9, pages 1339–1341 (2025)Cite this article

Subjects

The Original Article was published on 31 July 2023

Access through your institution
Buy or subscribe

arising from T. Webb et al. Nature Human Behaviour https://doi.org/10.1038/s41562-023-01659-w (2025)

In a recent study, Webb, Holyoak and Lu1 (henceforth WHL) demonstrated that a large language model (GPT-3, text-davinci-003) could match or even exceed human performance across several analogical reasoning tasks. This result led to the compelling conclusion that LLMs such as GPT-3 possess an emergent ability to reason by analogy. However, the findings were based on a single, proprietary model for which the releasing company provided limited public details and progressively restricted access to the internal probability distributions. Furthermore, text-davinci-003 was deprecated on 4 January 2024, and is no longer available through the OpenAI API. This poses a challenge to replicability in two ways. First, the lack of open access to the model and its recent deprecation make it difficult—if not impossible—for other researchers to verify or build upon the findings. Second, relying on a single model leaves open the question of whether the results can be extended to LLMs as a broader class of objects of scientific investigation. Without testing a diverse range of models, it is unclear whether the observed behaviours are specific to GPT-3 or represent a general property of comparable contemporary LLMs. Replicating experimental results based on proprietary models with public alternatives is thus crucial to ensure that the findings can be reproduced in the future2, generalized to new model instances, and, more generally, to adhere to transparency principles that are of paramount importance in scientific research3.

This is a preview of subscription content, access via your institution

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Performance of LLMs on analogical reasoning tasks.

References

  1. Webb, T., Holyoak, K. J. & Lu, H. Emergent analogical reasoning in large language models. Nat. Hum. Behav. 7, 1526–1541 (2023).

    Article  PubMed  Google Scholar 

  2. Hu, J. & Levy, R. P. Prompting is not a substitute for probability measurements in large language models. In Proc. 2023 Conference on Empirical Methods in Natural Language Processing (eds Bouamor, H. et al.) 5040–5060 (Association for Computational Linguistics, 2023).

  3. Binz, M. et al. How should the advancement of large language models affect the practice of science? Proc. Natl Acad. Sci. 122, e2401227121 (2025).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Brodeur, A., Dreber, A., Hoces de la Guardia, F. & Miguel, E. Reproduction and replication at scale. Nat. Hum. Behav. 8, 2–3 (2024).

    Article  PubMed  Google Scholar 

  5. Gunasekar, S. et al. Textbooks are all you need. Preprint at https://doi.org/10.48550/arXiv.2306.11644 (2023).

  6. Gemma Team. Gemma: open models based on Gemini research and technology. Preprint at https://doi.org/10.48550/arXiv.2403.08295 (2024).

  7. Touvron, H. et al. Llama 2: open foundation and fine-tuned chat models. Preprint at https://doi.org/10.48550/arXiv.2307.09288 (2023).

  8. Jiang, A. Q. et al. Mistral 7B. Preprint at https://doi.org/10.48550/arXiv.2310.06825 (2023).

  9. Apidianaki, M., Fourtassi, A. & Padó, S. Language Learning, Representation, and Processing in Humans and Machines: Introduction to the Special Issue. Comput. Linguist. 50, 1201–1210 (2024).

  10. Grattafiori, A. et al. The Llama 3 herd of models. Preprint at https://doi.org/10.48550/arXiv.2407.21783 (2024).

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, University of Milano - Bicocca, Milan, Italy

    Andrea Gregor de Varda, Chiara Saponaro & Marco Marelli

  2. Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA

    Andrea Gregor de Varda

Authors
  1. Andrea Gregor de Varda
    View author publications

    Search author on:PubMed Google Scholar

  2. Chiara Saponaro
    View author publications

    Search author on:PubMed Google Scholar

  3. Marco Marelli
    View author publications

    Search author on:PubMed Google Scholar

Contributions

A.G.d.V. conceptualized the project, developed the methodology, conducted investigation, curated data, performed visualization, formal analysis and validation, developed software and wrote the original manuscript draft. C.S. performed investigation and data curation, and reviewed and edited the manuscript. M.M. reviewed and edited the manuscript, and supervised and administered the project.

Corresponding author

Correspondence to Andrea Gregor de Varda.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Human Behaviour thanks Anna Ivanova and the other, anonymous reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Methods, Detailed Results, References and Tables 1–5.

Reporting Summary

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

de Varda, A.G., Saponaro, C. & Marelli, M. High variability in LLMs’ analogical reasoning. Nat Hum Behav 9, 1339–1341 (2025). https://doi.org/10.1038/s41562-025-02224-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue date:

  • DOI: https://doi.org/10.1038/s41562-025-02224-3

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics