Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • NEWS AND VIEWS

Expert-level test is a head-scratcher for AI

Conventional benchmarks are becoming less effective at assessing AI performance, but a multi-disciplinary test has set AI systems a fresh challenge.
By
  1. Katherine M. Collins

    1. Katherine M. Collins is in the Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, USA.

  2. Joshua B. Tenenbaum

    1. Joshua B. Tenenbaum is in the Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, USA.

The quest to build increasingly powerful artificial-intelligence systems demands a clear definition of what counts as intelligence, and how it should be measured. AI systems are typically assessed using tests called benchmarks. These are often sets of question–answer pairs in which each question has a definitive, verifiable answer that enables the AI tool to be scored automatically. Benchmarks have been used to assess how quickly frontier AI models (such as those behind OpenAI’s ChatGPT and Google’s Gemini systems) are improving in capacities ranging from general common sense1 and domain-specific knowledge2 to code generation3 and mathematical problem-solving4. However, over time, many benchmarks become less effective at identifying genuine advances — a phenomenon known as benchmark saturation.

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Nature 649, 1115-1116 (2026)

doi: https://doi.org/10.1038/d41586-025-04098-x

References

  1. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A. & Choi, Y. in Proc. 57th Annu. Meet. Assoc. Comput. Linguist. 4791–4800 (Association for Computational Linguistics, 2019).

    Article  Google Scholar 

  2. Hendrycks, D. et al. in 8th Int. Conf. Learn. Represent. (ICLR, 2020).

    Article  Google Scholar 

  3. Jimenez, C. E. et al. Preprint at arXiv https://doi.org/10.48550/arXiv.2310.06770 (2024).

  4. Cobbe, K. et al. Preprint at arXiv https://doi.org/10.48550/arXiv.2110.14168 (2021).

  5. Center for AI Safety, Scale AI & HLE Contributors Consortium Nature 649, 1139–1146 (2026).

    Article  Google Scholar 

  6. Collins, K. M. et al. Nature Hum. Behav. 8, 1851–1863 (2024).

    Article  PubMed  Google Scholar 

  7. Chu, J., Tenenbaum, J. B. & Schulz, L. E. Trends Cogn. Sci. 28, 628–642 (2024).

    Article  PubMed  Google Scholar 

  8. Getzels, J. W. in Frontiers of Creativity Research: Beyond the Basics (ed. Isaksen, S. G.) 88–102 (Bearly, 1987).

    Google Scholar 

Download references

Competing Interests

The authors know some colleagues who participated in the HLE question generation and review process.

Subjects

Latest on:

Nature Careers

Jobs

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing

Search

Advanced search

Quick links