Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Ignoring population structure in hominin evolutionary models can lead to the inference of spurious admixture events

Nature Ecology & Evolution volume 9, pages 225–236 (2025)Cite this article

Subjects

Abstract

Genomic and ancient DNA data have revolutionized palaeoanthropology and our vision of human evolution, with indisputable landmarks like the sequencing of Neanderthal and Denisovan genomes. Yet, using genetic data to identify, date and quantify evolutionary events—such as ancient bottlenecks or admixture—is not straightforward, as inferences may depend on model assumptions. In the last two decades, the idea that Neanderthals and members of the Homo sapiens lineage interbred has gained momentum. From the status of unlikely theory, it has reached consensus among human evolutionary biologists. This theory is mainly supported by statistical approaches that depend on demographic models minimizing or ignoring population structure, despite its widespread occurrence and the fact that, when ignored, population structure can lead to the inference of spurious demographic events. We simulated genomic data under a structured and admixture-free model of human evolution, and found that all the tested admixture approaches identified long Neanderthal fragments in our simulated genomes and an admixture event that never took place. We also observed that several published admixture models failed to predict important empirical diversity or admixture statistics, and that we could identify several scenarios from our structured model that better predicted these statistics jointly. Using a simulated time series of ancient DNA, the structured scenarios could also predict the trajectory of the empirical D statistics. Our results suggest that models accounting for population structure are fundamental to improve our understanding of human evolution, and that admixture between Neanderthals and H. sapiens needs to be re-evaluated in the light of structured models. Beyond the Neanderthal case, we argue that ancient hybridization events, which are increasingly documented in many species, including with other hominins, may also benefit from such re-evaluation.

Access through your institution
Buy or subscribe

This is a preview of subscription content, access via your institution

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Simplified representation of the structured model with no Neanderthal admixture.
Fig. 2: Performance of the structured model with no Neanderthal admixture.
Fig. 3: aDNA D statistics.

Similar content being viewed by others

Data availability

No genomic datasets were generated for this study. Some analyses relied on the public AADR v.54.1 dataset downloaded from https://reich.hms.harvard.edu/allen-ancient-dna-resource-aadr-downloadable-genotypes-present-day-and-ancient-dna-data.

Code availability

All the scripts used in this study (including, but not limited to, model simulation and plotting, statistics calculation, run selection, model comparison, figure plotting) as well as the demes YAML-formatted demographic histories of the 20 selected scenarios are publicly available at https://github.com/sunyatin/qna.

References

  1. Beaumont, M. A. Recent developments in genetic data analysis: what can they tell us about human demographic history? Heredity 92, 365–379 (2004).

    Article  CAS  PubMed  Google Scholar 

  2. Goldstein, D. B. & Chikhi, L. Human migrations and population structure: what we know and why it matters. Annu. Rev. Genomics Hum. Genet. 3, 129–152 (2002).

    Article  CAS  PubMed  Google Scholar 

  3. Harding, R. M. & McVean, G. A structured ancestral population for the evolution of modern humans. Curr. Opin. Genet. Dev. 14, 667–674 (2004).

    Article  CAS  PubMed  Google Scholar 

  4. Ramachandran, S. et al. Support from the relationship of genetic and geographic distance in human populations for a serial founder effect originating in Africa. Proc. Natl Acad. Sci. USA 102, 15942–15947 (2005).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Mazet, O., Rodríguez, W., Grusea, S., Boitard, S. & Chikhi, L. On the importance of being structured: instantaneous coalescence rates and human evolution—lessons for ancestral population size inference? Heredity 116, 362–371 (2016).

    Article  CAS  PubMed  Google Scholar 

  6. Rodríguez, W. et al. The IICR and the non-stationary structured coalescent: towards demographic inference with arbitrary changes in population structure. Heredity 121, 663–678 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  7. Arredondo, A. et al. Inferring number of populations and changes in connectivity under the n-island model. Heredity 126, 896–912 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  8. Ragsdale, A. P. et al. A weakly structured stem for human origins in Africa. Nature 617, 755–763 (2023).

  9. Scerri, E. M. L. et al. Did our species evolve in subdivided populations across Africa, and why does it matter? Trends Ecol. Evol. 33, 582–594 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  10. Scerri, E. M. L., Chikhi, L. & Thomas, M. G. Beyond multiregional and simple out-of-Africa models of human evolution. Nat. Ecol. Evol. 3, 1370–1372 (2019).

    Article  PubMed  Google Scholar 

  11. Prado-Martinez, J. et al. Great ape genetic diversity and population history. Nature 499, 471–475 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Lester, J. D. et al. Recent genetic connectivity and clinal variation in chimpanzees. Commun. Biol. 4, 283 (2021).

  13. Steux, C. et al. On the demographic history of chimpanzees and some consequences of integrating population structure in chimpanzees and other great apes. Preprint at bioRxiv https://doi.org/10.1101/2024.06.14.599042 (2024).

  14. Li, H. & Durbin, R. Inference of human population history from individual whole-genome sequences. Nature 475, 493–496 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Gutenkunst, R. N., Hernandez, R. D., Williamson, S. H. & Bustamante, C. D. Inferring the joint demographic history of multiple populations from multidimensional SNP frequency data. PLoS Genet. 5, e1000695 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  16. Currat, M. & Excoffier, L. Modern humans did not admix with Neanderthals during their range expansion into Europe. PLoS Biol. 2, e421 (2004).

    Article  PubMed  PubMed Central  Google Scholar 

  17. Beaumont, M. A., Zhang, W. & Balding, D. J. Approximate Bayesian computation in population genetics. Genetics 162, 2025–2035 (2002).

    Article  PubMed  PubMed Central  Google Scholar 

  18. Gerbault, P. et al. Storytelling and story testing in domestication. Proc. Natl Acad. Sci. USA 111, 6159–6164 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  19. Chikhi, L. Basato su una storia vera: come fanno i genetisti delle popolazioni umane a conoscere le storie che raccontano? in Evoluzione umana e origini di Homo sapiens - XLVII Seminario sulla Evoluzione biologica e i grandi problemi della biologia (ed Segre, B.) 181–211 (Bardi Edizioni, 2023).

  20. Wakeley, J. Nonequilibrium migration in human history. Genetics 153, 1863–1871 (1999).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Chikhi, L., Sousa, V. C., Luisi, P., Goossens, B. & Beaumont, M. A. The confounding effects of population structure, genetic diversity and the sampling scheme on the detection and quantification of population size changes. Genetics 186, 983–995 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  22. Battey, C. J., Ralph, P. L. & Kern, A. D. Space is the place: effects of continuous spatial structure on analysis of population genetic data. Genetics 215, 193–214 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  23. Green, R. E. et al. A draft sequence of the Neandertal genome. Science 328, 710–722 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Sankararaman, S., Patterson, N., Li, H., Pääbo, S. & Reich, D. The date of interbreeding between Neandertals and modern humans. PLoS Genet. 8, e1002947 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Eriksson, A. & Manica, A. Effect of ancient population structure on the degree of polymorphism shared between modern human populations and ancient hominins. Proc. Natl Acad. Sci. USA 109, 13956–13960 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  26. Eriksson, A. & Manica, A. The doubly conditioned frequency spectrum does not distinguish between ancient population structure and hybridization. Mol. Biol. Evol. 31, 1618–1621 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  27. Durand, E. Y., Patterson, N., Reich, D. & Slatkin, M. Testing for ancient admixture between closely related populations. Mol. Biol. Evol. 28, 2239–2252 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  28. Yang, M. A., Malaspinas, Anna-Sapfo, Durand, E. Y. & Slatkin, M. Ancient structure in Africa unlikely to explain Neanderthal and non-African genetic similarity. Mol. Biol. Evol. 29, 2987–2995 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  29. Lohse, K. & Frantz, L. A. F. Neandertal admixture in Eurasia confirmed by maximum-likelihood analysis of three genomes. Genetics 196, 1241–1251 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  30. Sankararaman, S. et al. The genomic landscape of Neanderthal ancestry in present-day humans. Nature 507, 354–357 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Racimo, F., Sankararaman, S., Nielsen, R. & Huerta-Sánchez, E. Evidence for archaic adaptive introgression in humans. Nat. Rev. Genet. 16, 359–371 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. Sankararaman, S. Methods for detecting introgressed archaic sequences. Curr. Opin. Genet. Dev. 62, 85–90 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. Amos, W. Signals interpreted as archaic introgression appear to be driven primarily by faster evolution in Africa. R. Soc. Open Sci. 7, 191900 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  34. Fu, Q. et al. An early modern human from Romania with a recent Neanderthal ancestor. Nature 524, 216–219 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Hajdinjak, M. et al. Initial Upper Palaeolithic humans in Europe had recent Neanderthal ancestry. Nature 592, 253–257 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  36. Notohara, M. The coalescent and the genealogical process in geographically structured population. J. Math. Biol. 29, 59–75 (1990).

    Article  CAS  PubMed  Google Scholar 

  37. Herbots, H. M. J. D. Stochastic Models in Population Genetics: Genealogy and Genetic Differentiation in Structured Populations. PhD thesis, Queen Mary Univ. London (1994).

  38. Wilkinson-Herbots, H. M. Genealogy and subpopulation differentiation under various models of population structure. J. Math. Biol. 37, 535–585 (1998).

    Article  Google Scholar 

  39. Takahata, N. Genealogy of neutral genes and spreading of selected mutations in a geographically structured population. Genetics 129, 585–595 (1991).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  40. Hudson, R. R. in Oxford Surveys in Evolutionary Biology Vol. 7 (eds Futuyama, D. & Antonovics, J.) 1–44 (Oxford Univ. Press, 1991).

  41. Tajima, F. Relationship between migration and DNA polymorphism in a local population. Genetics 126, 231–234 (1990).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. Bhatia, G., Patterson, N., Sankararaman, S. & Price, A. L. Estimating and interpreting FST: the impact of rare variants. Genome Res. 23, 1514–1521 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. Browning, S. R., Browning, B. L., Zhou, Y., Tucci, S. & Akey, J. M. Analysis of human sequence data reveals two pulses of archaic Denisovan admixture. Cell 173, 53–61 (2018).

    Article  Google Scholar 

  44. Keinan, A., Mullikin, J. C., Patterson, N. & Reich, D. Accelerated genetic drift on chromosome X during the human dispersal out of Africa. Nat. Genet. 41, 66–70 (2009).

    Article  CAS  PubMed  Google Scholar 

  45. Durbin, R. M. et al. A map of human genome variation from population-scale sequencing. Nature 467, 1061–1073 (2010).

    Article  CAS  PubMed  Google Scholar 

  46. Schaefer, N. K., Shapiro, B. & Green, R. E. An ancestral recombination graph of human, Neanderthal, and Denisovan genomes. Sci. Adv. 7, eabc0776 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  47. Skoglund, P. et al. Origins and genetic legacy of Neolithic farmers and hunter-gatherers in Europe. Science 336, 466–469 (2012).

    Article  CAS  PubMed  Google Scholar 

  48. Moorjani, P. et al. A genetic method for dating ancient genomes provides a direct estimate of human generation interval in the last 45,000 years. Proc. Natl Acad. Sci. USA 113, 5652–5657 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  49. Fu, Q. et al. Genome sequence of a 45,000-year-old modern human from western Siberia. Nature 514, 445–449 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  50. Lemaire, L., Jay, F., Lee, I., Csilléry, K. & Blum, M. G. Goodness-of-fit statistics for approximate Bayesian computation. Preprint at https://arxiv.org/abs/1601.04096 (2016).

  51. Gower, G., Picazo, P. I., Fumagalli, M. & Racimo, F. Detecting adaptive introgression in human evolution using convolutional neural networks. eLife 10, e64669 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  52. Ragsdale, A. P. & Gravel, S. Models of archaic admixture and recent history from two-locus statistics. PLoS Genet. 15, e1008204 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  53. Nordborg, M. On the probability of Neanderthal ancestry. Am. J. Hum. Genet. 63, 1237–1240 (1998).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  54. Edmonds, C. A., Lillie, A. S. & Cavalli-Sforza, L. L. Mutations arising in the wave front of an expanding population. Proc. Natl Acad. Sci. USA 101, 975–979 (2004).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  55. Currat, M. & Excoffier, L. The effect of the Neolithic expansion on European molecular diversity. Proc. R. Soc. B 272, 679–688 (2005).

    Article  PubMed  PubMed Central  Google Scholar 

  56. Chikhi, L. et al. The IICR (inverse instantaneous coalescence rate) as a summary of genomic diversity: Insights into demographic inference and model choice. Heredity 120, 13–24 (2018).

    Article  PubMed  Google Scholar 

  57. Currat, M. et al. Comment on “Ongoing adaptive evolution of ASPM, a brain size determinant in Homo sapiens” and “Microcephalin, a gene regulating brain size, continues to evolve adaptively in humans”. Science 313, 172 (2006).

    Article  CAS  PubMed  Google Scholar 

  58. Ahlquist, K. D. et al. Our tangled family tree: new genomic methods offer insight into the legacy of archaic admixture. Genome Biol. Evol. 13, evab115 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  59. Theunert, C. & Slatkin, M. Distinguishing recent admixture from ancestral population structure. Genome Biol. Evol. 9, 427–437 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  60. Vernot, B. & Akey, J. M. Resurrecting surviving Neandertal lineages from modern human genomes. Science 343, 1017–1021 (2014).

    Article  CAS  PubMed  Google Scholar 

  61. Beichman, A. C., Phung, T. N. & Lohmueller, K. E. Comparison of single genome and allele frequency data reveals discordant demographic histories. G3 7, 3605–3620 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  62. Zeberg, H. et al. A Neanderthal sodium channel increases pain sensitivity in present-day humans. Curr. Biol. 30, 3465–3469.e4 (2020).

    Article  PubMed  Google Scholar 

  63. Zeberg, H. & Pääbo, S. The major genetic risk factor for severe COVID-19 is inherited from Neanderthals. Nature 587, 610–612 (2020).

    Article  CAS  PubMed  Google Scholar 

  64. Talamo, S., Kromer, B., Richards, M. P. & Wacker, L. Back to the future: the advantage of studying key events in human evolution using a new high resolution radiocarbon method. PLoS ONE 18, e0280598 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  65. Prüfer, K. et al. A high-coverage Neandertal genome from Vindija Cave in Croatia. Science 358, 655–658 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  66. Kelleher, J., Etheridge, A. M. & McVean, G. Efficient coalescent simulation and genealogical analysis for large sample sizes. PLoS Comput. Biol. 12, e1004842 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  67. Baumdicker, F. et al. Efficient ancestry and mutation simulation with msprime 1.0. Genetics 220, iyab229 (2022).

    Article  PubMed  Google Scholar 

  68. Halldorsson, B. V. et al. Characterizing mutagenic effects of recombination through a sequence-level genetic map. Science 363, eaau1043 (2019).

    Article  CAS  PubMed  Google Scholar 

  69. Jónsson, H. et al. Parental influence on human germline de novo mutations in 1,548 trios from Iceland. Nature 549, 519–522 (2017).

    Article  PubMed  Google Scholar 

  70. Skov, L. et al. The nature of Neanderthal introgression revealed by 27,566 Icelandic genomes. Nature 582, 78–83 (2020).

    Article  CAS  PubMed  Google Scholar 

  71. Durvasula, A. & Sankararaman, S. Recovering signals of ghost archaic introgression in African populations. Sci. Adv. 6, eaax5097 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  72. Iasi, L. N. M., Ringbauer, H. & Peter, B. M. An extended admixture pulse model reveals the limitations to human–Neandertal introgression dating. Mol. Biol. Evol. 38, 5156–5174 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  73. Jacobs, G. S. et al. Multiple deeply divergent Denisovan ancestries in Papuans. Cell 177, 1010–1021 (2019).

    Article  Google Scholar 

  74. Kamm, J., Terhorst, J., Durbin, R. & Song, Y. S. Efficiently inferring the demographic history of many populations with allele count data. J. Am. Stat. Assoc. 115, 1472–1487 (2020).

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

We thank O. Mazet, S. Boitard, B. Parreira, A. Arredondo, P. Faux, B. Servin, F. Halkett, R. Leblois and members of the Population and Conservation Genetic group for their support and for useful discussions on this topic. We would also like to acknowledge the Bioinformatic Unit and the Informatics Team of the IGC, as well as CALMIP (project P23002) for their help and support with computational resources. We also thank O. Mazet and S. Boitard for their useful comments on the first version of this manuscript. L.C. and R.T. were funded by Fundação para a Ciência e Tecnologia (ref. PTDC-BIA-EVL/30815/2017, L.C.). This work was also supported by the LABEX entitled TULIP (ANR-10-LABX-41 and ANR-11-IDEX-0002-02, L.C. and R.T.), the IRP BEEG-B (International Research Project—Bioinformatics, Ecology, Evolution, Genomics and Behaviour, L.C.) as well as the DevOCGen project, funded by the Occitanie Regional Council’s ‘Key Challenges BiodivOc’ programme. We acknowledge an Investissement d’Avenir grant of the Agence Nationale de la Recherche (CEBA: ANR-10-LABX-25-01, L.C. and R.T.).

Author information

Authors and Affiliations

  1. UMR DIADE, Université de Montpellier, IRD, CIRAD, Montpellier, France

    Rémi Tournebize

  2. Instituto Gulbenkian de Ciência, Oeiras, Portugal

    Rémi Tournebize & Lounès Chikhi

  3. Centre de Recherche sur la Biodiversité et l’Environnement (CRBE) UMR 5300, Université de Toulouse, CNRS, IRD, Toulouse INP, Université Toulouse 3 Paul Sabatier (UT3), Toulouse, France

    Rémi Tournebize & Lounès Chikhi

  4. Centre for Ecology, Evolution and Environmental Changes (cE3c), Faculdade de Ciências da Universidade de Lisboa, Lisbon, Portugal

    Lounès Chikhi

Authors
  1. Rémi Tournebize
    View author publications

    Search author on:PubMed Google Scholar

  2. Lounès Chikhi
    View author publications

    Search author on:PubMed Google Scholar

Contributions

R.T. and L.C. designed the study. R.T. wrote the scripts, performed the simulations and analysed the data. R.T. and L.C. interpreted the results and wrote the manuscript.

Corresponding authors

Correspondence to Rémi Tournebize or Lounès Chikhi.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Ecology & Evolution thanks Vagheesh Narasimhan, Joshua Schraiber and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Text, Figs. 1–74, Tables 1–17 and References.

Reporting Summary

Peer Review File

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tournebize, R., Chikhi, L. Ignoring population structure in hominin evolutionary models can lead to the inference of spurious admixture events. Nat Ecol Evol 9, 225–236 (2025). https://doi.org/10.1038/s41559-024-02591-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue date:

  • DOI: https://doi.org/10.1038/s41559-024-02591-6

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing