Coalescent Methods in Phylogenetic Analysis

Coalescent methods provide a statistical framework for inferring species relationships by modelling the ancestral process that gives rise to gene genealogies within and between populations. Central to these approaches is the multispecies coalescent, which accounts for incomplete lineage sorting as a source of discordance between individual gene trees and the overarching species tree. Two broad classes of coalescent methods dominate current practice: full-likelihood or Bayesian approaches that jointly estimate gene trees, species trees and population parameters, and summary methods that combine independently inferred gene trees into a consensus species tree under coalescent theory. Advances in algorithmic efficiency and computational capacity have enabled scaling to genome-scale data sets, yielding more accurate divergence times, branch lengths in coalescent units and measures of node support. Applications span the reconstruction of rapid radiations, the detection of introgression and hybridisation events, and the calibration of molecular clocks. Emerging mixture models and quartet-based support metrics further refine our ability to capture heterogeneous evolutionary histories across loci and genomic regions.

Research from Nature Portfolio

No recent Nature Portfolio content available.

Research from all publishers

Recent developments in branch-length estimation under the multispecies coalescent have yielded methods that accommodate rate heterogeneity across the species tree. By deriving expected gene tree branch lengths in substitution units, new algorithms improve both computational speed and accuracy in estimating species-tree branch lengths directly from gene trees without concatenation.

A multi-tree mixture model has been introduced that allows concatenated alignments to be analysed under multiple predefined topologies, each with its own weight, branch lengths, substitution model and rate heterogeneity. Implemented in a leading phylogenetic software package, this approach accurately recovers signals of incomplete lineage sorting and admixture in primate and platyrrhine data sets, while enabling likelihood-based model comparison to reject single-tree assumptions.

A phylogenomic study of a rapidly radiated avian family has combined chromosome-level assemblies and high-coverage resequencing to reveal extensive gene tree-species tree discordance. The analysis identifies an empirical anomaly zone where the most common gene trees conflict with inferred species topologies, and shows that regions of low recombination retain more consistent coalescent signal, emphasising the need to integrate genomic architecture into phylogenetic inference.

Coalescent Methods in Phylogenetic Analysis publication trend

The graph below shows the total number of articles in coalescent methods in phylogenetic analysis across all publications each year (not limited to Nature Index journals).

Technical terms

Multispecies coalescent: A probabilistic model describing how gene lineages coalesce within diverging species lineages.

Incomplete lineage sorting: The persistence of ancestral genetic variation that causes gene trees to differ from the species tree.

Gene tree: A phylogenetic hypothesis for a single locus or genomic region.

Species tree: The branching diagram representing evolutionary relationships among species.

Anomaly zone: A parameter space in which the most probable gene tree topology does not match the species tree.

Mixture model: A statistical framework combining multiple phylogenetic histories weighted across sites or loci.

References

1. Phylogenomic branch length estimation using quartets. Bioinformatics (2023).
2. MAST: Phylogenetic Inference with Mixtures Across Sites and Trees. Systematic Biology (2024).
3. Gene flow and an anomaly zone complicate phylogenomic inference in a rapidly radiated avian family (Prunellidae). BMC Biology (2024).
4. Fast Coalescent-Based Computation of Local Branch Support from Quartet Frequencies. Molecular Biology and Evolution (2016).