Hybrid Sterility Mechanisms in Plant Species

Hybrid sterility represents a fundamental post-zygotic barrier that impedes gene flow between diverged plant lineages and underpins speciation. At its core lie genetic incompatibilities that disrupt gamete formation or function in F1 or later generation hybrids. Classical models invoke Bateson–Dobzhansky–Muller interactions, in which novel allelic combinations at two or more loci trigger pollen or ovule abortion. Increasingly, detailed molecular studies have revealed diverse mechanistic classes, including toxin-antidote systems that selectively eliminate specific gametes, gene-dosage effects arising from copy number variation or structural rearrangements, and segregation distorters that bias allele transmission. These genetic architectures often involve tightly linked multi-gene clusters whose evolutionary histories reflect adaptive divergence and balancing selection. Hybrid sterility mechanisms not only maintain species integrity in the wild but also pose substantial challenges and opportunities for crop improvement. By deciphering the molecular basis of sterility loci, researchers can develop bridge lines, neutral alleles or genome editing strategies to overcome barriers, thus enabling the introgression of beneficial traits from wild relatives. Examples in rice, Mimulus and other model systems illustrate how fundamental insights into incompatibility pathways translate into practical avenues for broadening genetic diversity, enhancing yield potential and ensuring food security on a global scale.

Research from Nature Portfolio

Recent studies have uncovered a toxin-antidote system controlling hybrid male sterility between cultivated and wild rice. One locus encodes a toxin expressed in sporophytic tissues and an antidote active during pollen development, selectively aborting half of the gametes in heterozygotes. Co-presence of distinct toxin-antidote pairs can stack reproductive barriers and reinforce species boundaries.

Structural variation and copy number changes at a key pollen-essential gene have been shown to drive hybrid male sterility in subspecific rice crosses. Tandem duplications of the gene in one subspecies produce a dosage imbalance that suppresses expression of the counterpart allele in pollen, causing selective abortion. Targeted knock-out of extra copies restores fertility and offers a path to bridge incompatible lines.

An asymmetric allelic interaction at a three-gene cluster in interspecific rice hybrids defines a killer-protector mechanism. The killer genes generate an abortion signal, while one gene provides protection only to gametes carrying the corresponding allele. Disruption of any component abolishes the barrier, highlighting opportunities for precision breeding to harness heterosis.

Hybrid Sterility Mechanisms in Plant Species publication trend

The graph below shows the total number of articles in hybrid sterility mechanisms in plant species across all publications each year (not limited to Nature Index journals).

Technical terms

Bateson–Dobzhansky–Muller incompatibility: A genetic model in which incompatible interactions between alleles at two or more loci lead to hybrid sterility or lethality.

Toxin-antidote system: A paired genetic mechanism where a toxin gene kills gametes lacking a linked antidote gene, causing non-Mendelian transmission.

Segregation distorter: A selfish genetic element that biases the usual 50:50 inheritance ratio in its own favour, often by impairing rival gametes.

Gene dosage effect: Sterility resulting from altered copy number of a gene, leading to imbalanced expression and disrupted gametogenesis.

Introgression: The incorporation of genetic material from one species into the gene pool of another through repeated backcrossing.

References

1. A toxin-antidote system contributes to interspecific reproductive isolation in rice. Nature Communications (2023).
2. Selfing Promotes Spread and Introgression of Segregation Distorters in Hermaphroditic Plants. Molecular Biology and Evolution (2024).
3. Genomic structural variation-mediated allelic suppression causes hybrid male sterility in rice. Nature Communications (2017).
4. An asymmetric allelic interaction drives allele transmission bias in interspecific rice hybrids. Nature Communications (2019).
5. Gene duplicates cause hybrid lethality between sympatric species of Mimulus. PLOS Genetics (2018).