Introduction

In cooperatively breeding species, offspring receive care not only from their parents but also from other group members, who are referred to as helpers1,2,3. Such complex social systems evolved independently multiple times across various taxa, including mammals, birds, fishes, and various invetebrates1,3,4, making them an excellent model to understand the evolution of cooperative and complex animal societies. According to classical theory, indirect benefits through kin selection were traditionally regarded as the principal evolutionary driver of cooperative breeding systems5,6,7. However, although the importance of direct benefits as a driving force in the evolution of cooperative breeding has gained increasing recognition recently7,8,9, there remains no broad consensus on how cooperative breeding evolved.

Recent advances in comparative methods have spurred active discussions on the ecological and social factors that promote the evolution of cooperatively breeding systems10,11,12,13. It is crucial to examine the relationship between life history traits and cooperative breeding across taxa to understand how cooperation evolves in response to social and ecological factors. While some studies have found evolutionary correlations between cooperatively breeding systems and life history traits13,14, others have yielded conflicting results13,15,16. For instance, a large-scale analysis of mammals has shown that cooperative breeding has evolved especially in taxa that produce multiple offspring in a single breeding event17. In contrast, clutch size (number of eggs released during a single breeding event) in cooperatively breeding bird species is smaller than that in non-cooperative species14. These findings suggest that there is a great variation in the evolutionary mechanisms driving and sustaining cooperatively breeding systems, depending on the taxonomic group. However, only a few studies have focused on the relationship between cooperative breeding and life-history traits in animals that are neither birds14,16,18 nor mammals13,17. Therefore, to fully understand the evolutionary relation between cooperatively breeding systems and life history traits, comparative studies in other animal taxa are needed.

Here, the ~90 lamprologine cichlid species of Lake Tanganyika represent a promising model system19,20. Lamprologine cichlids, most of which are endemic to Lake Tanganyika, show high variation in social and mating systems, ranging from monogamy and polygyny to polyandry, polygynandry, and highly complex cooperatively breeding systems21. Brood-care helpers in these fishes commonly engage in egg care, nest maintenance, and brood defense22,23,24,25,26. The combination of a well-resolved phylogenetic tree, their monophyletic origins, and their diverse ecological and social systems makes lamprologines a suitable model system for comparative analyses of social evolution11,12,19.

This study aimed to elucidate the evolutionary history of cooperative breeding and its correlation with life history traits through phylogenetic comparative analyses of lamprologine cichlid species endemic to Lake Tanganyika. First, the evolutionary trajectory of cooperative breeding in these cichlids was examined. Species were classified as either cooperative or non-cooperative breeding, and ancestral state reconstruction was conducted. Second, we tested the relationships among social system, body size, and habitat type as ecological factors that potentially drive the evolution of cooperative breeding among cichlids27,28,29. In particular, we focused on the relationship between the social system and body size, based on the premise that the smaller body size may increase predation risk. To complement these analyses, we also conducted stomach content analyses to assess whether predation risk is higher in smaller fish species. Finally, we tested whether the transition from a non-cooperative to a cooperative social system influenced patterns of egg investment, including clutch size and egg size.

Results and discussion

Repeated evolution of cooperative breeding in lamprologines

Previous studies8,27 suggest the convergent evolution of cooperative breeding in Tanganyikan cichlids, but the number of evolutionary events has remained unclear. Here, we infer the evolutionary history of this social system using the most recent phylogenetic tree based on whole-genome information for all cichlid species endemic to Lake Tanganyika19. To that end, ancestral state reconstruction with 1000 stochastic character maps was performed using 73 lamprologines (n = 29 cooperative breeders and n = 44 non-cooperative breeders; Supplementary Table S1). The results showed that cooperative breeding independently evolved 8–11 times (median = 9) from non-cooperatively breeding ancestors (Fig. 1a), and there was no evidence for any secondary loss of cooperative breeding. Cooperatively breeding cichlids first emerged roughly 4 million years ago from ancestral non-cooperatively breeding cichlids (Fig. 1b), and similar social evolution appears to have subsequently occurred in multiple clades.

Fig. 1: Phylogenetic distribution of cooperative breeding and diversity of life history traits among lamprologine cichlids.
Fig. 1: Phylogenetic distribution of cooperative breeding and diversity of life history traits among lamprologine cichlids.The alternative text for this image may have been generated using AI.
Full size image

a Blue and red circles indicate cooperative (n = 31 taxa from n = 27 species) and non-cooperative breeders (n = 50 taxa from 44 species), respectively. Variation in clutch size, egg size, body weight, total length, and habitat is indicated by color intensity. Pie charts represent the likelihood of the social system as estimated by ancestral-state reconstruction. Tips with unidentified social systems due to missing life-history data are depicted as circles with diagonal lines. Nodes of tips for which the ancestral state of the social system could not be estimated because of missing tip data are indicated by gray circles. Underlined taxon names denote taxa for which life-history trait data were obtained through standardized field sampling. The scale bar corresponds to one million years. b A lineage-through-time (LTT) plot visualizing the relationship between time (million years) and the cumulative number of taxa with non-cooperative (blue) and cooperative breeding (red). Each thin line represents a single LTT plot for 1/1000 stochastic character maps; the thicker central lines represent the average. The Bayesian MCMC tree was produced by Ronco et al., 2021.

The variation in model outcomes on the repeated evolution of cooperative breeding warrants further consideration. First, Neolamprologus niger, a basal species within the clade encompassing both non-cooperative and cooperative breeders (Fig. 1), was classified as a cooperative breeder in this study30, based on aquarium observations30 and personal observations in the field (M. Taborsky, personal observation). When N. niger was instead treated as an unclassified species in the dataset, the outcomes of ancestral state reconstruction shifted slightly from the original analysis, with cooperative breeding inferred to have evolved 7–10 times (median = 8) from non-cooperative ancestors (Supplementary Figs. S1 and S2). Alternatively, when N. niger was designated as a non-cooperative breeder, the number of transitions from non-cooperative to cooperative breeding was estimated at 7–11 (median = 8).

Moreover, several inferred independent transitions in ancestral state reconstructions rely on the presence of a single cooperative species (N. meeli, N. niger, N. buescheri, N. bifasciatus, and N. obscurus) (Fig. 1). As the results of ancestral state reconstruction were based on a published phylogenetic tree19, it was not possible to fully incorporate phylogenetic uncertainty in our analyses. Examination of the tree15 indicates that the clade comprising N. prochilus and N. obscurus is sister to the clade containing N. bifasciatus; however, this relationship is supported only weakly (bootstrap = 64%)19, leaving open the possibility that either the N. prochilus–obscurus clade or N. bifasciatus may instead form a sister group with N. buescheri. Under such a topology, the inferred number of evolutionary transitions to cooperative breeding would be reduced by one, and a subsequent loss of cooperative breeding could be detected. Regarding N. niger, uncertainty lies not only in the characterization of its social system but also in its placement in the phylogenetic tree19, both of which may influence the inferred frequency of cooperative breeding evolution. This differs from N. meeli, which represents an additional, independent evolutionary origin of a cooperative social system. Here, no other cooperatively breeding species are present in the Lepidiolamprologus clade; the effects of phylogenetic uncertainty are hence expected to be minimal.

Second, several studies have shown that the cooperative breeding system of Julidochromis ornatus and related species differs from that of other cooperatively breeding species. The helpers in this species are largely unrelated to dominants, and the reproductive skew is low24,31,32,33. Therefore, treating the cooperative systems of some cichlid species as equivalent to those of other cooperatively breeding taxa remains debatable. In addition to these interspecific variations in the reproductive skew in cooperatively breeding cichlids, a comprehensive comparative analysis of the prevalence of cooperative breeders, the modes of helping behavior (e.g., direct and indirect care), and their impacts on and responses from offspring would be essential34. Taken together, future revisions to behavioral descriptions of social structures and phylogenetic analysis among Lake Tanganyika cichlids may therefore necessitate corresponding updates to the conclusions drawn in this study.

Cooperative breeding and body size

Elucidating the evolutionary relationship between life history traits and social systems allows us to understand the factors that drive the evolution of cooperative breeding. Field data on three life-history traits (body size of male and female breeders, clutch size, and egg size) were obtained for 39 of the 86 taxa (14 cooperative and 25 non-cooperative breeders). These species well represented trait variation within the Lamprologini tribe (Fig. 1). The data on breeding habitats (classified into five types: crevices and caves, facultative shell-brooding, obligatory shell-brooding, rocky areas, and sandy or muddy substrates) were obtained from public databases. A phylogenetic generalized linear model (PGLM) with a binomial error structure, based on field data from these 39 cichlid taxa, revealed that the social system was significantly affected by the body size of mature females (z = –2.48, P = 0.013) (Fig. 2a and Table 1), with cooperative breeders exhibiting smaller body sizes than non-cooperative breeders (Fig. 3a). Furthermore, in terms of habitat effects, the intercept was significantly positive only for the category “crevices and caves,” indicating that habitat type modulates the relationship between social system and body size (Figs. 2a and 3a and Table 1). Similar results were obtained from an analysis using the body size of male breeders (Figs. 2b and  3b and Table 1).

Fig. 2: Evolutionary transition to cooperative breeding and its association with body size.
Fig. 2: Evolutionary transition to cooperative breeding and its association with body size.The alternative text for this image may have been generated using AI.
Full size image

Phylogenetic generalized linear models (PGLMs) were fitted separately for female (a) and male breeders (b). The models incorporated social system (1 = cooperative breeding; 0 = non-cooperative breeding) as a binary response variable, with body size and habitat as explanatory variables. The vertical axis represents each explanatory variable, while the horizontal axis depicts the z-values of their coefficients. Bars extending to the left and right indicate 95% confidence intervals ( ± 1.96).

Fig. 3: Body size differences between cooperatively breeding and non-cooperatively breeding cichlids in Lake Tanganyika living in five distinct breeding habitats.
Fig. 3: Body size differences between cooperatively breeding and non-cooperatively breeding cichlids in Lake Tanganyika living in five distinct breeding habitats.The alternative text for this image may have been generated using AI.
Full size image

a Depicts the differences in body weight of female breeders, b of male breeders. Female and male body weight was obtained for 39 species by standardized field sampling, with each point representing a species. c depicts differences in total length (TL) between cooperatively and non-cooperatively breeding cichlids, based on data obtained from FishBase. TL measurement methods were not fully standardized, and there was no distinction between sexes. Data for 61 species were included, with each plot representing a species. N. niger represents the basal species of the clade of cooperatively breeding cichlids; however, its status as a cooperatively breeding cichlid is based on anecdotal observations.

Table. 1 Results of Phylogenetic Generalized Linear Model (PGLMs) analyses of potential factors influencing the social system (cooperative or non-cooperative breeding) of lamprologine cichlids
Full size table

To assess the reliability of our analyses based on field-obtained data, we performed PGLMs using body size data from 61 taxa available on FishBase (https://fishbase.org/) and their phylogenetic tree. Also, this analysis revealed a significant association between social system and body size, as well as a significant effect of certain habitat types on the intercept (Supplementary Table S2). The relationship between body size and social system remained robust when N. niger was classified as a non-cooperative breeder (Supplementary Table S3) or excluded from the analysis altogether (Supplementary Table S4). These findings suggest that cooperative breeding evolved more readily in smaller species that might be more vulnerable to predation.

To further support this argument, we conducted stomach content analysis of predatory fish species of Lake Tanganyika ranging from 50 to 500 mm in standard length (SL). These analyses revealed that most prey were consistently small cichlids of less than 50 mm SL (up to ~90 mm SL) (Supplementary Fig. S3). Predators of larger fish should be scarce in the littoral zone of Lake Tanganyika35,36, and larger species, such as Lepidiolamprologus elongatus and L. cunningtoni, experience reduced predation pressures as they mature35. Conversely, smaller cichlid species are more susceptible to predation even after reaching sexual maturity35,37. Interpopulation comparisons in the cooperatively breeding cichlid N. pulcher revealed that group composition is strongly linked with the number of large fish predators, which poses a threat especially for smaller group members27. Moreover, in populations with higher predation risk, helpers tend to delay their dispersal28. By staying in their natal territory, these fish benefit from access to shelters28 as well as shared shelter maintenance and antipredator defense38,39,40. As a result, philopatric N. pulcher have higher lifetime fitness than dispersing individuals37.

Most large lamprologine cichlid species lay their eggs on rock surfaces41. This may be because their larger body size allows them to defend their offspring from predators regardless of nest location. In contrast, smaller species have limited ability to defend against predators, leading smaller lamprologines to build more complex nests constructed from rocks of various sizes, rock crevices, or gastropod shells to conceal their eggs and juveniles. The construction of such nests entails maintenance costs for territory holders42,43. Therefore, the costs of delayed dispersal for helpers may be compensated by the benefits of accessing shelters that are collectively maintained by group members. Shelter maintenance incurs energy costs42,43, and cooperatively working with group members can decrease these costs40. These findings indicate that breeders and helpers of smaller cichlids can gain more benefits from forming long-lasting cooperatively breeding groups.

The present analysis was predicated on the assumption that body size is strongly correlated with predation pressure. Indeed, consistent with patterns observed in many fish species, stomach content analyses indicated that smaller fish in Lake Tanganyika are subjected to disproportionately higher predation pressure than their larger counterparts. It should be noted, however, that body size is influenced by a multitude of ecological factors beyond predation44. What this study demonstrates is a phylogenetically controlled correlation between social system and body size—an association between social system and predation pressure that warrants further rigorous investigation.

Clutch size and egg size in relation to cooperative breeding

We examined whether clutch size or egg size is evolutionary related to the social system using data for 14 cooperative and 25 non-cooperative breeders, their breeding habitats, and the body size of female breeders. The Bayesian phylogenetic mixed model (BPMM) revealed a positive correlation between clutch size and body size (95% CI: 0.62–0.95) and revealed that cooperative breeders had smaller clutch sizes compared with non-cooperative breeders (95% CI: −0.87 to −0.42) (Figs. 4a and 5a and Table 2). Moreover, species inhabiting sandy or muddy areas had a larger clutch size than those inhabiting the other habitats (95% CI: 0.14–1.05) (Figs. 4a and 5a and Table 2).

Fig. 4: Evolutionary transition to cooperative breeding and its association with egg investment modes.
Fig. 4: Evolutionary transition to cooperative breeding and its association with egg investment modes.The alternative text for this image may have been generated using AI.
Full size image

Factors affecting the evolution of clutch size (a) and egg volume (b) among lamprologine cichlids analyzed using Bayesian phylogenetic mixed model analysis (BPMM). The density plot of the posterior distribution for each coefficient was shown. The colored area of the density plot indicates 95% CIs.

Fig. 5: Differences in egg investment by female breeders between cooperatively and non-cooperatively breeding cichlids in Lake Tanganyika.
Fig. 5: Differences in egg investment by female breeders between cooperatively and non-cooperatively breeding cichlids in Lake Tanganyika.The alternative text for this image may have been generated using AI.
Full size image

Relationships between a clutch size and egg size, and b egg volume and female body weight, and their habitat (crevices/caves, facultative and obligatory shell-brooding, rocky areas, and sandy/muddy substrates). The points represent individuals (n = 155 females from 39 species). Body weight, egg size, and clutch size were log10-transformed. Regression lines were estimated using the median of the posterior distribution derived from the Bayesian phylogenetic mixed models (BPMMs).

Table. 2 Results of Bayesian Phylogenetic Mixed Models (BPMMs) of potential factors influencing egg investment mode in lamprologine cichlids
Full size table

According to the life history hypothesis14, in an overcrowded breeding environment, resulting from a low annual mortality and low population turnover, natural selection may favor smaller clutch sizes in cooperatively breeding birds14. While recent comparative analyses for avian species do not support a part of this assumption16, the assumed relationship between social system and clutch size appears to be present in cichlid fishes. There are, however, alternative explanations for why cooperatively breeding cichlids have smaller clutches. In these species, young juveniles are exposed to predation pressure and are cared for by breeders and helpers45. While such social systems can increase the survival rate of offspring45, the number of family members within a breeding group may be limited by increased within-group aggression between differently sized and aged individuals4. Here, larger siblings behave aggressively and drive smaller siblings out of the breeding group when resource availability is low46,47. Such offspring will be exposed to repetitive harassment from siblings46 and/or high-predation pressure35,46,48,. If female breeders spawn large clutches in this situation, parents suffer from fitness loss49 as this would further fuel within-group aggression. This suggests that clutch size reduction in cooperatively breeding cichlids may contribute to mitigating intense competition among group members.

Based on previous studies, helpers in lamprologines have low dispersal even when the space for establishing an independent breeding territory is not limited50. This suggests that habitat saturation is not a key factor for delayed dispersal in cooperatively breeding cichlids in Lake Tanganyika8,28,50. Thus, a more feasible hypothesis is that both the carrying capacity of helpers in nests and the high survival rate of offspring before dispersal may limit the clutch size of cooperatively breeding cichlids.

In some cooperatively breeding species, such as superb fairy-wrens (Malurus cyaneus)51 and fairy cichlids (N. pulcher)52, egg size decreases when more helpers are present in the territory. In contrast to these within-species comparisons, the BPMM in the present study revealed that egg volume was associated with body size (95% CI: 0.07–0.35) but not with social systems (95% CI: −0.35 to 0.14) (Figs. 4b and 5b and Table 2). Similarly, in birds and mammals, there is no evidence that cooperatively breeding species generally have smaller offspring in comparative studies17,53. It is possible that cooperative breeders can change the egg investment at the individual level, but that benefits might be species-/lineage-specific.

How is the evolution of cooperative breeding linked to life history traits?

In this study, we found that two life history traits (body size and clutch size) were evolutionarily related to cooperative breeding (Figs. 3 and 5). A recent study reported that feeding ecology directly affects the evolution of cooperative breeding12. It was argued that a non-predatory or non-carnivorous diet promotes cooperative breeding because it leads to habitat saturation and constraints on dispersal due to increased population densities; abundant resources also decrease the costs incurred by dominants cohabiting with subordinate individuals12. Species with a higher reliance on carnivory, in turn, might tend to have lower population densities and are less likely to evolve cooperative breeding12. To shed more light on this relationship between previous and our findings, we obtained data from stable isotope analysis collected from a previous study19, including δ15N values, as a proxy for the trophic level or feeding ecology of individual species. Subsequently, a phylogenetic path analysis (PPA)54 was performed to examine the evolutionary transitions of life history traits, including clutch size and body size, feeding ecology, and social system (cooperative or non-cooperative breeding). The 16 candidate models incorporated causal hypotheses reflecting four primary predictions, grounded in the results of the BPMMs and previous studies: (Hypothesis A) body size influences the evolution of social systems and clutch size; (Hypothesis B) feeding ecology affects the evolution of social systems; (Hypothesis C) body size drives feeding ecology; and (Hypothesis D) social system drive the clutch size.

The most optimal PPA model showed that body size directly and significantly influenced social system, clutch size, and feeding ecology; cooperative breeding also resulted in a smaller clutch size (Fig. 6). Although we were unable to directly measure potential predation pressure occurring in the species in this study, body size can be seen as a life history trait that can be linked to predation pressure. Therefore, these results indicate that each relationship agrees with the following three hypotheses: Hypothesis A) predation risk of species with smaller body size promotes the evolution of cooperative breeding, and Hypothesis D) cooperative breeding restricts larger clutch sizes. Although body size influenced the feeding ecology (or trophic level) of each species (Hypothesis C), we could not find a significant relationship between social system and feeding ecology, as suggested by Dey et al.12. Instead, the described influence of diet on the evolution of cooperative breeding can be explained by the fact that piscivorous fishes are larger than non-piscivorous fishes.

Fig. 6: Evolutionary paths of the social system and associated life history traits in lamprologine cichlids estimated by phylogenetic path analysis.
Fig. 6: Evolutionary paths of the social system and associated life history traits in lamprologine cichlids estimated by phylogenetic path analysis.The alternative text for this image may have been generated using AI.
Full size image

Arrows indicate direct relationships between traits (blue =  positive effect; red = negative effect), while gray broken lines represent paths included in candidate models but not in the best (most optimum) model of phylogenetic path analysis selected by CICc comparison (see Supplementary Fig. S4 and Table S6 for statistics and assumed causal hypotheses in this analysis). The standardized path coefficient is shown near each arrow.

While cooperative breeding has evolved in 9% of all bird species55 and 1% of mammals56,57, it only occurs in less than 0.001% of teleost fish species (ca. 30 cichlid species in the tribe Lamprologini) found only in Lake Tanganyika30,58. However, cooperatively breeding lamprologine cichlids in Lake Tanganyika are receiving increasing attention as an evolutionary model of cooperative and complex societies that differs from those of birds and mammals12,58,59. Interestingly, no cooperatively breeding species have been reported among the mouth-brooding cichlids, which constitute approximately half of the cichlid species endemic to Lake Tanganyika. This may suggest that, in addition to the ecological factors examined in this study, certain preadaptations—such as the long-term maintenance of breeding nests—may facilitate the evolution of social complexity among fishes.

The findings of this study support an evolutionary association between cooperative breeding and two life history traits: body size and clutch size. Although these relationships should be re-evaluated in future studies with a broader range of species and detailed life history information, a plausible scenario is that high-predation pressure on smaller cichlid species promotes the evolution of cooperative breeding, and that constraints on the number of helpers within breeding groups result in clutch size limitation. These findings highlight the ecological factors that drive the evolution of social complexity and the consequent conflicts that arise among cooperative group members.

Methods

Data collection of the social system and life history traits

Data collection followed the taxonomic inventory of Ronco et al.16. This review suggested that single described species sometimes consist of multiple distinct sub-units (termed taxa from now on), including local varieties and undescribed species. Social system data were collected from previous studies (Supplementary Table S1). According to data collection, we determined that 31 taxa from 27 species were cooperative breeders, while 50 taxa from 44 species were non-cooperative breeders (Fig. 1). If a single species contained multiple taxa, we assumed they had the same social systems. The correspondence between scientific names and taxon names used in Ronco et al.19,20 is provided in Supplementary Table S1.

The type of substrates used for spawning was obtained from a public fish-specific database (FishBase; https://www.fishbase.se) and previous studies, and classified into five breeding habitat types (Rocky area, crevices/caves, sandy/muddy area, facultative shell-brooding, and obligatory shell-brooding) (Supplementary Table S1). Littoral rocky areas are the main breeding habitat for cichlids in Lake Tanganyika, with many species using the surfaces and undersides of rocks as breeding habitat. Among the species using rocky areas, those that use rock crevices and caves as breeding habitats, such as Julidochromis spp.24,32, are classified as species inhabiting “crevices/caves”. Some species of Tanganyikan cichlids used empty gastropod shells of Neothauma tanganyicense as shelters and breeding habitat to effectively avoid predation60. Two dependence types were classified based on shell brooding: (i) obligatory shell brooder in which species use shells throughout their lifespan; (ii) facultative shell brooder in which species use both shells and other substrates depending on the ecological situation and body size44,60. As the entrances of the shells are small, they limit the maximum body size of fish that can use them44. Telmatochromis temporalis and N. multifasciatus are sometimes described as facultative shell brooders because they use crevices between small rocks as breeding nests in some populations60; however, in this study, they were considered obligate shell brooders because they use empty shells in most populations while rock pupulations are an exception and because they use empty shells as breeding nests throughout their lifespan in the observed population44. These data were used to perform further analyses to reveal the evolutionary history of cooperative breeding and the correlation between social system and life history traits. The interplay between social and mating systems has been a subject of intense debate recently11,12,61. Nevertheless, this study refrains from considering this relationship because of the inadequate description of the mating system, high degree of plasticity, and ambiguity of relevant criteria.

We also collected body size data of lamprologines endemic to Lake Tanganyika from FishBase. Data on the maximum total length (in cm) were obtained from 66 lamprologines. Of these taxa, 61 taxa were matched to social system data (n = 22 cooperative breeders and n = 39 non-cooperative breeders). Note that FishBase provides a single species-level value of maximum total length without sex differentiation.

Field work and data collection for life history traits

Life history traits such as the clutch size (number of eggs released during a single spawning event), egg size, and body size of adult females were obtained for 14 cooperatively and 25 non-cooperatively breeding lamprologines. Sexually mature females were either obtained from local fishers and fish markets (n = 38 females from 14 taxa) or were caught using fine-mesh hand nets and/or gill nets while SCUBA diving (24 taxa (n = 82 females). Captured fish were sacrificed by an overdose of the anesthetic MS-222 (Wako Pure Chemical Industries). All specimens were preserved in a freezer before measurements. To increase the sample size, fish from previous collections stored at Osaka City University (n = 22 females of 10 species) and the Hokkaido University Museum (n = 13 females of 7 species) were also used for life history traits analyses. These samples were captured during previous field research and preserved in 10% formalin or 50% isopropanol. Only samples with mature eggs in their ovary were used.

The weight of the sample (in 0.001 g), and ovary weight (in 0.001 g) were measured for these female samples. For life history trait analyses, the total weight of the sample minus the gonad weight was taken as body weight (BW). This is because female body weight was greatly affected by changes in ovary weight due to the breeding cycle. Mature eggs in the ovary, which were distinguished from immature eggs based on coloration and size, were counted using a tally counter (clutch size). Eggs were subsequently preserved in 5% formalin. Up to five eggs were randomly selected from each clutch and photographed using a stereomicroscope (HD212; AMscope, USA) equipped with a digital camera (note that if the clutch size was smaller than five, all eggs were taken). From these images, the major and minor radii of each egg were measured with ImageJ ver. 1.52q58. Egg volume (mm³) was then calculated using the formula:\({{{\rm{Volume}}}}=\frac{4}{3}\pi \times ({{{\rm{major\; radius}}}})\times {({{{\rm{minor\; radius}}}})}^{2}\). The volumes of the five eggs obtained from each individual were averaged per individual.

To test the effects of body size differences between females and males, we collected male breeders from 39 taxa in the field. We conducted a short time (ca. 3 min) behavioral observation of a target species in order to determine the male status as a male breeder, and captured it using hand and gill nets when it exhibited behavior of roaming the territory or visiting the breeding nest of female breeders and participating in parental care (brood defense and digging for breeding nest) (n = 203 males from 30 taxa). After the sacrifice of males by an overdose of MS-222, we measured the weight of the sample (in 0.001 g). For the male analyses, the total sample weight (measured in 0.001 g units) was directly used as BW. Specimens preserved at the Hokkaido University Museum were also measured in the same manner (n = 148 males from 11 taxa). Their breeding status was recorded at the time of capture, and only individuals engaged in parental care were used for the measurements. In total, data were obtained from 155 females and 351 males of 39 taxa (n = 14 cooperative and n = 25 non-cooperative breeders).

Stomach content analysis

To ascertain whether smaller fishes are more vulnerable to predation, stomach content analysis was conducted on seven species of major piscivores of Lake Tanganyika. These predators were captured using hand nets or gill nets at Kasenga point (8°71’52.2“S, 31°14’18.6“E) and overdosed with an anesthetic (MS-222, Wako Pure Chemical Industries). Then, 38% formalin was injected into their abdominal cavity to fix the stomach contents. The SL of both predators and prey fishes was measured (impossible to measure the body weight of prey) and defined as body size for stomach contents analyses. In total, data on 279 prey fish in the stomach of 279 predators (L. lemairii, n = 3; L. elongatus, n = 62, L. kendalli, n = 68; L. mimicus, n = 32; L. profundicola, n = 38, N. fasciatus, n = 40, Mastacembelus spp., n = 36) were obtained. Species identification of prey fish in the stomach was often impossible and thus was not carried out. Note that there are several piscivorous catfish species (e.g., Dinotopterus cunningtoni) in Lake Tanganyika, but as they are nocturnal, they were not considered in this study.

Phylogenetic analyses

Ancestral state reconstruction

To account for the effects of phylogeny, we included a modified, time-calibrated phylogenetic tree of the Tanganyika cichlids19 in our analyses. To that aim, we pruned the Ronco et al. tree19 using the “droptip” function in R (ver 4.4.2)62 in the package ape (ver 5.8-1)63. Using the trimmed tree, ancestral states of social systems were assessed using phytools (ver 2.4-4)64. We included all 81 taxa (31 taxa from 27 cooperative breeders and 50 taxa from 44 non-cooperative breeders) in the analysis. We constructed two models for the probability of change between traits (All Rates Different [ARD] and Equal Rates [ER] models). In both models, the stationary probabilities of root traits were estimated using the maximum likelihood methods of FitzJohn (2012)65. Comparisons based on AIC (Akaike Information Criterion)66 values show that the ARD with FitzJohn root estimation provided the best fits to the data and represents the non-uniformity of evolutionary rates and heterogeneous transition probabilities between social systems (ARD, AIC = 60.088; ER, AIC = 61.426). Subsequently, based on the best model, 1000 stochastic character mappings were performed to estimate the probability of transitions between social systems64. Moreover, a lineage-through-time plot was created based on the simulation results to examine how the number of lineages in each social system changed over evolutionary time. As the breeding system of N. niger is only anecdotally described, we also performed a similar analysis, treating N. niger either as an undetermined social species (i.e., excluded from the dataset) or as a non-cooperatively breeding species.

In addition, although not reported in the results section, we calculated the phylogenetic signals for each trait (male and female body weight, total length, clutch size, and egg size) using the phytools package (ver 2.4-4)64. Significant phylogenetic signals were detected for female body size, total length, and clutch size. Detailed statistical results are provided in Supplementary Table S5.

Phylogenetic comparative analyses

We examined the relationships between the social system and body size, egg number, or egg size. Because the statistical analyses incorporated either 39 or 61 taxa, we pruned the MCMC tree from Ronco et al.19 to retain only the relevant taxa. When a species was represented by multiple terminal tips, the dataset corresponding to the tip geographically closest to the species’ natural distribution was used in the phylogenetic tree for this life-history trait analysis.

First, we used phylogenetic generalized linear models (PGLMs; R package phylolm (ver 2.6.5)67) to test the association between social system and body size across 39 taxa. Our analysis is grounded in the hypothesis that cooperative breeding is driven by predation pressure, under the assumption that body size and predation pressure are strongly correlated. Before fitting PGLMs, log-transformed body weights were averaged by taxa, separated by sex. The model was constructed with the social system (cooperative or non-cooperative breeding) as the response variable and body size (mean log10-transformed BW for females) and breeding habitat (with crevices/caves, facultative shell-brooding, obligatory shell-brooding, rocky area, or sandy/muddy area) as explanatory variables, applying a logistic regression framework (logistic MPLE method). A separate PGLM was fitted for the body size of mature males in the same manner.

In addition to PGLMs analyses for the life history traits (39 taxa) gained from field observation, we performed comparable analyses of the evolutionary relationship between body size and social system using a larger number of species (61 taxa) in the public database FishBase. While these data were collected using non-standardized methods, they provided a broader species representation than our field collection data. We used PGLMs, fitting the pruned phylogenetic tree with 61 taxa. In these models, body size (defined as log10-transformed total length in cm) was used as the response variable, with social system and breeding habitat included as explanatory variables. Given that the FishBase body size data included N. niger, we conducted separate PGLMs, treating N. niger either as a cooperatively breeding species, an undetermined social species (i.e., excluded from the dataset), or as a non-cooperatively breeding species.

Second, we performed a Bayesian phylogenetic mixed model analysis (BPMM; R package brms (ver 2.22.0)68 with Stan (ver 2.32.2)69) to determine the factors driving clutch size variation across 39 taxa using individual data of females (n = 155 females from 39 taxa). The model was constructed with log10-transformed clutch size as the response variable and included social system, breeding habitat, log10-transformed egg volume, and body size (log10-transformed BW) of female breeders as explanatory variables.

Third, we tested whether the social system influenced egg volume. In this model, log10-transformed egg volume was set as the response variable, with social system, breeding habitat (crevices/caves as the reference, along with facultative shell-brooding, obligatory shell-brooding, rocky area, or sandy/muddy area), log10-transformed clutch size, and body size (log10-transformed BW) of female breeders as explanatory variables.

In both BPMMs, the error distribution for continuous variables followed a Gaussian distribution. A Markov chain Monte Carlo (MCMC) simulation was performed with a total of 1000 iterations, including a burn-in of 500 iterations, a thinning interval of 1, and four chains were applied. We assessed model convergence with R-hat via the convergence diagnostic. All R-hats were kept less than 1.01. The correlation matrix of phylogenetic distance, assuming a Brownian motion model, was incorporated as a group random effect.

Phylogenetic path analysis

To determine the causal relationship between life history traits and the evolution of cooperative breeding, phylogenetic path analysis (PPA) was performed using the R package phylopath (ver 1.3.0)54,70. Dey et al. suggested that feeding ecology can affect the social system in Tanganyikan cichlids12. Therefore, the species mean δ15N value proxy for relative trophic level published by Ronco et al. was also included for the analysis19. Generally, predatory species have a high δ15N value, and this decreases with trophic level19.

The variables used were binary data of the social system (cooperative or non-cooperative breeder) and continuous data of body size (log10-transformed BW) of female breeders, clutch size (log10-transformed), and δ15N value. Sixteen candidate models, including a null model, were constructed from combinations of four hypotheses (Hypothesis A: cooperative breeding evolved in species with smaller body size due to predation pressure; Hypothesis B: feeding ecology influences the evolution of cooperative breeding; Hypothesis C: body size affects feeding ecology; Hypothesis D: cooperative breeding reduces clutch size; see Supplementary Fig. S4). For this analysis, we employed Pagel’s lambda model to estimate associations between continuous variables and the logistic MPLE method for those involving binary variables. The C-statistic Information Criterion corrected for small sample size (CICc) was calculated for model selection68. The best model selected by CICc comparison is shown in the results. Statistical information of all models is summarized in Supplementary Table S6.

Ethics statement

We have complied with all relevant ethical regulations for animal use. All experimental protocols were approved by the Animal Care and Use Committees of Osaka City University and Osaka Metropolitan University (approval no. S0091) and adhered to the ASAB/ABS guidelines for the treatment of animals in behavioral research. Our field research in Lake Tanganyika was conducted with permission from the Zambian Ministry of Agriculture, Food, and Fisheries, and complied with the current laws in Zambia.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.