Introduction

The social hierarchy serves as a fundamental organizational mechanism within most animal societies1. Hierarchical position of an individual strongly affects physiological responses, behavior, and mental health that will contribute to the overall emotional state2,3. Specifically, social hierarchy significantly influences susceptibility to stress and is recognized as a critical risk factor for the development of psychiatric disorders4,5. Consequently, an individual’s position within this hierarchy is often associated with a range of fitness-related outcomes. For example, this social structure influences access to resources and significantly affects health, survival, and reproductive success in both humans6,7 and animals8,9. At the population level, all group members derive benefits from the stability provided by dominance hierarchies10, which can mitigate agonistic interactions among all individuals, thereby conserving energy and diminishing the risk of physical injury11. Nevertheless, the neural mechanisms through which the brain regulates dominance hierarchy remain inadequately understood.

The regulation of social hierarchy is a multifaceted process that engages various brain regions, neural circuits, and neurotransmitter systems. Among these brain regions12,13,14, the dorsomedial prefrontal cortex (dmPFC) is identified as the principal regulatory center for social hierarchy12. It integrates information regarding social status from multiple brain areas and influences dominant behaviors through its projections to several downstream regions, including the dorsal raphe nucleus (DRN), basolateral amygdala (BLA), hypothalamus, striatum, and periaqueductal gray matter (PAG)12,15,16,17,18,19. Specifically, a recent study has shown that the dmPFC-DRN pathway promotes social competition20. The DRN serves as the primary source of serotonin (5-HT) within the brain21,22, highlighting the crucial role of 5-HT neurons in regulating diverse physiological functions and mental disorders23,24,25,26,27,28,29,30,31,32,33. The serotonergic system is closed linked to social rewards and the expression of social hierarchy across various species, such as crustaceans, reptiles, rodents, and primates4,13,34,35,36,37,38,39,40,41,42. For example, the injection of 5-HT into the hemolymph of both lobsters and crayfish diminishes the likelihood of retreat and prolongs the duration of aggressive encounters23,43. Similarly, dominant male vervet monkeys (Cercopithecus aethiops) exhibit approximately double the concentration of 5-HT compared to their subordinate counterparts44. Furthermore, experimentally enhancing serotonergic activity promotes the acquisition of dominance in monkeys, while reductions in this activity correspondingly diminish dominance39. Comparable effects have been reported in humans, where the administration of 5-HT similarly impacts social dominance45. These findings collectively highlight the potential involvement of DRN and the serotonergic system in the regulation of social dominance and status.

GABAergic neurons constitute the second most prevalent cell type within the DRN46,47. The amplitude of inhibitory postsynaptic currents elicited by local GABAergic neurons is approximately five times greater than that produced by various long-range afferents, indicating a pronounced inhibition of local GABAergic inputs on 5-HT neurons30,32,46. Additionally, in vivo stimulation of the mPFC leads to a reduction in the firing rates of DRN 5-HT neurons, suggesting that mPFC axons primarily synapse onto DRN GABAergic neurons, which subsequently inhibit 5-HT neuronal activity48. In contrast to the activity of DRN 5-HT neurons, DRN GABAergic neurons are activated by aversive stimuli but are inhibited during reward-seeking behavior40. This finding implies that GABAergic neurons may play a complementary role in the modulation of behaviors relative to 5-HT neurons. Supporting this notion, optogenetic activation of DRN GABAergic neurons has been shown to induce aversion and social avoidance in mice29,49. Conversely, optogenetic silencing of DRN GABAergic neurons results in disinhibition of adjacent 5-HT neurons and prevents the acquisition of social avoidance in mice faced with social threat29. Thus, it is reasonable to hypothesize that DRN GABAergic neurons are integral to the regulation of social hierarchy.

To examine this hypothesis, we employed optogenetics, chemogenetics, fiber photometry recordings, and behavioral assays to elucidate the role of DRN GABAergic neurons in regulating social hierarchy in mice. Our findings reveal that calcium signals in DRN GABAergic neurons increased during retreats and decreased during the initiation of pushes in the tube test. Notably, the optogenetic and chemogenetic activation of DRN GABAergic neurons led to a decrease in rank within the tube test, whereas their inhibition resulted in an elevation of social hierarchy. This study demonstrates the bidirectional regulatory role of DRN GABAergic neurons in social hierarchy and provides further insights into the neural mechanisms that govern this organizational structure in animal societies.

Results

Population activity of DRN GABAergic neurons increased during retreat and decreased during push-initiation in the tube test

To investigate the association between social competition and the activity of GABAergic neurons in the DRN, we employed a standardized behavioral paradigm, specifically the tube test50, to simulate a socially competitive environment. Using fiber photometry, we recorded neuronal calcium signals in mice during encountering with their cage mates in the tube test. The behaviors exhibited by the mice in the test can be categorized into push-initiation, push-back, stillness, resistance, and retreat50. To capture real-time population activity of DRN GABAergic neurons during competition, we administered an adeno-associated virus (AAV2/9) encoding the fluorescent calcium indicator GCaMP6s with GAD67 promoter (AAV2/9-GAD67-GCaMP6s-3xFLAG) and AAV2/9-GAD67-EGFP-tWPA into the DRN of distinct groups of mice (designated as DRN-GCaMP6s mice and DRN-EGFP mice, respectively; see Fig. 1a, b and Supplementary Fig. 1a). We subsequently implanted a fiber optic cannula into the DRN and secured it to the skull for in vivo fiber photometry recordings during the tube test (Fig. 1c). Compared to DRN-EGFP mice, DRN-GCaMP6s mice exhibited significantly enhanced calcium signals in DRN GABAergic neurons during retreat (Fig. 1d, e and Supplementary Fig. 1f), while demonstrating significantly reduced calcium signals during push-initiation (Fig. 1f, g and Supplementary Fig. 1f). Collectively, these findings indicate that DRN GABAergic neurons are activated during retreat yet exhibit suppressive activity during push-initiation, suggesting a potential negative correlation between DRN GABAergic neuron activity and social competition. Furthermore, we observed that calcium signals in DRN GABAergic neurons were diminished during effortful behaviors, including push-initiation, push-back, and resistance (Supplementary Fig. 1b-f).

Fig. 1: DRN GABAergic neurons demonstrated activation during retreat but inactivation during push-initiation in the tube test.
figure 1

a Schematic representation of the viral injection site in the DRN utilizing AAV2/9-GAD67-GCaMP6s-3xFLAG. b Representative photomicrographs of the DRN showing GCaMP6s (green, left) with the enlarged pictures for the boxed area illustrating GCaMP6s expression (green, second subplot), GAD67+ (red, third subplot), and their co-localization with DAPI (right). c Schematic of the fiber photometry setup (left) and a simplified diagram of the tube test procedure (right). d Ca2+ signals were aligned to the onset of retreats when DRN-GCaMp6s mice meet with their cage mates during the tube test. Upper panel, mean (blue trace) ± SEM (gray shading) showing the average Ca2+ signal transients for all trials (n = 6 mice). The red segment indicates a statistically significant increase from baseline (p < 0.05; permutation test). Middle panel, the heatmap of Ca2+ signals for all trials with each row representing an individual trial (n = 70 trials from 6 mice). Lower panel, the heatmap of Ca2+ signals averaged across all trials for each animal (n = 6 mice). e Ca2+ signals were aligned to the onset of retreats in DRN-EGFP mice meeting with their cage mates in the tube test (n = 124 trials from 7 mice). The three panels exhibit similarities to those in panels (d). f Ca2+ signals were aligned to the onset of push-initiations in DRN-GCaMp6s mice meeting with their cage mates in the tube test (n = 57 trials from 6 mice). The red segment indicates a statistically significant decrease from baseline (p < 0.05; permutation test). The three panels show resemblance to those depicted in panels (d). g Ca2+ signals were aligned to the onset of push-initiations in DRN-EGFP mice meeting with their cage mates in the tube test (n = 133 trials from 6 mice). The three panels exhibit similarities to those in panels (d).

Optogenetic activation of DRN GABAergic neurons resulted in loss in the tube test

To elucidate the role of DRN GABAergic neurons in regulating social hierarchy, we employed optogenetic activation of these neurons to monitor real-time changes in the social rank of mice during the tube test. Specifically, the viral vector AAV2/9-GAD67-hChR2(H134R)-EGFP-3xFLAG-WPRE was injected into the DRN (designated as DRN-ChR2 mice; Fig. 2a, b). Subsequently, a fiber optic cannula was implanted 500 μm above the injection site and affixed to the skull. At least four weeks post-injection, we stimulated the DRN in vivo to activate GABAergic neurons using pulses of 473 nm blue light (20 ms, 20 Hz, and 9-19 mW) before the mice entered the tube, maintaining light exposure throughout the duration of the test. Following this stimulation, the mice exhibited a loss of dominance when faced with previously subdominant opponents. Notably, the significant reduction in tube-test ranks persisted for a minimum of two days (Fig. 2c, d, f). Additionally, the optogenetic stimulation led to increased behaviors characterized by push-back, stillness, and retreat, in comparison to pre-photostimulation behaviors (Fig. 2g). However, optogenetic activation of DRN GABAergic neurons did not significantly influence locomotion, anxiety, aggression, social memory, or grip strength (Supplementary Fig. 2). Furthermore, the same photostimulation protocol did not influence the tube test ranks of mice injected with the AAV2/9-GAD67-mCherry-WPRE-hGH-pA viral vector into the DRN (designated as DRN-mCherry mice; Fig. 2e, f), as well as the behaviors in the tube test (Supplementary Fig. 3). Collectively, these findings suggest that optogenetic activation of DRN GABAergic neurons in mice results in a reduction of social dominance, a phenomenon that can be sustained to a certain extent.

Fig. 2: Optogenetic activation of DRN GABAergic neurons led to loss in the tube test.
figure 2

a Schematic depiction of the injection site in the DRN for the virus AAV2/9-GAD67-hChR2(H134R)-EGFP-3xFLAG-WPRE. b Representative photomicrographs of the DRN depicting ChR2 (green, left) with the enlarged pictures for the boxed area showing ChR2 expression (green, second subplot), GAD67+ (red, third subplot), and their co-localization with DAPI (right). c Example of the rank positions of a single cohort of mice tested daily over 7 days, indicating that the first-ranked mouse injected with the ChR2 virus fell to the third place following photostimulation. d Summary of rank changes in DRN-ChR2 mice before and after the optogenetic activation of DRN GABAergic neurons. Each line represents an individual animal. e No rank changes were observed in control mice before and after photostimulation. f Comparison of average rank changes in DRN-ChR2 mice following the optogenetic activation of DRN GABAergic neurons delivered throughout the tube test on Day 0 (Wilcoxon signed rank test; Day 0: Z6 = -2.640, p = 0.008; Day 1: Z6 = -2.070, p = 0.038; Day 2: Z6 = -1.633, p = 0.102; Day 3: Z6 = -1.633, p = 0.102). g Optogenetic activation of DRN GABAergic neurons significantly increased the number (Mann-Whitney U-test: U = 432, p = 0.038) and duration (U = 432, p = 0.039) of push-back, as well as the number (U = 223.5, p < 0.001) and duration (U = 362, p = 0.044) of stillness, and the percentage of time spent retreating (U = 60, p < 0.001). However, no significant differences were observed in the number (U = 445, p = 0.307) and duration (U = 415.5, p = 0.186) of push-initiation or the percentage of time spent resisting (U = 424, p = 0.108). “n” refers to the number of occurrences of the specific fine behaviors being measured. *p < 0.05, **p < 0.01, ***p < 0.001.

Chemogenetic activation of DRN GABAergic neurons led to loss in both the tube test and the warm spot test

To further validate the hypothesis that the activation of DRN GABAergic neurons leads to a sustained reduction in social hierarchy in mice, we employed DREADDs (designer receptors exclusively activated by designer drugs) technology, which allows for longer modulation periods and a greater number of manipulated neurons. To enhance the construct validity of social dominance assessments, we subsequently extended the findings from the tube test to the warm spot test. The viral vector AAV2/9-GAD67-hM3D(Gq)-mCherry-WPRE was injected into the DRN (referred as DRN-hM3Dq mice; Fig. 3a, b). Four weeks post-injection, Clozapine-N-oxide (CNO, 10 mg per kg) was administered intraperitoneally to one mouse in each cage, while its cage mates received an equivalent volume of saline injection. The tube test was conducted at 1-1.5, 3-5, 6-8, 24, 48, and 72 h post-injection. Compared to controls, CNO administration resulted in a significant decrease in tube test rankings in the CNO-injected mice, with the most pronounced effect observed between 3 and 5 h after injection (Fig. 3c-e). CNO-injected mice exhibited a marked increase in passive behaviors, including immobility and retreat (Fig. 3f, g). At 24 h post-CNO injection, most mice returned to their original ranking positions (Fig. 3d, e). In the warm spot test, DRN-hM3Dq mice exhibited a significant decrease in the duration of warm spot occupancy at two hours post-CNO injection when compared to saline-injected control subjects (Fig. 3h-j). Furthermore, the DRN-mCherry mice exhibited no significant alterations in social rank or behavior following either CNO or saline injection (Supplementary Fig. 4a-d and Supplementary Fig. 5). Thus, the chemogenetic activation of DRN GABAergic neurons resulted in a reduction of social rank, a finding that can be generalized across different contexts. Additionally, chemogenetic activation was associated with reduced locomotion in mice (Supplementary Fig. 4e, f).

Fig. 3: Chemogenetic activation of DRN GABAergic neurons resulted in a loss in both the tube test and the warm spot test.
figure 3

a Schematic representation of the injection site in the DRN for the virus AAV2/9-GAD67-hM3D(Gq)-mCherry-WPRE. b Representative photomicrographs of the DRN depicting hM3Dq (red, left) with the enlarged pictures for the boxed area showing hM3Dq expression (red, second subplot), GAD67+ (green, third subplot), and their co-localization with DAPI (right). c Example of the ranking positions for a cage of mice tested daily over a period of 7 days, indicating that the first-ranked mouse injected with the hM3Dq virus was relegated to the last position at 6-8 h post-CNO injection. d Summary of rank changes in DRN-hM3Dq mice before and after CNO injection. Each line represents an individual animal. e Difference in average rank change for DRN-hM3Dq mice following CNO injection (Wilcoxon signed rank test; at 1-1.5 h: Z5 = -1.414, p = 0.157; at 3-5 h: Z5 = -2.646, p = 0.008; at 6-8 h: Z5 = -1.342, p = 0.180; at 24 h: Z5 = -1.342, p = 0.180; at 48 h: Z5 = -0.816, p = 0.414; at 72 h: Z5 = -0.816, p = 0.414). f Behavioral annotations of two tube test trials between the same pair of mice before and after CNO injection. g Chemogenetic activation of DRN GABAergic neurons significantly increased both the number (Mann-Whitney U-test: U = 90, p = 0.043) and duration (U = 86, p = 0.040) of stillness, as well as the percentage of time spent retreating (U = 2, p < 0.001). However, no significant changes were observed in the number (U = 129, p = 0.512) and duration (U = 142, p = 0.864) of push-initiation, the number (U = 134, p = 0.551) and duration (U = 133, p = 0.522) of push-back, or the percentage of time for resistance (U = 146, p = 0.968). “n” refers to the number of occurrences of the specific fine behaviors being measured. h Picture for the warm spot test, wherein four mice competed for occupancy of a warm corner situated on an ice-cold floor. i Duration of warm spot occupancy (top; One-way repeated measures ANOVA with Bonferroni correction: F2,18 = 0.142; p = 0.868; n = 10) and rank in the warm spot test (bottom; Friedman test with Wilcoxon signed-rank test for multiple comparisons: χ2 = 0.677; p = 0.717; n = 10) for DRN-hM3Dq mice are reported for 1 day before, 2 h after, and 2 days after saline injection. Each gray folded line represents an individual animal, while the colored line indicates the mean. Animals of the same rank are arranged in parallel to facilitate the presentation of results. j Duration of warm spot occupancy (top; F2,14 = 11.785; p = 0.007; n = 8) and rank in the warm spot test (bottom; χ2 = 8.538; p = 0.014; n = 8) for DRN-hM3Dq mice are presented for 1 day before, 2 h after, and 2 days after CNO injection. *p < 0.05, **p < 0.01, ***p < 0.001.

Optogenetic inhibition of DRN GABAergic neurons induced win in the tube test

To ascertain the necessity of DRN GABAergic neurons in modulating social hierarchy, we employed an optogenetic strategy to selectively inhibit these neurons and subsequently assess changes in social rank among mice during the tube test. Specifically, we administered the viral vector AAV2/9-GAD67-eNpHR3.0-EGFP-WPREs into the DRN (hereafter referred to as DRN-NpHR mice; see Fig. 4a, b). Following this, a fiber optic cannula was implanted 500 μm above the viral injection site and affixed to the skull. At least four weeks post-injection, we stimulated the DRN in vivo to inhibit GABAergic neurons using 589 nm continuous orange light (9-19 mW) immediately prior to the mice entering the tube. The light remained activation throughout the duration of the test. In DRN-NpHR mice, this photostimulation protocol resulted in immediate victories over previously dominant opponents (Fig. 4c, d, f). Conversely, the same protocol did not alter the tube test rankings in DRN-mCherry mice (Fig. 4e, f). Additionally, optogenetic inhibition of DRN GABAergic neurons in subordinate DRN-NpHR mice led to a marked increase in inactivity and a reduction in retreat behaviors compared to the pre-photostimulation condition (Fig. 4g). Importantly, this inhibition did not induce significant changes in anxiety, aggression, social preference, or grip strength in DRN-NpHR mice (Supplementary Fig. 6a, c-f), as well as the behaviors of DRN-mCherry mice in the tube test (Supplementary Fig. 7). However, it did result in decreased locomotion among the mice (Supplementary Fig. 6b).

Fig. 4: Optogenetic inhibition of DRN GABAergic neurons induced winning in the tube test.
figure 4

a Schematic representation of the injection site in the DRN for the virus AAV2/9-GAD67-eNpHR3.0-EGFP-WPREs. b Representative photomicrographs of the DRN depicting eNpHR (green, left) with the enlarged pictures for the boxed area showing eNpHR expression (green, second subplot), GAD67+ (red, third subplot), and their co-localization with DAPI (right). c Example of the ranking positions of a cohort of mice tested daily over 7 days, demonstrating that the mouse initially ranked lowest, which received the eNpHR virus injection, subsequently ascended to the highest rank following photostimulation. d Summary of rank changes in DRN-eNpHR mice before and after photostimulation. Each line represents an individual animal. e No rank changes were observed in DRN-mCherry mice before and after photostimulation. f The average rank change for DRN-eNpHR mice following optogenetic inhibition of DRN GABAergic neurons, administered on Day 0 (Wilcoxon signed rank test; Day 0: Z6 = -2.636, p = 0.008; Day 1: Z6 = -1.414, p = 0.157; Day 2: Z6 = 0, p = 1; Day 3: Z6 = -1.732, p = 0.083). g Optogenetic inhibition of DRN GABAergic neurons significantly increased the number (Mann-Whitney U test: U = 301.5, p = 0.001) and duration (U = 166, p < 0.001) of stillness, along with the percentage of time dedicated retreat (U = 17, p < 0.001). In contrast, there were no significant changes observed in the number (U = 457, p = 0.168) and duration (U = 506.5, p = 0.463) of push-initiation, the number (U = 529.5, p = 0.520) and duration (U = 532, p = 0.551) of push-back, or the percentage of time spent in resistance (U = 428.5, p = 0.089). “n” refers to the number of occurrences of the specific fine behaviors being measured. **p < 0.01, ***p < 0.001.

Chemogenetic inhibition of DRN GABAergic neurons resulted in win in both the tube test and the warm spot test

To further validate the preceding findings, we employed chemogenetic inhibition to inactivate DRN GABAergic neurons and examined the alterations in social rank during both the tube test and the warm spot test. Specifically, the virus AAV2/9-GAD67-hM4D(Gi)-mCherry-WPREs was administered into the DRN (referred as DRN-hM4Di mice; Fig. 5a, b). Four weeks post-injection of the viral vector, CNO (10 mg per kg) was administered intraperitoneally to one mouse in each cage, while its cage mates received saline injections. The tube test was conducted at 1-1.5, 3-5, 6-8, 24, 48, and 72 h following CNO administration (Fig. 5c). Compared to controls, CNO treatment resulted in a significant increase in the tube test rankings of the CNO-injected mice, with the earliest rise observed between 1 and 1.5 h post-injection, peaking between 3 and 8 h post-injection (Fig. 5d, e). Notably, there was a significant increase in the frequency of push-initiation, push-back, and resistance, alongside a decrease in the frequency of retreat following CNO administration (Fig. 5f, g). By 24 h post-injection, most mice returned to their baseline rank positions (Fig. 5c-e). In the warm spot test, DRN-hM4Di mice injected with CNO for two hours demonstrated a significant increase in the duration spent in the warm spot, as well as an elevation in warm spot rank (Fig. 5i), while saline injection did not elicit a comparable change (Fig. 5h). In addition, chemogenetic inhibition did not affect locomotion in mice (Supplementary Fig. 4g, h). Consequently, the inhibition of DRN GABAergic neurons was observed to enhance social hierarchy, with this change in rank remaining consistent throughout the warm spot test.

Fig. 5: Chemogenetic inhibition of DRN GABAergic neurons produced a win in both the tube test and the warm spot test.
figure 5

a Schematic representation of the injection site in the DRN for the virus AAV2/9-GAD67-hM4D(Gi)-mCherry-WPREs. b Representative photomicrographs of the DRN depicting hM4Di (red, left) with the enlarged pictures for the boxed area showing hM4Di expression (red, second subplot), GAD67+ (green, third subplot), and their co-localization with DAPI (right). c Example of the ranking positions of a cohort of mice tested daily over 7 days, demonstrating that the last-ranked mouse injected with the hM4Di virus ascended to third place following CNO administration. d Summary of rank changes in DRN-hM4Di mice before and after CNO injection. Each line represents an individual animal. e Statistical analysis of mean rank change in DRN-hM4Di mice following CNO injection at 0 h (Wilcoxon Signed Rank Test; at 3-5 h: Z6 = -2.530, p = 0.011; at 6-8 h: Z6 = -2.070, p = 0.038). f Behavioral annotations of two tube test trials between the same pair of mice before and after CNO injection. g Chemogenetic inhibition of DRN GABAergic neurons significantly increased the number (Mann-Whitney U-test: U = 64.5, p < 0.001) and duration (U = 41.5, p < 0.001) of push-initiation, the number (U = 84, p < 0.001) and duration (U = 84, p < 0.001) of push-back, the percentage of time spent in resistance (U = 51, p < 0.001), and the number (U = 117, p = 0.011) but not the duration (U = 135, p = 0.050) of stillness. Moreover, the percentage of time spent in retreat was significantly decreased by chemogenetic inhibition (U = 18, p < 0.001). “n” refers to the number of occurrences of the specific fine behaviors being measured. h Duration of occupying the warm spot (left; one-way repeated measures ANOVA with Bonferroni correction: F2,14 = 0.769; p = 0.482; n = 8) and rank in the warm spot test (right; Friedman test with Wilcoxon signed-rank test for multiple comparison: χ2 = 0; p = 1; n = 8) for DRN-hM4Di mice assessed at 1 day before, 2 h after, and 2 days after saline injection. Each gray folded line represents an individual animal, while the colored line represents the average. Mice with the same rank are arranged in parallel to facilitate result presentation. i Duration of occupying the warm spot (left; F2,12 = 20.018; p < 0.001; n = 7) and rank in the warm spot test (right; χ2 = 9.579; p = 0.008; n = 7) for DRN-hM4Di mice at 1 day before, 2 h after, and 2 days after CNO injection. *p < 0.05, **p < 0.01, ***p < 0.001.

Discussion

The objective of this study was to elucidate the potential role of DRN GABAergic neurons in the regulation of social hierarchy. To achieve this, we employed optogenetic and chemogenetic techniques in conjunction with fiber photometry recordings. Our findings demonstrate that DRN GABAergic neurons exhibited activation pattern during withdrawal and showed reduced activity during the initiation of effortful behaviors. Furthermore, both optogenetic and chemogenetic activation of DRN GABAergic neurons resulted in a reduction of social rank, whereas inhibition of these neurons led to a corresponding increase in social rank. These results underscore the bidirectional regulatory role of DRN GABAergic neurons in the context of social hierarchy.

The current findings demonstrate that DRN plays a pivotal role in regulating social hierarchy, consistent with its well-documented roles in social interaction, reward processing, arousal, and the modulation of stress and anxiety27,31,40,46. In particular, a recent study has shown that the dmPFC-DRN pathway facilitates social competition20. The DRN, situated in the dorsal midbrain, encompasses at least four principal cell types that release various neurotransmitters, including 5-HT, GABA, glutamate, and dopamine51,52, and serves as the primary source of 5-HT within the brain16,21,22. Previous studies have established a close association between the serotonergic system and social rewards, as well as the acquisition of social dominance across diverse species4,13,34,35,36,37,38,39,40,41,42. For instance, the administration of 5-HT into the hemolymph of both lobsters and crayfish reduces the likelihood of retreat and extends the duration of aggressive encounters23,43. Similarly, the administration of 5-HT facilitates the attainment of dominance in both monkeys39 and humans45. Furthermore, dopamine within various brain regions, including the DRN, is implicated in the representation of social rank signals in green anole lizards (Anolis carolinensis)53. Collectively, these findings underscore the critical involvement of the DRN and its associated neurotransmitters in the regulation of social hierarchy.

Our findings suggest that GABAergic neurons in the DRN play a critical role in the regulation of social hierarchy. Specifically, activation of DRN GABAergic neurons was associated with an increased frequency of retreat behaviors, whereas inhibition of these neurons resulted in a decreased frequency of retreat in mice. These observations are consistent with previous studies indicating that optogenetic silencing of DRN GABAergic neurons leads to disinhibition of DRN 5-HT neurons, ultimately rescuing social avoidance behavior in mice29. GABA, the predominant inhibitory neurotransmitter in the central nervous system, is essential for determining the timing of neuronal firing and maintaining the balance between excitation and inhibition for information processing54,55,56,57,58. In the mammalian cerebral cortex, GABAergic interneurons are crucial for regulating this balance and dynamic organization of pyramidal neuron ensembles, which mediate diverse streams of information processing and output channels59,60,61. The connective projections between excitatory and inhibitory neurons within brain regions form neural microcircuits that regulate various social behaviors, including social dominance, social fear, exploration, memory, and preference62.

Consistent with this, serotonergic activity relies on a delicate balance between excitatory and inhibitory inputs to DRN 5-HT neurons46. Dynamic modulation of 5-HT neurons may be achieved through a push-pull mechanism (convergent excitation and inhibition) or feedforward inhibition provided by local GABAergic neurons22,32,63,64. For example, it has been demonstrated that local GABAergic inhibition of 5-HT neurons is fivefold stronger than that induced by external GABAergic afferents32, suggesting that DRN GABAergic interneurons exert a robust inhibitory influence on serotonergic firing29. This inhibition could diminish 5-HT production by DRN 5-HT neurons, leading to significant reductions in 5-HT levels in the PFC, hippocampus, and hypothalamus, ultimately influencing animal behaviors32,46,65,66. Research confirms that DRN 5-HT neurons in mice are activated by reward-related stimuli such as sucrose, food, and social interaction, while DRN GABAergic neurons respond to aversive stimuli including quinine and electrical stimulation, suggesting a functionally antagonistic relationship between these two neuronal types40. Together, these findings underscore the potential for GABAergic neurons to serve a complementary role in behavioral modulation relative to 5-HT neurons. Our current results indicate that both optogenetic and chemogenetic activation of DRN GABAergic neurons lead to a reduction in social rank, while inhibition of these neurons corresponds with an increase in social rank. These findings, which contrast with the effects observed when manipulating 5-HT neurons, further highlight the antagonistic interactions between GABAergic and 5-HT neurons in the regulation of social hierarchy.

Interestingly, in accordance with previous findings67, the present results indicate that optogenetic activation produces a more sustained effect on increases in social rank compared to chemogenetic activation. The establishment of a dominance hierarchy can be attributed to a reinforcing mechanism known as the “winner effect”, wherein animals exhibit an increased likelihood of subsequent victories following prior successes. Due to its millisecond-scale precision in neuronal activation, optogenetics enables researchers to target specific neural circuits during behaviorally relevant moments. This precision likely facilitates the winner effect, which may be underpinned by synaptic strengthening within these neural circuits67. Consequently, the enduring impact of optogenetics on social rank arises not from an extended duration of neuronal activity, but rather from its precise timing during critical behavioral events, thereby enhancing synaptic plasticity within the associated neural circuitry. In contrast, while chemogenetic activation may persist longer, it lacks the temporal specificity required to induce synaptic plasticity. Further research is warranted to elucidate whether the synaptic plasticity mechanisms within the circuit associated with the DRN also contribute to the winner effect.

Although our study provides insights into the bidirectional regulatory role of DRN GABAergic neurons in social hierarchy, two limitations in our experimental design and interpretation of results are unavoidable. Firstly, the dose of CNO administered in this study appears to be relatively high, which may potentially induce behavioral effects. Previous research has demonstrated that a 10 mg per kg dose of CNO does not yield brain concentrations exceeding 70 nM and is associated with subtle off-target effects in both pharmacokinetics and behavior68. In line with this, our current findings indicate that this dosage of CNO does not significantly alter mouse behavior in the tube test, except for a decrease in the number of stillness episodes. Given that changes in social ranks are primarily driven by effortful behaviors and retreat, this suggests that the observed limitation does not compromise the validity of our conclusions. Secondly, unlike some previous studies67,69, the optogenetic and chemogenetic strategies employed for the same manipulations (activation or inhibition) did not yield comparable changes in effortful behaviors during the tube test. This discrepancy may arise from differences in temporal resolution and spatial specificity between the two strategies, as well as the varying brain regions and neuronal populations targeted. This issue merits further investigation. Nevertheless, this limitation should not undermine our conclusions, as the same manipulations across different tools consistently induced similar effects on retreat and social hierarchy. This finding aligns with a more recent study indicating that optogenetic inhibition of various downstream circuits (including DRN) in the dmPFC primarily modulates social hierarchy through alterations in retreat, rather than other fine behaviors20.

In summary, our study highlights the critical role of DRN GABAergic neurons in regulating social dominance and status. GABAergic neurons within the DRN form local connections with various cell types and exhibit extensive ascending and descending projections to multiple long-range targets, thereby influencing relevant behaviors through both direct and indirect pathways. Consequently, the mechanisms by which DRN GABAergic neurons regulate social hierarchy may be diverse. First, considering the intricate reciprocal projection relationship between GABAergic and 5-HT neurons in the DRN46, we hypothesize that these neuron types may constitute a microcircuit that modulates behavioral outcomes. Second, at the whole-brain level, DRN GABAergic neurons demonstrate significant ascending projections to key forebrain and midbrain regions, including the prefrontal cortex (PFC), amygdala, nucleus accumbens (NAc), ventral tegmental area (VTA), lateral habenula (LH), bed nuclei of the stria terminalis (BNST), and hypothalamus65,70. Prior studies have indicated that the pathway from dmPFC to the anterior basolateral amygdala (dmPFC-aBLA), as well as mPFC-LH, mPFC-NAc, and mPFC-VTA circuits, are involved in social dominance and/or related behaviors20,71,72. Thus, the current findings suggest that the role of DRN GABAergic neurons in regulating social hierarchy may be shaped by long-range neural circuits. Therefore, further investigation is warranted to elucidate the neural circuits linking DRN GABAergic neurons to their long-range targets, as well as the local microcircuits that involve interactions between GABAergic and serotonergic or dopaminergic neurons in the DRN that contribute to social hierarchy regulation. Moreover, given that GABAergic neurons can be classified into multiple subtypes, it remains to be determined which specific subtype is involved in the regulation of social dominance and associated behaviors.

Methods

Animals

Male C57BL/6 mice, aged 8 to 12 weeks, were obtained from Chengdu Dossy Experimental Animals Co., Ltd. The mice were housed in an animal facility maintained at a controlled room temperature of 22 ± 1 °C, with a relative humidity of 60 ± 5%, and were subjected to a 12-h light/dark cycle (lights on from 08:00 to 20:00). The subjects were given ad libitum access to both food and water. All experimental procedures were approved by the Animal Care and Use Committee of Chengdu Institute of Biology, Chinese Academy of Sciences (permit number: CIBDWLL2020001), ensuring compliance with established ethical standards.

Virus injection and fiber implantation

The following viruses were utilized in this study: AAV2/9-GAD67-GCaMP6s-3xFLAG, AAV2/9-GAD67-hChR2(H134R)-EGFP-3xFLAG-WPRE, AAV2/9-GAD67-eNpHR3.0-EGFP-WPREs, AAV2/9-GAD67-hM3D(Gq)-mCherry-WPRE, AAV2/9-GAD67-hM4D(Gi)-mCherry-WPREs, AAV2/9-GAD67-EGFP-tWPA, and AAV2/9-GAD67-mCherry-WPRE-hGH-pA. These viruses were injected into the DRN at an angle of 15° (AP, -4.5 mm; ML, 0 mm; DV, -2.6 mm from lambda). Each injection delivered a volume of 300 nl of the viral solution. For the optogenetic manipulation experiments, the optical fiber was strategically positioned 0.5 mm above the DRN (angle, 15°; AP, -4.5 mm; ML, 0 mm; DV, -2.1 mm from lambda) to target a broader population of neurons. For fiber photometry recordings, the optical fiber was implanted directly at the DRN site (angle, 15°; AP, -4.5 mm; ML, 0 mm; DV, -2.6 mm from lambda), facilitating the acquisition of calcium signal recordings from neurons within the targeted brain region.

Behavioral tests

Tube test

The procedures of the tube test are described in detail in a prior study50. Briefly, to establish a stable social hierarchy, four adult male mice were co-housed in a single cage for a minimum duration of two weeks. Following this period, the mice underwent three consecutive days of handling, succeeded by three consecutive days of training in the tube test. An acrylic tube, measuring 3 cm in inner diameter and 30 cm in length, was utilized for training, with each animal performing ten entries per day (five from each end). During the tube test, pairs of mice were placed at opposite ends of the tube, meeting in the center. The mouse that exited the tube first was designated as the loser, while its counterpart was regarded as the winner. In cases where neither mouse exited the tube within a 2-min interval, the trial was repeated. After each trial, the tube and the surrounding experimental area were cleaned with bathroom paper moistened in 75% alcohol. Each animal participated in three pairwise interactions daily with its cage mates, resulting in a total of six pairs per cage. The ranks among the four mice were established based on the number of victories attained. Stable ranks were defined as those that remained unchanged over a minimum of three consecutive days. Animal behaviors during the tube test were systematically documented through video recording and subjected to frame-by-frame analysis using BORIS video analysis software (V7.12.2, University of Torino)73. Following the behavioral classification criteria outlined in the previous study50, the behaviors observed within the tube were categorized into distinct categories: push-initiation, characterized by actively initiating a push by shoving the head beneath the opponent; push-back, defined as a counter push wherein the subject mouse shoves its head beneath the opponent after being pushed; resistance, which involves maintaining territorial position while being pushed by the opponent, often with the head elevated; stillness, referring to remaining stationary except for occasional sniffing and grooming; and retreat, which encompasses moving backward after being pushed or voluntarily withdrawing, typically accompanied by a downward bend of the head. Each behavioral epoch was annotated using BORIS for subsequent analysis. To mitigate potential bias during the analysis, the scorer was blinded to the ranks and prior experiences of the mice while annotating their behaviors.

Warm spot test

Recognizing that each behavioral assay designed to measure social dominance, including the tube test, may be influenced by a range of factors, we utilized the warm spot test to assess whether dominance established in the tube test is transferable to other manifestations of dominant behavior, thereby enhancing the construct validity of social dominance assessments. A sufficient volume of water was introduced into a rectangular plastic container (dimensions: 28 cm × 20 cm × 20 cm), which was subsequently placed in a freezer. Before the experimental trials, a plastic film was affixed to the ice surface to maintain a temperature of approximately 0 °C throughout the testing period. A heating coil, encased in cardboard, was positioned in one corner of the container to elevate the local temperature to 34 °C. This configuration was designed to simulate a warm nest, with a diameter of 5 cm, suitable for accommodating a single adult mouse67. Before initiating the experiment, four mice from the same cage were housed in a box with an iced bottom but devoid of a warm nest for a duration of 20 min to lower their body temperature. Subsequently, the mice were transferred to the experimental container, which was equipped with ice and a warm nest. Their competitive behaviors for the warm nest were documented over a 20-min observation period through video recording.

The warm spot test was conducted on DRN-hM3Dq and DRN-hM4Di mice. One day after the initial warm spot test, each selected individual received an intraperitoneal (i.p.) injection of CNO at a dosage of 10 mg per kg, as previous reports suggest that this dosage is safe and associated with minimal off-target effects in both pharmacokinetics and behavior68,74. In contrast, their cage mates were administered an equivalent volume of saline solution. The warm spot test was subsequently repeated at 2 and 48 h post-CNO injection. We employed the BORIS software to document the timestamps for each mouse’s entry into and exit from the warm nest, allowing us to calculate the total duration of nest occupancy for each individual. In cases where multiple animals occupied the warm nest simultaneously, only the mouse occupying the largest area within the nest was recorded. Furthermore, the scorer was blinded to the hierarchical status and prior experiences of the mice.

Open field test

To evaluate the motivation for exploring the central region in a novel environment and to monitor the locomotor activity of the test subjects, each mouse was gently placed in the center of a 40 × 40 cm open field apparatus. Mice activity was recorded under dim lighting conditions during the dark phase for a duration of 10 min. The open field test was executed using a behavioral research system (CinePlex, Plexon, USA) for video recording and data acquisition. In optogenetic experiments, blue light (473 nm, 20 Hz, 20 ms, 9 mW; MBL-U-473-100mW, New Industry Optoelectronic Tech., Changchun, China) or continuous orange light (589 nm, 9 mW; MGL-F-589-100mW) was intermittently turned on and off in 1-min epochs. For chemogenetic experiments, the open field test was conducted 2 h post-administration of CNO. Tracking software (CinePlex Studio V3.5.0) was employed to quantify the number of entries into the central zone, the duration of time spent in the central zone (center time), and the total distance traveled (locomotion) for each minute.

Elevated plus maze

The Elevated plus maze (EPM) was utilized to assess anxiety-like behavior in mice, employing a configuration comprising two open arms and two closed arms. Each mouse was placed in the maze’s center, while its locomotor activity was recorded using the aforementioned system under dim light conditions during the dark phase for a duration of 9 min. For optogenetic stimulation, blue light (473 nm, 20 Hz, 20 ms, 9 mW) or continuous orange light (589 nm, 9 mW) was intermittently activated and deactivated in 3-min epochs. The time spent in the open versus closed arms was quantified using CinePlex Studio software.

Social memory test

The social memory assessment was conducted using an apparatus measuring 40 × 20 cm, comprising three distinct chambers. Initially, the test mouse was acclimatized in the central chamber for a duration of 10 min. Subsequently, an unfamiliar mouse, referred to as the object mouse, was introduced into one of the side chambers. Following a 5-min familiarization period with the object mouse, a novel object mouse was introduced into the opposing side chamber. During the subsequent 5-min observation period, the exploration behaviors of the test mice towards the familiar and novel object mice were recorded. For analysis, the time spent by the test mice in the interaction zone (defined as half of the central chamber adjacent to the side chamber, measuring 10 × 20 cm) was quantified in 1-min epochs. During these epochs, the mice were subjected to alternating light-on and light-off conditions, utilizing 473 nm blue light (20 Hz, 20 ms, 9 mW), or 589 nm continuous orange light (9 mW) that was intermittently activated and deactivated in 1-min intervals.

Resident-intruder test

The standard resident-intruder test was performed on the test mice. Prior to the initiation of the test, all cage mates were removed from the home cages, allowing the test subjects to acclimatize to their environment for a duration of 10 min. Following this acclimatization period, unfamiliar adolescent male C57BL/6 mice (aged 5-7 weeks) were introduced as intruders for a test period of 10 min. During this time, blue light (473 nm, 20 Hz, 20 ms, 9 mW) or continuous orange light (589 nm, 9 mW) was alternately turned on and off in 1-min epochs. The frequency and duration of aggressive behaviors (such as chasing and attacking) as well as non-aggressive social interactions (including sniffing and social grooming) were meticulously analyzed for each 1-min epoch, correlating with the light-on and light-off periods.

Grip-strength measurement

We employed a force gauge (DS2-50N, Puyan, Dongguan, China) to assess the forelimb grip strength of mice. Each mouse underwent measurements across three experimental blocks, with a 2-h interval between blocks. Within each block, mice participated in 10 trials conducted over approximately 6 min, divided into two equal 3-min periods corresponding to light-on and light-off conditions. The former conditions were illuminated by either blue light (473 nm, 20 Hz, 20 ms, 9 mW) or continuous orange light (589 nm, 9 mW). The maximum grip strengths recorded during both light-on and light-off conditions across the three blocks were subsequently compiled for statistical analysis.

Fiber photometry recordings

Mice were administered an injection of 300 nl of the AAV2/9-GAD67-GCaMP6s-3xFLAG virus, followed by the implantation of a fiber optic cannula into the DRN. Following viral expression, fiber photometry recordings were conducted utilizing a recording system (QAXK-FPS-SS-LED, ThinkerTech, Nanjing, China). The GCaMP6s virus was activated with 488 nm excitation light, while signals at 405 nm were subtracted from those at 488 nm to yield corrected signal outputs. The sampling frequency was set to 100 Hz, with the laser intensity maintained at a low level (40 µW) at the tip of the optical fiber to minimize photobleaching. Behavioral observations during the tube test were captured using a camera positioned adjacent to the tube and annotated by BORIS for subsequent analysis. To synchronize the fiber photometry recordings with the video data, an external trigger emitted a TTL pulse to the recording system while simultaneously activating a red light-emitting diode, which was visible to the camera. For each pair of opponents, six trials were conducted, with each side initiating three entries. To establish control measures, the mice were also allowed to traverse the tube alone across six trials, with each side initiating three entries.

The data were analyzed utilizing vendor-provided code from ThinkerTech Nanjing Biotech Co., Ltd., implemented in MATLAB (R2017b, MathWorks). The change in fluorescence, denoted as ΔF/F, was calculated using the formula (F - F₀)/F₀, where F₀ represents the baseline fluorescence signal averaged over a two-second interval preceding the initiation of a behavioral epoch. For the analysis of Ca²⁺ signaling, the onset of each behavior (namely push-initiation, push-back, resistance, and retreat) was standardized to time zero. In the assessment of control signals during the tube-walking task, fluorescence signals were synchronized to the moment when mice reached the midpoint of the tube. A permutation test was employed to evaluate the statistical significance of fluorescence changes associated with the behaviors, as previously documented40. In addition, the area under the curve (AUC) for the normalized ΔF/F values was calculated for various behaviors across different time windows of 1 s and 2 s with the onset of behaviors as the zero point.

Optogenetic manipulation of DRN GABAergic neurons in the tube test

The procedures employed in this study have been delineated in detail in previous studies50,67. Briefly, to minimize sample size and mitigate confounding effect arising from inter-subject variability, we employed a within-subjects design encompassing all viral types. Mice were administered one of the following viruses: AAV2/9-GAD67-hChR2(H134R)-EGFP-3xFLAG-WPRE, AAV2/9-GAD67-eNpHR3.0-EGFP-WPREs, or AAV2/9-GAD67-mCherry-WPRE-hGH-pA, via injection into the DRN, followed by the implantation of a fiber optic cannula. Once a stable tube test rank was established for a duration exceeding three days, all four mice were connected to dummy optic fibers and traversed a tube equipped with a 12 mm slit at the top for two days. Mice with comparatively lower ranks were subjected to optogenetic inhibition, while those with higher ranks underwent optogenetic activation. On the test day, the tube test commenced initially without light, followed by the application of photostimulation at 473 nm, 20 Hz, 20 ms (or 589 nm, continuous) prior to the mice entering the test tube. For each subject, photostimulation began at an intensity of 9 mW, with the tube test conducted first against their closest-ranked cage mate, followed by tests against increasingly distanced rank counterparts. If rank changes occurred, the mice exchanged starting positions to repeat the test. Each mouse was required to secure two out of three victories from each side of the tube. A successful inference of rank change due to photostimulation at this specific intensity was determined if the test mice won or lost from both sides of the tube. Otherwise, the light intensity was incrementally increased and the same procedure would be repeated until a rank change was achieved or the light intensity reached 20 mW. Only one mouse from each test cage served as the light-regulated subject, and the total number of victories induced by photostimulation for each test mouse was recorded.

Chemogenetic manipulation of DRN GABAergic neurons in the tube test

Mice were administered either AAV2/9-GAD67-hM3D(Gq)-mCherry-WPRE, AAV2/9-GAD67-hM4D(Gi)-mCherry-WPREs, or AAV2/9-GAD67-mCherry-WPRE-hGH-pA viral vectors into the DRN. Following a minimum four-week recovery period post-injection, all four mice housed together received an i.p. injection of saline. A tube test was subsequently conducted at 1-1.5, 3-5, 6-8, 24, 48, and 72 h post-injection. One week later, the test mice were administered an i.p. injection of CNO (10 mg per kg), while the control group received saline, and the tube test was repeated at the same aforementioned time intervals. Mice were randomly selected as test subjects. Statistical analyses were performed to evaluate rank changes for each mouse.

Immunofluorescence

Upon completion of all experimental procedures, mice were perfused with phosphate-buffered saline (PBS), followed by 4% paraformaldehyde (PFA). Following dissection, the brains were post-fixed overnight in 4% PFA and subsequently subjected to a dehydration protocol involving 15% sucrose in PFA for 24 h, followed by an additional 24 h in 30% sucrose in PFA. Coronal brain sections, each 30 μm thick, were serially prepared using a cryostat (CM1850, LEICA, Germany). The primary antibody, rabbit monoclonal anti-GAD67 (1:1000, ab213508, Abcam), was incubated overnight at 4 °C, after which sections were washed with PBS. Subsequently, Alexa-488 or Alexa-647 conjugated secondary antibodies, namely Donkey Anti-Rabbit IgG H&L (Alexa Fluor® 488) (ab150061, Abcam) and Donkey Anti-Rabbit IgG H&L (Alexa Fluor® 647) (ab150063, Abcam), were applied at a dilution of 1:500 for 2 h at room temperature. Following this treatment, cell nuclei were stained with DAPI for 5 min. Images of the stained sections were captured using a fluorescence microscope (Nexcope NE910, Ningbo, China). Additionally, the placements of optical fibers in the DRN for fiber photometry recordings and optogenetic stimulation using various viral vectors were evaluated (Supplementary Fig. 8).

Statistics and reproducibility

The normality of distributions and the homogeneity of variances for all values were assessed using the Shapiro-Wilk test and Levene’s test, respectively. In the analysis of fiber photometry recordings, the Wilcoxon signed rank test was utilized to assess the differences in AUC values across various behaviors. In the tube test, the Mann-Whitney U test was employed to evaluate differences in behavioral parameters across various light conditions and among distinct groups. For the warm spot test, one-way repeated measures ANOVA with Bonferroni correction was utilized to compare the duration of occupancy in the warm spot, while the Friedman test was applied to assess rank differences across multiple time points. In the open field test, one-way repeated measures ANOVA with Bonferroni correction was used to analyze behavioral parameter differences among groups subjected to chemogenetic manipulation. Additionally, paired-samples t tests were conducted to compare behavioral parameters between different light conditions in optogenetic manipulation. For the elevated plus-maze test, a two-way repeated measures ANOVA with Bonferroni correction was utilized, incorporating the factors of “light condition” and “group”. In the resident-intruder test, the Wilcoxon signed-rank test was utilized. For the social memory test, a two-way repeated measures ANOVA with Bonferroni correction was conducted, considering the factors of “zone” and “group”. Furthermore, two-sided paired-samples t-tests were performed to compare forelimb grip strength across different light conditions. Statistical analyses were carried out using SPSS software (version 23.0), with significance levels set at p < 0.05 and highly significant levels at p < 0.001. The sample size and the number of replications were determined by referencing previous studies in the same field.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.