Introduction

Depression and anxiety are among the most prevalent mental health disorders, affecting millions of individuals worldwide. According to the World Health Organization (WHO), depression affects over 260 million people, while anxiety disorders impact over 284 million people globally1. Depression and anxiety disorders exhibit substantial comorbidity, with meta-analytic evidence suggesting 50-60% of individuals with major depressive disorder also meet criteria for an anxiety disorder2. This overlap may stem from shared cognitive mechanisms - both conditions are characterized by attentional biases toward negative stimuli3, maladaptive interpretation patterns4, and ruminative processes that maintain symptom severity. The importance of addressing depression and anxiety lies not only in their high prevalence but also in their potential consequences, including lower quality of life, increased risk of suicide, and economic burden on society5. Therefore, raising awareness, promoting early intervention, and implementing effective treatment strategies are crucial steps toward improving mental health for individuals and communities.

Digital interventions, also known as digital therapeutics (DTx), have been developed incorporating various cognitive and behavioral interventions to treat depression and anxiety6,7. Cognitive Behavioral Therapy (CBT), focusing on identifying and correcting negative patterns of thought and behavior, has transdiagnostic benefits by targeting shared mechanisms like catastrophi­zing and emotional dysregulation. CBT has been successfully adapted to be delivered through the Internet and smartphones to treat depression and anxiety7,8. Leveraging the principle of cognitive restructuring (which is a core component in CBT), various forms of cognitive and attentional training schemes have also seen success in treating depression and anxiety. These methods not only correct maladaptive thought patterns that were common in individuals with depression and anxiety, but also enhance attentional control and modify action tendencies toward positive stimuli, demonstrating the comprehensive application of cognitive and behavioral principles in digital therapeutics9,10,11,12.

Although various cognitive and behavioral intervention techniques are effective in mitigating depression and anxiety, individual variations in response to treatment, symptom presentation, and cognitive biases remain a challenge. Personalization through reinforcement learning may be particularly valuable given the heterogeneous symptom profiles in comorbid anxiety-depression13. For example, patients with predominant anxiety might benefit from earlier exposure to threat disconfirmation modules, while those with anhedonic depression may require prioritized behavioral activation components. RL algorithms can dynamically optimize these sequences based on individual response patterns across both symptom domains14. Optimizing the treatment outcome of digital interventions requires a shift toward personalized treatment approaches. Reinforcement learning (RL) is a field of machine learning that focuses on learning from data to optimize sequential decision-making. RL algorithms can be used to improve the delivery and outcome of behavioral interventions14,15.

This pilot study is a randomized controlled trial on individuals with depression and anxiety symptoms. The primary objective of the study is to evaluate the feasibility and preliminary effectiveness of the game-based DTx and its personalization using RL in reducing symptoms of depression and anxiety.

Methods

Participants and procedure

This study was a pilot randomized controlled trial (RCT) with one experimental group and two control groups (an active control and a no-intervention control). As the study is a pilot trial, the sample size was determined to be 70 for each group based on feasibility and practicality. The study was registered at clinicaltrials.gov (NCT06301555) on March 4th, 2023. The study was approved by the Ethics Committee of the First Hospital of China Medical University (KLS 2023#53).

Participants were recruited from the Internet by posting advertisements on Chinese social media, forums, and classifieds websites. The advertisements invited individuals who thought they were experiencing depression and/or anxiety symptoms to participate. Upon clicking the advertisement page, participants were redirected to a page that collects demographic information (age and sex) as well as PHQ-9 and GAD-7 assessments. A detailed questionnaire asked the participants for their vision acuity (after correction), psychiatric history, current psychological or pharmacological treatments, cognitive impairments, substance use, and pregnancy or postpartum status. Participants were not required to have a clinical diagnosis of depression and/or anxiety disorders. The inclusion criteria include: (1) having a PHQ-9 score \(\ge 10\) or a GAD-7 score \(\ge 8\) upon enrollment; (2) owning an Android or iOS smartphone and being fluent in smartphone use; (3) having normal vision after correction. Exclusion criteria include (1) clinical diagnoses of severe psychiatric conditions such as schizophrenia or bipolar disorder; (2) at high risk of suicide or self-harm; (3) having significant cognitive impairments; (4) active substance abuse or dependence; (5) currently undergoing other psychological or pharmacological treatments for depression or anxiety; (6) pregnant individuals or those with postpartum depression, and those with serious physical health conditions that could impact mental health. Eligible participants were randomly assigned to the RL algorithm group, the no-intervention control group, and the no-algorithm active control group. All participants provided informed consent before participation. The randomization was performed using a computer-generated random sequence to ensure equal chances of assignment and avoid selection bias.

Intervention design

The DTx intervention program was crafted to resemble a role-playing game where the user navigates a character to explore the virtual world, complete quests and challenges, and interact with non-player characters (NPCs). To enhance user engagement, we have incorporated a questioning system. Periodically, users would receive quests that challenge them to complete a suite of training modules encompassing CBT, Visual Search Attention Training (VSAT), Approach Bias Modification (ApBM), and Cognitive Bias Modification Interpretation (CBM-I). The intervention was designed to last four weeks. Participants were encouraged to engage with the intervention by completing quests, which were released on a daily basis. New quests would only become available once the participant had completed the currently active ones. The users would receive a push notification if their quests of the day have not been completed by 7 pm. This design aimed to promote consistent engagement, with the expectation that participants would use the app daily to progress through the intervention. The gamification approach in the intervention program fosters a sense of accomplishment and progression, ultimately increasing motivation and adherence to the intervention. As an added incentive, we rewarded users with achievement badges, experience points, and reward points (which can be redeemed for outfits and equipment for their character) upon completion of their quests. In the RL algorithm group, the personalization of the quests was achieved through a proprietary reinforcement learning algorithm that maximizes user engagement. In the No Algorithm group, the quests were predetermined as a fixed sequence.

The game incorporated concepts, knowledge, and techniques in cognitive behavioral therapy into the interactions with the NPCs. Within the game, the player would encounter NPCs and converse with them to learn about CBT and use CBT concepts and techniques. The game also allowed for role-playing scenarios where the player could interact with the NPCs and apply CBT skills in a safe, virtual environment to build confidence for real-life situations. The game also included combat scenes, where the player controls the main character to combat various monsters representing negative emotions, thoughts, and behaviors. The combats leveraged CBM-I, VSAT, and ApBM to correct biased cognitive patterns (see Fig. 1 for the screenshots of the modules). CBM-I focuses on altering biased interpretations of ambiguous information using training exercises that promote more positive and healthy interpretations9.VSAT is an attentional training designed to enhance individuals’ abilities to control their attention and to visually search for information, and it can significantly improve the attention of individuals who suffer from depression and anxiety, particularly in terms of their ability to focus on positive information11,12,16. ApBM aims to reduce approach tendencies by modifying the patients’ action tendencies toward positive stimuli5. The design of the CBT and CBM-I modules incorporated examples from real-world social interactions, such as workplace communications and interactions within intimate relationships and with family members. These modules not only provided in-game practice of cognitive restructuring and interpretation corrections but also included strategies in the form of cards to encourage participants to apply these skills in their daily lives. For instance, after completing a module on reframing negative thoughts, participants were encouraged to reflect on recent real-life situations where they could apply the same strategy. Additionally, behavioral activation and journaling tasks were provided to help them record how they used these skills in real-world interactions. By embedding these real-world examples and encouraging reflection and practice beyond the game environment, the intervention aimed to promote the generalization of new cognitive and behavioral strategies. This approach ensured participants could develop greater confidence in using these skills independently.

Fig. 1: Screenshots of the intervention modules.
figure 1

a On the left: screenshot of CBM-I; b inthe middle: screenshot of VSAT; c on the right: screenshot of ApBM.

Full size image

Reinforcement learning is a machine learning framework for sequential decision-making problems in which an agent must choose an action given contextual information (also known as state). Each action is associated with a probability distribution that governs the generation of rewards, which is influenced by the contextual information. The agent’s goal is to learn, given contextual information, which action should be chosen to maximize the accumulated rewards17.

In the DTx intervention program, each quest corresponded to an action, which was a suite of training modules. Each action contained a different set of modules to simulate different dosages and compositions. A weighted sum of the app usage time and reductions in PHQ-9 and GAD-7 scores was used as the reward for the RL model. The usage time reflected the engagement and adherence of the participant to the intervention program, while the reductions in PHQ-9 and GAD-7 scores indicated an improvement in symptoms of anxiety and/or depression. The weight of each component can be adjusted to prioritize certain outcomes, depending on the goals of the intervention. For example, if increasing engagement is a primary objective, a higher weight can be assigned to the usage time component. Conversely, if symptom improvement is the main focus, greater emphasis can be placed on the reduction in PHQ-9 and GAD-7 scores. By using this reward structure, the reinforcement learning algorithm can learn to make decisions that maximize the overall effectiveness of the intervention program. The contextual information used in the RL algorithm included demographic information (e.g., age, gender), initial PHQ-9 and GAD-7 scores, usage patterns (e.g., cumulative usage time of each component), and performance on the various training modules. By incorporating this information, the RL algorithm can personalize treatment plans and optimize the training modules offered to each user based on their characteristics. Overall, our algorithm allows for a dynamic and personalized intervention that maximizes user engagement and efficacy in treating depression and anxiety.

Outcome measures

The Patient Health Questionnaire-918 (PHQ-9) was used to assess symptoms of depression at baseline and post-intervention, with a cut-off point of ≥10 indicating that the patient is a clinical case in a primary care population. The 7-item Generalized Anxiety Disorder scale (GAD-7) was used to assess symptoms of anxiety at baseline and post-intervention19. A cut-off point of GAD-7 ≥ 8 indicated that the patient is a clinical case in a primary care population. These cut-off points followed the Improving Access to Psychological Therapies program of the UK’s National Health Service. For PHQ-9 and GAD-7, we used the standard definitions of response, i.e., \(\ge 50 \%\) reduction, to define whether a participant responds to the intervention. We followed the definition used in the Improving Access to Psychological Therapies program of the UK’s National Health Service to define recovery based on the PHQ-9 and GAD-7 scores20. A patient was considered recovered if they were a clinical case at the start of treatment (\(\ge 10\) on PHQ-9 and/or \(\ge 8\) on GAD-7) but fell below the threshold to be considered a clinical case at post-intervention. The post-intervention assessments took place at the end of the 4-week intervention.

Statistical analysis

All statistical analyses were conducted using R version 4.3.221. Statistical significance was set at p < 0.05, and all tests were two-tailed. A complete case analysis was performed using the data from participants who had completed the intervention as well as PHQ-9 and GAD-7 assessments at both pre-intervention and post-intervention time points. An intend-to-treat analysis was performed using all eligible and randomized participants regardless of their adherence and assessment completion. Missing assessment data were imputed using multiple imputations.

Using the complete case data, we performed three multivariate logistic regression analyses to predict the three binary outcomes of interest, namely, PHQ-9 response, GAD-7 response, and recovery. The predictor variables were the baseline PHQ-9 and GAD-7 scores, age, sex (female as the reference level), and group (no-intervention control as the reference level). The odds ratios, their 95% confidence intervals, and p-values were calculated and reported.

For the intend-to-treat analysis, we performed a bootstrapped multiple imputation to impute the missing variables of the eligible and randomized participants (n = 223) using the bootImpute R package (ver 1.2.1)22. For each bootstrap sample, we imputed the missing variables by group using the available variables. The number of bootstrap samples was set to 2000, and each bootstrap sample was imputed 5 times. We estimated the response rates (PHQ-9 and GAD-7) as well as the IAPT recovery rate using the bootstrapped multiple imputed data and calculated the standard errors as well as the 95% confidence intervals.

Results

Recruitment and participants

Recruitment started in November 2023, as recruitment flyers and posters were posted. A total of 254 individuals were screened, and recruitment ended in December 2023. After screening, 31 individuals were excluded due to low depressive and anxiety symptoms (initial PHQ-9 < 10 and GAD-7 < 8). Table 1 shows the baseline characteristics of the study sample by group. Differences in baseline characteristics between groups were examined using Pearson’s Chi-squared test (for the categorical variable, i.e., sex) and one-way analysis of means (for continuous variables, i.e., age and PHQ-9 and GAD-7 scores at baseline).

Table 1 Participant characteristics
Full size table

Adherence and dropout

Figure 2 shows the CONSORT flow diagram illustrating the participants’ progression through the study. In the RL Algorithm group, 37 (50.68%) participants failed to commence training, and 4 participants had poor adherence (having completed less than 20 quests during the 4-week intervention period). The No Algorithm group had 48 (64.86%) participants who failed to commence training, and 2 (2.70%) participants with poor adherence. Moreover, there were 3 (4.11%) and 5 (6.76%) participants who did not complete the assessments at post-intervention in the RL Algorithm and the No Algorithm groups, respectively. In the no-intervention control group, 20 (26.32%) participants did not complete the post-intervention assessments. The overall dropout rates (including failure to commence, poor adherence, and missing outcome) were 60%, 73%, and 26% in the RL Algorithm, the No Algorithm, and the no-intervention Control groups, respectively. The participants in the RL algorithm group and the No Algorithm group completed on average 62.4 (SD = 18.1) and 63.1 (SD = 17.8) quests (out of a maximum 4 weeks × 5 daily = 140 quests) throughout the 4-week study period.

Fig. 2: CONSORT flow diagram.
figure 2

Number of patientsscreened, randomized, and analyzed shown in the figure.

Full size image

Complete case analysis

For the PHQ-9 response, the RL Algorithm group had a rate of 44.8%, the No Algorithm group 38.1%, and the Control group 18.9%. In terms of GAD-7 response, the RL Algorithm group achieved 31.0%, the No Algorithm group 23.8%, and the Control group 11.3%. For IAPT recovery, the rates were 55.2%, 47.6%, and 30.2% for the RL Algorithm, No Algorithm, and Control groups, respectively. For the PHQ-9 response, the RL Algorithm group had a rate of 44.8%, the No Algorithm group 38.1%, and the Control group 18.9%. In terms of GAD-7 response, the RL Algorithm group achieved 31.0%, the No Algorithm group 23.8%, and the Control group 11.3%. For IAPT recovery, the rates were 55.2%, 47.6%, and 30.2% for the RL Algorithm, No Algorithm, and Control groups, respectively. Table 2 presents the logistic regression results. In comparison to the no-intervention control group (which served as the reference level with odds ratios (OR) of 1) the RL algorithm group exhibited significantly higher odds ratios in accurately predicting PHQ-9 response (OR = 3.77, 95% CI [1.22, 12.40], p = 0.024) and GAD-7 response (OR = 4.05, 95% CI [1.16, 15.60], p = 0.032). The No Algorithm group did not display statistically significant odds ratios in predicting changes in symptoms.

Table 2 Logistic Regression Results
Full size table

Intend-to-treat analysis

Table 3 shows the bootstrap multiple imputation estimates. When comparing the Complete Case Analysis to the ITT (Intention-To-Treat) analysis, the overall pattern remained consistent, with the RL Algorithm group showing better outcomes. However, the estimates were somewhat lower, and the CIs were wider. In the ITT analysis, the estimated PHQ-9 response rate was 45% (95% CI: 26–65%) for the RL Algorithm group, compared to 41% (95% CI: 19–63%) for the No Algorithm group, and 19% (95% CI: 8–30%) for the Control group. Similarly, the GAD-7 response estimates were 31% (95% CI: 15–48%) for the RL Algorithm group, 27% (95% CI: 11–44%) for the No Algorithm group, and 12% (95% CI: 3–20%) for the Control group. For IAPT recovery, the estimates were 54% (95% CI: 35–72%), 42% (95% CI: 21–64%), and 28% (95% CI: 16–40%) for the RL Algorithm, No Algorithm, and Control groups, respectively.

Table 3 Bootstrapped multiple imputation estimates
Full size table

Discussion

The present study aimed to evaluate the effectiveness of the gamification and personalization of a digital therapeutics intervention for reducing depressive and anxiety symptoms. Specifically, the study investigated whether incorporating a reinforcement learning algorithm into the intervention program could enhance its efficacy. Only the RL algorithm group demonstrated consistent significant effect sizes across outcomes (PHQ-9: OR = 3.77; GAD-7: OR = 4.05; Recovery: OR = 3.05) compared to the control. The pattern of larger effect sizes in the RL group suggests potential value in algorithmic personalization that warrants further investigation in adequately powered trials. The results suggested that the game-based DTx intervention with algorithmic personalization had the potential to be an effective treatment option for individuals with depression and anxiety symptoms.

These findings aligned with previous research that has demonstrated the benefits of cognitive bias modification training and cognitive behavioral therapy for reducing symptoms of depression and anxiety. CBM techniques have shown promise in previous studies, suggesting that they can lead to significant reductions in symptoms of depression and anxiety and improvements in attentional control and interpretations of emotional stimuli11,12,16. The inclusion of CBT in the game-based DTx intervention is consistent with previous studies that have demonstrated the efficacy of internet-based CBT programs in reducing symptoms of anxiety and depression7,23. This approach not only provides a convenient and accessible platform for individuals to engage in CBT but also fosters a sense of accomplishment and progression through the game mechanics, which can enhance motivation and adherence to the intervention8.

The study’s use of reinforcement learning algorithms showed a novel approach for personalizing digital interventions. RL algorithms have been widely used to optimize decision-making processes based on feedback in various contexts. In this study, an RL algorithm was employed to personalize the DTx intervention by dynamically adjusting the training modules offered to each user based on their individual needs and preferences. The algorithm took into account contextual information, such as demographic characteristics, baseline symptom scores, usage patterns, and performance in the various training modules. By considering these factors, the RL algorithm aimed to optimize the intervention program to maximize user engagement. The results of this study provide preliminary evidence supporting the effectiveness of the game-based DTx intervention, particularly when combined with the RL algorithm. The higher odds ratios and estimated response rates observed in the RL algorithm group suggested that the personalized approach facilitated by the RL algorithm may enhance the effectiveness of the intervention in reducing symptoms of anxiety and depression. The findings also imply the potential utility of RL algorithms in tailoring interventions to individual needs and optimizing treatment outcomes.

The active control group in the study adopted a fixed treatment scheme and did not show significant treatment efficacy. This observation can be attributed to the following: first, the fixed treatment protocol likely produced more variable outcomes due to its “one-size-fits-all” approach, resulting in substantial positive effects (OR = 2.60 for PHQ-9, OR = 3.53 for GAD-7) but with wider confidence intervals crossing the significance threshold. Second, the RL algorithm’s dynamic adaptation to individual usage patterns likely created more consistently effective treatment experiences, reducing response variability and increasing therapeutic precision. Third, our sample size was sufficient to detect the larger, more consistent effects in the RL algorithm group but may have been underpowered to detect the more variable effects in the fixed treatment group, despite their promising effect sizes suggesting clinical relevance.

The digital intervention in this study utilized gamification elements to enhance user engagement. Gamification has been shown to be effective in promoting learning, behavior change, and treatment adherence across various domains, including mental health interventions24,25. By integrating gamified features, such as quests, rewards, and achievements, the game-based DTx intervention provided a more immersive and motivating experience for participants, potentially enhancing their willingness to actively participate and persist in the intervention.

The study has the following limitations. First, the participants were not clinically evaluated or diagnosed but self-assessed with PHQ-9 and GAD-7 scales. Second, the sample size was small. Third, since the setting of the study was entirely online, the study saw dropouts higher than expected. Fourth, due to resource constraints, long-term follow-ups were not conducted. Fifth, the study was not designed to detect differences between the two active intervention arms, limiting our ability to draw definitive conclusions about the added value of RL-based personalization. Future directions include studying whether certain demographics (e.g., teenagers) respond better to the gamified approach than others.

In conclusion, this pilot study provides preliminary evidence supporting the effectiveness of personalized, gamified DTx interventions for reducing symptoms of depression and anxiety. The use of RL algorithms for personalization and gamification for engagement shows promise in enhancing the efficacy of digital mental health programs. Future studies could explore whether specific demographic characteristics, such as age, gender, or cultural background, influence the effectiveness of gamified interventions. For example, younger participants may be more receptive to game-based elements, given their familiarity with digital environments. Identifying which demographic groups benefit most from such interventions will allow for further personalization and optimization of digital therapeutics, potentially enhancing treatment adherence and outcomes.