Introduction

The global pandemic has acted as a catalyst and highlighted the changes required in health and social care systems to ensure the ongoing well-being of the population, with an emphasis on mental health1. This is especially true for children and adolescents, with the prevalence of anxiety and depression symptoms doubled during the pandemic: clinically elevated anxiety symptoms during the COVID-19 pandemic was 21%, compared to a pre-pandemic estimate of 12%2. These increases, particularly in older and female adolescents, highlight an urgent need for mental health help (such clinical services as outpatient care, including psychiatric consultations, diagnostic assessments, psychotherapy, psychopharmacology, and neuropsychological testing, as well as hospitalization) and equitable early detection efforts to mitigate the long-term impact3. Even without the additional load, comprehensive screening for pediatric mental health concerns is challenging (due to several factors, such as multiple informants, overlapping symptoms across different mental health conditions, as well as time constraints and limited mental health training)4,5,6. While Artificial Intelligence (AI) offers considerable potential to enhance access to mental health diagnostics, its intended role is primarily as a supportive tool for initial screening. Rather than replacing clinical expertise, AI systems are designed to assist clinicians by streamlining workflows and identifying individuals who may benefit from further evaluation. Given the complexity of psychiatric assessment and the risk of bias amplification, it remains essential to maintain clinician oversight and ensure that AI integration reinforces the quality and equity of care7.

However, despite the promise of AI in supporting mental health, the development and deployment of machine learning models in real clinical environments remain limited7,8. One of the potential reasons is the risk of propagating harmful biases. Additionally, clinicians often have concerns about the interpretability of predictions from “black box” models, obscuring model suggestions9.

One crucial aspect that influences the development and implementation of trustworthy AI models in clinical settings is the availability of high-quality data in sufficient volumes10. This is particularly challenging in mental health care, where the primary source of information is unstructured free text, clinical notes that often contain highly sensitive and nuanced details. However, the available data are often sparse and biased because the process by which individuals enter the healthcare system reflects underlying inequities. These include structural barriers such as who has access to care, who seeks care, and how symptoms are recognized and documented, all of which contribute to selection bias.

In the context of mental health, AI models are especially vulnerable to amplifying existing biases due to the subjective nature of psychiatric assessment, historical underrepresentation, and the complexity of symptom expression across diverse populations. When trained on such data, models often underperform for marginalized groups, leading to inequitable outcomes and reinforcing existing disparities3,11. Several studies have highlighted this issue within mental health applications3,12. For example, Yates Coley et al. found that suicide risk prediction models performed poorly for underrepresented demographic groups, including Black, Native American, and Alaskan patients12.

There has been extensive research on assessing bias in AI algorithms in machine learning (ML)13,14,15,16 and natural language processing (NLP)17,18,19 as part of broader research on fairness (equitable and non-discriminatory outcomes). However, these general techniques are only beginning to be applied in fields like medicine, and potential remains largely unexplored in the area of mental health3. Traditionally, bias mitigation techniques in ML are broadly divided into: pre-processing techniques, learning algorithm modifications, and post-processing techniques. Pre-processing techniques include approaches like down-weighting the biased instances during training to discourage the model from exaggerating related effects20,21, as well as resampling methods such as under-sampling or oversampling to balance classes in the data, and data augmentation, which synthesizes samples for underrepresented groups22,23. There are also some de-biasing techniques specifically developed in the NLP domain, such as attribute swapping, which creates new text examples by swapping words indicating sensitive attributes in sentences24.

Learning algorithm modifications include approaches like adversarial de-biasing, which uses adversarial networks to penalize predictions influenced by protected attributes25,26. Other approaches also include modifying objective functions to promote fairness during learning (e.g., by penalizing performance differences across demographic groups27,28). Recently, self-supervised pre-trained AI models have shown promise in reducing performance disparities across demographic groups29. Post-processing techniques include approaches such as calibration, which aligns predicted probabilities with actual observed distributions30. Popular post-processing approaches transform linguistic embeddings by removing their projections onto the semantic subspace of the demographic aspect17.

As these methods are applied in the healthcare field, including mental health, several important factors must be taken into account. In healthcare, observed differences between demographic groups can generally be characterized as follows3: (a) genuine differences based on biological influences on disease risk (e.g., presence of hormones contributing to breast or prostate cancer), which should be preserved; (b) disparities shaped by non-biological differences (e.g., variations in clinical visit frequency and the writing style of clinical notes resulting from visiting different experts), which should be when possible reduced; and (c) false differences caused by flawed measurement, such as over-diagnosis in certain groups due to misperceptions (e.g., female depression31). These errors are primarily annotation flaws that need correction.

This categorization establishes a hierarchy of disparities, ranging from acceptable biological variation to deeply problematic systemic bias, and serves as a framework for guiding how bias should be identified and addressed in clinical AI. For example, different presentations or manifestations of disease may fall under genuine differences if biologically driven, or under contextual disparities if shaped by access or reporting practices. While females are not underrepresented in terms of prevalence of anxiety, they may be underrepresented in fair representation within data, particularly when symptoms are documented differently. In the pediatric primary care setting, these challenges are amplified by the need to integrate input from multiple informants (e.g., parents, teachers, and the child), variability in symptom interpretation, and overlapping diagnostic criteria. These challenges complicate consistent and equitable screening4,5,6. These complexities also limit the applicability of many existing de-biasing methods: for instance, swapping gendered terms or removing gender components from embeddings can distort clinical meaning, while algorithmic modifications may lead to overfitting.

In this study, we refer to gender when analyzing linguistic features such as pronouns and personal names, which are typically associated with gender identity rather than biological sex. In NLP, language often reflects gendered patterns and social roles more than biological attributes.

Bias in clinical text remains understudied and poses unique challenges, often reflecting non-biological differences rather than true biological variation. Therefore, specialized mitigation strategies are needed to selectively reduce non-biological bias while preserving essential clinical content.

This study aims to detect and mitigate disparities in the textual training data for AI models intended to assist in pediatric mental health screening. These disparities are caused by biological differences and differences in social circumstances (male and female patients with different symptoms were treated in different healthcare sites with different reporting practices) across sex groups and lead to biased predictions.

To address this, we first assess the presence of bias by evaluating outcome parity across sex subgroups in an AI model trained to predict pediatric anxiety. Observing disparities in model performance indicates bias, which may disproportionately impact underrepresented groups. Next, we identify sources of bias by examining how linguistic and statistical properties in clinical text contribute to unequal outcomes. Using interpretability techniques, we analyze the impact of gender-related language (e.g., first and last names, gender pronouns) on model predictions to trace the influence of potentially biased terms. Finally, we develop and evaluate a data-centric de-biasing method specifically adapted for mental health text. This method includes normalizing information density to reduce bias and replacing biased words with neutral alternatives. Our approach complements general de-biasing techniques and contributes to best practices in the field, particularly as Large Language Models (LLMs) continue to grow in prominence.

We analyze ~20,000 pediatric anxiety cases and matched controls (ages 5–15) from Cincinnati Children’s Hospital records, observing a shift in gender prevalence from male-dominant in early childhood to female-dominant in adolescence. Our anxiety prediction model uses a Transformer architecture optimized for computational efficiency. We find measurable disparities in anxiety prediction across sex subgroups, with females showing 4% lower accuracy and 9% higher false-negative rates. These biases arise from differences in clinical notes, such as word distribution and information density linked to reporting practices. Our data-centric approach, combining information density filtering and gender-neutral word substitutions, reduces diagnostic bias by up to 27%. Further validation across AI models and clinical settings remains essential.

Methods

Our dataset originates from the Cincinnati Children’s Hospital Medical Center (CCHMC). We apply a range of representative NLP methods to analyze this data.

Datasets

As part of this study, we created a foundational database consisting of Electronic Health Record (EHR) data from the CCHMC’s Epic Link. This database has approximately 1.3 million unique patients seen at CCHMC between January 1, 2009, and March 31, 2022, with 63 million clinical notes (see Fig. 1).

Fig. 1: Flowchart illustrating the selection and binning of 73,288 anxiety cases from the Cincinnati Children’s EHR.
Fig. 1: Flowchart illustrating the selection and binning of 73,288 anxiety cases from the Cincinnati Children’s EHR.
Full size image

Cases were identified using ICD codes and filtered for patients with at least one note >30 days before diagnosis. All the anxiety cases were divided into 25 age bins and split into 80% training and 20% test sets. Controls (drawn from the same patient pool) were matched 1:1 by age (±30 days) and sex, with no prior anxiety diagnosis. On average, 25 notes per patient were retained, duplicates were removed.

We define anxiety patients as any patients who have received any of the diagnostic codes listed in Supplementary Table 1. Additional selection criteria are that the patient must have had at least one documented healthcare contact in the EHR in the 18 months prior to the anxiety diagnosis. As a result, there were 1,383,145 total patients in the CCHMC EHR, 84,426 total anxiety cases that passed our selection criteria, 77,187 total anxiety cases with at least 1 note, 73,288 total anxiety cases with at least 1 note >30 days before their first anxiety diagnosis, and 7,810,849 notes for these 73,288 patients.

This cohort contains demographic data (age, sex, and race) and the timeline of clinical textual notes provided by different care providers (progress notes, telephone encounters, plan of care notes, patient instructions, etc.). When we apply our additional selection criteria on note types (we select progress notes and telephone encounters as deemed the most informative by internal experts), this is reduced further to 4.3 million notes.

Age binning

All 73,288 patients who matched the selection criteria were successfully binned by age (in years) at the time of their anxiety diagnosis. We used a data-driven approach, ensuring that no eligible data was discarded during the process. For example, the 5-year-old age group consists of patients who received an anxiety diagnosis between their fifth and sixth birthdays. The controls were matched one-to-one with cases by age and sex. Matched controls (1) were born within 30 days of the case and (2) were of the same sex. Additional selection criteria required that each patient had never received one of the anxiety diagnoses at the time of the matched case’s anxiety diagnosis and had at least one documented healthcare contact in the EHR within the 18 months preceding their anxiety diagnosis, ensuring sufficient clinical history for analysis. Only data prior to the first anxiety diagnosis were used for our analysis.

We used Bins of age 5, 8, 10, 12, and 15. This selection of Bins is not common for pediatrics, but gives us a comprehensive selection of datasets with diverse percentages of female patients varying from 36% to 69%.

Each final Bin contained the training set of ~3700–5064 cases and controls, and the testing set included ~852–1278 cases and controls (sampled as 20% of each Bin, refer to Table 1 and section Descriptive Analysis in Results).

Table 1 Demographic statistics and properties of textual notes across Age Bins (measured for the training data)
Full size table

Data cleaning

To remove duplicates in progress notes and telephone encounters (duplicate notes are common due to clinical practices such as copy-pasting previous entries), we first tokenized our notes by removing punctuation and stopwords using the NLTK toolkit32. We then vectorized the notes using the CountVectoriser from the Scikit-learn toolkit33. Duplicate notes with cosine similarity ≥0.8 were removed. Finally, we selected the 25 most recent notes in each patient’s history (average minimum count of notes per patient across the Bins).

This study was approved by the Institutional Review Board of Cincinnati Children’s Hospital (STUDY2020-0942) and was conducted in accordance with the Declaration of Helsinki. Informed consent and assent to participate in the study were obtained from all participants and their parent/legal guardians.

Anxiety prediction models

We built our anxiety prediction models by fine-tuning the representative Transformer-based Clinical-BigBird model34 as imported from HuggingFace35. Clinical-BigBird is pre-trained on MIMIC-III36, a large, publicly available dataset of de-identified EHRs, providing strong domain alignment with our task. One of its key advantages is the ability to process long input sequences of up to 4096 tokens, which is particularly beneficial in clinical NLP, where patient histories often span multiple notes. This capability stands in contrast to models like BERT37, which are limited to 512 tokens and may truncate important context. Clinical-BigBird uses a computationally-efficient sparse attention mechanism. While our fine-tuning data is narrower in scope than MIMIC-III, it remains domain-consistent, consisting of mental health text notes from pediatric settings.

Our motivation for selecting Clinical-BigBird was threefold: (i) it belongs to a family of compact, discriminative pretrained models that remain widely used in mental health contexts, where efficiency and robustness are essential (they are less prone to hallucinations)38,39; (ii) such models have been shown to perform on par with, or in some cases outperform, much larger generative models on in-domain classification tasks40; and (iii) its low computational requirements allowed us to run systematic experiments in the limited computational environment.

We followed the best practices in the domain and fine-tuned the model for 2 epochs. We limited the input length of notes to 1000 tokens (since considering longer inputs did not result in further improvement), even though the average concatenated sequence of the 25 most recent notes per patient exceeds 4000 tokens. Hence, 728 tokens on average were retained per patient in the model inputs (see Table 1). We used AdamW optimizer41 with a learning rate of 1e-5 and a batch size of 8 (these were the best-performing set of parameters out of the sets recommended for fine-tuning).

Bias assessment

In this work, we focus on two key types of bias in predictive models: selection bias and overamplification bias. Selection bias occurs when training or testing data are not representative of the population, which can lead to systematic disparities in model performance. Within selection bias, we specifically address textual bias, a subtype that arises in NLP when differences in word usage across demographic subgroups influence predictions. For example, terms like “football” may appear more frequently in notes describing male patients, while “cheerleading” is more common in notes about female patients.

Overamplification bias occurs when models learn spurious correlations between patterns in the data (such as biased words) and target labels, amplifying existing disparities rather than relying on clinically relevant information. For instance, predictive models may incorrectly associate gendered pronouns (e.g., “she”) or occupation-related words (e.g., “teacher”) with higher risk of certain conditions.

Figure 2 illustrates the broader taxonomy of predictive bias in clinical NLP models, including label bias (introduced through human annotation) and linguistic bias (arises from general linguistic variation). While our framework acknowledges all four types, it specifically targets textual bias through data-centric interventions, such as neutralizing biased language and normalizing information density while preserving clinically relevant content.

Fig. 2
Fig. 2
Full size image

Types of predictive bias and their origin. We consider four types of bias in clinical Natural Language Processing (NLP) models, as outlined by Shah et al. (2020)62: a selection bias, arising from training or testing data that are not representative; b label bias, introduced through human annotation; c linguistic/textual bias, which refers to systematic differences in word distributions resulting from general linguistic variation or within a particular dataset that influence model predictions; and d overamplification bias, where statistical models intensify discrepancies present in the training data. While our framework acknowledges all four, it specifically targets textual bias, as this is the most directly addressable through data-centric interventions.

According to established practices in machine learning15,16, model bias is typically assessed through three key dimensions: classification parity (ensuring equal error rates across groups), anti-classification (evaluating the influence of protected attributes on predictions), and calibration (comparing predicted risks with actual outcomes).

In this work with predictive models, we focus on classification parity. Several metrics are commonly used in the fairness literature to evaluate classification parity. These are accuracy equality (equal accuracy between groups), equal opportunity (equal false negative rate (FNR), orientation towards recall), or predictive equality (equal false positive rate (FPR) between groups, orientation towards precision)42,43. Following Feldman et al., we also use the balanced error rate (BER), which is the unweighted average of FPR (precision) and FNR (recall)42:

$$BER=\frac{(\frac{FP}{FP+TN})+(\frac{FN}{FN+TP})}{2}$$
(1)

To assess bias, we compute the ratio of Balanced Error Rate (BER) scores by comparing the non-privileged subgroup to the privileged subgroup. A ratio > 1 indicates that the model performs better for the privileged group, while a ratio < 1 suggests better performance for the non-privileged group. Following Feldman et al. (2015), we interpret ratios > 1.25 or < 0.80 as indicating meaningful differences in model performance between groups. Specifically, a ratio > 1.25 reflects a substantial disparity favoring the privileged group, whereas a ratio < 0.80 signals a reverse disparity favoring the non-privileged group. We use FPR, FNR, and BER ratio to assess the bias of our five representative Transformer-based Clinical-BigBird34 prediction models built using Bins 5, 8, 10, 12, and 15 with 1000 tokens input sequence length.

Anti-classification measures are also suitable to assess textual bias. For example, predictive models can erroneously rely on biased words (e.g., relying on the pronoun “she” indicating gender to predict the increased probability of depression or on the word “teacher” carrying the semantic of a female in its pre-trained representation44). In this work, we use the state-of-the-art interpretability analysis to verify if our models erroneously rely on gender words (first and last names, as well as gender pronouns). Note that this reliance could also be considered the sub-case of overamplification bias (picking up on imperfect evidence to predict outcomes), but while working with text, we consider it as textual bias. We leave the investigation of calibration measures to future work.

Explainability

In addition to measuring the performance of our models with the standard accuracy measure, we perform qualitative analysis of words our models rely on while making predictions. The Local Interpretable Model-Agnostic Explanations (LIME) technique45 enables this qualitative analysis by offering local explanations and pinpointing specific words that influenced the model decisions. These explanations are “local” because they relate to the model’s behavior for each specific incoming note. Note that globally important features (for example, weights that a model assigns to words in its vocabulary) might not be precise enough for the local prediction context.

In our study, we used the LIME methodology to highlight influential words in order to verify whether our prediction models are functioning correctly. LIME is designed to uncover undesirable behaviors in AI models that might seem efficient based on standard metrics. For example, Ribeiro et al. (2016) showed that a model, despite a 94% accuracy in differentiating documents on Atheism versus Christianity, relied on irrelevant words such as “posting”, “host”, and “re”. These words were wrongly associated with Atheism due to their frequent appearance in the training data45.

We selected LIME for local explainability due to its speed and intuitive output, which makes it particularly suitable for communicating model behavior to non-technical stakeholders in clinical settings. While alternative methods such as SHAP46 offer more theoretically grounded attributions, they are computationally more intensive and conceptually complex, hence less practical for our use case.

Text de-biasing methods

Motivated by our observations over the textual distributions across demographic subgroups, we propose two following bias mitigation methods. These are adaptations of existing techniques, developed by us to address the challenges of mental health clinical text:

  1. 1.

    Information density filtering (tf-idf filtering): We perform normalization of content, which involves filtering sentences from concatenated notes using their importance scores. Those importance scores are computed as averaged sum of word-level TF-IDF scores per sentence. TF-IDF scores help identify most salient words in a document by multiplying the frequency of the word in that document by its rarity across all documents (the more rare is the word the higher is the score) so that document keywords (e.g., term “myopia”) that may appear multiple times in one document but not in the others receive higher score than auxiliary words such as articles ("the”, “a”) or verbs ("am”, “have”).

    Our information density filtering builds on the established technique of information filtering, which is widely used across domains47. However, its effectiveness depends heavily on how it is adapted to the specific context. In our case, we apply sentence-level TF-IDF scoring to clinical notes to retain the most informative content. Importantly, there is no universally accepted “go-to” benchmark for information filtering in clinical text. Because filtering strategies must be customized to the data, comparisons across methods are often not straightforward. To provide a meaningful baseline, we benchmark our approach against random information filtering.

  2. 2.

    Gender-word debiasing (gender debias): Following the best de-biasing practices from the NLP domain48, we focus on names and pronouns as gender-biased attributes in text. We automatically detect those biased words and replace them with relevant neutral versions. In particular, we detect proper nouns (first and last names) using the off-shelf Stanza tool49. We extract unique names and group them based on the character similarity, ensuring that variations of names are considered equivalent (for example, “Jonathan”, “Johnathan”, and “Johnatan”). This grouping facilitates the creation of a mapping system where each name group is replaced by a generic identifier, such as “person1”, “person2”, etc. (enumeration is maintained per note).

    Following the name replacement, the text undergoes pronoun substitution, where gender-specific pronouns from a dictionary (for example, “she” and “he”) are replaced with their gender-neutral counterparts using a predefined pronoun mapping (for example, “she” and “he” are replaced with “they”). Note that this approach maintains the integrity and coherence of the original text while achieving certain gender neutrality. This also avoids reinforcing societal or cultural biases associated with specific names.

    We avoid techniques such as gender-swapping (e.g., replacing “he” with “she”), which may introduce clinically implausible scenarios. For example, the sentence “She reported experiencing abdominal pain and mood changes around her period” would become “He reported experiencing abdominal pain and mood changes around his period”, a biologically unrealistic sentence. Our method is designed to mitigate bias without compromising the authenticity of symptom descriptions, which is especially critical in sensitive domains like mental health.

Both de-biasing methods were tested independently and in combination. For example, substitution of biased words could be applied to the original text or after the removal of the least informative sentences. We aimed to preserve baseline accuracy while reducing bias, rather than to optimise overall performance.

Statistics and reproducibility

This study was designed as a feasibility proof-of-concept, and available resources did not permit high-throughput statistical validation. We selected five age bins with ~4000 of cases and matched controls per bin, providing sufficient sample size for training Transformer-based models and ensuring demographic diversity.

All models used in this work were downloaded and installed locally, and all experiments were executed on NVIDIA A100-SXM4-40GB GPUs. To address reproducibility, we have released a public version of the code for our models and bias detection metrics, along with synthetic data, in a public repository: https://github.com/julia-ive/bias-pediatric-anxiety50.

Results

We elaborate on our findings per each research objective below.

Objective 1: assessing the presence of bias by evaluating outcome parity across sex subgroups

Overall, performance of our models is reported in Table 2 across Bins and Tables 3 and 4 across demographic subgroups. Our model achieves the performance of 0.61 accuracy on average (this performance is above the random 0.50 accuracy). We also report the percentage of uncertain predictions as an indicator of the model’s confidence in its decisions. Standard AI predictive models issue a probability distribution over their outcomes. If the predicted probability is ≥0.5, the instance is classified as the positive class. Uncertain predictions (from the borderline probability zone [0.4, 0.6]) are less stable and reliable. If the percentage of such predictions is high, a model can hardly be considered useful.

Table 2 Classification parity results for models trained on original and de-biased notes
Full size table
Table 3 Classification parity results across sex groups for models trained with original and de-biased data
Full size table
Table 4 Classification parity results across race groups for models trained with original and de-biased data
Full size table

Looking closer at the bias within demographic subgroups using BER, we observe that our Bin 10 model is slightly more biased towards male patients with the BER ratio of 1.33 (see Table 3). BER ratios, however, do not tell us the whole story. Looking at FNR scores (equal opportunity), we can see that female patients are consistently under-diagnosed (+0.09 FNR average higher for females than for males; an increase of +0.13 is observed for Bins 5 and 15). This trend is observed across cohorts and is consistent even if the percentage of female patients changes. FPR values do not exhibit such a systematic increase for males or females and stay roughly the same. Also, the percentage of uncertain predictions is systematically higher for females than for males (+5%), confirming that our models “hesitate” more for female patients. Finally, accuracy differences exhibit a decrease of −0.04 points on average for female patients. In summary, FNR (equal opportunity) is the most expressive measure revealing relevant predictive bias in our anxiety models. It correlates well with differences in textual statistics from the Descriptive Analysis subsection. Bias is most pronounced for female patients in cohorts 5 and 15, where the similarities in textual distributions are the lowest. Note that similar tendencies are observed across race subgroups: we observe an average increase in FNR of +0.05 for the non-privileged race group. FPR values exhibit an increase towards the privileged class (+0.05 FPR). Also, a decrease of -0.03 in accuracy is observed for other races (Table 4).

For the anti-classification analysis (reliance on biased words for prediction), we check the top ten examples with the highest model confidence (most useful predictions) in a case or a control prediction (20 examples per age Bin, 100 examples in total across Bins). We extract five words per example that influence the model decision the most and then collate them into a frequency vocabulary per predicted class. The results of this analysis demonstrate that the models mostly rely on relevant words both for cases and controls across low and high confidence groups (Table 5). However, there is some tendency to rely on irrelevant gender-biased words: for example, for cases, 10% from the collated dictionary is biased.

Table 5 Top-10 most influential words and Top-10 most frequent words as extracted by the state-of-the-art interpretability LIME tool45 from 100 examples across Bins 5, 8, 10, 12, and 15
Full size table

Objective 2: identifying bias sources through textual distribution analysis

To understand the reasons for performance differences, we analyzed primarily the sex subgroups of the dataset and the properties of the relevant texts. In particular, we considered the following characteristics of textual distributions commonly applied in the NLP domain:

  1. 1.

    Average length of a patient note in words (without tokenization, i.e., separation of punctuation). To characterize the typical volume of clinical documentation per patient, we assessed the average length of patient notes by aggregating all valid entries into a single text string per individual.

  2. 2.

    Percentage of medical terms, in each note on average. We extract biomedical named entities using the state-of-the-art Stanza tool51. Those in-domain entities are extracted for ten standard categories, such as Observation, Treatment, Anatomy, Procedure, etc.

  3. 3.

    Percentage of gender-biased words, in each patient note on average. Using the best practices from NLP48, we extract all the proper nouns (first and last names) using the off-shelf tool49, as well as gender pronouns (he, she, his, her, him, hers). We focus on this semantic group as it is most relevant to our cohort design, which involves matching patients by age and sex.

  4. 4.

    Jaccard index measures the similarity between two vocabularies by estimating the portion of common words. It is calculated by dividing the number of words shared by both vocabularies by the total number of words in both vocabularies combined. We mainly compare vocabularies for male/female.

  5. 5.

    Familiarity score assesses the ratio of common words to unique words. It is calculated dividing the proportion of words that occur in both sets by the proportion of words that occur only in one of the sets. Again, we mainly compare vocabularies male/female.

The results of our analysis across Bins are presented in Table 1 and across demographic groups within each Bin in Table 6. Our initial observations indicate notable changes in demographics and text properties across age bands (seen in Table 1). The ratio of females is consistently growing (from 36% for Bin 5 to 69% for Bin 15), while the percentage of the other races is stable (around 30%). The average concatenated note length remains constant at around 4000 tokens.

Table 6 Class statistics and properties of textual notes across sex and race demographic subgroups in Bins
Full size table

The differences between demographic groups within the Bins are evident (see Table 6). The percentage of cases is always 50% across sex subgroups, which is predefined by the cohort design. This percentage is typically lower for the non-privileged race subgroup, as race was not factored into the cohort design (43% on average for other races). There are also obvious differences in the length of the notes. Notes for the male subgroup are on average 543 words longer (161 words longer for white patients).

The word distributions for male/female show the similarity of 0.54 Jaccard on average (average Familiarity 2.4). The lowest similarity for male/female is observed for Bins 5 and 15 (average Jaccard 0.43, Familiarity index 1.75). The percentages of terms and biased words for male/female are roughly the same. The distributions of terms male/female are even more dissimilar (average Jaccard 0.34, average Familiarity 1.5). Relatively low similarity values for word distributions are also observed across race subgroups (average Jaccard index 0.34, average Familiarity index 1.52). Though other races make only 30% of the patients, these values are not directly comparable to those for sex subgroups.

Given that the female subgroup is well-represented (49% of the examples on average), the differences we observe cannot be explained by scarcity and suggest qualitative differences in the content of the notes across demographic subgroups (as evidenced by rather low similarity scores). The volume of diagnostic and gender-biased content remains constant across sex groups. This suggests that the differences in the length of the notes across subgroups are caused by other than diagnostic content and may be filtered. These content differences may be due to the fact that notes for female/male patients tend to come from different care sites following different reporting standards. For example, the male notes in Bin 5 come from more than 400 care sites, while the female notes come from 337 sites. Female notes mostly come from General, Developmental, and Behavioral Pediatrics, whereas male notes very often come from more specialized departments such as Neurology and Gastroenterology.

In summary, our cohort matching procedure, considering age/sex ensures we have equivalent representation of each between cases/controls. However, statistical differences in textual distributions for sex subgroups persist. It is important to note that these differences are influenced by symptomatic variations and social circumstances of patients of different sexes who are seen at different care sites. These differences are difficult to control (it is difficult to control lengths of notes across care sites or choose only the records without gender-related words or names), but could be more easily manipulated with the help of text pre-processing techniques. We propose two techniques like this for bias mitigation in subsection Bias Mitigation above.

Objective 3: developing and evaluating a de-biasing method for mental health text data

We have already seen that there are considerable differences in information density and textual distributions of notes for sex subgroups, in particular, those differences naturally involve gender words, which our models can erroneously rely on. Our core hypothesis is that, by eliminating less relevant sentences (tf-idf filtering), we can balance density in the notes and reduce bias in our models. Regarding gender words, we have already seen that male and female notes contain similar percentages of such words. Hence, we have developed an approach to neutralize those words via substitution with their gender-neural versions rather than their removal (gender debias, see subsection Text De-biasing Methods in Methodology).

We applied our de-biasing methods as described in the Methodology section above to modify our training data. The underlying anxiety classification model is based on pretrained BERT representations, which were learned from the original, unaltered clinical text. De-biasing is applied only during the fine-tuning phase, where the model learns task-specific representations. This approach allows us to assess the impact of gender-neutral language on downstream predictions. In particular, we applied the random filtering method and removing 20% of sentences at random as our baseline. This choice was deliberate, aiming to represent the cheapest realistic mitigation strategy that could be implemented without requiring additional data or compute, domain-specific or complex modeling. We compared it to the performance of our approach tf-idf filtering, which removes 20% sentences according to their informativeness score (threshold defined empirically). We applied the gender debias method to both the original notes and the notes filtered using tf-idf filtering to trace the combined effect of the two models. Our approach is compared against a baseline method (random filtering), which removes 20% of sentences selected at random.

Each time we modified the training data, we re-trained our models to obtain new models, which were then tested on the original test data. Our results show that the de-biased models in general maintain the performance of the original models (Table 2): both gender debias and random filtering slightly increase the performance by +0.5 accuracy, while tf-idf filtering decreases the performance on average by -1 accuracy. The mixed method tf-idf filtering + gender debias maintains the original performance with negligible changes. While maintaining performance, our de-biasing methods positively impact the performance by reducing the percentage of uncertain predictions by −8% on average, with the highest reduction for the tf-idf filtering + gender debias of −12%.

In terms of the reduction of bias as measured by FNR, tf-idf filtering is a clear winner (see Table 3). It outperforms the random sentence removal baseline random filtering, reducing the FNR gap by -0.024 (27%, initial average gap 0.09) point on average, with the highest reduction by -0.11 point for Bin 5 (from 0.13 to 0.02) and −0.06 point for Bin 15 (from 0.13 to 0.07). Random filtering baseline does not exhibit any consistent behavior and does not influence the gap across Bins. Similarly, our gender debias approach, when applied in isolation, is not effective in narrowing the FNR gap. On average, we even observe a small increase in this gap rather than a decrease (+0.008). tf-idf filtering + gender debias has roughly the same performance as tf-idf filtering, with the average decrease of -0.022 points. Both tf-idf filtering and tf-idf filtering + gender debias approaches manage to maintain the BER ratios within the acceptable level, and even decrease the BER ratio for Bin 10 from 1.33 to the acceptable values of 1.24 and 0.98 for tf-idf filtering and tf-idf filtering + gender debias, respectively. tf-idf filtering reduces the increase in uncertain predictions for females by 50%, while tf-idf filtering + gender debias fully eliminates this dis-balance. tf-idf filtering + gender debias also demonstrates the ability to mitigate bias for race subgroups, reducing the FNR gap by -0.034 points on average. For Bin 10, we also reduce the bias towards the non-privileged class (from BER ratio 0.74 to the acceptable level of 0.95).

We also observe a positive effect of our de-biasing methods in terms of the words our classifiers rely on (anti-classification, see Table 5). Our first observation is that both tf-idf filtering and tf-idf filtering + gender debias improve the generalizability of our models. We observe less terms appear in the statistics, signifying more reliance on context (e.g., words “complaint”, “presents”, “no”) while making predictions rather than overfitting to keywords (e.g., “anxiety”, “depression”, etc.). Furthermore, our de-biasing techniques the proportion of biased words that the models rely on. For instance, tf-idf filtering reduces this percentage from 10% to 3% for cases and from 4% to 2% for controls. tf-idf filtering + gender debias further halves the frequency of biased words used by tf-idf filtering. This is a positive outcome, as these residual biased associations become less systematic compared to those observed with tf-idf filtering alone.

Discussion

This study aimed to detect and mitigate linguistic bias in the training data of AI models, with a particular focus on sex-based bias. First, we found measurable disparities in model performance across sex subgroups, highlighting predictive bias that disproportionately affects females. This was evident in the lower classification parity (4% lower accuracy for females than for males across age groups) and higher false negative rates for female patients (9% higher on average across age groups), suggesting that the model was less accurate in diagnosing anxiety in this subgroup. Second, we identified intrinsic differences in textual properties between male and female patient notes, such as variations in note length (notes for males are 500 words longer), word distribution (low similarities for male/female word distributions of 0.54 Jaccard index, whereas values above 0.7 are considered indicative), and information density (low similarities for male/female term distributions of 0.34 Jaccard index). These differences are likely linked to reporting practices and medical documentation styles, which contribute to biased outcomes in AI predictions. Third, our data-centric approach to mitigate this bias using information density filtering and gender-neutral word substitutions improved classification parity by up to 27%, particularly benefiting the non-privileged subgroup (females).

This study supports previous findings that AI models trained on clinical data can perpetuate biases present in the original data, disproportionately affecting underrepresented groups, e.g., the study of Obermeyer et al. (2019) has found that commercial prediction algorithms used in healthcare to identify patients with complex needs exhibit pronounced racial bias, as they predict healthcare costs instead of illness severity, resulting in Black patients being under-identified for additional care despite having more severe health conditions52. Similarly, our findings align with prior research showing that linguistic patterns, such as gendered language, contribute to bias in natural language processing models used in healthcare19,53.

While previous studies have shown that AI models can produce biased outcomes across demographic groups, addressing bias in healthcare data presents unique challenges. In healthcare, it is important to retain biological differences in the training data that reflect actual patient needs, while mitigating biases that arise from non-biological factors, such as cultural or provider-based documentation differences. Traditional NLP methods, such as swapping gendered words24 or removing gendered meanings from word representations17, are not suitable for this purpose in healthcare, as they could lead to inaccurate data representations. Moreover, healthcare notes vary widely across providers, introducing additional complexity. Our method addresses these challenges by selectively de-biasing data: it maintains information relevant to clinical care while reducing the influence of biased language and normalizing information density across records.

This study has several strengths. First, this study focuses on a data-centric approach, emphasizing the quality and relevance of data rather than improving algorithms (model-centric AI)54,55. Second, the de-biasing methodology developed here is specifically adapted for heterogeneous healthcare text data from different clinical sites. Our dataset includes records from over 600 care locations within a single institution, ranging from primary care clinics to specialized and school-based centers. This tailored approach is particularly effective in pediatric mental health, where reliable and equitable early detection is critical. This approach not only clarifies how specific language influences model predictions but also demonstrates practical effectiveness: bias mitigation techniques, such as word substitution and information density balancing, reduced diagnostic bias by up to one-third for systematically under-diagnosed female patients. Third, this study creates a pathway to further exploration of complimentary de-biasing techniques specific to AI in the mental health domain.

The study has some limitations. The focus of the study relies on the quality and consistency of the EHR text. Variability in note quality between providers may influence the model’s ability to generalize across different clinical settings, which is a common challenge in EHR research. We also acknowledge that matched controls may have had clinical notes for reasons unrelated to anxiety, and that sex-based differences in healthcare-seeking behavior could introduce bias into the models. To minimize this risk, controls were drawn from the same patient pool as the anxiety cases, ensuring a comparable clinical context between groups. Our model was therefore designed to distinguish the early clinical presentation of anxiety from the point at which a formal diagnosis is recorded.

This study focuses on demographic bias within a male/female pediatric population, with an emphasis on gender. Our decision to prioritize gender was guided by well-established clinical evidence showing that females have a higher lifetime prevalence of anxiety disorders than males, a disparity that becomes especially pronounced during adolescence56,57. While our framework is designed to be generalizable across clinical domains, this focus emphasises its particular relevance for adolescent mental health research. The drivers of gender-related anxiety differences remain insufficiently understood and are likely shaped by a complex interplay of biological, psychological, and social factors. Addressing gender bias in pediatric anxiety research is therefore critical, both to prevent AI-driven tools from reinforcing existing disparities and to enable fair models that can support abductive reasoning and the generation of new hypotheses about the underlying mechanisms of these patterns.

Finally, another limitation of this study is the reliance on Clinical-BigBird, a relatively small discriminative language model. While such models are well-suited for in-domain classification tasks, often outperforming larger generative LLMs due to their efficiency and reduced risk of hallucinations40, they may not fully leverage the extended context capabilities offered by newer LLM architectures (such as MedGemma58 or Med-PaLM259).

Moreover, Clinical-BigBird is also representative in that its performance is modest (diagnostic accuracy of 0.61). This reflects the reality of many clinical NLP applications. Also, as prior work suggests60,61, models with lower absolute performance can sometimes make subgroup disparities more apparent, since their predictions are more sensitive to input features. We therefore used Clinical-BigBird as a diagnostic bias tool.

Our proposed de-biasing framework has the potential to be generalizable across different model types. However, future work is needed to empirically validate this. Our de-biasing methods represent accessible and cost-effective mitigation strategies that could be implemented without additional data, heavy computation, or complex modeling.

Conclusion

Anxiety disorders are a leading cause of disability in children and adolescents worldwide, with rising rates among minority groups56. AI can play a transformative role in early mental health detection, but its success depends on reliable, unbiased data. This study presents a data-centric de-biasing approach designed to address disparities in AI model performance in clinical text, especially among under-diagnosed groups like female patients. By balancing information density and neutralizing biased terms, our approach reduced diagnostic bias by up to one-third. These findings underscore the importance of bias-aware data processing to create fair and effective AI tools in mental health.