Introduction

Parkinson’s disease (PD) is a highly heterogeneous neurodegenerative disorder that presents with diverse symptoms, underlying causes, and individual patient experiences1. While historically, PD research has predominantly focused on its motor symptoms, it has become increasingly evident that PD is not just a motor disorder but also a complex neuropsychiatric condition2. Neuropsychiatric symptoms, such as mood changes, depression, anxiety, apathy, impulse control disorders, and cognitive impairment, have an equal or even greater impact on patients’ quality of life than motor symptoms3.

Dopaminergic replacement therapies are effective not only in alleviating the motor symptoms of PD but also play a critical role in managing common neuropsychiatric symptoms.

Nonetheless, as the disease progresses, the therapeutic window of optimal symptom control narrows. Consequently, the long-term use of dopaminergic medication and disease progression often lead to fluctuations in symptoms, which in the case of neuropsychiatric symptoms are commonly called neuropsychiatric fluctuations4. During “OFF” medication states (low levels of dopaminergic medication), patients frequently report fatigue, low mood, lack of initiative, and difficulty generating new ideas. Conversely, during “ON” medication states (high levels of dopaminergic medication), patients may exhibit increased talkativeness, euphoria, impulsivity, and rapid generation of new ideas5,6. Recognizing these fluctuations is essential since they can substantially impact patients’ daily functioning and overall well-being4.

Timely adjustments to treatment can prevent the emergence of more severe behavioral issues, such as apathy or impulse control disorders, which can profoundly impact both the patient’s and their family’s lives6,7,8. Therefore, at this stage of PD, the patient’s quality of life heavily relies on the clinician’s ability to optimally adjust dopaminergic treatment, minimizing “OFF” periods and mitigating medication peaks9. However, the precise management of these therapies is often challenging due to the limited and infrequent assessments in clinical practice. Additionally, patients usually find it difficult to retrospectively report past neuropsychiatric fluctuations, as well as their previous mood and behavior states.

Speech analysis offers a promising solution to overcome the challenges of monitoring PD symptoms10. Previous studies have demonstrated that paralinguistic speech features (e.g., prosodic, respiratory, phonatory, and averaged speech spectral features) are valuable digital biomarkers for assessing motor severity, monitoring disease progression, and tracking changes in motor symptoms due to dopaminergic effects11,12,13,14. Speech can also provide rich insights into cognition and emotions, with prosodic features being the most studied paralinguistic domain for emotion recognition15,16,17,18. However, prosodic features in PD are heavily influenced by core motor symptoms, and the resulting dysarthria may alter emotional perception, often skewing it towards negative emotions, such as sadness19.

Recent advances in natural language processing have facilitated the exploration of language patterns in PD, revealing linguistic abnormalities in the language of patients with PD compared to healthy individuals20,21,22. Most existing studies have primarily evaluated linguistic features extracted from spontaneous discourse as markers of cognitive functions23,24. However, language conveys a broader range of valuable information, including feelings and emotions25,26. By analyzing “what patients say”, we can move beyond standardized patient-reported outcomes, gaining deeper insights into patients’ individual experiences and identifying the most bothersome symptoms in PD26.

In this study, we used natural language processing techniques to assess neuropsychiatric changes induced by dopaminergic medications, integrating recent advances in generative artificial intelligence (AI) and large language models (LLMs). First, we analyzed the semantic meaning conveyed by the free speech samples to discriminate between “ON” and “OFF” medication states using traditional machine learning (ML) algorithms and recent LLMs. Then, we evaluated the ability of these models to predict a neuropsychiatric state score under the two medication states.

Results

Participants

This study involved 35 patients with PD who completed the speech assessment during the levodopa challenge and provided written consent for biomedical research. One patient was excluded due to a revised diagnosis after the levodopa assessment. After revising speech recordings, another patient was excluded as their answers consisted solely of single words.

The participants’ mean age was 64 ± 7.7 years, and disease duration were 10.3 ± 3.4 years. Their detailed demographic and clinical information are provided in Table 1.

Table 1 Demographic and clinical characteristics of study participants
Full size table

Figure 1 shows the variability in the neuropsychiatric scores between the ON and OFF medication states, with a mean difference of 31.2 ± 14.9 out of a maximum difference of 60. The average speech length was comparable between the ON and OFF medication states (ON 43. ± 11.8 vs. OFF 45.9 s ± 17.6, W = 263.0, p = 0.76, Wilcoxon signed-rank test).

Fig. 1: Neuropsychiatric State Score during OFF and ON medication states.
figure 1

The dots represent the neuropsychiatric state score for each participant during the OFF-medication condition (purple dots) and ON-medication condition (green dots). The values on the gray segments indicate the difference between the scores in the two states. The vertical lines and the shaded areas represent our sample’s average neuropsychiatric state score and the standard deviation in the two conditions. ID identifier, STD standard deviation.

Full size image

ON-OFF medication state classification

Table 2 presents the best results obtained with each method. The semantic search approach achieved a classification accuracy of 0.85 when the “multi-qa-mpnet-base-dot-v1” model was used to compute the text embeddings, and the top five most similar NFS items were used as voters. Figure 2a shows the occurrences of the five most frequent similar NFS items retrieved by this method.

Fig. 2: Additional metrics for best models in each approach.
figure 2

a Illustrates the frequencies of the top five most similar items of the Neuropsychiatric Fluctuation Scale (NFS) retrieved using the semantic search approach. The x-axis shows the 20 NFS items while the y-axis shows their normalized frequencies, indicating how frequently each item was selected as one of the top five most similar to the recordings. This approach predominantly assigned OFF items as most similar to the language in the OFF-medication state and ON items to recordings in the ON-medication state. b Shows the receiver operating characteristic (ROC) and the corresponding area under the curve (AUC) values obtained with the top performing machine learning models and the LLMs.

Full size image
Table 2 Comparison of the top-performing models for each approach in classifying medication state
Full size table

In the second approach, we used three ML classifiers (RF, SVM, and Gaussian NB) to classify the speech transcriptions. An accuracy and F1-score of 0.92 were obtained when the RF and the SVM classifiers were applied to the “Alibaba-NLP/gte-Qwen2-1.5B-instruct” text embeddings.

In the third approach, we instructed three different LLMs to classify the transcriptions directly. The LLMs Gemma-2 (9B) and Llama-3.1 (8B) achieved an accuracy of 0.98 in the few-shot setting. Specifically, the Gemma-2 (9B) and Llama-3.1 (8B) achieved the best results when the six and ten most similar recordings to the query were used in the prompt, respectively (Supplementary Fig. 2).

The best models were further evaluated by plotting the ROC curves and computing the area under the ROC curve (AUC) (Fig. 2b). The LLMs (Gemma-2 (9B) and Llama-3.1 (8B)) achieved an AUC of 0.99, followed by the machine learning methods, which achieved an AUC of 0.94 with RF and 0.97 with SVM.

The LLMs (Gemma-2 (9B) and Llama-3.1 (8B)) misclassified only one recording. For example, the Gemma-2 (9B) misclassified one sample belonging to the OFF-medication class, assigning the label “ON” with a probability of 0.73. This recording belonged to patient 31 (Fig. 1), who had a neuropsychiatric state score of 49 in the OFF-medication state versus 45 in the ON-medication state. This patient was the only one with a higher ON-score than the OFF-score and was among three patients with a fluctuation of < 10/60, indicating the absence of clinically meaningful neuropsychiatric fluctuations.

Neuropsychiatric state score predictions

We used similar methods to predict the neuropsychiatric state score. The top-performing models for each approach are shown in Table 3. The semantic search approach obtained an RMSE of 14.0 when the top five most similar patients’ recordings were considered. The RF model receiving the “Alibaba-NLP/gte-Qwen2-1.5B-instruct” embeddings as input achieved an RMSE of 11.2, MAE of 8.68, and a median absolute error of 6.23.

Table 3 Comparison of the top-performing models across the three approaches for predicting the neuropsychiatric state score
Full size table

The LLM “gemma-2-9b” in the nine-shot setting resulted in an RMSE of 10.66 and MAE of 8.1. Figure 3 shows scatter plots showcasing the relationships between predicted and actual neuropsychiatric state scores with the best-performing models for each approach. The LLM Gemma-2 (9B) predictions showed the strongest correlation with the actual neuropsychiatric state scores (Spearman’s ρ = 0.81, p < 0.001; Fig. 3c), followed by the Random Forest model (ρ = 0.76, p < 0.001; Fig. 3b) and the semantic search (ρ = 0.64, p < 0.001; Fig. 3a).

Fig. 3: Scatter plots of predicted vs. actual neuropsychiatric state score.
figure 3

The gray dotted line represents perfect correlation. a Shows the results of the semantic search approach based on the similarities between speech transcriptions. b Reports the score estimates of the machine learning model (Random Forest) applied to gte-Qwen2-1.5B-instruct embeddings. c Illustrates the predictions computed by Gemma 2-9b. The colored dots represent the average estimates of 9 predictions. RMSE Root Mean Squared Error, R2 coefficient of determination.

Full size image

The RF model predicted similar scores for the transcriptions recorded in the OFF-medication state, assigning values between 15 and 25. All the approaches better estimated the neuropsychiatric scores in the ON-medication state. The MAEs were 6.6, 7.5, and 11.1 in the ON-medication state and 10.7, 8.6, and 12.2 in the OFF-medication state for the ML model (RF), Gemma-2 (9B) LLM and semantic search, respectively.

Discussion

This study explored multiple linguistic approaches to predict medication states in patients with PD using a spontaneous free speech paradigm. The LLMs Gemma-2 (9B) and Llama-3.1 (8B) achieved the highest accuracy (0.98) and AUC (0.99) for classifying the medication state in the few-shot setting. The other approaches also performed strongly, with text embedding combined with RF and SVM achieving an accuracy of 0.92 and an AUC of 0.94 and 0.97, respectively.

The medication states were classified based on the assumption that most patients with PD experience improved neuropsychiatric symptoms in the “ON” medication state (indicated by higher neuropsychiatric state scores) and worsening symptoms in the OFF medication state (indicated by lower neuropsychiatric state scores). However, given the heterogeneity of PD, it is unsurprising that in rare cases, patients may not exhibit sharp neuropsychiatric fluctuations across medication states or may even experience worsening neuropsychiatric symptoms during the ON medication state. These scenarios may lead to model misclassifications but reflect the actual symptom patterns of the patients. For example, the Gemma-2 (9B) LLM misclassified one patient with consistently high neuropsychiatric state scores (neuropsychiatric state score >40) in both medication states, as this patient showed minimal neuropsychiatric changes between states. These cases highlight variability in PD symptomatology but demonstrate that Gemma-2 (9B) LLM performed robustly even in complex situations.

Performance metrics alone did not clarify whether the models based their predictions on the neuropsychiatric content of patients’ transcriptions or other symptom domains. One possibility to enhance the transparency of the black-box LLMs is to generate self-explanations along with their predictions. Therefore, we “asked” the LLMs to explain their predictions, including the most discriminative words for the medication states. The most frequent words or phrases self-retrieved by the model were “feel better” and “relaxed” for the ON medication state and “don’t feel well” and “tired” for the OFF medication state. While these terms primarily relate to neuropsychiatric symptoms, in some cases, the generated self-explanations just reflected the prompts we provided. This finding is consistent with previous research, which showed that although plausible, the faithfulness of generated explanations can be unreliable27. This issue is particularly evident when using few-shot prompting, where the LLM models tend to mimic the provided human explanation rather than mirroring their inner decision-making28.

Moreover, in Supplementary Material (Section F), we included a calibration analysis of the best-performing models, which were not well calibrated. While our primary goal was to demonstrate the potential of using NLP techniques and generative AI for detecting neuropsychiatric fluctuations, calibration remains an important factor to consider when transferring these techniques in clinical practice. As an outlook for future studies, post-processing methods such as temperature scaling29 for LLMs could be explored to improve calibration and, thus, confidence reliability.

These caveats underscore the importance of using complementary approaches. For example, while the semantic search approach achieved a slightly lower accuracy of 0.88, it provided additional value by retrieving NFS items closely related to each transcription. Interestingly, the most frequent NFS items used to classify ON states (Fig. 2a) —“I have a feeling of well-being” and “I feel talkative, I want to communicate”—and OFF states—“I have jelly legs, trembling,” “I lack energy for everyday activities,” and “I feel low”— aligned with the most sensitive items for distinguishing medication states in the original NFS validation study30. This observation suggests that the items retrieved by the semantic search approach partially mirrored those most effective in differentiating medication states in the NFS validation. Additionally, while the item “I feel lethargic” was frequently noted in both medication states, it occurred more frequently in the OFF state.

Only a few studies have focused on medication state classification. For example, Norel et al.31 recently combined linguistic (semantic word embeddings) and acoustic features from three speech tasks to differentiate between ON and OFF medication states. The greatest accuracy (0.89) was achieved with a picture description task, where PD patients were observed to use fewer action-related words in the OFF state, consistent with previous research. However, their work could not disentangle whether the medication prediction was driven by changes in motor, emotional, or cognitive symptom domains since it lacked a measure for neuropsychiatric fluctuations. Similarly, our study could not disentangle emotional and cognitive symptom domains, as both are represented in the NFS. However, using AI to analyze semantics seems to provide a good measure of the cognitive-emotional state while traditional acoustic analysis instead focuses on motor states. Based on our new findings, speech content analysis appears to provide an even superior prediction of medication states than the traditional speech acoustic analysis10. Combining both approaches with the same speech recordings might be of value, considering that acoustic analysis does not rely on a black box with potential sources of bias, unlike LLMs.

Restricting the classification of the neuropsychiatric state in patients with PD to only ON and OFF medication states may overlook intermediate states, as some patients may not experience a sharp neuropsychiatric transition between them. To account for this continuum, we evaluated the accuracy of the different models in predicting the neuropsychiatric state scores. Consistent with the medication classification results, the LLMs outperformed the other approaches, particularly Gemma-2 (9B), which achieved the lowest RMSE of 10.6 and MAE of 8.1, showing a strong correlation (ρ = 0.81) between predicted and actual neuropsychiatric state scores. Promising accuracies in neuropsychiatric state score prediction were also achieved using RF (RMSE = 11.22), outperforming the LLMs in the zero-shot scenario and the other traditional ML models. Moreover, it achieved the lowest MAE in estimating the recordings in the ON medication state.

The LLMs performed best when a few of the most semantically similar transcriptions were included as input (few-shot setting). A significant advantage of LLMs is their ability to adapt to new tasks by “learning” from a few relevant examples32. This is extremely valuable in the medical field, where small sample sizes are common and pose a challenge for traditional ML algorithms. Additionally, we used nine paraphrases of the input with the same in-context examples for each transcription to achieve stable and consistent predictions across different prompts33. Indeed, LLMs have the drawback of being extremely sensitive to the prompt’s format and content34. Varying the prompts, especially changing in-context examples, resulted in different classification outcomes and neuropsychiatric state score predictions. We observed that incorporating semantically similar transcriptions into the prompt, instead of including random examples, reduced the prediction errors and drastically decreased the variance in the repeated estimates (Supplementary Fig. 5). Moreover, throughout the prompt tuning process, various issues with LLM-generated outputs occurred, including inconsistent output formats (particularly in the zero-shot setting), the generation of multiple labels instead of a single one, and unstructured text arising from a reluctance to classify the neuropsychiatric state of a person. These issues were overcome using the reported final version of the prompt. Such findings underscore the importance of carefully engineering prompts and selecting in-context examples when utilizing LLMs for neuropsychiatric assessment in patients with PD. By optimizing these parameters, we improved the predictive performance and reliability of the models, further validating the use of LLMs to capture neuropsychiatric fluctuations associated with dopaminergic medication states.

Despite our promising results, our study had several limitations. Firstly, its sample size was relatively small for ML applications, and the cohort comprised patients with PD with moderate to severe fluctuations, to whom a suprathreshold dose of medication was administered, which could make neuropsychiatric fluctuations more pronounced. Secondly, patients were assessed in a fixed sequence: initially after overnight withdrawal of medication and subsequently after administration of a suprathreshold dose of dopaminergic medication. Although we recognize the possibility of a repetition effect due to increased familiarity with the task in the second assessment, we believe its impact is likely minimal. This assumption is supported by data presented in Supplementary Material (Section G), where repeating the experiments twice in the ‘off’ state for 20% of the patients did not yield significant differences between the two trials conducted under the same conditions. Moreover, all the included patients also underwent comprehensive clinical assessments, which allowed our models to train on high-quality, comprehensive patient data with significant variability between medication states. Nevertheless, our findings must be replicated in larger populations to validate our methods further.

Thirdly, our approach to estimating neuropsychiatric fluctuations was based on the NFS scale, which has shown reliability in quantifying acute neuropsychiatric fluctuations30,35. However, large-scale validation of the neuropsychiatric state score has not yet been reported (ClinicalTrials.gov: NCT04455074), limiting the generalizability of our findings.

Fourthly, our study relies on Whisper-generated automatic transcriptions, which can sometimes produce hallucinations. While Whisper performs well, even with Swiss German dialects36,37, concerns remain about its reliability in high-risk settings, with around 1% of transcriptions affected by hallucinations38. In our study, occasional hallucinations (around 2 out of 66 sentences were affected), specifically as repeated phrases at the end of transcriptions. These were easily identifiable and did not alter semantic meaning. However, a thorough quantitative evaluation, especially for speech from patients with Parkinson’s disease, would be important to detect other potential hallucinations at lexical and semantic levels.

Fifthly, these methods were based on the assumption that the speech contained emotion-enriched content. Notably, the models predicted neuropsychiatric state scores more accurately for ON recordings. This observation could be explained by the fact that, in the OFF state, some patients tended to focus more on motor symptoms and provide limited information about their neuropsychiatric condition, potentially affecting the neuropsychiatric state score predictions. While this could be considered a limitation, it also underscores the robustness of our methods in isolating neuropsychiatric features from other symptom domains.

Sixthly, while translating the transcriptions into English allows the application of these methods across different languages, it could introduce some additional biases and not fully capture the nuances of the cultures or expressions39. For example, the NFS item “At the moment, I have jelly legs, trembling” may carry a predominantly neuropsychiatric connotation in French but can be interpreted as a motor symptom in German. Future work could explore using original transcriptions and multilingual LLMs to better capture these cultural nuances of different languages.

Additionally, both LLMs (in the few-shot setting) and ML models made their predictions based on examples from other patients, which vary in their characteristics. Collecting repeated recordings from the same patient could lead to the development of more personalized, patient-specific methods, potentially improving long-term monitoring.

Finally, we selected relatively small LLMs due to limited computational power. Models with a larger size should be considered to further improve the predictions, leveraging techniques such as quantization to reduce computational and memory requirements40,41.

By analyzing patients’ own words describing their condition, our methods enhance the sensitivity and specificity of symptom detection and provide a foundation for developing real-time monitoring systems of neuropsychiatric fluctuations. Based on brief talks, our approach may provide a more natural and engaging method for collecting neuropsychiatric self-reported outcomes than traditional tools. Indeed, conventional clinical instruments used in routine clinical practice and research settings rely on pre-defined, structured responses, which can limit patients’ ability to fully express their experiences and potentially lead to inaccurate symptom reporting (e.g., underreporting). These tools are also impractical for frequent daily use, making them less suitable for capturing the fluctuating symptoms of PD outside clinical settings. In contrast, smartphone-based speech assessments, based, for example, on daily calls42, offer the potential to remotely and unobtrusively monitor PD progression10. Given the minimal effort required from patients, our approach could be easily adapted for home use by developing smartphone applications, enabling continuous, unobtrusive, remote monitoring of patients’ neuropsychiatric symptoms10. Moreover, our methods based on speech content analysis provide the additional advantage over acoustic analysis of being less sensitive to the quality of smartphone microphones10.

In conclusion, our proof-of-concept study demonstrates the feasibility of using the patients’ speech content to accurately differentiate between medication states and predict neuropsychiatric fluctuations. This approach can lead to more timely and personalized adjustments of therapeutic strategies, allowing for the detection of neuropsychiatric fluctuations. These fluctuations significantly impact patients’ quality of life directly4 and underlie behavioral sensitization, driving potentially devastating behavioral complications of dopaminergic treatment, such as impulse control disorders, punding, and dopamine dysregulation syndrome8,43.

Methods

Study design

This observational study included patients diagnosed with fluctuating PD according to Movement Disorders Society criteria44. These patients underwent a levodopa challenge as part of the routine evaluation for advanced PD therapies, which included systematic recording of spontaneous speech in their native languages. Patients with PD clinically diagnosed with dementia, according to the Diagnostic and Statistical Mental of Mental Disorders, Fifth Edition45, were excluded. The patients were recruited between 2021 and 2023 at the University Hospital Inselspital in Bern, Switzerland.

The participants were evaluated in ON and OFF medication states. Initially, they were assessed after an overnight OFF, practically defined as a withdrawal of at least eight hours for levodopa and/or 48 h for dopaminergic agonists. Then, they were reevaluated in the ON medication state, occurring 30–60 min after taking a fast-acting compound of levodopa/benserazide 100/25 mg, equivalent to 150% of the patient’s usual dopaminergic equivalent morning dose. Levodopa equivalent doses were calculated according to previously described conversion factors46. Additionally, to ensure that the results reported were not due to a learning effect and more familiarity with the task and the interviewer, a subgroup of 7 PD patients (about 20% of the total sample) was assessed three times. Two times in the practically OFF medication condition with an interval of 15 min, and one time in the practically defined ON condition.

The retrospective analysis of the data collected from patients who provided general consent for biomedical research was approved by the local ethics committee (KEK 2023-01427) and conducted according to the Declaration of Helsinki.

Clinical examination

The neuropsychological fluctuations were quantified using the Neuropsychiatric Fluctuations Scale (NFS)47 in both ON and OFF medication states. The NFS is a self-report questionnaire designed to assess the momentary neuropsychiatric symptoms of PD, demonstrating good sensitivity in detecting acute changes in neuropsychiatric symptoms35,47. It consists of 20 items rated on a scale from 0 (does not describe how I feel right now) to 3 (describes a lot of how I feel right now), with 10 items reflecting typical ON symptoms (e.g., increased well-being, self-confidence) and the other 10 corresponding to OFF symptoms (e.g., tiredness, anxiety). A neuropsychiatric state score was calculated from the NFS items using a formula previously described by Magalhães et al.35, ranging between 0 and 60. A score of 0 indicates a very low mood, whereas a score of 60 corresponds to a very high mood.

Motor symptom severity was assessed in both ON and OFF medication states by the same trained clinician using the Movement Disorder Society–Unified Parkinson’s Disease (MDS-UPDRS) Part III48. Non-motor symptoms (MDS-UPDRS Part I), activities of daily living (Part II), and severity of motor complications in daily life (Part IV) were evaluated exclusively in the ON condition.

Global cognitive functioning was assessed using the Montreal Cognitive Assessment (MoCA)49.

Depressive and anxiety symptoms were measured using the Hospital Anxiety and Depression Scale (HADS)50, apathy was assessed using the Starkstein Apathy Scale (SAS)51, and impulse control behavioral disorders were evaluated using the Questionnaire for Impulsive-Compulsive Disorders in Parkinson’s Disease Rating Scale (QUIP-RS)52. These assessments, which evaluate chronic cognitive and neuropsychiatric symptoms, were conducted only once during the ON medication state.

Speech examination

Participants’ speech was recorded in a quiet room using a head-mounted condenser microphone (Shure Beta 53; Shure, Niles, IL, USA) positioned 5 cm from the mouth. The recordings were sampled at 48 kHz with 16-bit resolution53. Speech recording was conducted prior to the motor examination to minimize potential biases in participants’ descriptions toward motor symptoms.

Participants were asked to speak freely in a monologue task designed to assess their neuropsychological state under different medication states. To elicit responses rich in emotional and cognitive content, participants were informed of the importance of describing not only their current motor state but also their emotional and cognitive state in detail. The task began with the same open-ended question: “How do you feel right now?”. To replicate a conversational setting, the examiner, who was alone in the room, was facing the participant and maintained an encouraging demeanor. If the initial response was too short, after a pause of 10 s, the examiner would prompt further elaboration to ensure the collection of additional speech. The target cumulative duration for the monologue was 60 s (i.e., upper limit) of total recorded speech length, including pauses and hesitations (not accounting for net connected speech time). This task was conducted under both ON and OFF medication states.

Speech transcription and English translation

The recorded speech samples were transcribed into text using an automatic speech recognition (ASR) tool developed by OpenAI, USA (Whisper)54. This ASR tool was selected based on a comparative performance evaluation of different state-of-the-art speech-to-text models55. The recorded speech samples included three languages spoken in Switzerland: French, Italian, and German or their respective Swiss dialects. Since a standardized written form does not exist for the Swiss-German dialects, ASR for Swiss-German usually transcribes it into standard High-German text. The selected ASR has shown impressive zero-shot performance for Swiss-German speech36, further improved by fine-tuning techniques37. Therefore, to transcribe the Swiss-German recordings, we fine-tuned (https://huggingface.co/blog/fine-tune-whisper) the ASR tool (Whisper-medium version) on a publicly available annotated Swiss-German speech dataset (STT4SG-350)56. French and Italian recordings were transcribed using the generic version of the ASR tool (Whisper-large-v3 version (https://github.com/openai/whisper)). Then, all the transcriptions were translated from their original languages into English using Whisper’s translation task since multilingual autoregressive language models usually demonstrate superior performance in English57.

Additionally, paralinguistic elements of speech, such as filled pauses (e.g., “ehm”), were removed as they do not carry any semantic value. Furthermore, we removed all the references to medication intake or condition in the transcriptions. Utterances spoken by the examiner (i.e., prompts for elaboration following a 10 s pause) were removed from the transcriptions.

ON/OFF medication state classification using semantic search

Three approaches were used to classify the medication state from speech recordings. In Fig. 4a, the semantic search approach is presented using the patients’ transcriptions and the NFS items.

Fig. 4: Schematic representation of the three approaches for the ON/OFF medication state classification of speech recordings.
figure 4

The English transcription of each speech recording was the input for all the approaches. a The transcriptions (query) and the 20 items (corpus) of the Neuropsychiatric Fluctuation Scale (NFS) are first mapped into numerical vectors. Each recording transcription is treated as a query, and the five most similar NFS items are retrieved using cosine similarity. The query is then classified based on the most frequent label (ON or OFF) among the retrieved items. b The medication state is predicted by applying machine learning (ML) models on linguistic features represented as text embeddings. c English transcriptions are directly included in a decoder-only Large Language Model (LLM) prompt.

Full size image

A key strength of the NFS questionnaire is that its items were derived from various psychiatric questionnaires completed by a large cohort of PD patients with fluctuations, reflecting the most common neuropsychiatric symptoms of PD47. We used this strength to classify participants’ speech based on similarity with the NFS items. First, we performed text normalization; specifically, the speech transcriptions and NFS items were converted to lowercase, punctuation, and specific stop-words (pronouns, conjunctions, articles, and prepositions) were removed while negations were kept. Next, the normalized transcriptions and NFS items were converted into numerical vectors (i.e., text embeddings). Then, the English texts were mapped to a numerical representation using a Bag of n-grams (n = 1,2,3) model, implemented with the class CountVectorizer from the Python’s scikit-learn module (version 1.3.0)58. The CountVectorizer, together with the TfidfVectorizer were used as baseline methods. Next, more recent Language Models from Python’s “sentence-transformer” module59 and an open-source repository (HuggingFace Hub (https://huggingface.co/docs/hub/index)) were used to capture the meaning of the transcriptions. Then, the “multi-qa-mpnet-base-dot-v1” model from “sentence-transformer”, the “dunzhang/stella_en_1.5B_v5” and the “and “Alibaba-NLP/gte-Qwen2-1.5B-instruct” models from the HuggingFace Hub were used to map the text into dense vectors of size 768, 1024, and 1536 respectively. These embedding models were selected based on a tradeoff between GPU memory requirement and their semantic search/retrieval task performance (as reported on the HuggingFace leaderboard (https://huggingface.co/spaces/mteb/leaderboard) and in the “sentence-transformer” documentation (https://sbert.net/docs/sentence_transformer/pretrained_models.html).

Each transcription was used as a query, and the cosine similarity was calculated to measure its similarity to each NFS item embedding. For classification, we adopted a simple major voting rule, requiring an odd number of voters to avoid ties. In this approach, the voters corresponded to the NFS items most similar to the query. The optimal number of voters was determined empirically based on classification accuracy (Supplementary Fig. 1). The query was then classified as OFF or ON depending on whether most of the top five most similar NFS items were OFF or ON items. All the results obtained using different text embeddings are reported in Supplementary Table 1.

ON/OFF medication state classification using machine learning models

First, the speech recordings were mapped into text embeddings as described for semantic search. Given the high dimensionality of the numerical vectors generated from these models (minimum dimension of 768) and the limited data available, we used principal component Analysis (PCA) to the standardized embeddings and Truncated singular value decomposition for the CountVectorizer to reduce their dimensionality. By setting an explained variance threshold of 0.9, the dimensions of the vectors were reduced to 40 for the “multi-qa-mpnet-base-dot-v1” and for the “dunzhang/stella_en_1.5B_v5” and 45 for the “Alibaba-NLP/gte-Qwen2-1.5B-instruct” embeddings, respectively.

Binary Classification was performed by applying different ML models to the computed text embeddings, such as naïve Bayes (NB), support vector machines (SVM), and Random Forest (RF). The models’ hyperparameters were optimized using the Grid Search approach.

A comprehensive overview of the results obtained using the different text embeddings and machine learning models is reported in Supplementary Table 2.

This approach is graphically represented in Fig. 4b. This ML approach is similar to methods already explored in psychology and mental health to predict depression or dementia from Tweets or speech60,61.

ON/OFF medication state classification using LLMs

Finally, we explored the performance of decoder-only models in classifying different medication states (ON vs. OFF) directly using the transcriptions as input. Different prompts were tested before obtaining consistent and coherent results (see supplementary material section B). In order to instruct the LLMs, the ON and OFF conditions were explained using the NFS items. The final version of the prompt was the following: “You will be provided with transcription of speech from people with Parkinson’s Disease. Please assign the class ON or OFF to each text. Patients in the OFF condition may experience tiredness, lack of energy, inability to relax, trembling legs, and feel lethargic and low. In the ON condition, they usually feel full of energy, sure of themselves, talkative, and a sense of well-being. Please provide only the class estimate without any additional text.”

We selected different open-source LLMs based on a tradeoff between benchmark performance according to the HuggingFace Leaderboard and the computational power needed since all the computations were performed locally to ensure the privacy of sensitive data. The selected models are listed in Table 4. We tested these models in both zero-shot (i.e., prompts without any example transcripts included) and few-shot settings (i.e., prompts including some representative transcripts). In the few-shot prompting, in addition to the recording to classify, we included its most semantically similar transcriptions from the other patients, selected using the cosine similarity between the text embeddings calculated by the “multi-qa-mpnet-base-dot-v1” model32,62. The optimal number of examples for each LLM was empirically determined by assessing the model performance when different numbers of speech transcriptions were included in the input (Supplementary Fig. 2).

Table 4 The names and corresponding IDs in the Huggingface Hub of the three LLMs used in this study
Full size table

Moreover, to test the consistency of the predictions, we used multiple paraphrases of the final prompt. The final assigned class corresponded to the most frequent one in the various predictions33.

Chat template (https://huggingface.co/docs/transformers/main/en/chat_templating) was used for the prompts given to the Llama-3.1-Storm (8B) model. The prompts used for each model are reported in the Supplementary Material section B.

Using a similar approach described by Portillo Wightman et al.33, we computed the “ON” and “OFF” confidence scores by averaging the softmax probability scores of the first token (“either ON or OFF”) provided by the models Llama-3.1 (8B) and Gemma-2 (9B).

As a final experiment, we “asked” the LLMs to explain their outcomes by defining the most influential words in the provided transcription that drove their predictions. Therefore, we replaced the last sentence of the previous prompt, “Please provide only the class estimate without any additional text,” with “In addition to the class prediction, please identify and provide a list of the keywords or phrases that contributed most to the classification decision.”

Neuropsychiatric state score predictions using semantic search

As an initial approach to estimating the neuropsychiatric state score, we analyzed the similarities between patients’ speech transcriptions. Following the methodology described above for the semantic search, we used cosine similarity as a metric to quantify the similarity between a patient’s transcription (the query) and a corpus of transcriptions from other patients. To predict the score of a given query, we calculated a weighted average of the neuropsychiatric state scores of the top-K most similar transcriptions, using their similarity scores as weights. The results obtained using different text embeddings are reported in Supplementary Table 4.

Neuropsychiatric state score predictions using machine learning models

We performed regression analysis to predict the neuropsychiatric state score during the two medication states. Text was mapped into embeddings using the same approach described above for the semantic search. Different regression models were used, particularly RF, ridge regression, SVM light gradient boosting machine, and extreme gradient boosting (Supplementary Table 5). We also merged the regression predictions with ensemble learning, using the bagging approach (i.e., uniformly averaging the prediction from the two best base models applied to different text embeddings)63. Some models predicted a neuropsychiatric state score > 60; in these cases, the values were clipped to remain consistent with the maximum possible score.

Neuropsychiatric state score predictions using LLMs

As a third approach, we tested the capability of LLMs to perform a regression task when relevant in-context examples were provided64. For each LLM, we empirically investigated the optimal number of patients’ recordings (in-context examples) to include in the prompt based on the models’ performance (Supplementary Fig. 4). The K included in-context examples were the K-most similar patients’ recordings to the given query, selected using the same semantic approach explained in section above for the semantic search.

The neuropsychiatric state score was calculated as the average of nine model outputs obtained using different paraphrases of the prompt with the same in-context examples. More detailed information about the prompts used for this regression task is provided in the Supplementary Material (Section B).

Performance evaluation

The models described above were evaluated using leave-one-out cross-validation, where two recordings from the same unseen patient served as the hold-out set.

Classification performance was assessed using accuracy, precision, recall, F1-score, and receiver operating characteristic (ROC) curve. We also provide the confidence calibration analysis of the top-performing models in Supplementary Material (Section F). Including confidence scores around a model’s prediction is especially crucial in sensitive domains like healthcare. Indeed, it has been reported that LLMs can generate incorrect answers with high confidence65. Therefore, calibration analysis is thus needed to assess the trustworthiness of the predictions. A model generates a well-calibrated confidence score if its predicted probabilities for a class are aligned with the actual likelihood of being correct. Calibration was tested by computing the calibration plot, the Brier score, the maximum calibration error and the expected calibration error66,67.

Regression performance was evaluated by computing the root mean squared error (RMSE), mean absolute error (MAE), median, interquartile range of the absolute errors, Spearman Correlation and the coefficient of determination (R2).

All the experiments were performed on consumer hardware equipped with a 4090 NVIDIA RTX GPU and Python version 3.11. Python’s scikit-learn library was used to implement ML models and evaluation metrics.