Introduction

Diagnosis and assessment of health outcomes hold valuable impact on healthcare, medical research, and policy1. A perfect diagnostic test has the possibility to entirely differentiate people with and without the outcome of interest (i.e., the disease), without any misclassifications2. A perfect diagnostic test, a gold standard, does not exist in a real life setting and thus diagnostic tests could only partially differentiate people with and without disease2. The estimation of diagnostic accuracy of health outcome tests can be performed by estimating the diagnostic sensitivity, specificity, and positive and negative predictive values3. Estimating the diagnostic sensitivity and specificity requests a diagnostic standard, which serves as a reference4. In the case of imperfect tests, i.e. diagnostic sensitivity and/or specificity being lower than 100%, misclassifications will occur. However, for many outcomes, such diagnostic standards are lacking. This is particularly true in mental health and subjective outcome measures5. The lack of standard diagnostic or screening criteria makes it challenging to assess the validity of newly developed or existing tests6, leading typically to underestimations of their diagnostic accuracy as all discrepancies in test results between the new and the standard test will always be considered as misclassifications of the new test7. To address this issue, Bayesian Latent Class Modelling (BLCM) was proposed to assess the performance of diagnostic tests in the absence of a perfect gold standard8.

Following a BLCM approach, we can observe the combination of observed diagnostic test outcomes and accuracy and prior knowledge of the outcome prevalence to obtain a posterior knowledge of the accuracy of the diagnostic tests in the populations of interest9. In summary, prior information about the diagnostic sensitivity and specificity is linked with observed data (e.g. diagnostic test outcomes) to acquire a posterior distribution of the variable10. Hui and Walter11 presented the first Latent Class Analysis (LCA) model to estimate diagnostic test accuracy using two imperfect tests applied to two populations with different outcome prevalence. An important method that increased the implementation of Bayesian statistics, in the case of lack of a gold standard and use of imperfect tests, is the Gibbs sampling14. Gibbs sampling is a Markov Chain Monte Carlo algorithm which uses simulations to manage a complex integration process essential for a full Bayesian analysis. The first user-friendly Bayesian statistical software package using Gibbs sampling was “WinBUGS”. WinBUGS permitted the user to indicate the likelihood for the data and the distributions of the prior for each parameter in the model allowing to derive the posteriors. With the WinBUGS package, the Hui − Walter model could be estimated in a Bayesian framework with a user-friendly interface. Later, other statistical software packages based on this methodology (e.g., JAGS, OpenBUGS) were introduced8. A key assumption of the original Hui − Walter model is (I) that sensitivity and specificity are the same in all populations, and (II) that the diagnostic tests under assessment are independent and conditional on the true outcome status of a person11. However, these assumptions are rather difficult to validate in most cases. Soon after the Hui-Walter model was first described, the effect of conditional dependence between diagnostic tests on the evaluation of diagnostic accuracies, i.e. under- or overestimated was stated12. BLCM models that model conditional dependence between tests using covariance terms were later developed to soften Hui-Walter’s assumption of conditional independence13,14. These more complex BLCM models allow for more precise estimation of diagnostic test accuracies8.

In this paper we will use the example of two diagnostic tests to assess burnout as an illustration of the usefulness of BLCM application in the absence of a gold standard in mental health research. Burnout is an interesting example because despite extensive research, no consensus still exists regarding its measurement15, similar to other psychiatric and psychological phenomena. Currently, burnout is mainly evaluated through Patient-Reported Outcome Measures (PROMs). PROMs measure a patient’s health status or health-related quality of life at a single point in time, and are collected through short, self-completed questionnaires. Oldenburg Burnout Inventory (OLBI) is one of the widely used available PROMs to assess burnout16. Yet, to improve and facilitate the burnout measurement by health professionals, the Early Detection Tool of Burnout (EDTB) was recently developed as the first published hetero-assessment test17. Hetero-assessment instruments are filled out by one person (i.e., physician or clinical psychologist) about another (i.e., patient) and encompass evaluations of the individual’s work capacity, attitude, performance, and other characteristics18.

We have previously assessed the diagnostic accuracy of EDTB in Belgian and Swiss study samples, using OLBI as a comparator assumed to be a gold standard19,20. The Belgian and Swiss studies were independent, each having its own study sample, but used the same tests. However, these studies suffer from an important limitation due to the lack of a gold standard test for measuring burnout, which may introduce bias when comparing EDTB’s performance to an imperfect comparator (OLBI). In the present study, we used a completely different methodological approach (LCM) and analytical paradigm (Bayesian) that we applied to study sample comprising both Belgian and Swiss samples merged. This study aims to demonstrate the advantages of using BLCM by comparing the diagnostic accuracy of two burnout assessment tests without considering any of them as a reference test.

Results

The results of BLCM showed no difference between models with and without conditional dependencies (Supplementary Tables S1 & S2). The 95% credibility intervals of covariance terms to model conditional dependencies between diagnostic sensitivities and specificities included 0. Using the BLCM without conditional dependencies, the diagnostic sensitivity and specificity of EDTB were 0.91 and 0.82, respectively (Table 1). The diagnostic sensitivity and specificity of OLBI were 0.73 and 0.73, respectively (Table 1). The estimated prevalence of burnout was 52% in the Belgian study population and 82% in the Swiss study population. The effective sample size (SSeff > 1000) and Gelman-Rubin statistic (psrf < 1.05) validated the model fit (Table 1). By plotting the results, we observed that the two chains converged well for the diagnostic sensitivity and specificity of EDTB and OLBI (Supplementary Figures S1 & S2). We also observed that the autocorrelations decreased after a small number of iterations (Supplementary Figures S1 & S2). The estimated Positive Predictive Value (PPV) in the Belgian population was 84.5% and the estimated Negative Predictive Value (NPV) was 89%. The estimated PPV in the Swiss population was 95.8% and the estimated NPV was 66%.

Table 1 Results of the bayesian latent class modelling using minimally informative priors and without conditional dependency.
Full size table

The sensitivity analysis did not yield changes in the diagnostic sensitivity and specificity of EDTB or OLBI (Supplementary material S3-S10). In all models, the diagnostic sensitivity was never below 0.82 and the diagnostic specificity was never below 0.78. Yet, in some models (Supplementary material Table S5), the credibility intervals for the diagnostic specificity of EDTB became wider. The lowest estimated diagnostic sensitivity of OLBI was 0.69 whereas 0.67 was the lowest estimated diagnostic specificity of OLBI.

Discussion

We contrasted the results obtained through the BLCM approach used in this study with those from the conventional approach applied in the earlier studies using the classical gold standard approach18,19,20. Thus, this study aimed to illustrate the application of BLCM in diagnostic mental health research when a gold standard is lacking. For this, we used burnout measurement as example and assessed the diagnostic performance of two burnout measures: EDTB and OLBI independently. The findings of the present study indicate that BLCM is a relevant and feasible approach that helps evaluate the diagnostic sensitivity and specificity of tests particularly in the absence of gold standards. The model performance was sufficient, and the two chains converged well, and the Gelman-Rubin statistics indicated convergence which indicate the reliability of the results despite the small sample size. Based on this study results, the diagnostic sensitivity and specificity of the EDTB are better compared to the estimates reported in previous studies that used the classical approach assuming that the reference test is a gold standard, i.e., being 100% sensitive and specific. In the Swiss study20, the reported diagnostic sensitivity and specificity of the EDTB were 0.88 and 0.29, respectively. In the Belgian study19, these estimates were 0.76, and 0.60, respectively. Regarding the diagnostic accuracy of OLBI, the estimated diagnostic sensitivity and specificity in this study was slightly higher compared to the Belgian study (0.70, 0.67)19, whereas the Swiss study did not report these values precluding the comparison20. The estimated diagnostic sensitivity and specificity of the EDTB and OLBI using BLCM encourage their use in the assessment of diagnostic accuracy of health outcome measures for which there is no diagnostic standard.

This study provided a proof of concept of the use of BLCM in the absence of a gold standard, which is often the case for mental health outcomes. For example, the quality of life outcome assessment is difficult because this outcome is a latent variable which is not directly observable and there is no gold standard for its assessment21. The example of burnout assessment presented in this paper is especially sound because this phenomenon has significant consequences both at the individual and organizational levels22. The standardization of burnout detection is imperative to support high-quality research that facilitates evidence-based interventions to protect workers and reduce this phenomenon23. The importance of utilizing standardized methods in assessment is widely recognized as the foundation of evidence-based practice24. However, the current assessment of burnout is mainly reliant on self-reported instruments16 and therefore there is a need for a structured interview instrument (i.e., EDTB) for the prognostic and interventional activity and research on burnout. Hence, a re-assessment of the diagnostic accuracy of EDTB is still essential with larger representative samples.

The study aimed to demonstrate the feasibility and relevance of BLCM in burnout research by utilizing data from two previous studies with small study samples. This constitutes a limitation of the study, as it may impact the precision of the results. The assumption that the diagnostic sensitivity and specificity of the two burnout assessment measures were comparable across the Swiss and Belgian populations could also be considered as a limitation. However, considering the similarities between the Swiss and Belgian populations, we may assume that the diagnostic sensitivity and specificity of EDTB and OLBI were comparable. This study serves as a proof of concept and focuses on replacing the practice of gold standard assumption when determining the accuracy of a new test. Nevertheless, performing a thorough quality and validity assessment of the tests remains crucial although it is beyond the scope of this paper.

The question of whether burnout should be considered a continuous or dichotomous variable still remains open25. In our data set, we used dichotomous test results, but BLCM approaches modeling continuous test results are also available26. Schaufeli et al., already discussed the question of continuous versus dichotomous results in 2008 “Medical practitioners favor dichotomous diagnoses, especially when informing decisions on treatment or disability insurance claims. In this way the definition of burnout is shaped by practical questions– Who is to be treated? Who is to receive financial compensation?”27. Schaufeli et al., described the dichotomization of burnout as an expansion from a psychological phenomenon to a medical diagnosis at least in some European countries such as The Netherlands and Sweden and using statistical or clinically validated cut-offs27. More recently, a study by Guseva Canu et al. showed that 14 out of 34 European and neighboring countries including Turkey, Denmark, Lithuania, to cite just a few, recognize and compensate burnout as an occupational disease28. Finally, the most recently developed burnout measurement test (Burnout Assessment Tool) provided clinical cut-off values to facilitate dichotomization and case identification29. The debate on dichotomization of an evolving condition is not unique to burnout but shared by several psychiatric and psychological phenomena in mental health research e.g., psychosis29.

It is important to highlight the underlying assumption of the BLCM that a positive (“+”) or negative (“−“) result holds the same interpretation for both the OLBI and the EDTB, an assumption that remains open to debate. Strictly speaking, however, this assumption is also made in the classical approach in diagnostic test evaluation, when a new test is evaluated against an assumed perfect reference test. In contrast to this, BLCM is a more realistic approach, allowing the reference test to be imperfect. Moreover, the cut-offs we used to dichotomize burnout measured using OLBI could be considered arbitrary and future research should address this issue either by validating clinical cut-offs or using continuous scales in BLCM.

This study provided a practical example of the feasibility and relevance of BLCM for diagnostic research. To our knowledge, this is the first study conducted using BLCM to assess the diagnostic sensitivity and specificity of two burnout measures. The findings of this study encourage the use of BLCM in future, methodologically sound studies and for other outcomes particularly subjectively measured ones such as: quality of life, and wellbeing.

The EDTB’s diagnostic sensitivity and specificity re-assessed using BLCM are better and less biased compared to the previous studies conducted using the classical gold standard approach. This study demonstrated the practical feasibility and scientific relevance of BLCM for diagnostic research especially for subjectively measured outcomes for which there are no standard diagnostic criteria, such as burnout.

Methods

Study populations

The present study utilized data from two previous studies19,20. The population 1 consisted of 42 Swiss patients (mean age 46 years, 74% women), who completed the OLBI before their “Work and Suffering” consultation at Unisanté between 2010 and 2013 and for whom the EDTB was completed retrospectively by the occupational physicians in charge of the consultation, based on their clinical reports. The population 2 comprised 123 Belgian patients (mean age 44 years, 50% women), who consulted an occupational physician (n = 100) or a general practitioner (n = 23) in 2018. The physicians had to complete the EDTB online, either during or right after the consultation. In addition to this clinical judgment tool, physicians asked each patient to complete a paper version of the OLBI after the consultation. The results of the Belgian study showed that occupational physicians had better estimates of diagnostic sensitivities and specificities compared to general practitioners or both types of physicians. As a result, we used data from 100 Belgian and 42 Swiss patients who had a consultation with an occupational physician. Both the Belgian and Swiss studies utilized convenience sampling as the recruitment method19,20.

Occupational burnout tests

The EDTB: In 2020, the Belgian Federal Public Service Employment, Labour and Social Dialogue (BFPSELSD) introduced the first tool for early detection of occupational burnout (EDTB) to hetero-assess burnout by health professionals17. Two recent studies19,20 have examined the diagnostic accuracy of the EDTB by comparing it to a valid self-reported burnout test, the Oldenburg Burnout Inventory (OLBI)30. The EDTB is a concise three-page document that encompasses various topics17. These include general complaints such as sleep disturbances, stress, workload, and workplace conflicts; symptoms of burnout across physical, cognitive, emotional, and behavioral domains; work-related factors like the origins of complaints, potential risk elements, job demands, and available job resources; sociodemographic details; data on absenteeism; specific diagnoses like burnout, anxiety, and depression; and any additional remarks. Health professionals have the option to select one or multiple items under each category based on the worker’s feedback. Ultimately, the health professional’s conclusion, specifically whether burnout is present or absent, serves as the primary piece of information17.

The EDTB was validated in French and Dutch languages19.

OLBI consists of two dimensions30: exhaustion (8 items) and disengagement (8 items). The items of OLBI are scored on a Likert scale where responses range from 1 to 4, typically with “Strongly agree” at one end and “Strongly disagree” at the other end. Example of items are: “I always find new and interesting aspects in my work” and “During my work, I often feel emotionally drained”. The sum of the responses is calculated, given that each item has four possible answers (1, 2, 3 or 4), we obtain a score ranging from 16 to 64. To obtain a dichotomized status regarding burnout, the Swiss and Belgian studies used a cut-off of 44, which means that scores higher than 44 represent high-severity burnout. This cut-off is based on the findings from the previous studies, because OLBI does not classify burnout in a dichotomous outcome but instead provides a global score ranging from 16 to 64. In our previous analysis20, we evaluated two cut-offs values of OLBI and decided to use the cut-off of 44 because it was more reliable according to the Power et al. criteria31. The validity and reliability of OLBI was reported to be sufficient in a recent systematic review16. There are Dutch and French validated versions of OLBI which were used in this study32.

There was a fair overall agreement between OLBI and EDTB, with a significant but fair to moderate kappa (κ = 0.36, 95%CI = 0.20, 0.54)19.

Analysis

We applied BLCM using the Hui-Walter framework11 with Markov Chain Monte Carlo (MCMC) simulation to two tests and two populations. We used minimally informative priors in models, with or without conditional dependencies between diagnostic sensitivities of OLBI and EDTB or specificities of OLBI and EDTB. The model code (Supplementary material) was obtained with the function “auto hui-walter” of the runjags package33 using the freely available software R (R Core Team, 2024 Version-4.4.1, https://www.r-project.org/). We set the burn-in to 5000 and the sample to 10,000 iterations for two independent chains. To assess convergence, we plotted the trace plots and considered the potential scale reduction factor (< 1.05). Model selection, i.e., in- or exclusion of conditional dependencies was based on the 95% credibility intervals (including 0 or not). The prevalence we measure in this study is the prevalence in our study population and not in the general population. Indeed, out target population are workers exposed to job stress and therefore at risk of burnout. Our source population is the workers who came to the consultation either at Unisanté center of occupational medicine in Switzerland or during general practitioners and occupational physician’s consultations for work-related health problems in Belgium and who completed the OLBI test of burnout. We computed Positive Predictive Value (PPV) and Negative Predictive Value (NPV) for Belgian and Swiss populations. PPV was computed for a test (i) in a population (k) using the formula: Pk​Seik​​/(Pk​Seik​+(1 − Pk​)(1 − Spik​)). NPV was calculated using the formula: ​(1 − Pk​)Spik/(Pk​(1 − Seik​)+(1 − Pk​)Spik).

We further performed sensitivity analysis by replacing one of the minimally informative priors with the distribution1,2 at each time for the six priors. These six priors were: diagnostic sensitivity and specificity of EDTB, diagnostic sensitivity and specificity of OLBI and prevalence of burnout in each study population (Swiss and Belgian samples). Moreover, we performed the analysis using informative priors for OLBI based on the data from the literature34,35.

We reported the results of this study following the Standards for the Reporting of Diagnostic accuracy studies that use Bayesian Latent Class Models (STARD-BLCM) guidelines36,37. All codes and analyses are available online from: DOI:https://doi.org/10.16909/DATASET/35.