Introduction

Staphylococci, namely, Staphylococcus aureus and coagulase-negative staphylococci (CoNS), play a major role in bloodstream infections, and S. aureus bacteremia (SAB) is associated with a high mortality risk and frequent recurrence1. The management of SAB relies on adherence to treatment recommendations, including the judicious selection of antibiotics2,3,4,5,6. Conversely, CoNS bloodstream infections pose a clinical challenge, as many episodes involve mere contaminants, leading to unnecessary antibiotic use, subsequent adverse events, antibiotic resistance, and increased costs7,8. Optimal management for staphylococcal bloodstream infections must meet two essential criteria: adherence to recommended treatment principles and prompt administration. Ideally, the immediate involvement of an infectious disease specialist (IDS) ensures prompt and optimal treatment for every patient with a staphylococcal bloodstream infection5,9,10,11. However, these specialists are scarce in many parts of the world. One solution to mitigate this scarcity is the adoption of algorithm-based therapy, as explored by Holland et al.12. Their study demonstrated that algorithm-determined antibiotic treatment resulted in a noninferior clinical outcome compared to the established standard of care (SOC) but did not improve adverse events. Unsolicited IDS telephone consultations for nonacademic hospitals were suggested as a middle ground between comprehensive IDS service and pure algorithmic support. However, this approach did not lead to a reduction in 30-day all-cause mortality compared to SOC without consultation among SAB patients in a recent clinical trial13, which was attributed to the time delay between the onset of bacteremia and the IDS consultation. Based on these findings, we propose that combining a CDSS with IDS consultation could overcome the limitations of each approach. The CDSS could guide physicians until an IDS steps in by bridging the gap between infection detection and specialized consultation. In the HELP trial (Hospital-wide Electronic computerized decision support system to improve outcomes of Patients), we investigated the use of a CDSS to assist physicians in implementing best practices for patients with staphylococcal bacteremia. The “HELP CDSS” aims to optimize IDS allocation and patient outcomes while also serving as a use case for the newly established Data Integration Centers (DIC)14 within the Medical Informatics Initiative Germany15. This initiative seeks to enable the integration of diverse clinical routine data for the enhancement of patient care and scientific applications. Given the logistical constraints of having to implement the CDSS across entire wards, the HELP trial was designed as a stepped-wedge cluster randomized trial.

Results

Patient and ward flow throughout the trial

In the five DIC, 5056 patients with Staphylococcus bacteremia were identified across 126 out of the 134 initially randomized wards. No patients were found in the remaining 8 wards during the study period. Among the identified patients, 812 were diagnosed with SAB, and 4244 were diagnosed with CoNS (Fig. 1). The baseline characteristics of the patients are detailed in Table 1, and SAB- and CoNS-specific data are provided in Supplementary Tables 2 and 3.

Fig. 1: Recruitment CONSORT flowchart.
figure 1

Due to the stepped-wedge design, each ward took part in the SOC phase at the beginning of the trial and switched to the CDSS phase at a randomized crossover point in time, thus contributing patients from both phases to the analysis. Numbers of patients are displayed for each type of bacteremia separately.

Full size image
Table 1 Total sample characteristics
Full size table

Patient-related outcomes

Overall hospital mortality was 27% (1365 of 5056 patients), with 26% (797 of 3026 patients) during the SOC phase and 28% (568 of 2030 patients) during the CDSS phase. Among the SAB patients, the 90-day mortality was 32% (262 of 812 patients), with 31% (138 of 448 patients) and 34% (124 of 364 patients) during the SOC and CDSS phases, respectively. The 90-day relapse rate for SAB patients was 7% (54 of 812 patients), with 7% (33 of 448 patients) during the SOC phase and 6% (21 of 364 patients) during the CDSS phase. Figure 2 displays the GLMM-estimated probability differences of all outcomes and their confidence intervals.

Fig. 2: Differences in endpoint probability between the SOC and CDSS phases.
figure 2

The gray boxes show the area of noninferiority (=5% of the SOC phase probability). Overlapping CIs indicate that noninferiority could not be established, while the CIs left of the gray box indicate noninferiority. Confidence intervals: 90% for noninferiority hypotheses and 95% for superiority hypotheses.

Full size image

The primary endpoint analysis revealed a hospital mortality of 28% in the SOC phase compared to 25% in the CDSS phase, indicating noninferiority of the CDSS phase (difference −3%; 90% CI −6% to 0%), given that +5% was not covered by the CI. With respect to the SAB patients, we next tested the subordinate primary endpoint of 90-day mortality or relapse, which was 46% in the SOC phase and 43% in the CDSS phase (difference -0.03; 90% CI −0.11 to 0.06), indicating that noninferiority was not met. Following the confirmatory test strategy, the subordinate primary endpoint limited to CoNS patients is reported only descriptively. Among patients with CoNS, 14% (351 of 2578 patients) were identified as having true CoNS infections in the SOC phase, as opposed to 17% (283 of 1666 patients) in the CDSS phase. We expected lower vancomycin use in the CDSS phase but found similar proportions in both phases: 26% in the SOC phase versus 24% in the CDSS phase (difference −2%; 95% CI −5% to 1%; p = 0.29). Additionally, the cumulative vancomycin dosage per patient did not significantly differ between phases: 10,298 mg in the SOC phase versus 12,251 mg in the CDSS phase (difference 1953 mg; 95% CI −120 mg to 4026 mg; p = 0.06). An overview of all the estimates is shown in Table 2. Figure 2 also summarizes the prespecified secondary endpoints. Although all three point estimates favored the CDSS, none of the estimates showed a significant effect. Sensitivity analyses, including COVID-19 diagnosis and site effects, yielded similar estimates, except for hospital mortality, where including site as a fixed effect changed the estimated difference in mortality (difference −4%; 90% CI −7% to −1%; p = 0.03). To investigate potential differences in hospital mortality between bacteremia types we conducted an exploratory post-hoc stratified analysis extending beyond our pre-registered plan. For SAB, crude mortality rates were 40% (129 out of 319 patients) in the SOC phase vs. 51% (123 out of 241 patients) in the CDSS phase. For CoNS patients, crude mortality rates were 35% (668 out of 1910 patients) in the SOC phase vs. 36% (445 out of 1221 patients) in the CDSS phase. Table 3 displays the results of the corresponding exploratory GLMM analysis.

Table 2 Model estimates for primary and secondary endpoints
Full size table
Table 3 Exploratory model estimates for hospital mortality stratified by type of bacteremia
Full size table

Additional site effect information can be found in Supplementary Table 5 and the crude mortality/relapse by site is available in Supplementary Tables 6 and 7.

CDSS usage

Forty physicians from all five hospitals participated in a poststudy survey to assess CDSS usage and satisfaction. Only 19 physicians used the CDSS, with 16 using it in just 25% of the bacteremia cases they encountered. The survey, which was conducted six months after this study, might have missed some CDSS users due to high staff turnover rates. Despite the low usage rate of the CDSS, Fig. 3 shows that the system had consistently high usability ratings, suggesting that the issue was not CDSS usability. Instead, free-text responses pointed to the limited accessibility of the CDSS due to suboptimal clinical IT service integration. Nevertheless, Table 4 (first line) indicates a 14% reduction in IDC during the CDSS phase versus the SOC phase, while most infection management parameters remained consistent, suggesting that the CDSS could uphold treatment adherence in these patients. This finding aligns with the clinicians’ subjective assessment (Fig. 4), which showed high compliance with the CDSS recommendations.

Fig. 3: Usability rating of the HELP CDSS.
figure 3

Physicians rated 10 statements on the CDSS’s usability, using a scale from “strongly agree” to “strongly disagree.” Each bar shows the percentage of physicians who agreed with each category for each statement.

Full size image
Table 4 Management of SAB infections
Full size table
Fig. 4: Impact of the CDSS on clinical decisions and patient safety.
figure 4

Physicians rated 10 statements on their clinical decisions and patient safety, using a scale from “strongly agree” to “strongly disagree.” Each bar shows the percentage of physicians who agreed with each category for each statement.

Full size image

Discussion

The HELP trial evaluated the safety and benefits of CDSS support for treating Staphylococcus bacteremia. The CDSS phase was not inferior to the SOC phase in terms of hospital mortality. However, noninferiority for 90-day mortality/relapse in SAB patients could not be confirmed, and vancomycin use was not significantly reduced in CoNS patients. These results should be interpreted with caution, as the proportion of patients for whom the system was actually used can only be estimated. Originally, it was planned to monitor CDSS usage digitally, however this was not possible due to resistance from staff councils. Thus, we can only rely on the data from the poststudy survey, which suggest low usage. This can be attributed to various factors. First, the low self-reported usage of the CDSS may have stemmed from challenges in accessing the system, which is a common obstacle that has been identified in prior studies of CDSSs16,17. In anticipation of this challenge, we planned to integrate a HELP CDSS access point into all microbiology reports featuring Staphylococcus bacteremia, thus enabling immediate app launches with relevant patient data. However, this strategy was unfeasible due to IT security concerns. Second, all participating hospitals provided high-quality infectious disease management as part of the SOC, with dedicated infectious disease services that proactively approached relevant patients. Thus, matching and improving the already high SOC was difficult to achieve. The value of a CDSS might be greater in smaller hospitals without in-house IDSs, where it could bridge the gap between infection detection and telephone IDS consultation, as shown in the study by Weis et al.13. Although IDS consultation was theoretically mandatory for all SAB patients as part of the center’s standard of care (SOC), our data indicate that not all SAB patients actually received a consultation. This discrepancy can likely be attributed to two factors. First, some patients may have died before the consultation could occur. Second, the study was conducted during the COVID-19 pandemic, which has been reported to negatively impact the quality of care, particularly for SAB patients, as IDS were heavily engaged in managing COVID-19 cases18. Third, the COVID-19 pandemic caused major disruptions and limited the attention given to and thus usage of the HELP CDSS. Due to these challenges, the effect estimates of our primary analysis should be interpreted with caution. Nevertheless, the HELP trial demonstrated the potential of a CDSS to support Staphylococcus bacteremia treatment. Even in its technically simple form, physicians found the CDSS useful, suggesting a general appreciation for algorithmic decision support. The straightforward presentation of guideline information as an interactive decision tree may even be an advantage over more sophisticated yet less explainable artificial intelligence applications. In their recent editorial, Khera et al.19. caution against the hasty adoption of artificial intelligence-driven CDSSs, pointing out the potential risks of automation bias that could adversely affect doctors’ clinical decisions instead of enhancing them. The use of simpler and consequently more transparent systems, such as the HELP CDSS, may offer safer alternatives while still providing valuable assistance. This study has several limitations. First, the data collection process relied on post hoc identification of patients with Staphylococcus bacteremia through the integration of EHR data collected in the DIC. Because DIC infrastructures were still under development during the HELP trial, it is possible, albeit unlikely, that we missed patients in our analysis. Second, this study was conducted in academic hospitals with a high possibility of carry-over effects from physicians switching between wards in different phases. Finally, due to restrictions set by staff councils and the hospitals’ IT departments, we could not directly monitor CDSS usage for individual patients and therefore could not distinguish between intention-to-treat and per-protocol analyses. Overall, this stepped-wedge cluster randomized trial revealed that CDSS support for the treatment of Staphylococcus bacteremia was noninferior to SOC in terms of hospital mortality but did not significantly reduce vancomycin use. With its simple and MDR-compliant design, the CDSS received positive ratings from surveyed physicians, suggesting that it may improve patient outcomes if barriers to its use can be removed. Furthermore, the HELP study employed a hybrid data collection method that integrated the documentation of a traditional randomized controlled trial into clinical practice, contributing to the development of a learning healthcare system.

Methods

HELP computerized computerized decision support system

The HELP CDSS assigns patients to one of two arms: one for CoNS bacteremia (including Staphylococcus intermedius) or one for S. aureus bacteremia (including Staphylococcus lugdunensis). In the CoNS arm, the CDSS checks the number of independent blood cultures. If only one positive blood culture is available without evidence of catheter infection by the same pathogen (clinical judgment by the physician in charge), the CDSS recommends a follow-up blood culture to rule out contamination before starting antibiotics. If two separate blood cultures yield positive results for the same CoNS species with matching antibiotic susceptibility profiles, the CDSS suggests investigating a possible source of infection and considering antibiotic therapy. Blood cultures were considered separate if they were collected from different sites and at different times (2 h to 5 days). A negative follow-up blood culture suggested contamination, prompting a reevaluation of antibiotic therapy if it was already initiated.

The HELP CDSS for S. aureus bacteremia is based on treatment recommendations published by Hagel et al.20. It was developed by a panel of IDS and medical microbiologists in collaboration with medical informaticians from all study sites. A simplified schematic representation is shown in Fig. 5.

Fig. 5: Schematic representation of the HELP CDSS.
figure 5

Rough overview of the decision algorithm that was the basis for the HELP CDSS.

Full size image

For SAB, the CDSS distinguishes between uncomplicated and complicated cases based on criteria such as positive follow-up blood cultures, permanent foreign bodies, prolonged fever, or vasopressor use. Treatment of complex SAB exceeds the capabilities of the CDSS, so immediate IDS consultation is advised. This also applies to polymicrobial bacteremia when more than one pathogen is detected in addition to staphylococci. For uncomplicated SAB treatment, guidance is offered, including source control, diagnostic imaging, and appropriate antibiotics.

Except for complicated SAB episodes, the CDSS can recommend treatment options independently of an IDS consultation. However, all hospitals maintained an IDS consultation service as part of their SOC, which remained operational throughout the treatment phase. Thus, the main difference between the two phases was that the CDSS provided supplementary information and acted as a bridge when immediate IDS consultation was not available. The CDSS was originally designed for seamless integration with hospital clinical information systems, but the development of this system faced challenges due to the European Medical Device Regulation (MDR)21, which was introduced in May 2021 and classified the original CDSS as a medical device requiring a highly complex and time-consuming certification process that was not compatible with the schedule. To ensure viability within our trial setting, the CDSS was transformed into a user input-reliant version that avoids automated patient-specific conclusions, functioning as an interactive decision tree accessible on smartphones, desktops, or laptops; an archived version of the CDSS is available online22.

Study design

The HELP trial was designed as a stepped-wedge cluster randomized trial (SW-CRT)23 conducted at five German university hospitals: Jena, Leipzig, Aachen, Halle, and Essen. Following the methods outlined by Hussey and Hughes24, we calculated a sample size of 135 wards enrolling at least 2700 patients based on a 90-day mortality of 30% among SAB patients, as reported by Mejer et al.25. For details, see the published trial protocol26. However, only 134 wards met the inclusion criteria described later and were included at randomization. All 134 wards started in the control phase (SOC) and transitioned to the treatment phase (HELP CDSS application) in a stepwise manner (Supplementary Fig. 1). The timing of crossover was determined by randomization, which was stratified by hospital and ward type (critical care units vs. general wards). Randomization lists generated using R27 were integrated into the technical infrastructure deploying the CDSS. This ensured that only physicians in the intervention wards could access the CDSS. There were nine randomization steps, with 2-month intervals for Jena, Leipzig, and Aachen and 1.5-month intervals for Halle and Essen due to legal difficulties in the pilot phase that necessitated a later start and thus a shorter overall data collection period. The trial adhered to Good Clinical Practice (where applicable) and the Declaration of Helsinki28, with approval from Jena University’s Ethics Committee (2018-1264_3-BO) and the respective center-specific ethics committees (Aachen University Ethics Committee, Halle University Ethics Committee, Essen University Ethics Committee, Leipzig University Ethics Committee.) The trial was registered at the German clinical trials register (www.drks.de/DRKS00014320) on June 6th 2019.

Recruitment of wards and patients and CDSS rollout

We included all sufficiently technically equipped wards at each site, excluding maternity, psychiatry, and pediatric units. Physicians in intervention wards were informed about the impending CDSS rollout and received microbiology reports containing references to the CDSS, while wards in the control phase did not have CDSS access. We included all adult patients with a positive blood culture for S. aureus/S. lugdunensis or CoNS, except for CoNS patients who passed away within 72 h of the initial positive blood culture. Patients were included as new patients if they were discharged and then readmitted to the hospital or met the inclusion criteria more than 30 days after their initial inclusion. Study nurses monitored blood cultures and completed an electronic case report form (eCRF) for each HELP patient. Routine documentation collection did not require informed consent, except for SAB patients, who were scheduled for a 90-day telephone follow-up, with prior notification and the option to decline participation.

Outcomes

The coprimary outcomes were hospital mortality for all patients, 90-day mortality/relapse for SAB patients and cumulative vancomycin use in milligrams for CoNS patients. Relapse was defined as the recurrence of S. aureus bacteremia or the occurrence of any related secondary complication within 90 days of the onset of initial SAB. We chose a pragmatic definition of relapse because without genotyping, it is difficult to ascertain whether a second infection is due to the same strain or infectious focus (i.e., relapse) or whether it is a reinfection with a different strain. Most S. aureus reinfections within 90 days are due to the same strain, suggesting the relapse of endogenous infection29,30. Secondary outcomes included the occurrence of acute renal dysfunction according to the Kidney Disease: Improving Global Outcomes (KDIGO)31 criteria, transesophageal echocardiography (TEE) usage and the use of seven additional antibiotics (Supplementary Table 1).

Data collection

We implemented hybrid data collection, combining documentation in eCRFs with electronic health record (EHR) data collected in the five participating DIC. To enable secondary data use, DIC integrate EHR data from different clinical information systems and transform them into Fast Healthcare Interoperability Resources (HL7® FHIR®). This approach allowed the HELP trial to blend the characteristics of traditional randomized controlled trials using data from eCRFs with the practicality of being entirely embedded in the clinical routine. While this setup places restrictions on data availability and quality, it is much closer to a real-world clinical scenario. In addition, this approach has the potential to reduce the documentation burden associated with traditional trials, offering a more pragmatic study design within a learning healthcare system.

To identify patients with positive Staphylococcus blood cultures from participating wards in the DIC data, we sourced admission, discharge, and transfer (ADT) data and integrated them with data from the laboratory information system (LIS). Subsequently, all other data needed for downstream statistical analyses were extracted. To protect patient privacy, we used a distributed analysis approach in which R scripts using the fhircrackr32 package were sent to the DIC for preprocessing and local data aggregation, followed by central statistical analysis in Jena. Figure 6 shows the extraction process in more detail: we provided R code via GitLab to each of the five Data Integration Centers (DIC), tasked with extracting FHIR data from the FHIR server and converting it into tabular format. To ensure patient privacy, all identifying data were removed, and information relevant to clinical descriptions was aggregated. These anonymized data, depicted in the upper part of the figure, were subsequently sent to Jena. The creation of individual R scripts for each site necessitated extensive collaboration between the data analyst and the DIC employees, as outlined in the lower part of the figure. This process included verifying the executability of R scripts in the local IT environment, ensuring the availability and plausibility of FHIR data, and mapping relevant microbiology reports from the Laboratory Information System (LIS) to the associated electronic Case Report Form (eCRF) and admission, discharge, and transfer (ADT) data. The patient cohort was defined by selecting individuals with positive Staphylococcus blood cultures who were admitted to a HELP study ward when the first preliminary microbiology report was issued. Subsequently, data for this patient cohort were extracted from the FHIR server, checked for plausibility, and transmitted to Jena. Each of these steps required multiple iterations to ensure accuracy and functionality. After the study, we conducted an online survey incorporating the “System Usability Scale“33 among participating physicians to assess HELP CDSS usage and user satisfaction (see the Supplementary Materials).

Fig. 6: Data extraction process.
figure 6

Starting with FHIR data available in each DIC, each step of the data extraction required iterative adaptions until the local analysis result was ready to be sent to the central analysis site.

Full size image

Statistical analysis

The coprimary outcomes were tested in a hierarchical, confirmatory fashion, meaning that each hypothesis test could only be interpreted as confirmatory if the null hypothesis for the preceding endpoint was rejected. For mortality outcomes, we used a noninferiority hypothesis with a 5% margin, indicating that our expectation was not to exceed a 5% increase in mortality in the CDSS vs. SOC phase. We employed binomial generalized linear mixed models (GLMMs) to estimate death/relapse probabilities and two-sided 90% confidence intervals (CIs). Regarding vancomycin use, our superiority hypothesis anticipated lower antibiotic usage in the CDSS phase. Extending the protocol descriptions, we used a two-part regression model. This included a binomial GLMM for vancomycin administration probability and a lognormal GLMM for cumulative dosage per patient. Analysis of the secondary outcomes followed the same modeling approach. In all the models, “treatment” (HELPS-CDSS vs. SOC) and “time since study initiation” were included as fixed effects; “ward” (cluster) was included as a random effect. To address potential site effects, we performed sensitivity analyses by adding “site” as a fixed effect in all the models and “coronavirus disease (COVID-19) diagnosis” as a fixed effect in all the mortality outcome models. Complying with privacy protection rules, we aggregated the variables used for the description of additional patient characteristics locally at each site and combined them using the meta-analytic random effects inverse variance model. All the statistical analyses were carried out with R version 4.2.1 using the R packages lme434, marginaleffects35 and meta36.