Introduction

Late-onset sepsis is a life-threatening condition that occurs in up to 15% of premature infants with very low birth weight (VBLW, <1500 g) and is characterized by a systemic inflammatory response to infection that often includes changes in heart rate and oxygenation patterns.1,2,3 Early detection of sepsis can allow for early initiation of antibiotics and other interventions to prevent disease progression.4 We previously developed and externally validated the Pulse Oximetry Warning Score (POWS), a predictive model that detects abnormal patterns in continuous HR and SpO2 data during the early phase of sepsis. In this prior work, we trained the POWS model to predict the risk of blood culture-positive late-onset sepsis in the next 24 h.5 We then evaluated the performance of the model to predict sepsis across three NICUs. POWS performed well, having good calibration and with area under the receiver operator characteristic curve (AUC) ranging from 0.782 to 0.827 across sites.

While these AUC’s indicate good discrimination between time with and without confirmed sepsis, rare events inevitably lead to some false positive predictions. In the case of late-onset sepsis, the average risk in any 24 h period is 0.25%. The systemic inflammatory response to sepsis causes autonomic dysfunction leading to the cardiorespiratory instability and altered vital signs detected by POWS,6,7 but this pathophysiology is not specific to bloodstream infection. Blood culture-negative infections and respiratory diseases not related to infection can cause similar physiological alterations due to systemic inflammation.8 Thus, while POWS was developed and validated for predicting late-onset sepsis with bacteremia, we hypothesized that acute rises in POWS would also correspond with events of cardiorespiratory deterioration due to conditions other than confirmed sepsis.6,8,9

In this study, we sought to understand the clinical context around the time of a high POWS score. We used a two-NICU retrospective cohort to quantify the occurrence of documented clinical events near an acute rise or “spike” in POWS over the prior baseline in preterm VLBW infants as compared to times without a rise in POWS. Our goal was to understand the clinical associations of a significant rise in the predicted risk.

Methods

Patient population and study design

We conducted a retrospective cohort analysis using data from two-level IV academic NICUs (University of Virginia Children’s Hospital and Washington University/St. Louis Children’s Hospital). The study was approved at each institution under a waiver of consent by their respective IRBs. At each center, heart rate (HR) and oxygen saturation (SpO2) are recorded every one or two seconds from the bedside monitor and archived from NICU admission through discharge. In the current study, we excluded VLBW infants with major congenital or chromosomal anomalies, as many have abnormal cardiorespiratory patterns at baseline. All remaining VLBW infants for whom archived HR and SpO2 data were available to calculate POWS were included. Missing HR and SpO2 data occurred at random for various technical reasons. Infants who died within 3 days of a POWS spike were excluded.

POWS calculation and spikes

We calculated model input features (mean, standard deviation, skewness, kurtosis, and cross-correlation of HR and SpO2) every 10 min from the raw monitoring data. We used the calculated features to obtain hourly POWS risk predictions from 3 days of age to hospital discharge for all infants where data were available.5 We excluded the first 3 days of data to eliminate high POWS scores related to early-onset sepsis or transitional physiology. The POWS value represents the hourly fold-increased risk of late-onset sepsis diagnosis in the next 24 h compared to the baseline sepsis risk for all VLBW infants in our cohort at all times. For example, a POWS value of 3 indicates an infant has a 3-fold increased risk of imminent sepsis diagnosis compared to the baseline risk at any time for a VLBW infant. This POWS value is based on the model detecting abnormal HR and SpO2patterns.

We evaluated a range of high POWS thresholds to define a spike by analyzing the resulting frequency and sensitivity for blood culture-positive sepsis. We selected a high threshold at a POWS of 6, A POWS threshold of 6 represents only 1.9% of observed scores and is associated with a 58% sensitivity for culture-positive sepsis. A lower threshold would have higher sensitivity, but we focused on an extremely high-risk value to narrow our analysis to the most significant events. We further defined POWS spikes as rising above the threshold of 6 after remaining below 6 for the previous 5 days. All POWS spikes were identified, and a random sample, allowing one per infant, was selected for chart review. Every POWS spike date and time was matched with a control date and time at equal gestational and postnatal ages whose POWS score near that time did not surpass the threshold. When infants could not be matched by both GA and postnatal age, we matched by GA and selected a control timestamp at the nearest postnatal age with POWS < 6. Infants with POWS spikes were not excluded from control matching, so infants with a POWS spike could also have a control time matched with a POWS spike in another infant, as long it occurred 10 days or more from the POWS spike time.

Chart review

A list of dates and times corresponding to POWS spikes or control times was provided to reviewers at each site who were not involved in the statistical analysis and were blinded to the spike type (POWS spike or control time). Reviewers were instructed to record any significant event documented in the medical record within 3 days, defined as the day of the timestamp, −2 calendar days, +2 calendar days. We included days before and after the timestamp because the timing of clinical diagnosis is variable and cannot be determined precisely from chart review. Events could include a change in patient status, change in management, new diagnosis, or significant abnormality in laboratory or radiography results. Data sources included progress notes, problem lists, radiographs, operative notes, nursing and respiratory care flowsheets, and laboratory results during the five days centered on each POWS spike or control time. A second blinded reviewer at each site audited the first reviewer’s work for clarifications and questions.

Together, the two reviewers (still blinded) determined the primary clinical correlation with the spike or control time as falling into one of nine pre-defined infection-related and non-infection-related categories (Table 1). For times when more than one clinical correlation occurred, the primary clinical correlation was designated as infection-related over non-infection-related categories if one was present. For example, when respiratory deterioration prompted blood culture and sepsis diagnosis, the primary clinical correlation was categorized as sepsis and not respiratory deterioration. Of note, POWS scores were calculated for research purposes after infants were discharged from the NICU and thus could not have impacted clinical care.

Table 1 Definitions of clinical diagnoses assigned after chart review.
Full size table

Statistical analysis

We used summary statistics to analyze cohort characteristics. Frequencies of the categories of primary clinical correlations assigned for POWS spikes and control times were compared using Fisher’s exact test. We also analyzed clinical correlations of POWS spikes and control times according to whether there was a blood culture for suspected or confirmed sepsis and whether there was respiratory deterioration, taking events with both types of clinical correlation into consideration. We calculated the positive predictive value of POWS spikes to detect A) suspected or confirmed infection, B) respiratory deterioration, and C) any clinically significant event.

Results

Patients and POWS spikes

We analyzed data from 806 VLBW infants, 50% male, with a mean gestational age of 28.0 ± 2.8 weeks and mean birth weight of 997 ± 293 g. Of these, 285 infants (35%) had one or more hourly POWS values that met the criteria for a spike during their NICU stay (Table 2). The relative frequency of POWS scores > 6 constitutes approximately 2% of all recorded values. In this cohort, 475 POWS spikes were present in 30,666 patient days of data. To give a sense of the spike frequency, in a NICU with an average daily census of 20 VLBW infants, this would equate to one spike every 3.2 days.

Table 2 Characteristics of infants included in medical record review of POWS spike or control times.
Full size table

Clinical associations of POWS spikes

We randomly selected 111 POWS spike times and matched them with 111 control times from 192 VLBW infants (Table 2). Fig. 1a shows the average POWS scores around the time of a spike for infants in the POWS spike group. Fig. 1b displays the average POWS scores for the matched controls. Infants with POWS spikes had similar sex, race, and ethnicity as infants selected as controls, but they had slightly lower mean birthweight despite matching for gestational age (Table 2). Postnatal age at the timestamp used for chart review was also similar between the two groups (17 days old at POWS spikes vs 20 days old at control times, Table 2). The average POWS score was 7.0 (SD = 1.6) at spike times and 1.3 (SD = 0.9) at control times. Clinically significant events were significantly more likely to occur within 3 days of POWS spike times compared to control times (spike times 85/111 (77%) with identified events; control times 56/111 (50%) with identified events (p < 0.001, Fig. 2). Table 3 shows the frequencies of the primary clinical correlation identified for POWS spikes and control times. The PPV of a POWS spike was 52% for suspected or confirmed sepsis, 34% for respiratory deterioration, and 77% for any significant clinical event.

Fig. 1: Average POWS scores around POWS spikes and control times.
figure 1

Plots of the average hourly POWS scores from 3 days before to 3 days after a (a) POWS spike and (b) control time matched for gestational age and postnatal age.

Full size image
Fig. 2: Clinical associations with POWS spikes and matched control times.
figure 2

Clinical conditions defined in Table 3 were grouped into the major categories of suspected or culture-confirmed infection, respiratory deterioration (including increased apnea), and other clinical events.

Full size image
Table 3 Clinical correlations of POWS spike times and matched control times.
Full size table

An evaluation for suspected sepsis or NEC occurred within 3 days of a higher proportion of POWS spikes than control times (58/111 (52%) vs. 27/111 (24%), p < 0.001). Of the 58 POWS spikes identified close in time to blood cultures for suspected sepsis or NEC, 32 (55%) preceded the blood culture order with a median lead time of 18.5 h (IQR 13–41 h). Respiratory deterioration, with or without suspected infection, also occurred near a higher proportion of POWS spikes than control times (38 (34%) vs. 20 (18%), p = 0.009). Respiratory deterioration or increased apnea without suspected infection occurred at similar frequencies (POWS spikes, 27 (24%) vs. control times, 28 (25%)). Of the 38 POWS spikes near respiratory deterioration or increased apnea, 28 (82%) of the spikes occurred on or before the day of the clinical diagnosis.

Discussion

We quantified clinical events near a rise in POWS above a high threshold to understand the clinical context and, thus, how the risk score might inform decision-making. Infection and respiratory deterioration were the most common clinical events identified at the time of high POWS scores and occurred significantly more often near POWS spikes than control times. Considering the rare incidence of cardiorespiratory deterioration throughout the NICU hospitalization in VLBW infants, a POWS spike had a relatively high PPV for infection, respiratory deterioration, and other significant clinical events.

This work draws from our experience with heart rate characteristics (HRC) monitoring for late-onset sepsis in VLBW infants.8,10,11 Both POWS and HRC monitoring use physiologic data and predictive modeling for early warning of sepsis, but POWS incorporates SpO2 analytics with HRC, better capturing the cardiorespiratory deterioration associated with sepsis and sepsis-like syndromes. The POWS algorithm also includes the cross-correlation of HR and SpO2, which increases when an infant is experiencing increased apnea, HR decelerations, and oxygen desaturation.6,12 Respiratory deterioration and increases in apnea are common early signs of infection.9 However, such events also occur in premature infants without infection due to lung disease and immature respiratory control.13

Abnormal cardiorespiratory patterns occur during neonatal sepsis due to the autonomic response to systemic inflammation. Afferent parasympathetic nerves are stimulated by pathogen toxins and inflammatory molecules in the bloodstream or tissues and relay signals to the brainstem, triggering a cholinergic anti-inflammatory response, which is a critical host defense mechanism.14 Efferent vagus nerve signals act on muscarinic cholinergic receptors on leukocytes to decrease proinflammatory cytokine production. Vagus nerve firing at the sinoatrial node triggers transient heart rate decelerations.15 Cytokines and prostaglandin-E2 bind to receptors in the brainstem that control respiratory rhythmogenesis, resulting in apnea, which is often accompanied by bradycardia and desaturation.7,16,17 POWS uses features that detect patterns of decreased heart rate variability, increased bradycardia events, increased intermittent hypoxemia, and increased apnea events.5 While these physiologic disturbances are frequent signs of sepsis, they are non-specific. Non-infectious conditions in preterm infants, such as respiratory distress syndrome and chronic lung disease, are associated with inflammation and cardiorespiratory instability.18 The results of this study suggest that while infection is more common when POWS is high, respiratory deterioration without infection can be included in the differential diagnosis when evaluating a patient with a significant rise in the score over their prior baseline. Thus, while developed for sepsis, knowing that POWS can provide early warning of cardiorespiratory deterioration with or without sepsis may broaden the clinical utility of predictive monitoring in the NICU.

A limitation of our approach is the relatively small sample size of POWS spike time periods reviewed but a strength is the careful medical record review of the 222 spike and control times by two investigators and the inclusion of two NICUs. The retrospective nature of the study limited the ability to identify undocumented clinical events. However, we note that events documented in daily progress notes are generally more likely to be clinically significant, and other corroborating sources in the medical record, such as flowsheets, laboratory test results, radiographs, and nursing notes were also reviewed. Despite matching control times to have similar gestational and postnatal age as POWS spikes, the mean birth weight of the infants with POWS spikes was lower by ~70 g, a magnitude unlikely to have a significant impact on the analysis. The limitations of this study will inform prospective analysis, which will be necessary to determine the clinical impact and utility of such a system.

We focused the current analysis on a high threshold occurring at low frequency to capture the clinical context of infrequent but extremely high-risk POWS scores. Our intention was not to analyze the sensitivity and specificity of a specific alert threshold for implementation in the clinical setting, as this would have required a detailed analysis of every clinically significant event throughout the NICU course. Importantly, a POWS threshold lower than 6 would yield higher sensitivity for infection or respiratory deterioration and may still occur at a clinically reasonable frequency that would not contribute to alarm fatigue.19,20

A predictive monitoring system without threshold-based alerts avoids the issue of alarm fatigue and false positives but requires users to be proactive rather than reactive in evaluating the scores and interpreting the information.21,22 On the other hand, implementing a risk score with alert thresholds can trigger clinical response guidelines intended to prompt action at a discrete level of predictive accuracy. Our analysis of a clinical prediction model using in-depth chart review and matched controls is unique and provides a foundation for implementation in a clinical trial setting. Further evaluation is necessary to understand the impact POWS will have on clinical care, including risks and benefits.

Conclusions

In VLBW infants at 2 NICUs, a spike in the predicted risk of late-onset sepsis using POWS, a continuous model based on cardiorespiratory data was associated with infection, respiratory deterioration, and other significant clinical events. A POWS spike had a 52% PPV for suspected or confirmed infection and 77% PPV for any clinically significant event. These results will inform the next steps towards implementation of POWS, such as prospective pre-implementation evaluation and clinical trials.