Introduction

Frailty is increasingly being recognized as a risk factor for mortality, prolonged hospitalization, readmission, and poor quality of life after discharge in critically ill older adults1,2,3,4,5,6,7. Up to 24% of critically ill patients may be frail at baseline prior to admission8. Most studies have prospectively assessed frailty using the Canadian Study of Health and Aging Clinical Frailty Score (CFS)4,8,9,10. However, most frailty scores (i.e., CFS, Fried’s frailty phenotype, Edmonton Frail Scale) have limited use in administrative databases (i.e., Canadian Institute for Health Information Discharge Abstract Database, United States [US] Centers for Medicare & Medicaid Services databases), as these databases do not contain the information necessary to calculate these scores10,11,12,13. Hence, our understanding of frailty in critical illness has been limited to prospective studies.

Consequently, frailty scores for administrative databases have been developed, as interest in big data research increases. The electronic Frailty Index (eFI) was developed for use in primary care electronic records14. The modified Frailty Index (mFI) has been studied in Brazilian intensive care units (ICUs); however, it requires the measurement of functional capacity15. Recently, the Hospital Frailty Risk Score (HFRS) was derived using ridge regression and 109 International Classification of Diseases, Tenth Revision Clinical Modification (ICD-10-CM) codes, from a cohort of > 20,000 hospitalized older adults16. The HFRS has been validated to predict the risk of 30-day mortality, prolonged hospitalization, and 30-day emergency hospital readmissions in older hospitalized patients16, and it has since been validated in other hospitalized population databases17,18,19,20. However, its validity in critically ill patients has been questioned in a single center study21. Thus, there is a need to study the validity of the HFRS in large administrative databases of critically ill patients. The primary goal of this study was to externally validate the HFRS among a nationally representative US sample of older adults receiving mechanical ventilation.

Methods

This study was reported using the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement22. It was exempted by the Saskatchewan Health Authority Research Ethics Board (SHA-REB-20–77), as de-identified information was used, and it was performed in accordance with all institutional guidelines and regulations.

Data source

We extracted information from the Nationwide Readmissions Database (NRD) from January 1, 2016, to November 30, 2018. The NRD is the largest all-payer US readmission database from the Healthcare Cost and Utilization Project (HCUP), and it includes hospitalizations of both insured and uninsured patients from 28 different state databases23. It samples > 15,000,000 unique hospitalizations annually, representing > 36,000,000 weighted hospitalizations, including general ward, intermediate care, and ICU patients23. The NRD accounts for 60% of the total US population and 59% of all hospitalizations, allowing for national estimates.

Study population

We included all hospitalizations of older adults (≥ 75 years old) receiving mechanical ventilation, using a validated administrative definition in the ICD-10 procedure coding system (ICD-10-PCS) (Electronic Supplementary Material (ESM) eTable 1)24. We excluded patients who left against medical advice and hospitalizations with missing information for length of stay, time to next visit, and December admissions, as the NRD is unable to follow these patients beyond the calendar year. We also excluded hospitalizations of non-residents of the state, as the NRD does not have any linking state identifiers.

Measurements

The covariates in the NRD included age, biological sex, hospital characteristics (teaching status, size), income quartile, primary insurance status (Medicare, Medicaid, private insurance, self-pay, or other) and Elixhauser-van Walraven comorbidity index score25. The ICD-10-CM and ICD-10-PCS codes were used to classify comorbidities (ESM eTable 1). We determined the primary reason for admission of the index hospitalization and readmission, using the first listed diagnosis (DX1) and aggregate groups of the Clinical Classifications Software Refined (CCSR) developed by HCUP (ESM eTable 2)26. Hospital costs were determined using total hospital charges multiplied by the all-payer cost-to-charge ratio, then inflation-adjusted to 2018 US dollars using the US Bureau of Labor Consumer Price Index for medical care27,28. Linked visits were identified through a linking variable.

Frailty risk

Frailty risk was assessed using the HFRS developed by Gilbert et al. (ESM eTable 3)16. We classified patients as either low-risk (score < 5), intermediate-risk (score 5–15), or high-risk (score > 15) for frailty, based on the original HFRS study and subsequent validation studies16,17,18.

Outcome(s)

We evaluated the performance of the HFRS to predict in-hospital mortality, as the primary outcome. The predictive performance of the HFRS for prolonged hospitalization and 30-day emergency hospital readmissions were evaluated as secondary outcomes. We only evaluated in-hospital all-cause mortality instead of 30-day mortality (inpatient or outpatient) because the NRD only records in-hospital deaths. We defined prolonged hospitalization as a hospital length of stay > 10 days and only evaluated 30-day emergency hospital readmissions, similar to Gilbert et al16,17.

Statistical analysis

All statistical analyses were performed using Stata/MP 15.1 (College Station, Texas, US). A two-sided p value < 0.05 was considered statistically significant. We accounted for the complex sampling design of the NRD using sampling weights provided by HCUP23. Categorical variables were presented as unweighted numbers and weighted percentages. Continuous variables were presented as either means (standard deviation [SD]), or medians (interquartile range [IQR]), following testing for normality. Survey-specific Rao-Scott tests were used to compare nominal data. Survey-specific linear regression was used to compare continuous data, using the geometric means for non-normal data. Missing data were present in < 5% of all patient visits. As a result, a complete case analysis was performed for all analyses given the complex sampling design29,30.

We assessed the validity of the HFRS for predicting in-hospital mortality, prolonged hospitalization, and 30-day emergency hospital readmission, using unadjusted and adjusted logistic regression. For in-hospital mortality and prolonged hospitalization, we performed adjustment for age, biological sex, income quartile, insurance status, do-not-resuscitate status, admission diagnosis, hospital characteristics, and year. For 30-day emergency hospital readmissions, we performed adjustment for the same variables, also including hospital disposition. Model discrimination was assessed with the c-statistic and calibration with the Brier score31,32. Calibration plots additionally were constructed.

Sensitivity analyses

We performed several sensitivity analyses to assess the robustness of our findings. First, we re-evaluated our findings using the HFRS as a continuous variable and using restricted cubic splines with five knots33. Next, we performed survey-specific Cox proportional hazards regression for in-hospital mortality and 30-day emergency hospital readmissions34. Subsequently, we derived 30-day in-hospital mortality, using hospitalization data from the NRD, and re-performed our primary analysis. We performed additional post hoc analyses, restricting the population to those who only received mechanical ventilation for greater than 24 h and restricting the population to only those who were admitted emergently. Additionally, subgroup analyses were performed for patients who received major operative procedures and those who did not. We also performed an additional sensitivity analysis adjusting for time receipt of mechanical ventilation. We then performed multiple imputation with chained equations for missing data using 10 imputations, and repeated the primary analysis with the imputed dataset35. Finally, as a post hoc analysis, we evaluated the total population of older adults in the NRD, independent of receiving mechanical ventilation, to determine whether our findings held for the entire older adult population.

Ethics approval and consent to participate

This study was reviewed by the Saskatchewan Health Authority Research Ethics Board (REB-20–77) and was considered exempt under the TCPS2, with a waiver of consent.

Preprint

A previous version of this manuscript was published as a preprint doi: https://doi.org/10.21203/rs.3.rs-1086390/v3, https://www.researchsquare.com/article/rs-1086390/v3

Results

There were 371,410 hospitalizations of older adults receiving mechanical ventilation, representing 649,330 weighted hospitalizations (3.4% of all weighted hospitalizations in the database) (Fig. 1). A summary of baseline characteristics is described in Table 1 and ESM eTable 4. Missing data are described in ESM eTable 5. Of the hospitalizations, 50.0% had female patients, the median (IQR) age was 81 (78–86) years old, and the median (IQR) Elixhauser-van Walraven comorbidity index score was 18 (12–25). Infection-related diagnoses (30.5%) were the most common primary diagnoses. Many patients had primary or secondary diagnoses of severe sepsis (32.8%), shock (40.2%), and acute kidney injury (51.5%). Referral to palliative care occurred in approximately 26.8% of hospitalizations, with the high-risk for frailty group receiving the most referrals (p < 0.001).

Figure 1
Figure 1
Full size image

Flow chart of hospitalizations of older adults receiving invasive mechanical ventilation.

Table 1 Characteristics of the population.
Full size table

The median (IQR) HFRS was 10.8 (7.7–14.5) (ESM eFigure 1). Of all hospitalizations, 9.5% were classified as low-risk, 68.3% as intermediate-risk and 22.2% as high-risk.

Prevalence of mortality, long hospital length of stay and 30-day hospital readmissions

In-hospital mortality occurred in 45.3% of all hospitalizations, and prolonged hospitalization occurred in 41.1% of all hospitalizations (Table 1). Of survivors, 20.3% were readmitted to hospital by 30 days. Among high-risk for frailty hospitalizations, they had an increased incidence of prolonged hospitalization and 30-day emergency hospital readmissions (all p < 0.001) compared to the low-risk for frailty group. However, they had a reduced incidence of in-hospital mortality compared to other frailty groups (p < 0.001).

Assessment of model performance

Model performance was assessed for in-hospital mortality, prolonged hospitalization, and 30-day emergency hospital readmission (Table 2). In the unadjusted analysis, the intermediate- and high-risk groups were associated with reduced risk of in-hospital mortality, prolonged hospitalization, and increased risk of 30-day emergency hospital readmission. After adjustment, the intermediate- and high-risk for frailty groups were associated with reduced in-hospital mortality in this patient population (aOR 0.79 [95% CI 0.77–0.82] for intermediate-risk and aOR 0.46 [95% CI 0.45–0.48] for high-risk, c-statistic 0.712, Brier score 0.214), compared to the low risk for frailty group. Additionally, they were associated with prolonged hospitalization (aOR 2.61 [95% CI 2.46–2.78] for intermediate-risk and aOR 5.59 [95% CI 5.24–5.97] for high-risk, c-statistic 0.694, Brier score 0.216) and increased risk for 30-day emergency hospital readmission (aOR 1.18 [95% CI 1.12–1.24] for intermediate-risk and aOR 1.20 [95% CI 1.13–1.27] for high-risk, c-statistic 0.595, Brier score 0.162) after adjustment. Model calibration assessed using calibration plots (Fig. 2) visually demonstrate good calibration of the adjusted models.

Table 2 Model performance of HFRS subcategory and outcome in mechanically ventilated older adults.
Full size table
Figure 2
Figure 2
Full size image

Calibration plots for logistic regression models. The red line refers to the reference slope or perfect calibration. The blue line refers to actual calibration slope of the model of interest. The dots refer to the observed frequency of events per tenth of predicted risk. (A) Unadjusted model for in-hospital mortality. (B) Unadjusted model for long hospital length of stay. (C) Unadjusted model for 30-day emergency hospital readmissions. (D) Adjusted model for in-hospital mortality. (E) Adjusted model for long hospital length of stay. (F) Adjusted model for 30-day emergency hospital readmissions.

Sensitivity analyses

Detailed information on the sensitivity analyses is available in the ESM eResults and in eTable 6-e16. We performed several different analyses to evaluate the robustness of our analysis method, including re-analyzing our data using the HFRS as a continuous variable (ESM eTable 6) or using restricted cubic splines with five knots (ESM eTable 7, Fig. 3), performing Cox proportional hazards regression (ESM eTable 8), evaluating in-hospital 30-day mortality (ESM eTable 9), and performing multiple imputation with chained equations (ESM eTable 15). These additional analyses did not alter our overall findings.

Figure 3
Figure 3
Full size image

Association of the hospital frailty risk score (HFRS) with outcome of interest, using restricted cubic splines with five knots and an HFRS of 5, as the reference category. All models adjusted for age (continuous variable), Elixhauser-van Walraven comorbidity index score (continuous variable), do-not-resuscitate status, biological sex, insurance status, income quartile, year of study, hospital teaching status, hospital size, and admission diagnosis category. All spline graphs truncated at an HFRS of 30. (A) In-hospital mortality. (B) Longer hospital length of stay. (C) 30-day emergency hospital readmission, with additional adjustment for hospital disposition.

Discussion

In this study, the primary objective was to externally validate the HFRS to accurately predict in-hospital mortality in a large nationally representative cohort of older adults receiving mechanical ventilation. In its current form, the HFRS could not be successfully validated for use in this population. As expected, we found that patient hospitalizations receiving mechanical ventilation with intermediate- and high-risk for frailty, as categorized by the HFRS, were associated with increased risks of prolonged hospitalization and 30-day emergency hospital readmissions, compared to low-risk hospitalizations. Counterintuitively, they were inversely associated with in-hospital mortality when compared to the low-risk hospitalizations, suggestive of a potential spurious relationship. Regardless, the HFRS had only moderate discrimination and accuracy in predicting any of these outcomes. Using the HFRS as a continuous variable or with splines did not provide additional value over using the HFRS subcategories of low-, intermediate-, and high-risk. Our findings would suggest that clinicians and researchers should avoid using the HFRS when conducting big data research with administrative datasets of critically ill patients.

Comparison with previous studies

Prior HFRS studies focused on validating it in general hospitalizations, including non-ICU and ICU patients17,18,19,36,37,38,39. Recently, there has been interest in externally validating the HFRS in ICU administrative databases, as interest in big data frailty research increases21,40,41. A German ICU study of 1,498 patients evaluated the HFRS to predict a combined endpoint of mortality and risk of readmission and found no association after adjustment for severity of illness21. In a large Wales population study, the HFRS had only moderate ability for predicting inpatient, 6-month, and 1-year mortality in hospital and ICU patients41. Conversely, a US study of 12,854 patients, using the single-center Medical Information Mart for Intensive Care (MIMIC-III) database, found that higher HFRS was associated with an increased risk of 28-day mortality40,42.

In our study, we found that critically ill older adult hospitalizations receiving mechanical ventilation were at high-risk of poor outcomes, including prolonged hospitalization (41%), 30-day in-hospital mortality (44%), in-hospital mortality (45%), and 30-day emergency hospital readmission (20%). Unsurprisingly, palliative care utilization was very high at 26.8%, with higher use in the high-risk frailty groups. The overall readmission rate was high in the patients of this study, suggestive of current difficulties in transitions in care for these patients and potential room for quality improvement.

Prior studies of critically ill patients have established that frailty is associated with increased risks of mortality3,4. Counterintuitively, we found that the HFRS was inversely associated with mortality in the NRD (i.e., lower HFRS was associated with the highest risks of in-hospital mortality). To ascertain this surprising and unnatural finding, we performed a post hoc analysis on the entire NRD population of older adults, independent of the receipt of mechanical ventilation, and found that the HFRS performed well on the whole population (i.e., higher HFRS was associated with the highest risks of in-hospital mortality in all older adults) (ESM eTable 13).

There may be some possible explanations for this unusual phenomenon, including selection biases and coding biases. In Gilbert et al.’s original study, they validated the HFRS in a general hospitalized population to predict in-hospital mortality16. In general, critically ill patients are at higher risk of death compared to a general hospitalized population, representing a surrogate endpoint. Therefore, by limiting our population to mechanically ventilated patients, selection bias may have been introduced, potentially altering the true association of the HFRS and mortality. Coding biases may also occur as critically ill patients who had prolonged hospitalizations and/or survived their hospitalization may appear to more “frail,” as they accrue more ICD-10-CM secondary diagnoses captured in their medical records. In the NRD, most of the hospitalizations of older adults receiving mechanical ventilation were in the intermediate-risk frailty group, and most hospitalizations in the high-risk group had significantly more ICD-10-CM codes captured compared to hospitalizations in the low-risk group. Finally, frail patients with higher severity of illness or those with treatment limitations may choose less invasive treatments, introducing further selection bias. We did adjust for do-not-resuscitate status; however, this may not fully capture all treatment limitations.

These biases and differences in the ICU patient population from the original development cohort could potentially explain why the HFRS had mixed performances for predicting in-hospital mortality in an ICU patient population, as seen in this study and others.

Strengths and limitations

Our study had several strengths including the use of a large multicentre dataset, comprising close to 650,000 weighted hospitalizations. To our knowledge, our study represents one of the largest studies of critically ill patients examining the use of the HFRS, allowing for generalizability of our findings to critically ill older adults receiving mechanical ventilation. Unlike prior external validation studies in critically ill administrative databases, we evaluated the HFRS to predict prolonged hospitalization and 30-day emergency hospital readmissions. Additionally, we assessed both model discrimination and calibration, allowing for confidence in the results presented. Finally, our study performed several sensitivity analyses to verify our findings.

However, our study has limitations. As discussed previously, selection bias may have occurred in our selection of a mechanically ventilated population. The NRD was not designed specifically to flag admissions for critical care. Hence, the identification of critically ill patients was done through ICD-10 codes, specific to mechanical ventilation. Other codes, such as vasopressor use, are known to be significantly undercoded in administrative databases43. As the HFRS is derived from a composite of ICD-10 codes, coding practices and biases may affect the relative prevalence of admission comorbidities, diagnoses, and treatments. Some important codes to the determination of the HFRS, such as dementia in Alzheimer’s disease (F00) or care involving the use of rehabilitation procedures (Z50), were undercoded (ESM eTable 3). This is similarly seen in other databases including the Centers for Medicare & Medicaid Services and National Inpatient Sample databases36,37. Other databases of critically ill patients may perform differently, depending on their coding practices.

Additionally, the NRD does not have sufficient information to determine ICU severity of illness, such as the sequential organ failure assessment (SOFA) or Acute Physiologic Assessment and Chronic Health Evaluation II (APACHE II) scores. We are therefore unable to verify whether the HFRS would perform better after adequate adjustment for severity of illness; however, other studies would suggest that the HFRS does not perform well even after adjustment for severity of illness21. Likewise, the NRD does not capture detailed clinical information (i.e., patient weight, vasopressor dosing), and while it collects information on length of mechanical of ventilation, this information is often incomplete. Furthermore, it does not record out-of-hospital deaths, limiting our ability to only evaluate in-hospital mortality. Finally, we did not evaluate other scores as this was beyond the scope of our study. These limitations highlight the difficulty in applying the HFRS to datasets of critically ill patients and further support our caution on avoiding the use of the HFRS to predict these outcomes.

Clinical implications, research implications, and future directions

Clinicians need to have accurate predictions of frailty and outcomes to identify patients who would benefit from early geriatric medicine referral, as well as to engage with patients and their families in shared decision-making, goals of care discussion, and end-of-life planning, and/or palliative care referral. Likewise, healthcare administrators need to have accurate estimates of the number of frail patients to plan and allocate healthcare services. Big data researchers need accurate scores to classify patients correctly.

While the HFRS may have utility in non-ICU databases, our study demonstrates its limitations in critically ill patients. The mFI is a promising alternative; however, it needs further development and validation for use with ICD-10-CM codes15,44. Perhaps the better solution for clinicians, researchers, and administrators would be to adapt and transform existing databases for frailty research. With other well-validated frailty scores such as the CFS, there is a compelling argument for its integration into routine clinical practice and inclusion in data capture. Future research should be performed to re-develop the HFRS or other scores with different weighting specifically for critically ill patients.

Conclusion

In this large nationally representative external validation study of older adults receiving mechanical ventilation, the HFRS could not be validated to predict in-hospital mortality in this population. While the HFRS may predict prolonged hospitalization and 30-day emergency hospital readmissions, its use should be avoided in the critically ill. Further research with administrative databases is necessary to develop accurate, intuitive frailty scores in critically ill patients.