Background

Patients are often concerned about their risk of experiencing a specific health outcome within a certain time period [1] and risk assessment models (RAMs), otherwise known as prognostic or prediction models, may be helpful in this regard. RAMs consider multiple variables to calculate an individual patient’s absolute risk or probability of developing an outcome [1, 2] and play an integral role in precision medicine [3]. Also, RAMs can be used in health care institutions as part of a shared decision-making process to individualize population level recommendations and optimize patient management. In this editorial, we discuss the strengths and limitations of RAMs, and what factors clinicians may consider when selecting a RAM for use in practice. We will use examples from a United Kingdom (UK) study on models to estimate progression of diabetic retinopathy [4] and from a study that developed a RAM to predict the risk of venous thromboembolism (VTE) and bleeding among hospitalized patients that presents novel methods [5].

Features of an ideal risk assessment model

A RAM should include all important prognostic factors and have undergone external validation and impact assessment [1]. Additionally, models should be clinically relevant, easy to use in practice, and cost-effective [1]. For example, an ideal RAM for diabetic retinopathy progression should be able to accurately identify patients at high risk of progression to visual loss and reliably exclude those who are at low risk [4]. This ensures optimal selection of patients who could benefit from treatment to prevent loss of vision [4].

Common limitations in the development of risk assessment models

Inclusion of large numbers of factors can render RAMs difficult to implement in clinical practice due to excessive patient and care giver burden. For example, the authors of the UK study identified 78 prognostic factors for diabetic retinopathy progression that have been included in previously published prediction models [4]. A related concern is when factors are selected for RAMs based only on statistically significant associations with the outcome of interest, without consideration of the magnitude of association, which can lead to burdensome instruments [2, 3, 6]. The threshold for including a predictor should be informed by how much increased risk (in absolute terms) most clinicians or patients expect to see in order to change their decisions around the management approach.

Second, RAMs that include highly correlated factors will impair model performance [3]. For example, a model that included both diabetic nephropathy and chronic kidney disease would largely be measuring the same predictor twice [4]. Prognostic model developers should test inter-item correlations and, if found, either collapse related factors or remove redundancy [3].

A third consideration is the generalizability of patient data used to develop a RAM to patients that are typically seen in clinical practice. Most RAMs to predict progression of diabetic retinopathy were developed using data from low risk of progression to visual loss patients that may bias the predictive power of the developed RAM if used to assess outcomes in the higher risk category [4]. Specifically, underestimating the risk of the outcome of interest in higher risk patients may lead to under-treatment.

Fourth, the identification of potential prognostic factors for RAMs should be informed by comprehensive systematic reviews [7]. However, most RAMs are developed using data registries that typically comprise of routinely collected data and are unlikely to include all factors of interest [7].

Fifth, predictors are sometimes extrapolated from unrelated outcomes inappropriately. For example, the UK study found that predictors of macrovascular outcomes (e.g., stroke) were used in some RAMs to assess risk of retinopathy progression [4].

Sixth, many RAMs have not undergone validation and impact assessment, which means their clinical utility in practice has not been established. Going back to our example, the UK study found 14 published models for progression of diabetic retinopathy, of which only half were externally validated and none had undergone impact assessment [4].

Selection of a risk assessment model for use in practice

Clinicians may justifiably reject use of prognostic models due to a lack of clinical credibility and lack of evidence that they improve patient care [8]. Below we outline strategies to address these concerns.

Method of development

Rigorous development of a RAM should entail a systematic review of prognostic factors to ensure that all established predictors have been assessed and the consideration of clinical relevance [9]. Such an approach, building on what is known in clinical practice and on evidence base, prevents the inclusion of nonsensical, less relevant, more invasive, or costly variables or variables that are more difficult to measure in a RAM. A novel method considering these essential factors was developed based on the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach [5]. Development of the RAM included mixed methods where, first, a systematic review was conducted to identify all previously recognized factors and assess the certainty of evidence for each one [10, 11]. Next, considering this evidence, an expert panel was selected to make judgments on whether to include, potentially include, or exclude factors in a RAM, based on the domains of the GRADE approach [5]. A similar approach was used by the UK study who first conducted a systematic review of the literature and identified 78 potential predictors, that were then supplemented with an additional eight candidate predictors [4]. Forty-four clinical experts than considered this evidence and underwent four nominal group meetings to select come to consensus on a final set of 19 predictors of diabetic retinopathy progression for inclusion in a RAM to identify higher-risk patients referred to hospital eye services [4].

RAM validation and impact assessment

Before being adopted into clinical practice, RAMs must demonstrate how well they predict the outcome of interest (validity), which can be done by applying the model to a subset of the data from which it was developed (i.e. internal validity) [12]. Further, external validity should be established in a population and context that is similar to the clinical practices in which it will be implemented [13].

Another consideration is whether the RAM has been shown to provide benefits to patients (impact assessment). Ideally, a clinical trial will be available in which clinicians are randomized into two arms: one in which they administer the RAM to patients and use the results to guide practice, and the other in which usual care without the RAM is provided. Adoption of the RAM into clinical practice will be supported if use of the prediction model demonstrates important benefits to patients [2].

Conclusion

When considering whether to adopt a RAM into practice, clinicians may consider evaluating whether it has been constructed in a rigorous manner and includes all known important predictors. Inclusion of statistically significant factors that show trivial associations with the outcome of interest may result in a tool that is overly burdensome to patients. RAMs should also have demonstrated an ability to accurately predict the outcome of interest in populations that are representative of the clinical practices into which they are going to be utilized, and that their use will result in important benefits to patients’ beyond usual care.