Table 1 Pre-deployment and operationalization on Mayo platform of ECG AI-Guided Screening for Low Ejection Fraction (EAGLE).

Pre-deployment

Impact evaluation

A problem is identified, and a proposed solution is evaluated by a cross-disciplinary team.

Prior to deployment, the proposed EAGLE model is judged on (1) potential clinical value, and (2) potential for impactful operationalisation given existing infrastructure and clinical environment. In this case, discovering hidden diagnoses from complex data would provide new diagnostic and screening capabilities that are currently unavailable in the given environment.

Data lifecycles

Availability of suitable datasets and data flows are identified.

The team ensures that data flows are available for training, for prospective validation, and for safe monitoring of outcomes. In this case, interoperability between ECG devices and other clinical data within the platform (“Gather”) means that suitable datasets can be curated, accessible in a training environment (“Discover”). Real-time data flows can be easily established for prospective validation, production, and observation. Model output data can be messaged back to end-users at point-of-care.

Model-building

Training a model on data directly curated from real-world pathways

Having considered the above, a model trained on the platform can emerge ‘production-ready’. Established data aggregation and quality assurance pipelines on the Mayo platform means accurate and useful labels, allowing EAGLE to be benchmarked in under-represented groups (“Validate”). A well-calibrated model can be taken to prospective validation on live data flows. While in a research container, EAGLE performance can be silently observed against other gold standard diagnostic indicators (such as echocardiography) in the same environment.

Production

Infrastructure that is ready to receive a trained model

Positioning of devices and EHR, in parallel to data flows and the model-building environment, means the EAGLE model can be moved directly into a production environment without significant reconfiguration (“Deliver”). Helped by early in-situ end-user involvement, EAGLE outputs will appear directly at a suitable moment on a clinical pathway.

Operationalization

Impact evaluation +

Data lifecycles +

Model re-validation +

Production

Deployment supported by all components

With all components in place, a trained model can be operationalized in a live pathway. Components work symbiotically to support the deployment:

 1) Adjacency of analysis and production environment allows users to monitor real-time model outputs. Chosen outcome measures can be observed during a clinical trial⁵¹.

 2) Wider data flows monitored for intended and unintended clinical impacts, contributing to pre- and post-market quality management and compliance with regulatory requirements across the product lifecycle⁶⁶.

 3) Containers are created for users to observe data and model output distributions. Early safety signals can trigger model re-validation. Over time, new and manually validated data will enrich the original training dataset.

 4) Adjacency of training and production environments, and use of established data flows, means re-validation cycles (and future adaptive AI) are easy to implement.

 5) In-situ end-user interactions in development, and once operationalized, allows for direct feedback into usability. Production environment supports responsive updates.