Extended Data Fig. 2: Multi-disease 5-year prediction.

(a) We fine-tune Merlin for predicting chronic disease onset in otherwise healthy patients within 5-years. (b) We compare Merlin to other baseline model variations fine-tuned for the same task. We find that with both 100% and 10% of downstream training data, Merlin outperforms the other model variations. Bars represent the mean performance; error bars denote 95% confidence intervals evaluated on the test set (n = 1,243 independent CT scans). (c) Comparison of Merlin chronic disease prediction performance to a model trained using only phenotypes (EHR Pretraining), an ImageNet I3D initialized model, and a randomly initialized model. (d) An ablation study that measures the impact of various aspects of Merlin’s training strategy. We find that training with EHR and radiology reports, using staged training (Stg.) or multi-task learning (MTL), and training with radiology reports only (Rpt.), all outperform training with EHR only. Data are shown as mean ± 95% CI; n = 1,243 CT scans. The icons in a were adapted from the Noun Project (https://thenounproject.com/) under a royalty-free licence.