Fig. 1: Schematic overview of MAARS.
From: Multimodal AI to forecast arrhythmic death in hypertrophic cardiomyopathy
MAARS has three input branches for three types of inputs: LGE-CMR images (left middle, green), clinical covariates from EHRs (left top, blue) and measurements from a CIR, which includes CMR and echocardiogram reports (left bottom, orange). The LGE-CMR images are processed to obtain the left ventricle as a region of interest and then used as input by a 3D-ViT. The EHR and CIR covariates are both structured tabular data and are used as input by dedicated FNNs. The ends of the three input branch networks are connected to a multimodal fusion module, which uses an MBT to fuse knowledge and learn to predict patient-specific SCDA risk scores (see the Methods for detailed explanations). Echo, echocardiogram; ROI, region of interest; METS, metabolic equivalents; SBP, systolic blood pressure.
