Fig. 1: The GMLF multimodal deep learning framework of Histology and Gene Expression Integration for Predicting Response to NAC. | npj Digital Medicine

Fig. 1: The GMLF multimodal deep learning framework of Histology and Gene Expression Integration for Predicting Response to NAC.

From: Predicting response to neoadjuvant chemotherapy in muscle-invasive bladder cancer via interpretable multimodal deep learning

Fig. 1: The GMLF multimodal deep learning framework of Histology and Gene Expression Integration for Predicting Response to NAC.The alternative text for this image may have been generated using AI.

Our model uses two paired data types from bladder cancer samples: gigapixel whole-slide images from routine Hematoxylin and Eosin (H&E) stained slides, and gene expression data from tissue microarrays. Our GMLF model consists of three branches: (1) WSI Neural Embeddings Branch: a GNN-based branch processing attributed graphs with nodal features as neural embeddings extracted by ResNet50 from WSIs, (2) WSI Cell-type and Morphological Branch: another GNN-based branch for graphs with nodal features comprising cell type and morphological features extracted by HoVer-Net from WSIs, and (3) Gene Expression Branch: a multilayer perceptron that processes the gene expression vector. Each branch i of the model yields a scalar score Si. We employ a multimodal late fusion strategy, aggregating these branch-level scores through summation, followed by Platt scaling to generate a prediction value. This value represents a probability between 0 and 1, where 1 indicates a complete response (pCR) to NAC.

Back to article page