Graph attention network-based multimodal approach for lung diseases classification

Abstract

Lung diseases remain one of the major causes of mortality worldwide. Medical experts usually require information from multiple sources, including clinical symptoms, laboratory and pathological tests, in combination with chest X-rays, to confirm a lung disease diagnosis. However, the manual interpretation of information collected from various sources is time-consuming and may lead to misdiagnosis of lung disease due to structural overlap and similar symptoms. An automated classification framework that integrates information from multiple sources, such as clinical notes, disease symptoms, and chest X-ray images, could enhance diagnostic precision and optimize clinical workflow efficiency. Therefore, this study proposed a multimodal approach based on the Graph Attention Network that integrates the pre-trained Clinical ModernBERT and RAD-DINO models, allowing for granular cross-modal interactions and providing robust multimodal representation learning to advance lung disease classification. The experiments are conducted using a publicly available lung disease dataset, which consists of chest X-ray images and their corresponding clinical text. For multiclass lung disease classification, the proposed multimodal approach achieved an accuracy of 95.73%, precision of 95.75%, recall of 95.72%, and F1-score of 95.70%, along with an expected calibration error of 0.0124. The proposed graph attention network-based multimodal framework outperformed existing state-of-the-art methods and effectively integrated imaging and textual information at the token level, offering an accurate and reliable automated alternative for lung disease diagnosis.

Introduction

Lung diseases remain one of the most common causes of morbidity and mortality around the world. It is well known that millions of individuals worldwide are affected by various lung diseases. Some of these conditions can be effectively treated if the disease is identified in its early stages, before it becomes life-threatening^1,2,3. The early and accurate detection is crucial for effective treatment and optimal patient outcomes. The traditional methods of diagnosing thoracic diseases predominantly depend on medical imaging techniques (such as X-rays and CT scans) or clinical findings (such as medical history, symptoms, and signs)^4,5,6. However, the overlapping and intricate symptoms of various lung diseases make lung disease diagnosis challenging. To accomplish this task effectively and accurately, highly qualified and experienced medical professionals are required. The lack of competent and experienced radiologists, coupled with manual interpretation, often leads to misdiagnoses^1,5,7. Additionally, observations and information gathered from a single modality are usually inadequate and may not provide sufficient information to accurately and efficiently diagnose lung diseases^8,9. It is crucial to merge information from multiple data sources to ensure an accurate, comprehensive, and effective diagnosis of lung diseases. These sources include laboratory tests, clinical notes, disease manifestations, and additional relevant information. To automate and improve lung disease detection, a deep learning approach is required that leverages and consolidates multimodal data from multiple sources^4,7,10.

Recently, deep learning strategies, particularly multimodal learning that incorporates visual and textual data, have gained considerable attention and impact in the medical domain, where datasets typically consist of paired images and related clinical text^11,12. Therefore, multimodal learning has emerged as a dominant strategy for acquiring extensive and detailed medical knowledge by integrating data from multiple modalities to improve the effectiveness of subsequent multimodal tasks^13,14. However, multimodal fusion techniques still face numerous challenges. It remains a considerable challenge to align and merge heterogeneous data from diverse modalities, such as imagery and textual information. Simple concatenation or cross-modal attention techniques are often incapable of capturing the complex relationships among various modalities, leading to the loss of fine-grained semantic information. Furthermore, end-to-end training is challenging due to high computational costs and unstable training dynamics, particularly when employing large backbone models. Additionally, the fine-tuning of pre-trained backbones may result in the loss of valuable pre-trained knowledge^4,13,15. To preserve pre-trained knowledge and reduce training costs, the benefits of partially freezing components, such as language models or vision backbones, within multimodal pre-training frameworks are explored^13,15,16, but not both components simultaneously.

Deep learning approaches based on graph learning technology have been extensively applied in various medical-related tasks, including segmentation¹⁷, multimodal fusion¹⁸, and disease classification^19,20. Therefore, this study proposes a novel multimodal approach based on graph structure, specifically the Graph Attention Network (GAT), that leverages chest X-ray (CXR) images and clinical text to classify lung diseases. The proposed approach integrates domain-specific pre-trained models: RAD-DINO is used to extract high-quality visual features, while Clinical ModernBERT is utilised to derive semantically rich textual embeddings. Keeping both components in an entirely frozen state simultaneously, the study presents a robust and efficient multimodal approach that retains domain-specific knowledge and eliminates the high computational demands usually associated with fine-tuning and training. The high-dimensional textual and visual embeddings are projected into a common latent space and subsequently transformed into a cosine-similarity-based multimodal Top-K graph, enabling effective multimodal fusion and enhancing the comprehension of the complex relationships between the various modalities. A multimodal graph structure is created by defining relationships among the visual and textual Top-K most semantically similar features based on cosine similarity, enabling the model to capture meaningful cross-modal relationships at the token level. The GAT module allows feature interactions within and across modalities and retains attention weights information for neighbourhood nodes. A public dataset consisting of CXR images and their associated clinical texts is used to evaluate the proposed approach. The empirical findings show that the suggested GATMM framework outperformed other multimodal frameworks. The study presents several key contributions:

• 1. Developed a multimodal framework using a multimodal graph and GAT module to improve the interaction between imaging features and clinical text embeddings. By combining complementary information from visual and textual sources, the proposed multimodal framework is designed to improve lung disease classification accuracy.

• 2. The proposed approach extracts rich embeddings from both image and text modalities using pre-trained, domain-specific encoders (RAD-DINO and Clinical ModernBERT), which are frozen simultaneously throughout training. Therefore, the proposed strategy is more efficient and robust because it eliminates the need for domain-specific knowledge and reduces computational overhead.

• 3. The suggested approach uses a cosine-similarity-based bipartite Top-K graph construction mechanism to capture fine-grained interactions across modalities. This technique optimally integrates the most relevant features of both modalities, reducing information loss and lowering computing complexity.

• 4. The proposed multimodal approach is validated through extensive experiments on a public benchmark dataset that includes CXR images and clinical text. Compared with existing methods, the presented framework achieved state-of-the-art performance, demonstrating the effectiveness of the developed multimodal fusion strategy for lung disease classification.

The article is structured as follows: Sect. “Related work” offers a comprehensive analysis of related work, highlighting key developments and identifying research gaps that motivate our study. Extensive descriptions of the proposed methodology are provided in Sect. “Methodology”, including the datasets, data preprocessing, overall framework design, the feature extraction processes, graph-based fusion mechanisms, and evaluation metrics. In Sect. “Experimental setup and results analysis”, the experimental setup and implementation details are described, followed by an exhaustive analysis of experimental results. Section “Discussion” describes the implications, strengths, and comparisons with other related methods. Section “Limitations and challenges” highlights the limitations and challenges of the proposed framework, whereas Sect. “Conclusion” concludes the study and outlines future research directions.

Related work

The lung disease classification and detection are crucial and widely researched areas of medical research. Recently, multimodal learning has captured the interest of researchers and emerged as an effective tool for improving disease diagnosis accuracy and reliability through the use of diverse, complementary clinical data from various sources. CXR is one of the most widely used imaging modalities for evaluating lung diseases, attributed to its low cost, ease of use, and wide range of clinical applicability. The combination of CXR images with computed tomography (CT) images, clinical texts, and lung sounds has been extensively explored by researchers to enhance the diagnostic reliability and accuracy of lung-related abnormalities. In the published literature, numerous studies have investigated the classification of lung diseases using multimodal techniques. However, this section discusses the most relevant studies. The study elucidates the strengths and weaknesses of related studies and explains the need to develop a more reliable, efficient, and robust multimodal framework for interpreting lung diseases using CXR and clinical text data.

Research studies^21,22,23,24 have combined the CXR images with other imaging modalities, such as CT scans, to improve the classification reliability and accuracy of lung diseases. Farhan et al.²¹ used the CT scans in combination with CXR images to develop a multimodal framework for the identification and classification of lung diseases. For chest radiograph feature extraction, the convolutional neural networks (CNNs) were used. The features derived from CXR and CT images were passed to Decision Tree, SVM, and LSTM models to classify lung diseases. The proposed model gained an accuracy of 98.78% accuracy, 98.99% precision, 95.89% recall, and 97.41% F1-score. The research study²² combined multiple imaging modalities (CXR, CT, and PET) to design a multimodal approach for lung disease classification. The feature maps from multiple modalities were collected through modality-specific feature extraction and fusion strategies. The authors employed CNN and pooling layers to extract high-level abstractions, subsequently using the fully connected layers for the classification of lung diseases. Varunkumar et al.²³ introduced a multimodal method that combined the CT scans and histopathological images to enhance the screening of lung diseases. The features of CT scans were extracted using a SimSiam-based self-supervised learning method, while histopathological features were obtained with the MIXTURE model. The collected features of both modalities were then passed to a Random Forest classifier to predict lung diseases. The study²⁴ proposed a novel multimodal framework, VisCapsNet, that combines a Vision Transformer with a Capsule Network for optimal feature extraction from CT and CXR images. Multimodal features were fused using a soft attentive dense weighted fusion mechanism, followed by a SoftMax classifier. However, the multimodal approaches based on imaging modalities have demonstrated improved performance in analyzing lung diseases, but due to the reliance on visual data, these approaches lack semantic richness to contextualize visual findings.

The research studies^25,26,27,28 explored multimodal learning approaches that combine chest radiographs with lung or cough sounds to improve the analysis of lung diseases. Issahaku et al.²⁵ presented a multimodal deep learning framework to detect lung diseases using CXR images and cough sounds. Modified VGG16 networks were used to extract features from both CXR images and cough sound spectrograms. To fuse the multimodal features, an attention mechanism is applied, while a modified Faster R-CNN is used for disease detection. The study²⁶ presented a multimodal technique for early detection of COVID-19 employing breathing sounds and CXR images. The CXR images were processed using Anisotropic Diffusion, Fast Guided Filter, U-shaped Dual Swin Attention Transformer, and Deep Graph Regularized Nonnegative Matrix Factorization. In parallel, tri-gaussian filters with mel frequency cepstral coefficient features were employed for sound signal processing. The progressive split deformable field fusion module integrates visual and audio features, which are then classified using a dual sampling residual attention CNN. The proposed framework achieved accuracy, precision, recall, F-score, and AUC values of about 98%. In the study²⁷, an innovative approach named FuzzyGuard was introduced, which utilized CT scans and audio signals (such as cough and lung sounds) for the early identification of chronic obstructive pulmonary disease. A Neuro-fuzzy Random Vector Functional Link network was used to extract features across all modalities, and a weighted sum-fusion technique was utilized to fuse these features. Malik et al.²⁸ developed a multimodal framework based on CNNs for the classification of nine lung diseases. The proposed model comprises batch normalization layers, Max-pooling, rank-based average pooling, dropout, and the multiple-way data generation module. For lung disease classification, the proposed method achieved 99.01% accuracy, 98.98% precision, 98.30% recall, 98.12% F1-score, and 93.7% AUC. While image-sound-based multimodal frameworks typically integrate heterogeneous features through the implementation of fusion layers or attention mechanisms, the availability of large and well-labeled lung sound datasets poses a significant challenge to these approaches. In addition to taking chest radiographs and recording lung or cough sounds at different times, matching up the times between them is a challenging task.

The research studies^29,30,31 examined multimodal learning approaches that integrated chest radiographs with tabular patient metadata to enhance lung disease analysis. Shimbre et al.²⁹ introduced ChestXFusionNet, an efficient multimodal deep learning framework that combined CXR images with patient metadata for the accurate prediction of lung diseases. The model applied 2D CNNs to extract visual features from CXR images, whereas a parallel dense deep learning branch encoded auxiliary patient information, including age, gender, and view position. After extracting features from both modalities, the extracted features were combined to predict lung disease. In addition, a blended map of Image activation was employed as an additional step for further optimization. For binary lung disease classification, the presented approach obtained an accuracy of 92.4%, a precision of 92.2%, a recall of 93.5%, and an F1-score of 92.3%. The study³⁰ proposed a novel transformer-based multimodal framework to improve the classification of lung disease, leveraging CXR images and patient metadata. In the proposed model, visual features are extracted using a vision transformer, and non-imaginary clinical data is incorporated using a cross-attention mechanism. The transformer encoders were used to merge information across modalities, followed by multilayer perceptrons to predict disease. Althenayan et al.³¹ developed a novel multimodal framework for the classification of lung diseases by integrating CXR images with tabular medical information. After extracting features using both ResNet-like and VGG-like architectures, early fusion was performed in order to combine both modalities. Synthetic samples were generated using Generative Adversarial Networks fr balancing the dataset. For multiclass lung disease classification, the proposed approach gained an accuracy of 95.97%, a precision of 96.01%, a recall of 95.97%, an F1-score of 95.98%, and an AUC of 95%. For lung disease classification, methods based on CXR and patient metadata showed the best results. However, using metadata may be inappropriate or suboptimal in certain scenarios, particularly if it is not specific to radiographic findings. Furthermore, the diagnosis of most lung diseases is not based on demographics or general patient information.

Research studies^32,33,34,35 investigated the most promising and rapidly advancing research direction that combines chest radiographs and clinical textual data to enhance the lung disease classification. Kumar et al.³² proposed an innovative multimodal fusion architecture that integrates CXR images and clinical data to detect lung disease. To extract image features, three modified CNN architectures were employed (DenseNet121, DenseNet169, and ResNet50), while clinical information was processed using an LSTM with a self-attention mechanism. Using the SVM model, feature vectors from both modalities are aligned and concatenated. For classification, the fused features are passed through ReLU and Softmax layers. The presented framework achieved 93.85% accuracy, 93.80% precision, and 94.74% F1-score. In the study³³, a transformer-based multimodal approach was introduced, which incorporated CXR images and clinical data to enable accurate diagnosis of lung diseases. To extract CXR features, a CNN model was used, whereas to extract clinical data features, a denoising autoencoder model was utilized. The proposed method merged features from diverse modalities for disease classification by using a cross-attention transformer module. Yang et al.³⁴ implemented a multimodal approach named MCX-Net, which integrates CXR with clinical texts to aid in the diagnosis of multi-label lung diseases. The MCX-Net used pretrained BERT as a text encoder, BEIT-v2 as an image encoder, and as a cross-modal image-text transformer module. The study³⁵ introduced a multimodal approach based on early and late fusion techniques to enhance lung disease classification, integrating clinical text and CXR images. The LLaMA-2 model is used for text data, while the 2D CNN is used for images. Both modalities were fused using cross layers (consisting of attention blocks). For multiclass lung disease classification, the proposed early fusion-based multimodal approach achieved an AUC of 97.10%. The image-text fusion methods have shown promising results. Nevertheless, these methods often overlook interactions at the token and region levels, potentially missing crucial semantic information. Furthermore, employing large backbone models requires substantial computational resources, which could hinder efficient training, fine-tuning, and deployment in clinical environments.

Recent multimodal learning methods for lung disease classification have demonstrated the benefits of integrating chest radiographs with additional data sources, including lung sounds, metadata, and clinical text. However, these methodologies usually used simple fusion procedures that may fail to capture cross-modal dependencies. Furthermore, training or fine-tuning complex, large deep learning architectures on limited medical datasets considerably increases the risk of overfitting, particularly for models with a massive number of parameters. This gap underscores the urgent need for an advanced, robust multimodal approach. Therefore, this study introduced a GAT-based multimodal (GATMM) strategy that explicitly captures interactions between the visual features of CXR images and the textual embeddings of clinical text at the token or pitch level. The semantically sparse Top-K multimodal graph and the GAT module enable fine-grained, interpretable interactions between visual and textual modalities. By leveraging pre-trained domain-specific models, such as RAD-DINO for visual feature extraction and Clinical ModernBERT for textual embeddings, the proposed multimodal approach not only enhances classification performance but also reduces the high computational burden. The proposed GATMM method offers a viable, scalable alternative with superior performance for the classification of multiclass lung-related diseases.

Methodology

This study introduced a multimodal deep learning framework that used pre-trained domain-specific vision and language models, a multimodal cosine-similarity-based Top-K graph, and two GAT layers to integrate CXR images and clinical text for lung disease analysis. The developed multimodal framework benefits from the complementary strengths of domain-specific pre-trained models and the diverse nature of visual and textual information derived from CXR images and clinical text. The subsequent subsections provide a comprehensive description of the proposed GATMM framework.

Dataset

This research work used a public dataset “Lung X-Ray Image + Clinical Text Dataset”³⁶, which consists of about 80,000 CXR images and clinical texts. The dataset has eight subfolders, such as Obstructive Pulmonary Diseases, Higher Density, Lower Density, Chest Changes, Encapsulated Lesions, Degenerative Infectious Diseases, Mediastinal Changes, and Normal. Each subfolder originally contains 10,000 images related to a particular lung condition. The CXR images were obtained from the dataset “X-ray Lung Diseases Images”³⁷, which has CXR images without patient information, deidentified by radiologists, and made publicly available for researchers. The clinical texts are collected from public datasets and medical literature sources, which include the symptoms related to each lung condition. All CXR images are pre-processed, normalized, and resized to 224 × 224 pixels before being fed to the model. The clinical texts are cleaned and pre-processed using regular expressions, and for tokenization, the Clinical ModernBERT model tokenizer is used. The study used stratified sampling by class label, with a fixed random seed (42) to split the data into three subsets (training, validation, and test), which helps to maintain class balance across splits and ensure reproducibility. A 60:20:20 ratio is used for the initial dataset splitting.

In public datasets, clinical text may show partial reuse or templating across its samples. As mentioned, the study uses a public dataset, so this issue is addressed by implementing a multi-stage leakage-control technique to ensure rigorous separation across the training, validation, and test data. Initially, to identify exact reuse or duplication across the splits, three basic filtering techniques (normalized image identifiers, normalized clinical text, and joint image-text identifiers) are used. The result of pairwise overlap comparison between splits such as training-validation, training-test, and validation-test is 0 for all three techniques, indicating that no exact instances are shared across the three subsets of the dataset. To further reduce hidden textual redundancy or templating, the normalized clinical texts are encoded using term frequency-inverse document frequency representations with a maximum of 50,000 features and a minimum document frequency of 2. Encoding is only applied to the training set to avoid the contamination of evaluation data. Cosine similarity is computed between each validation and test sample and all training samples. The validation and test samples with a similarity greater than or equal to 0.95 are excluded. In addition, the test samples having a similarity of 0.95 or higher to the filtered validation set are removed to preserve rigorous distinction between splits and eliminate near-duplicate textual redundancy among all splits of the dataset. Most importantly, only the validation and test sets are filtered, but the training set remains unchanged. After the extensive filtering operations, the final filtered dataset contains the training instances of 48,000, the validation instances of 7619, and the test instances of 5688. The subsequent consistency analysis verified that there are no exact or near-duplicate samples across all subsets. Table 1 shows the distribution of the final cleaned and filtered dataset across classes and splits.

Table 1 Final class-wise distribution of the cleaned Lung X-ray image–clinical text dataset after data preprocessing. Significant values are in [bold].

Proposed GAT-based multimodal approach

The lung disease classification task based on a multimodal approach is formulated as a multiclass classification problem. The goal of this study is to leverage both textual reports (T) and medical images (V) associated with lung disease samples (N) to predict their diagnostic categories accurately. The GATMM framework is developed to address the underlying challenges related to heterogeneous modalities and learn robust joint representations. The framework processes multimodal input in four distinct stages: Feature Encoding from text and image data, multimodal space projection and multimodal graph construction, graph attention network module, graph-level pooling, and classification head. Each component of the GATMM approach is discussed in the following sections. Figure 1 shows the proposed GATMM approach based on pre-trained domain-specific encoders, multimodal graph structure, and a GAT layer for the classification of multiclass lung diseases.

Fig. 1

Proposed GATMM approach for lung disease classification.

Visual and textual feature encoders

Visual feature encoder

For the visual modality, the input CXR image V =  {p₁, p₂, …, p_n }, representing the entire image at the pixel or pitch level, is processed to extract feature embeddings. For this task, the study utilized a domain-specific pre-trained model, RAD-DINO³⁸, as a Vision Encoder Model (VEM). The RAD-DINO is a pre-trained self-supervised learning based vision transformer model, DINOv2³⁹, which has been trained and fine-tuned to encode CXR images. The last hidden state represents the output of the vision model, which is a sequence of embeddings that represents the visual semantics of the image. Mathematically, the visual encoder output can be expressed as follows:

$$v_{i} = VEM(V)$$

(1)

where v_i denotes the sequence of pitch-level visual embeddings {v₁, v₂, …, v_n} for the i^th input sample obtained from the RAD-DINO encoder, these embeddings contain both semantic and spatial information captured from various regions of CXR images and serve as the foundation for subsequent multimodal graph generation. For visual tokens, these raw embeddings derived from CXR images have a dimension of 768, using the RAD-DINO model.

Textual feature encoder

For the textual modality, the clinical text is interpreted as a sequence of tokens or words, T = {w₁, w₂, …, w_l }, related to the lung disease instance. The pre-trained domain-specific language model, Clinical ModernBERT⁴⁰, is utilized as a Language Encoder Model (LEM) to capture rich semantic and syntactic representations. Clinical ModernBERT is a state-of-the-art encoder-based transformer model, particularly designed to handle clinical and biomedical texts. It is based on ModernBERT⁴¹ and incorporates domain-specific knowledge to generate a semantically rich representation of biomedical literature and clinical descriptions. As an output, the study extracted token-wise embeddings of the last hidden state of the textual encoder. Mathematically, the textual encoder output can be expressed as follows:

$$t_{j} = LEM(T)$$

(2)

where t_j denotes the sequence of token-level textual embeddings, {t₁, t₂, …, t_l} for the jth input sample extracted from the Clinical ModernBERT encoder model, these embeddings capture both semantic and contextual information from the clinical text. These raw textual embeddings have a dimension of 768 for textual tokens. Based on these processed textual features, the set of textual nodes for graph construction can be established for subsequent multimodal graph construction.

Multimodal space projection and multimodal graph construction

Multimodal space projection

To achieve optimal multimodal representation learning, it is crucial to address the heterogeneity gap that arises due to the distinct modalities and their specialized encoders. To bridge this gap and enable semantically meaningful unified joint learning, the study project high-dimensional features from the visual and textual encoders into a 512-dimensional embedding space using modality-specific linear layers. Mathematically, it can be presented as follows:

$$PN_{i}^{v} = LayerNorm(LinearLayer(v_{i} )),\forall v_{i} \subset \{ v_{0} ,v_{1} ,v_{2} , \ldots ,v_{n} \}$$

(3)

$$PN_{j}^{t} = LayerNorm(LinearLayer(t_{j} )),\forall t_{j} \subset \{ t_{0} ,t_{1} ,t_{2} , \ldots ,t_{l} \}$$

(4)

where $PN_{i}^{v}$ and $PN_{j}^{t}$ indicate the projected visual and textual feature representations, respectively. The projection layers transform the heterogeneous output distributions of pre-trained encoders into a unified space. It also enables seamless and effective joint representation learning, addresses the heterogeneity gap, and serves as a base for efficient multimodal Top-K graph generation.

Multimodal graph construction

To capture semantically rich relationships between visual and textual projected embeddings, a semantically sparse cross-modal Top-K multimodal graph, G_MM = (N_MM, E_MM), is constructed. In the Top-K multimodal graph, the nodes N_MM are created by concatenating the projected textual and visual token embeddings for each input sample. Mathematically, it can be represented as follows:

$$N_{MM} = \left\{ {PN_{0}^{v} ,PN_{1}^{v} , \ldots ,PN_{n}^{v} ,PN_{0}^{t} ,PN_{1}^{t} , \ldots ,PN_{l}^{t} } \right\}$$

(5)

Each node N_MM in the constructed graph represents a token-level embedding from either the image or the text modality. The nodes in the graph encode local, token-level contextual and semantic information from their respective modalities, enabling the graph to model fine-grained cross-modal interactions.

To determine the edges E_MM, the study initially computed cosine similarity between the projected and normalized feature vectors of the visual and textual nodes. The cross-modal cosine similarity for the feature vectors of the i_th visual node and the j_th textual node can be expressed mathematically as follows:

$$S_{vt} (i,j) = \cos \left( {PN_{i}^{v} ,PN_{j}^{t} } \right) = \frac{{PN_{i}^{v} \cdot PN_{j}^{t} }}{{\left\| {PN_{i}^{v} } \right\|2 \cdot \left\| {PN_{j}^{t} } \right\|2}},i = 1,2, \ldots ,n,\;j = 1,2, \ldots ,l$$

(6)

where $PN_{i}^{v} .\;PN_{j}^{t}$ represents the inner dot product of feature vectors, and $\left\| {PN_{i}^{v} } \right\|2 \cdot \left\| {PN_{j}^{t} } \right\|2$ is the ℓ₂-normalization.

The cosine similarity is computed exclusively across modalities, resulting in a bipartite similarity matrix S_vt (i, j). In the bipartite similarity matrix S_vt, each entry represents the cosine similarity between the i_th visual node and the j_th textual node, effectively determining their cross-modal semantic relationship. The edges are induced using the reciprocal Top-K neighbors selection strategy. For each image node ${\text{i}} \in \left\{ {{\text{1,}} \ldots {\text{,n}}} \right\}$, directed edges are formed toward its Top-K most similar text nodes, and reciprocally, for each text node ${\text{j}} \in \left\{ {1,{\text{ }} \ldots ,{\text{ }}l} \right\}$, edges are created toward its Top-K most similar image nodes. This bidirectional Top-K graph construction ensures balanced cross-modal interactions while avoiding intra-modal connections and domination by a single modality. The reciprocal Top-K selection process helps to determine cross-modal semantically similar relations, ensuring balanced bidirectional interaction while maintaining sparsity and robust graph construction, even when either modality contains fewer tokens than the K parameter. To accommodate variable token lengths across samples, the effective neighbor counts that adapt to token availability can be mathematically defined as follows:

$$k_{v} = \min (K,l),\;k_{T} = \min (K,n)$$

(7)

Here, k_v and k_t indicate the optimal numbers of neighbors selected by the Top-K operations, which are adapted according to the availability of tokens. The hyperparameter K is the sparsity parameter that controls the total number of neighbors, which is set to 12 in the experiments of this study. To construct a sparse yet semantically meaningful graph, edges are created only among visual and textual nodes (bipartite topology) after determining the Top-K most semantically similar neighbors. For each image token i, directed edges are formed toward its Top-K most similar text tokens. In contrast, each text token j independently selects its Top-K most similar image tokens, which can be defined mathematically as follows:

$$E_{{v \to t}} = \left\{ {\left( {i,j + n} \right)\left| {j \in {\text{TopK}}\left( {S_{{vt}} \left[ {i,:} \right]} \right)} \right.} \right\}$$

(8)

$$E_{{t \to v}} = \left\{ {\left( {j + n,i} \right)\left| {i \in {\text{TopK}}\left( {S_{{vt}} \left[ {:,j} \right]} \right)} \right.} \right\}$$

(9)

where n is corresponding to visual tokens and the indices n, …, n + ℓ-1 represent text tokens in the shared multimodal node space. The final cross-modal edge set can be represented mathematically as follows:

$$E_{MM} = E_{v \to t} \cup E_{t \to v}$$

(10)

This reciprocal graph construction process naturally introduced bidirectional connectivity among cross-modal semantically most similar neighbor nodes, enabling balanced cross-modal information sharing while maintaining graph sparsity and enhancing computational efficiency. Self-loops are added to all nodes in the graph to ensure robust, stable message passing across the GAT layers. Mathematically, this can be defined as follows:

$$E_{{self}} = \left\{ {\left( {u,u} \right)\left| {u = 1, \ldots ,n + l} \right.} \right\}$$

(11)

The final edge set of multimodal used is the union of the cross-modal edges and the self-loop edges, which can be demonstrated mathematically as follows:

$$E_{MM} \leftarrow E_{MM} \cup E_{self}$$

(12)

The multimodal graph construction topology used detached token embeddings to avoid unstable gradients caused by Top-K discrete operations. However, message-passing operations still use the original (non-detached) node features. This graph construction method enables end-to-end training with a stable graph topology while maintaining the differentiability of feature representations. The cosine-similarity-based cross-modal Top-K graph enables semantically aligned connections, retains the most meaningful Top-K multimodal relationships, supports directed message passing, and ensures the stability of self-loops in the GAT layers. The cross-modal graph constructed with this strategy is a sparse, semantically meaningful graph that captures fine-grained cross-modal token interactions while maintaining computational efficiency for multi-head attention GAT layers.

For optimal computation, the above graph-construction approach is applied to each sample in the mini-batch, generating a list of individual graphs [G₁, G₂, …, G_B], where B denotes the batch size. Using the dgl batching operation, the per-sample single graphs are converted into a batched graph, which facilitates efficient parallel computation across all samples within the batch using the proposed GAT framework. In the constructed multimodal graph, nodes represent text tokens and image regions, while edges encode the most salient adaptive cross-modal relationships, which offer a semantically aligned and computationally efficient graph structure for downstream multimodal learning.

Graph attention network

A two-layer Graph Attention Network (GAT) module⁴² is applied to the developed cosine-similarity-based Top-K multimodal graph G_MM = (N_MM, E_MM) to generate graph-level embeddings and learn rich multimodal representations. The GAT layers utilize the self-attention mechanism to compute the hidden features of each node by aggregating neighbourhood information. The GAT layer improves the representation of a node by adjusting its features and collecting the most relevant neighborhood information. A GAT layer takes a set of node features h as input and generates a refined set of features h′ as output. The attention weight e_ij between node i and node j is computed with a learnable linear transformation weight matrix W, which is applied across all nodes. The attention coefficients represent the significance of node j to node i, which is determined as follows:

$$e_{ij} = a\left[ {Wh_{i} \left\| {Wh_{j} } \right.} \right]$$

(13)

where h_i and h_j refer to the feature vectors of nodes i and j, respectively, ∥ refers to the concatenation operation, and W is a learnable weight matrix. The a represents the attention coefficient between nodes i and j. The GAT layer enables each node to attend to every other node, computing e_ij for nodes j ∈ N_i, where N_i refers to the most semantically similar neighbors of node i. To reduce the risk of overfitting during model training, after computing the e_ij, it is activated employing the nonlinear LeakyReLU function and normalized by using the SoftMax function. Mathematically, it can be expressed as follows:

$$a_{ij} = {\text{softmax}}_{j} (e_{ij} ) = \frac{{\exp ({\text{LeakyReLU}} (e_{ij} ))}}{{\sum_{{k \in N_{i} }} {\exp } ({\text{LeakyReLU}} (e_{ik} ))}}$$

(14)

To improve the learning stability of the GAT, the multi-head attention mechanism of the GAT layer is extended and formulated as follows:

$$h_{i}^{\prime } = \mathop {||}\limits_{k = 1}^{K} \sigma \left( {\sum\limits_{{j \in N_{i} }} {a_{ij}^{k} } W^{k} h_{j} } \right)$$

(15)

where $a_{ij}^{k}$  represents the attention coefficient between nodes i and j for head K^th, the W^k is a learnable weight matrix, σ is the activation function Elu, and ∥ refers to concatenation across heads. Using multi-head attention mechanisms, the output features generated by each attention head are concatenated and passed to the subsequent layers. After propagating through the stacked GAT layers, when the multi-head attention mechanism operates in the final layer, the concatenation operation is substituted by an average operation. Mathematically, it can be represented as follows:

$$h^{\prime}_{i} = \sigma \left( {\frac{1}{K}\sum\limits_{k = 1}^{K} {\sum\limits_{{j \in N_{i} }} {a_{ij}^{k} } } W^{k} h_{j} } \right)$$

(16)

The multi-head attention mechanism could be used to explore diverse strength-factor alternatives and enhance training stability. Employing multi-head attention mechanisms, the model can capture complex multimodal interactions by integrating data from multiple, diverse sources. This study used two GAT layers, each with eight heads, to classify lung diseases. The first layer uses concatenation, whereas the second layer averages information from multiple heads. Hence, the output of the first GAT layer is concatenated from K = 8 attention heads, each with 64-dimensional embedding nodes. The second GAT layer outputs from the K = 8 attention heads are averaged along with the head dimensions, yielding a 256-dimensional embedding per node.

Graph-level pooling and classification head

Graph-level pooling operation

Following the acquisition of updated node-level features via the GAT multi-head attention module, the study applied global max node pooling operations for graph-level representation, which can be represented mathematically as follows:

$$h_{GMM} = \mathop {\max }\limits_{i \in N} h_{i}^{\prime }$$

(17)

where $h_{i}^{\prime }$ is the output feature vector of node i, N is the total number of nodes in the graph, and h_GMM is the global max-pooled graph embedding. This graph-level max pooling over all node embeddings captures the most salient cross-modal features across the graph, generating a highly effective and discriminantive graph-level representation for the downstream classification.

Classification head

As the final stage of the proposed GATMM architecture, the study employs a fully connected classification head for the classification of lung diseases. The learned multimodal graph-level representation h_GMM, which encapsulates semantic information derived from both text and image modalities, is used as input to this module. The classification head consists of two linear layers. For generalization and training stability, the first linear layer maps the input features to a 128-dimensional latent space, followed by a ReLU activation, a Dropout layer, and Layer Normalization. In the second linear layer, the intermediate representation is projected into the output space with dimensions equal to the number of target classes (in our case, eight classes are used for classifying multiclass lung diseases). A SoftMax function is used on raw logits to transform them into class probabilities.

$$p_{i} = \frac{{\exp (z_{i} )}}{{\sum_{k = 1}^{C} {\exp } (z_{k} )}},{\text{ for }}i = 1,2,3, \ldots ,C$$

(18)

where z refers to the raw logits predicted by the method for the i^th instance, C represents the total number of classes, and pi is the predicted probabilities after computing the SoftMax method.

During training, the proposed multimodal architecture is trained by reducing the standard multiclass cross-entropy loss. For a single sample, the multiclass cross-entropy loss can be formulated mathematically as follows:

$$L(y,p) = - \sum\limits_{i = 1}^{C} {y_{i} } \cdot \log (p_{i} )$$

(19)

where C refers to the number of classes, y_i represents the one-hot encoded of the true label, and p_i denotes the predicted probability for class i calculated from the SoftMax activation function applied to the raw logits z of the model.

In multimodal representation learning, modelling complex inter-modal relationships remains a significant challenge. This study proposes a cosine-similarity-based Top-K multimodal graph and a GAT multi-head attention framework that selectively preserves the top-most semantically meaningful edges between visual and textual nodes, thus reducing noise and redundancy associated with fully connected or static-threshold graphs. The Top-K sparsity of the graph reduces the number of parameters and mitigates human bias by eliminating manual coding and arbitrary graph construction. The GAT multi-head attention module efficiently weights the contributions of neighboring nodes in the semantically reciprocal Top-K sparse multimodal graph. The proposed method aims to learn robust and semantically rich multimodal representations to improve the performance of lung disease diagnosis.

Evaluation metrics

The performance evaluation metrics, such as AUROC, Area Under the Precision-Recall Curve (AUPRC), accuracy, loss, precision, recall, and F1-score, are used to assess the effectiveness of the proposed approach. By considering these performance evaluation indicators, it is possible to determine the model’s potential to identify and classify lung diseases. These evaluation metrics assess the classification performance of a model across all classes. The TN illustrates true negatives, TP indicates true positives, FN denotes false negatives, and FP represents false positives.

$${\text{Accuracy = }}\frac{{\text{TP + TN}}}{{\text{TP + FN + TN + FP}}}$$

(20)

$${\text{Precision}} = \frac{{{\text{TP}}}}{{{\text{TP}} + {\text{FP}}}}$$

(21)

$${\text{Recall}} = \frac{{{\text{TP}}}}{{{\text{TP}} + {\text{FN}}}}$$

(22)

$${\text{F1}} - {\text{score}} = \frac{{{\text{Recall}} \times {\text{Precision}}}}{{{\text{Recall}} + {\text{Precision}}}}$$

(23)

Experimental setup and results analysis

This section presents a comprehensive description of the experimental setup and the analysis of results obtained from various experiments conducted in this research for the multiclass lung disease classification task.

Experimental setup

The effectiveness of the developed GATMM method is assessed using a publicly available image-text dataset named “Lung X-Ray Image + Clinical Text Dataset,” which is one of the most frequently downloaded multimodal datasets for the classification of lung diseases. The data is split into three distinct subsets to conduct experiments: the training set is used for training the model, the validation set for validation, and the test set is kept unseen during training and validation and used for testing the model. The hyperparameters of the proposed multimodal approach are adjusted based on the average loss. Cross-entropy loss is employed as a loss function in the experiments performed in this study. The AdamW is applied as an optimizer with weight decay and a multiplicative factor of the learning rate. To minimize the risk of overfitting, the proposed approach is implemented with an early stop and L2 regularization. Pre-trained RAD-DINO is used as a vision encoder, and pre-trained Clinical ModernBERT is used as a textual encoder. The multimodal graph is constructed and passed to the GAT module to enhance cross-modal interaction. For development and experimental purposes, Python 3.9, PyTorch 2.2, DGL graph library, Anaconda 3, TensorFlow 2.10, and CUDA 12.6 are installed on a Windows 11 computer with an Intel(R) Core (TM) i7-10700 K 3.8 GHz processor, NVIDIA RTX 2080 Ti GPU 11 GB, and 32 GB of RAM. All model implementations are carried out using the PyTorch library. The Concatenation-based Multimodal (CMM) fusion approach contains approximately 224.0164 million parameters, the Cross-model Multi-head Attention-based Multimodal (CMAMM) has about 225.067 million parameters, the Graph Convolution Network-based Multimodal (GCNMM) contains approximately 224.378 million parameters, and the proposed GATMM has about 226.384 million parameters. The inference time during testing for all 12,000 test instances is approximately 350–460 s for all models. Table 2 presents the hyperparameter settings used for experiments and model training in this study.

Table 2 The hyperparameter setting employed in the experiments of this study.

Experimental results analysis

Detailed descriptions of the experimental results for the proposed approach are provided in the following subsections. As performance evaluation metrics, the confusion matrix, loss, accuracy, precision, recall, F1-score, AUROC, and AUPRC are considered. Each evaluation metric is described in detail, accompanied by a visual representation, in the following sections.

Confusion matrix

In this subsection, the study provides a comparative analysis of the main multimodal fusion strategies using confusion matrices to evaluate their performance on the test dataset. For the classification of eight lung disease classes, each fusion approach reveals distinctive characteristics in terms of accuracy and error rates. Among 5688 test instances, 5303 are correctly classified by the CMM fusion approach, whereas 385 are incorrectly classified. The CMAMM fusion approach correctly classified 5383 samples and incorrectly classified 305 samples out of the total 5688 test samples. The GCNMM fusion method correctly identified 5394 cases while misclassifying 294 samples from 5688 test samples. On the other hand, out of 5688 test instances, the proposed GATMM method correctly classified 5445 instances while misclassifying only 243 instances. The proposed strategy reduces false negative and false positive rates and outperforms other competing multimodal strategies, indicating its reliability and robustness. The confusion matrices show that classes (0, 5, and 6) are classified with high confidence, while classes (4 and 7) exhibit a high misclassification rate due to confusion with classes (2 and 3). Figure 2 shows the comparison of the confusion matrix for the developed GATMM strategy with other comparative multimodal fusion strategies employed for the classification of multiclass lung diseases in this research study.

Fig. 2

The comparison of the confusion matrices of all multimodal strategies.

Training and validation loss, accuracy curve

Training and validation curves for all multimodal fusion strategies are discussed in this subsection. These curves represent the learning and generalization power of a method. The research evaluates various multimodal fusion strategies, including CMM, CMAMM, GCNMM, and GATMM, for the classification of lung diseases. The CMM approach achieved training and validation accuracies of 0.9116 and 0.9313, respectively, with training and validation losses of 0.3745 and 0.2684. The CMAMM method got a training accuracy of 0.9243 and a validation accuracy of 0.9442, along with a training loss of 0.3276 and a validation loss of 0.2355. For the GCNMM method, the training accuracy is 0.9264, and the loss is 0.3220, whereas the validation accuracy is 0.9474, and the loss is 0.2282. In contrast, the proposed GATMM framework achieved a training accuracy of 0.9403 and a loss of 0.2227, as well as a validation accuracy of 0.9553 and a loss of 0.1373.

The evaluation findings indicate that the GATMM approach performed much better during training and validation. It illustrates the effectiveness of the proposed strategy for representing the complicated inter- and intra-modal interactions. The closely aligned and minimal variation between the training and validation curves of the GATMM demonstrate its generalization potential. Furthermore, the training and validation loss curves of the proposed approach remain constant and decrease gradually in both phases, demonstrating that no overfitting has occurred. According to this comparison of training and validation curves, the GATMM strategy outperformed the other methods and retained excellent generalization to unseen instances. Figure 3 shows the training and validation curves for loss and accuracy of all multimodal strategies evaluated in this research study.

Fig. 3

The training and validation loss and accuracy curves for all multimodal strategies.

Prediction accuracy, precision, recall, and F1-score

The evaluation of experimental results and comparative performance analyses is presented in Table 3. According to Table 3, the experimental findings demonstrate impressive improvements in prediction performance and efficiency for different multimodal fusion approaches for lung disease classification. It also shows the implications of architectural design choices on multimodal efficiency and performance. The CMM architecture achieved an accuracy of 93.23 ± 0.65%, a precision of 93.44 ± 0.64%, a recall of 93.21 ± 0.65%, and an F1-score of 93.23 ± 0.65%. The CMAMM framework obtained an accuracy of 94.64 ± 0.59%, a precision of 94.65 ± 0.58%, a recall of 94.2 ± 0.59%, and an F1-score of 94.62 ± 0.59%. The GCNMM method gained 94.83 ± 0.58% accuracy, 94.81 ± 0.58% precision, 94.81 ± 0.58% recall, and 94.79 ± 0.58% F1 score. The proposed GATMM method outperformed other traditional techniques for the classification of lung diseases, with an accuracy of 95.73 ± 0.53%, a precision of 95.75 ± 0.52%, a recall of 95.72 ± 0.53%, and an F1 score of 95.70 ± 0.53%. The superior performance of the GATMM model could lead to a considerable reduction in misdiagnosis. The proposed automated multimodal approach could serve as a robust and reliable alternative clinical tool for the diagnosis of lung diseases. In the clinical setting, high precision is required because false negatives have a detrimental impact on patient outcomes and can cause delays in the treatment process.

Table 3 Prediction accuracy, precision, recall, and F1-score of all multimodal architectures. Significant values are in [bold].

Receiver operating characteristic curve and precision–recall curve analysis

The AUROC and AUPRC are used to evaluate the performance of the proposed multimodal approach. The AUROC graph describes the correlation between the True Positive Rate (TPR) and the False Positive Rate (FPR), while the AUPRC curve summarizes the trade-offs between precision and recall. These metrics provide the most meaningful insights for the performance evaluation of a classification method⁴³. The AUROC and AUPRC for the multimodal approaches used in this study are shown in Figure 4. The multimodal frameworks, such as CMM, CMAMM, and GCNMM, obtained AUROCs of 99.56 ± 0.17%, 99.64 ± 0.16%, and 99.70 ± 0.14%, while the AUPRCs of 98.03 ± 0.36%, 98.53 ± 0.31%, and 98.73 ± 0.29%, respectively. Conversely, the GATMM approach achieved the highest AUROC of 99.79 ± 0.12% and AUPRC of 99.06 ± 0.25%, indicating its reliability in distinguishing among various lung conditions.

Fig. 4

Comparison of AUROC (left) and AUPRC (right) for all multimodal architectures.

Additionally, the study calculated per-class AUROC and AUPRC for the GATMM approach. Per-class AUROC and AUPRC provide a more comprehensive and clinically significant assessment of the prediction model, revealing how well it distinguishes each disease category from all others. Figure 5 shows the proposed GATMM approach per-class AUROC and AUPRC. Class 6 achieved the highest AUROC of 100.00 ± 0.01%, whereas class 7 had the lowest AUROC of 99.35 ± 0.21%. Furthermore, class 6 also achieved the highest AUPRC of 99.98 ± 0.04%, and class 7 recorded the lowest AUPRC of 97.40 ± 0.41% for the proposed GATMM approach. According to the findings, the GATMM approach achieved optimal AUROC and AUPRC for each class, indicating its reliability in distinguishing among different classes.

Fig. 5

Proposed GATMM framework per-class AUROCs (left) and AUPRCs (right) visualization.

The interpretability and error analysis

For the predictive transparency of the proposed GATMM framework, the study employed gradient-based feature attribution on projected image and text embeddings to examine the influence of each feature on predictions. To highlight regions and influential clinical terms involved in the prediction, pitch-level heatmaps for the CXR image and ranked saliency scores for the clinical text are generated. This highlighting approach for image regions and influential terms in clinical text provides reliable, consistent representations for qualitative error analysis. Figure 6 illustrates the visual attributions, salient clinical tokens, and both ground-truth and predicted labels for the example test instances.

Fig. 6

Representative qualitative examples with visual and textual explanations.

Probabilistic calibration and predictive uncertainty analysis

The reliability and uncertainty characteristics of the proposed GATMM model are evaluated on the test dataset. The probabilistic calibration of the proposed model is evaluated using the Expected Calibration Error (ECE), the Negative Log-Likelihood (NLL), and the multiclass Brier score. According to Table 4, the proposed multimodal approach achieved an ECE of 0.0124, an NLL of 0.1431, and a Brier score of 0.0656, which indicates meaningful probability estimates with minimal calibration error. In addition, the epistemic uncertainty is determined by applying the Monte Carlo Dropout with 20 forward iterations. The proposed model achieved a mean predictive entropy of 0.4612, an epistemic variance of 0.0025, and a mean mutual information of 0.0540. The mutual information for misclassified cases is often higher than for correctly classified cases (0.2009 vs. 0.0481). These empirical findings show that the proposed approach effectively identifies error-prone predictions while producing robust and ambiguity-aware outputs.

Table 4 The probabilistic calibration and predictive uncertainty analysis of the proposed GATMM approach on the test dataset.

In addition, Fig. 7 shows the visual representations of reliability, predictive entropy, epistemic variance, and mutual information for the proposed approach. The reliability curve shows a close correlation between the predicted confidence and the empirical accuracy across the confidence interval. On the other hand, the distributions of predictive entropy, epistemic variance, and mutual information highlight cases that are uncertain or misclassified. These results jointly show that the proposed model provides more reliable probability estimates and clinically useful uncertainty measures that could help to enhance transparency in confidence analysis and improve the reliability of automatic clinical decision-making systems.

Fig. 7

The probabilistic calibration and uncertainty analysis for the proposed GATMM model (a) Reliability diagram. (b) Predictive entropy. (c) Epistemic variance. (d) Mutual information.

Ablation study

It is crucial to perform an ablation study to evaluate the impacts of different components on the performance of the proposed approach. In this study, a series of ablation experiments is conducted to validate the effectiveness of the proposed GATMM architecture. The experiments for the ablation study are performed across multiple variants of the proposed approach, including the number of GAT layers, graph-level global pooling techniques, graph creation strategies (cosine-similarity-based and Top-K), and the Top-K values. Initially, the proposed GATMM backbone is evaluated using varying numbers of GAT layers. Next, the performance of the proposed approach is evaluated using different graph-level global pooling techniques (attention pooling, mean pooling, and max pooling). Afterward, the performance of GATMM is evaluated for graph construction topologies, including the cosine-similarity-based graph and the Top-K graph. Finally, the proposed framework is evaluated for various Top-K values. The findings demonstrate that the proposed GATMM architecture, which employs two layers of GAT (each with 8 heads), global max-pooling at the graph-level, and a cosine-similarity-based Top-K graph with K = 12, improved classification performance. These findings confirm that the number of GAT layers, the Top-K value, the global pooling strategy at the graph level, and cosine-similarity-based Top-K graph construction are critical factors for improving the performance of the proposed GATMM framework in the analysis of lung diseases. Table 5 presents the ablation study results conducted for the proposed approach.

Table 5 Ablation study results for GATMM-based multiclass lung disease classification (the bold means best).

Proposed model generalization test

To evaluate the generalization and robustness of the proposed approach, the study performed generalization test experiments using a single-label ten-class lung-related subset collected from the most frequently downloaded publicly accessible MIMIC-CXR-JPG dataset⁴⁴. The collected subset comprises 45,358 CXR images and corresponding findings sections from clinical reports for single-label lung diseases, with an imbalanced class distribution, various CXR image views, and diverse disease classes, which makes it different from the primary public dataset used in this study. These sorts of variations usually impose considerable generalization challenges for a medical image classification model. The experiments are conducted on the subset dataset, using the same preprocessing, training, validation, and evaluation procedures as the primary dataset. However, the dataset is split on the patient-based approach and uses a weighted cross-entropy loss. The diseases included in the subset are Atelectasis, Consolidation, Edema, Lung Lesion, Lung Opacity, Pleural Effusion, Pleural Other, Pneumonia, Pneumothorax, and No Finding. The down-sampling technique, which uses random state 42, is employed to randomly select No Finding cases from all No Finding samples. Table 6 shows the dataset distribution, splits, and number of instances for each class.

Table 6 Subset dataset distribution, splits, and per-class counts of samples.

The proposed method achieved an accuracy of 80.52%, precision of 80.19%, recall of 80.13%, and F1-score of 80.52%. Also, the GATMM framework achieved an average AUROC of 95.88% and an AUPRC of 86.67%, with ECE of 0.0763, NLL of 0.6675, and Brier score of 0.3109 for the subset. The decrease in performance is expected because accuracy is heavily influenced by class imbalance and differences in disease prevalence across the dataset. However, the higher AUROC and AUPRC scores indicate that the proposed GATMM approach retains strong discriminative power even when distributions shift and class imbalance is present. The higher values of AUROC and AUPRC suggest that the model maintains the high predictive performance across diseases, which is crucial for disease screening. The performance of the GATMM approach on the external Mimic subset is lower compared to the primary dataset, but it exhibits excellent and clinically useful generalization, with high AUROC and AUPRC values despite significant distribution change and class imbalance. Figure 8 shows a visual representation of the GATMM model performance achieved on the single-label subset of the MIMIC-CXR-JPG dataset.

Fig. 8

GATMM performance achieved on the single-label subset dataset.

Discussion

The proposed study leverages the complementary strengths of pre-trained domain-specific models, such as RAD-DINO and Clinical ModernBERT, to extract visual and textual features from CXR images and clinical texts, respectively. In the proposed approach for capturing the inter-modal interactions at the token level and aligning textual and visual features, semantically sparse Top-K multimodal graphs and the GAT layers are used. A comprehensive comparison of various fusion techniques and performance metrics demonstrated the effectiveness of the suggested multimodal approach. Experimental results reveal that the proposed GATMM framework performed remarkably well for multiclass lung disease classification. The developed GATMM method achieved the highest prediction accuracy of 95.73 ± 0.53%, precision of 95.75 ± 0.52%, recall of 95.72 ± 0.53%, F1 score of 95.70 ± 0.53%, AUROC of 99.79 ± 0.12%, and AUPRC of 99.06 ± 0.25% for the classification of multiclass lung diseases. On the other hand, the traditional multimodal methods, such as CMM, CMAMM, and GCNMM, achieved performance ranging from 93.23 ± 0.65% to 94.83 ± 0.58% across all evaluation metrics. The GATMM approach achieved optimal performance as compared to the other methods, indicating its robustness and effective generalization.

Furthermore, compared with conventional fusion methods, the GATMM approach leverages multimodal graph structures and multi-head node-wise attention mechanisms to capture complex relationships between visual and textual modalities, enabling important inter-modal representations. Thus, multimodal features are better aligned, represented, and classified. The proposed method ensures the smooth and effective integration of heterogeneous data sources, providing a multimodal framework that serves as a powerful tool for automating the analysis of lung diseases. Figure 9 compares the proposed GATMM approach with other fusion approaches used in this research study.

Fig. 9

Comparison of the proposed GATMM approach with other fusion approaches used in this study.

A comparative analysis of the proposed multimodal approach with other related state-of-the-art approaches is presented in Table 7. According to Table 7, the previous research reported average accuracy, precision, recall, and F1-score values ranging from 77 to 95.59%. However, the proposed GATMM approach outperforms these traditional multimodal integration techniques, achieving a classification performance of about 95.73 ± 0.53% for multiclass lung diseases. Furthermore, the integration of frozen-pretrained domain-specific RAD-DINO for visual feature extraction and Clinical ModernBERT for textual feature representation provides substantial benefits by leveraging rich semantic representations while reducing the computational resources required for model fine-tuning and training from scratch. The proposed approach could also help to minimize the risk of overfitting when training large models from scratch on limited medical datasets. The empirical findings of the proposed approach validate the effectiveness of the GATMM framework in the medical domain. Combined with the advantages of transfer learning and graph learning, the developed framework presents a promising solution to advance automated lung disease diagnosis and other multimodal healthcare AI challenges by representing complex cross-modal relationships between heterogeneous data sources.

Table 7 Comparison of the proposed approach with other related research studies. Significant values are in [bold].

Limitations and challenges

The GATMM framework is presented in this study, which obtained competitive performance on the image-text dataset, making significant contributions to the automated classification of lung diseases and establishing a solid foundation for future research in this area. However, the model is evaluated on public and curated datasets that may not reflect the true complexity and variation of medical data. The clinical texts in the main dataset are disease-related symptoms collected from public datasets and the literature, and may not reflect the actual complexity of clinical reports. So, the generalizability analysis of the framework requires further validation through cross-institutional studies and prospective clinical trials. It is also necessary to optimize the computational complexity of the model to adapt it for real-time use. As part of the proposed framework, future work will focus on expanding its application to additional modalities as well as on improving its interpretability to enhance clinical trust.

Conclusion

The study presented a GATMM framework that integrates pre-trained domain-specific RAD-DINO and Clinical ModernBERT models with the GAT module for the classification of lung diseases. A publicly available multimodal CXR and clinical text dataset is used to evaluate the efficiency of the developed framework. The proposed approach addresses the limitations of previous approaches, such as the heterogeneity gap in multimodal medical data and existing solutions that used complex fusion techniques to integrate the different modalities, thereby increasing the risk of overfitting. Unlike previous approaches, our approach represents multimodal data in the form of a graph, with image and clinical features represented as nodes and edges, allowing fine-grained inter-modal interactions. For eight classes of lung diseases, the proposed GATMM approach achieved 95.73 ± 0.53% accuracy and 99.79 ± 0.12% AUROC. The results show that the proposed GATMM outperformed the traditional fusion methods across various performance evaluation metrics. The proposed architecture offers an efficient multimodal solution for automating the diagnosis of lung diseases, potentially leading to enhanced patient outcomes and reduced missed diagnoses.

In future research, the capabilities and clinical relevance of the GATMM approach will be examined and enhanced. The proposed approach will be evaluated on the most famous and clinically relevant CXR and clinical text datasets, such as MIMIC-CXR-JPG and CheXpert Plus, to rigorously validate and establish its robustness and generalizability. Furthermore, the proposed method will be improved to integrate multimodal data sources, including laboratory results, audio data, and temporal clinical information, which is crucial for capturing a wide range of patient diagnostic information. Future research should integrate advanced explainable AI tools to improve the reliability and explainability of the proposed approach.