Intra-scale interaction and cross-scale fusion network for detecting the progression of neurodegeneration in Alzheimer’s disease

Abstract

Alzheimer’s disease (AD), one of the most widespread neurodegenerative disorders, can be mitigated through early recognition and treatment. Recent research has shown that multimodal fusion is effective for the early-stage diagnosis of AD. However, most existing methods do not adequately account for the differences in data modality domains, their interconnections, and relative relevance.In this paper, we introduce a robust Intra-scale Interaction and Cross-scale Fusion Network (ISI-CSFN) for AD progression detection. The proposed model employs a linearized convolutional attention module to enable interaction between global information captured by the Cascaded Transformer (CTransformer) and local features extracted by the Depthwise Separable Convolution Network (DSCN). This mechanism enhances the discriminative ability for AD progression detection by allowing each modality-specific branch to incorporate complementary contextual representations from the others while maintaining the integrity of its own features.Furthermore, the model integrates background (BG) information with multimodal temporal data for simultaneous prediction of several cognitive score variables. The proposed method achieves strong results for both regression and multi-class progression tasks. Specifically, the accuracy of our approach is 97.26% for NC vs. AD, 89.25% for NC vs. sMCI, and 84.74% for NC vs. pMCI classifications. In addition, the model attains the highest correlation coefficient and the lowest root mean square error (RMSE) across several clinical score regression tasks.

Introduction

Alzheimer’s disease (AD) is the most common neurodegenerative disorder, influencing nearly 60% of individuals over the age of 60¹. Neurodegeneration in sporadic AD is defined by gradual brain atrophy and anatomical abnormalities². AD is portrayed by both movement-related and systemic manifestations because of the progressive death of Neurons involved in dopamine transmission in different areas of the brain³. Even though there is no known cure for AD, cognitive damage can be postponed in the early stages. To ensure a higher standard of living of patients and to improve prevention and treatment, it is very important to identify AD in its early stages and identify its pathological biological markers⁴.

Mild cognitive impairment (MCI) is a transitional stage between cognitively normal (CN) state and AD⁵. MCI is diagnosed based on memory impairment on standard tests when there is no discernible impairment in everyday life activities or dementia. According to severity, MCI can be categorized as either progressing (pMCI) or stable (sMCI)^6,7. A number of machine learning techniques are currently being used for diagnosing AD and identifying relevant disease biomarkers based on neuroimaging data from MRI, fMRI, PET, CT, EEG, SPECT, and MEG^8,9,10,11,12. The positron emission tomography (PET) and magnetic resonance imaging (MRI) have been extensively used in the literature to identify AD and cognitive decline^13,14. However, in the early stages, even when indicators are clearly visible on MRI and/or PET data, it is still difficult to detect progression from MCI to AD dementia in clinical practice¹⁵.

The number of research on automated MCI conversion prediction methods has increased in recent years. Some focused on short-term prognosis, although it provided encouraging outcomes, was of little therapeutic importance because it was linked to subsequent treatments and it could not repair the neurological damage. Some conventional machine learning (ML)-based techniques in long-term predictive studies required complex feature engineering, which led to the exclusion of significant pathological signals^16,17. These drawbacks can be addressed by end-to-end deep neural network (DNN) techniques, which also have a lot of potential for clinical decision assistance.

Deep learning-based models outperform classical models in AD detection tasks because of their remarkable nonlinear mapping capacity, which allows them to capture intricate texture features and fine image details¹⁸. For example, high-level abstract characteristics are extracted from multimodal or longitudinal neuroimaging data using convolutional neural networks (CNNs), stacked autoencoders (SAEs), and deep belief networks (DBNs)^19,20,21. These models facilitate the identification of pathological changes by generating feature vectors or differential representations between time points or systems. The generated features are then added to clustering or classification structures to diagnosis Alzheimer’s disease. However, even though the aforementioned DNNs are fully utilized, models that rely only on single biomarkers are not sufficient to predict MCI change²².

Furthermore, single-task models cannot provide clinical professionals with relevant information about the patient’s potential cognitive behavior during progression. According to some research, estimating MCI development and some cognitive tests, such as the Mini-Mental State Examination (MMSE) and the Alzheimer’s Disease Rating Scale (ADS), are the signs of progression of AD²³. Recent research on multimodality data has gathered a lot of interest since it can effectively overcome the drawbacks of a unimodal data, increasing the precision of prediction models used in disease detection²⁴. Multi-level features extracted from MRI and PET images, as well as features that differ between the two modalities, are combined using an attention mechanism in multimodal data analysis. This is then utilized for feature reconstruction related to disease^25,26. However, when attention is paid to reweighting the linked data across channel or spatial dimensions, it is still difficult to capture long-range relationships across methods. Combining local and global features allows deep learning models to capture fine-grained, region-specific changes (local features) as well as overall brain-level patterns (global features), which is critical for accurately detecting early-stage Alzheimer’s disease and differentiating it from normal aging²⁷.

Even though earlier methods have shown encouraging outcomes, diagnosing AD with multimodality data still presents several difficulties. One major issue is that these methods frequently ignore interactions across multimodal data scales in favour of concentrating on interactions between Multi-modal inputs at one resolution. These arguments encourage us to suggest a new fusion network for the identification AD. The major contributions of this paper are provided as:

• To propose a new dual-backbone framework that combines Depthwise Separable Convolution Networks (DSCN) and Cascaded Transformers (CTransformer) for efficiently extracting complementing features from MRI and PET images.

• To suggest a novel Intra-Scale and Cross-Scale Fusion Mechanism that facilitates the efficient integration of multi-scale data and rich interactions between modality-specific aspects at the same scale. This could improve the discriminatory power of combined traits for detecting the progression of AD.

• To integrate the FreeSurfer-derived anatomical characteristics and baseline demographic data into a single fully integrated architecture for boosting the performance across classification and regression tasks linked to AD development.

The rest of the paper is structured as follows: Sect. 2 reviews the existing papers on AD progression detection. Section 3 details about the proposed method. Section 4 validates the proposed approach using simulations. Section 5 provides the conclusion.

Literature survey

The effectiveness of AD prediction models with missing value imputation was influenced by the manually or statistically calculated static values such as zeros and means. Some of the missing value imputation are Mean imputation with LSTM (LSTM-M), Forward imputation with LSTM (LSTM-F), Zero imputation with Peephole LSTM (PLSTM-Z), Multi-directional RNN (MRNN), and Model filling with MinimalRNN (MinimalRNN)^28,29,30. In Alzheimer’s data, these models frequently introduced bias or skew temporal dynamics. Sequence modeling was improved by more complicated models like MRNN and MinimalRNN, however they may have limited representation power, excessive complexity, or a danger of data leaking.

A new computational approach was presented by Jung et al.³¹ that can forecast cognitive scores, disease progression pathways, and MRI phenotypic biomarkers at several future time points. However, this usually encountered a large number of unexpected observations when using temporal data. These missing values ​​can be identified by calculating temporal and multi-variable correlations in the time series data. To address the problems of missing value imputation, phenotypic measurement forecasting, cognitive score progression modeling, and patient health status forecasting based on time-series imaging markers, a deep recurrent network (DRN) was specifically designed. Particularly, well-defined loss function was used to train the hyperparameters of every models using the morphological features and cognitive scores as input.

Zuo et al.³² suggested a unique model called the cross-modal transformer generative adversarial network (CT-GAN) to successfully combine the structural and functional information found in diffusion tensor imaging (DTI) and functional MRI imaging (fMRI). From multimodal imaging data, the CT-GAN efficiently learned topological characteristics and provided multimodal connectivity from start to finish. Additionally, the shifting bi-attention mechanism was created to successfully improve the complementing features between modalities and gradually align shared features. The CT-GAN model detected AD-related brain connections by examining the produced connectivity features. But data modality types, their linkages, and differences in their relative relevance have not been taken into account.

Lu et al.³³ have illustrated the importance of multimodal fusion in early AD identification by presenting a Hierarchical Attention-Based Multimodal Fusion framework (HAMF). It used clinical, genetic, and imaging data to detect AD early. Additionally, it made use of deep neural networks, stacked denoising autoencoders, and CNNs. Through hierarchical attention, the HAMF framework used attention strategies to acquire the proper weightage for each data source and comprehend how modalities interact. Sheng et al.³⁴ suggested a multimodal ML architecture for improved AD characterisation by fusing cerebrospinal fluid (CSF), PET, and MRI. For concurrent feature selection and categorization, this model utilized a hybrid method that combined the Kernel Extreme Learning Machine (KELM) classifier with an improved Harris Hawks Optimization (HHO) algorithm known as ILHHO.

Abdelaziz et al.³⁵ created a multimodal and multi-scale DL (MMSDL) that successfully takes advantage of the relationship between the multi-source and multi-resolution features of the neurological imaging data. Initially, they embedded every resolution level of the multimodal images using a CNN. Then both multi-head self-attention and multi-head cross-attention were used in a multimodal scale fusion process. It improved scale-aware feature extraction and correlation of MRI and PET images by capturing global relations between the embedding features and weighing the contribution of each modality to another. After that a cross-modality fusion module was presented for promoting global features from the earlier attention layers and merge MRI and PET image at different scales. In order to distinguish between the different stages of AD, all of the elements from each scale were finally combined. When compared to other traditional deep learning models such as ResNet, ResNext, MobileNet, ShuffleNetv2, and EfficientNet, this MMSDL produced the greatest results.

Recent progress on Alzheimer’s disease (AD) diagnosis has explored several deep learning architectures integrating multimodal data to enhance efficiency and accuracy. The AD-Transformer³⁶ utilizes a single transformer-based architecture that encodes sMRI, clinical, and genomic data through a Patch-CNN for image tokenization and a linear projection layer for non-image data to realize state-of-the-art AUC scores on prediction of AD and MCI. Alternatively, through the integration of multimodal MRI and PET information through middle-fusion with cross-attention mechanisms, light-weight cortical surface-based models³⁷ provide computationally efficient substitutes for traditional volumetric methods.Through integrating cross-sectional and temporal data between MRI, PET, and CT information through attention-driven feature fusion, lightweight topologies, and GAN-based data augmentation, FusionNet³⁸ significantly enhances the accuracy of early AD diagnosis. EffiSwin-MCI³⁹ merges EfficientNet and Swin Transformer with sliding-window spatial and temporal attention for discovering progressive neurodegenerative changes from longitudinal sMRI scans, surpassing current CNN and Transformer models to deal with temporal dynamics.

According to this analysis, current models for AD detection have achieved great advances in multimodal fusion (e.g., CT-GAN, HAMF, MMSDL) and handling of missing data (e.g., DRN, MRNN, MinimalRNN). However, they still have serious drawbacks, such as inadequate cross-scale and cross-modal interaction, a lack of specialized backbone architectures for modality-specific strengths, and end-to-end fusion insufficiencies. The majority of existing fusion frameworks (such as CT-GAN, HAMF, and MMSDL) do not specifically represent intra- and inter-scale interactions between fine-grained and global data across MRI and PET modalities. Techniques such as DRNs and LSTM-based imputation models can analyze time-series data, but they are not able to fully utilize spatially detailed neuroimaging information, especially functional patterns from low-resolution PET and local anatomical structures from high-resolution MRI. In general, existing methods do not take advantage of modality-specific network architectures, lack strong cross-scale and intra-modality interactions, and frequently do not include domain information from pre-processing tools such as FreeSurfer. These drawbacks highlight the requirement for more context-aware, scalable, and specialized frameworks. The proposed ISI-CSFN tackle these issues, through dedicated intra- and inter-scale feature modeling, modality-tailored backbones, and unified end-to-end learning.

Proposed method

In general, neurodegeneration is the progressive loss of a neuron’s structure or function. It occurs in a variety of illnesses, including Parkinson’s and Alzheimer’s. Among neurodegenerative diseases, AD is the most prevalent. Additional information about the brain is provided by multimodal neuroimaging data, such as PET and MRI. In this paper, a novel Intra-scale Interaction and Cross-scale Fusion Network (ISI-CSFN) is proposed for AD progression detection. First, the ADNI 1, ADNI GO, and ADNI 2 datasets are used to access the data. Next, the FreeSurfer is utilized for extracting the neuroimaging features of the MRI and PET modalities. Additionally, two sub-networks including Depthwise separable convolution network (DSCN) and cascaded Transformers (CTransformer) are used to concurrently extract fine-grained local features and global features from pre-processed MRI and PET, respectively as shoen in Fig:1. DSCNs effectively capture local spatial patterns through convolutional filters, so they are suitable for MRI images. CTransformers model long-range dependencies over the entire brain without depending on local structure, so they are more appropriate for PET imaging.

Fig. 1

Workflow of multimodal data processing and feature fusion for cognitive score and disease stage prediction.

Furthermore, an intra-scale cross-interaction and an inter-scale feature fusion module are introduced for successfully fusing multiple scale feature maps mined from the two backbones. It can fuse feature maps of various scales and impose interaction of various feature maps at the same scale. Finally, the feature maps are combined with the baseline background data and the features extracted by FreeSurfer. The classification task is then handled by the Softmax, and the regression task by the Sigmoid as in Fig. 2.

Fig. 2

Proposed network structure for Alzheimer’s disease classification and regression.

Data pre-processing

Initially, the input MRI and PET images are pre-processed using FreeSurfer to extract neuroimaging features. FreeSurfer is a neuroimaging tool that divides the brain into anatomical regions (such as cortical and subcortical ROIs). Morphometric features such as cortical thickness, surface area, volume, depth, and so on will be retrieved for each cortical and subcortical ROI. The collected features and baseline data are then pre-processed to remove missing data and standardize the data.

Missing data handling: The features retrieved from freesurfer and bassline data are numerical and unrestricted. Hence, the missing values are processed appropriately. For the baseline static data, features with excess of 35% missing elements was removed. Next, the k-nearest neighbors (KNN) algorithm is utilized for imputing missing values⁴⁰, which were then substituted with data from other subjects with the same diagnosis.

Data standardization: The accessible participant data for the baseline and time series are of varied magnitudes. It is difficult to achieve convergence when training an ML model using this data. The z-score method was utilized to standardize features in the data so that they all had the same amount of relevance. The z-score approach reduces outliers by converting data to a 0 mean and unit standard deviation.

Intra-scale interaction and cross-scale fusion network (ISI-CSFN)

This section introduces ISI-CSFN framework that primarily composed of the three steps as shown in Fig. 3. Initially, two sub-networks are used to extract highly detailed local features and label-level global features in parallel: the depth-wise separable convolution (DSCN) network and the cascaded transformer (CTransformer). DSCN and C-Transformer are used for MRI and PET feature extraction, respectively, because each modality has distinct characteristics. MRI data benefit from DSCN’s ability to capture local spatial patterns, while PET data, being more functional and temporal in nature, are better represented using C-Transformer, which can model long-range dependencies effectively. Using a single architecture for both modalities may not fully exploit these modality-specific properties, potentially limiting performance. Furthermore, an ISI-CSFN module are suggested to fuse feature maps of various scales and apply contact of various feature maps at the same scale. These modules can successfully fuse multi-scale feature maps mined from the two backbones.

Fig. 3

ISI-CSFN framework.

Multi-scale local and global pattern mining

In this paper, two networks namely convolutional networks and Transformers are used simultaneously for obtaining multiple scale local and global feature maps, correspondingly. Especially, the widely used DSCN⁴¹ is used to obtain the multi-scale feature maps in the CNN branch and the enhanced CTransformer in the Transformer branch. Fig. 4 CNNs, frequently use the DSCN to lower the computational load of a typical multiple channel convolution. The depthwise convolution kernel and the pointwise convolution kernel are the two distinct kernels that are created from a normal 2D convolution kernel. When compared to a conventional convolution, the DSC minimizes computation by requiring fewer input changes as in Fig. 3.

Fig. 4

DSCN principle (a) Depthwise convolution (b) Pointwise convolution.

The CTransformer uses patch-wise linear embedding to project images into token sequences. The network design of the CTransformer is depended on the well-known Swin Transformer, although it includes numerous enhancements to compensate for the shortcomings of the conventional Swin Transformer. As shown in Fig. 5, the normalization layer, activation function, and spatial data fusion are different from the original Swin transformer module. The concatenating layer and residual learning are replaced by the SK fusion and soft reconstruction layers. Inspired by SKNet⁴², the Selective Kernel Fusion (SKF) layer uses channel attention to fuse several branches.

Fig. 5

CTransformer (a) network structure (b) optimized swin transformer.

The normalization layer is essential to deep learning model because it steadies the network’s training. Nevertheless, LayerNorm continues to eliminate the feature map’s mean and standard deviation. Therefore, Rescale Layer Normalization (RescaleNorm) is introduced and it adds the mean and standard deviation at the residual block’s output stage. RescaleNorm can be expressed as follows:

$$\:z=F\left(\frac{y-\mu\:}{\sigma\:}.\delta\:+\alpha\:\right).\left(\sigma\:{\omega\:}_{\delta\:}+{\beta\:}_{\delta\:}\right)+(\mu\:{\omega\:}_{\alpha\:}+{\beta\:}_{\alpha\:})$$

(1)

Where $\:y$ represents the feature map resolution. µ and σ represent the mean and standard deviation, $\:\delta\:$ and $\:\alpha\:$ are trained scaling parameter and bias. It utilizes two linear layers with weights $\:{\omega\:}_{\delta\:}$, $\:{\omega\:}_{\alpha\:}$and biases $\:{\beta\:}_{\delta\:}$, $\:{\beta\:}_{\alpha\:}$for transforming µ and σ through $\:\left(\sigma\:{\omega\:}_{\delta\:}+{\beta\:}_{\delta\:}\right)+(\mu\:{\omega\:}_{\alpha\:}+{\beta\:}_{\alpha\:})$.

Despite having derived GELU, ReLU, and LeakyReLU, some new transformer-based image processing networks pose a gradient inversion problem due to their non-singularity. Hence, SoftReLU is suggested and it is a straightforward smooth approximation to the ReLU:

$$\:softReLU\left(y\right)=\frac{y+\sqrt{{y}^{2}+{\epsilon\:}^{2}-\epsilon\:}}{2}$$

(2)

where $\:\epsilon\:$ is a shape parameter. Swin Transformer implements effective batch computing for shifted window partitioning using cyclic shift with masked MHSA. This mask changes the window size along the image boundary to be less than the specified window size. This work suggests using reflection padding to accomplish effective batch computing for shifted window subdivision without creating unwarranted inter-patch interactions.

Intra-scale interaction

Even though the feature maps produced by the two pathways are inherently in tensor dimension, the cross-attention calculations need a series of vectorized input. Every feature map must be anticipated into 3 transitional elements, i.e. query$\:\stackrel{-}{Q}$, key $\:\stackrel{-}{K}$and$\:\stackrel{-}{V}$, utilizing the spatially normalized features$\:{F}_{D}\in\:{\mathfrak{R}}^{\hslash\:\times\:w\times\:{\mathbb{C}}_{1}}$and $\:{F}_{T}\in\:{\mathfrak{R}}^{\hslash\:\times\:w\times\:{\mathbb{C}}_{2}}$, and the three learned linear transformations$\:{P}^{\stackrel{-}{Q}}$, $\:{P}^{\stackrel{-}{K}}$ and $\:{P}^{\stackrel{-}{V}}$. In the proposed model, $\:{F}_{D}$and $\:{F}_{T}$ are processed together through parallel operations. In particular, the 1 × 1 convolution process is carried out for unifying and minimizing the channel size, followed by the traditional positional embedding process to give the positional data. The tensor flattened $\:{\stackrel{-}{Q}}_{x}$ from DSCN and CTransformer are alternately utilized to complete feature fusion in order to enforce the data collaboration between $\:{F}_{D}$ and $\:{F}_{T}$. The same self-care mechanism module is used for simplicity. It should be noted that the traditional process uses Eq. 3 to formulate a scaled dot-product operation.

$$\:SA\left(\stackrel{-}{Q},\:\stackrel{-}{K},\stackrel{-}{V}\right)=softmax\left(\frac{\stackrel{-}{Q}{\stackrel{-}{K}}^{T}}{\sqrt{m}}\right)\stackrel{-}{V}$$

(3)

Where $\:T$denotes the transpose operator, the input series $\:\stackrel{-}{Q},\:\stackrel{-}{K},\stackrel{-}{V}\in\:{\mathfrak{R}}^{M\times\:m}$, $\:M=\hslash\:\times\:w$and $\:m$ is the condensed size. The suggested model achieves collaborative proliferation between the feature maps of the DSCN branch and the CTransformer branch on the same resolution level by factorizing the attention map and computing the second matrix multiplication (key $\:\:\stackrel{-}{K}$and value $\:\stackrel{-}{V}$). Equation (4) provides the specific operation.

$$\:LA\left(\stackrel{-}{Q},\:\stackrel{-}{K},\stackrel{-}{V}\right)=\frac{\stackrel{-}{Q}}{\sqrt{m}}softmax{\left(\stackrel{-}{K}\right)}^{T}\stackrel{-}{V}$$

(4)

Here softmax(·) is applied element-wise on all tokens of the array.

Additionally, the suggested model added a Relative Position-Aware Convolution (RPC) to the Conv Attention module, which means that the query and the adjacent features influence a token’s output. In particular, as demonstrated by Eq. 5, the proposed model describes the position-centric interactions between $\:\stackrel{-}{Q}$ and $\:\stackrel{-}{V}$ using the exceptional instance of relative position encoding implemented with depthwise convolution.

$$\:RPC\left( {\mathop Q\limits^{ - } ,\:\mathop V\limits^{ - } } \right) = \mathop Q\limits^{ - } \cdot DSC\left( {\mathop V\limits^{ - } } \right)$$

(5)

where $\circ$ denotes the Hadamard product. The full Linear convolutional attention layer consists of 2 sub-modules, namely linear attention and RPC in order to accomplish the collaborative interaction of various branches and produce multiple scale interactive feature maps $\:{F}_{1}$and $\:{F}_{2}.$ The whole structure is defined is as follows:

$$\:ConvAtt\left(\stackrel{-}{Q},\stackrel{-}{K},\stackrel{-}{V}\right)=LA(\stackrel{-}{Q},\:\stackrel{-}{K},\stackrel{-}{V})+RPC\left(\stackrel{-}{Q},\:\stackrel{-}{V}\right)$$

(6)

Here a Conv Attention layer is employed for strengthening interaction effectiveness. The Fusion unit aggregate the two outputs $\:{F}_{1}$ and $\:{F}_{2}$ after cross-interaction to further explore the extraction of admirable features and eliminate the interference of inappropriate features.

Cross-scale fusion network

A masked aggregation unit is used by this cross-scale fusion network to draw attention to relevant properties of interest in the reconstructed features $\:{F}_{1}$and $\:{F}_{2}$. Here, the feature maps are convolved using Hadamard product to generate mask maps $\:{MK}_{1}$and $\:{MK}_{2}$. It supresses vagueness introduced by the unnecessary features and activating the task-specific ones.

$$\:{MK}_{1}=UpSamp\left(Conv\right(BNorm\left(Conv\right({F}_{L}\left)\right))$$

(7)

$$\:{MK}_{2}=Conv\left({F}_{H}\right)$$

(8)

$$g_{1} = \text{Re} LU(MK_{1} \circ F_{H} )$$

(9)

$$g_{2} = \text{Re} LU(MK_{2} \circ UpSamp\left( {BNorm\left( {Conv\left( {F_{L} } \right)} \right)} \right)$$

(10)

$$\:g=ReLU(BNorm\left(Conv\left(Concatenate\left({g}_{1},{g}_{2}\right)\right)\right)$$

(11)

where $\:{F}_{L}$ and $\:{F}_{H}$ represent the lower-scale and higher-scale feature maps. $\:Conv$, $\:BNorm$, $\:ReLU$, $\:UpSamp$, and $\:Concatenate$ stand for the convolution operation, batch normalization, activation function, upsampling technique, and aggregation. Furthermore, $\:{g}_{1}$ is generated by applying $\:{MK}_{1}$attained from the $\:{F}_{L}$to $\:{F}_{H}$, with an emphasis on gathering features such boundaries and spatial organization. On the other hand, background noise may affect the learned features. Additionally,$\:\:{MK}_{2}$ is derived from the $\:{F}_{H}$to add rich semantic information to $\:{F}_{L}$, helping to identify and reduce noise for $\:{g}_{2}$. Consequently, the output $\:g$ integrates a number of admirable attributes into multi-scale features.

Also, a spatial alignment unit is presented using the attention strategy to develop $\:{F}_{3}$through the fusion of $\:{F}_{1}$ with local features and$\:{F}_{2}$with global features. The total spatial attention distribution is modelled by merging the various scale features $\:{F}_{L}$ and $\:{F}_{H}$

$$\:{g}_{merge}=BNorm\left(Conv\left(UpSamp\left({F}_{L}\right)\right)\right)+BNorm\left(Conv\right({F}_{H}\left)\right)$$

(12)

$$\:\gamma\:=sig(BNorm\left(Conv\left(ReLU\left({g}_{merge}\right)\right)\right)$$

(13)

$$\:g=ReLU\left(BNorm\right(Conv\left(Concatenate\right(UpSamp\left({F}_{L}\right),\gamma\:.{F}_{H}$$

(14)

Where$\:\:\gamma\:$ stands for the attention coefficient and $\:sig$ is the activation function. Target areas in$\:\:\gamma\:$ end to have values of one, whereas background regions tend to have values of zero. These regions are in charge of capturing task-related aspects and reducing the interference of unimportant information.

Multi task cost function for ISI-CSFN

The suggested ISI-CSFN employs CTransformer for global features and DSCN for fine-grained local features from input PET and MRI images, respectively. For deeper feature learning, the learnt features from these blocks are then fed into two dense layers. Three dense layers then combine the output of the all streams to create deeper and more recognizable features. Furthermore, baseline information is used to improve the precision and assurance of the training procedure. This baseline information includes patient’s demographics and a few statistical characteristics taken from patient’s longitudinal time series data. A feed-forward neural network is utilized for extracting additional deep and unique features from the baseline data independently. To discover finer features for the categorization and regression tasks, a pair of shared dense layers fuses the output of these two-feature extraction phase.

The suggested model simultaneously learns a number of related tasks, such as four regression problems and a multiple class classification problem. We anticipate that this kind of data is essential for doctors to belief the outcomes of DL frameworks. The proposed framework makes use of shared parameters of the DL model for multitask learning.

The suggested DL framework uses a set of modalities $\:N$ to mutually learn a set of associated tasks$\:\:S$. Let $\:P=\left\{{P}^{\left(1\right)},\dots\:,{P}^{\left(N\right)}\right\}$ represent $\:N$ modalities of data and $\:S=\left\{{s}^{\left(1\right)},\dots\:,{s}^{\left(L\right)}\right\}$ represent multiple tasks to be learned. $\:{S}^{\left(i\right)}=\left\{{s}_{1}^{\left(i\right)},\dots\:,{s}_{M}^{\left(i\right)}\right\},\:i\in\:\{1,\dots\:,L\}$ describes every $\:i$th task. Every modality $\:{P}^{n}$is described as $\:{P}^{n}=\left\{{p}_{1}^{\left(n\right)},\dots\:.{p}_{j}^{\left(n\right)},\dots\:{p}_{M}^{\left(n\right)}\right\}$for $\:M$ patient samples, each of which is a multivariate time series $\:{p}_{j}^{\left(n\right)}=\left\{{p}_{j{t}_{1}}^{\left(n\right)},\dots\:.{p}_{j{t}_{2}}^{\left(n\right)},\dots\:{p}_{j{t}_{g}}^{\left(n\right)}\right\}$ for $\:t=\text{1,2},\dots\:T$ time steps and $\:g$sets of univariate time series.

Let $\:\left\{{p}_{j}^{\left(1\right)},\dots\:,{p}_{j}^{\left(n\right)},\dots\:.{p}_{j}^{\left(N\right)},{s}_{j}^{1},\:{s}_{j}^{2},\dots\:\:{s}_{j}^{L}\:\right\},\:j=1,\dots\:.,M,\:{s}_{j}^{1}\in\:\left\{\text{1,2},\text{3,4}\right\};\:j\in\:\{2,\dots\:.L\}$be the label for the $\:t$th task of $\:M$ patients. Here, both task-specific ($\:{\varnothing\:}^{t}$) and shared $\:\left({\varnothing\:}^{s}\right)$ parameters need to be improved. The task-specific loss functions are $\:{\mathbb{L}}^{t}\left(.,.\right):{s}^{\left(t\right)}\to\:{\mathfrak{R}}^{+}$and the parametric theory for every task is$\:{g}^{t}\left(p,{\varnothing\:}^{s},{\varnothing\:}^{t}\right):P\to\:{s}^{\left(t\right)}$. Gradient-based multiobjective optimization of task-dependent losses is a general optimization problem.

$$\:\underset{\begin{array}{c}{\varnothing\:}^{s}\\\:{\varnothing\:}^{1},\dots\:{\varnothing\:}^{L}\end{array}}{\text{min}}\mathbb{L}({\varnothing\:}^{s},{\varnothing\:}^{1},\dots\:{\varnothing\:}^{L})=\underset{\begin{array}{c}{\varnothing\:}^{s}\\\:{\varnothing\:}^{1},\dots\:{\varnothing\:}^{L}\end{array}}{\text{min}}\left({\widehat{\mathbb{L}}}^{1}\left({\varnothing\:}^{s},{\varnothing\:}^{1}\right),\dots\:,{\widehat{\mathbb{L}}}^{T}\left({\varnothing\:}^{s},{\varnothing\:}^{T}\right)\right)$$

(15)

$$\:{\widehat{\mathbb{L}}}^{T}\left({\varnothing\:}^{s},{\varnothing\:}^{T}\right)=-\left[\frac{1}{M}\sum\:_{j=1}^{M}\frac{1}{Q}\sum\:_{q=1}^{Q}{\omega\:}_{D}I({s}_{q}=Q)log\left(\frac{\text{exp}\left({\varnothing\:}_{q}^{L}{P}_{j}+\beta\:\right)}{\sum\:_{i=1}^{Q}\text{exp}\left({\varnothing\:}_{q}^{L}{P}_{j}+\beta\:\right)}\right)\right]+$$

$$\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\frac{1}{L}\sum\:_{t=2}^{L}\frac{1}{M}\sum\:_{j=1}^{M}{\left({s}_{j}^{t}-{\widehat{s}}_{j}^{t}\right)}^{2}+\frac{\tau\:}{n}\sum\:_{i=1}^{n}{\varnothing\:}_{i}^{2}$$

(16)

Where $\:{\widehat{\mathbb{L}}}^{T}(.,.)$ denotes an application-driven loss function described as $\:{\widehat{\mathbb{L}}}^{T}\left({\varnothing\:}^{s},{\varnothing\:}^{T}\right)=\frac{1}{M}\sum\:_{j}\mathbb{L}({g}^{T}\left({p}_{j};{\varnothing\:}^{s},{\varnothing\:}^{T}\right),{s}_{j}^{T})$. Both classification and regression tasks are performed by the suggested framework. The first part in loss function denotes the Class-weighted cross-entropy of multiple class categorization and the 2nd part denotes the mean squared loss for 4 regression problems.

Dataset description

This study uses data collected from ADNI 1, ADNI GO, and ADNI 2. Here, 1539 participants divided into 4 groups depending on their patient-specific clinical diagnoses at baseline and subsequent visits. Participants are divided into four groups. The first category contains 420 participants who were diagnosed with CN at baseline. The second category contains 474 participants who were classified with sMCI across all time points during the trial. The third class contains 306 subjects with pMCI at the baseline visit and who converted to AD at any point during the study. Finally, the fourth class contains 339 subjects with consistently diagnosed with AD at all visits⁴³. Table 1 shows the subjects’ demographic and clinical data.

Table 1 Demographic and clinical data of subjects under study.

Results and discussion

The performance of the proposed model is validated by simulating it in python programming language. Also, the Keras library with TensorFlow is utilized as the backend for our tests on a computer system running Ubuntu, 506 Nvidia GTX Titan Xp x2, and an i7-6800 K. The network parameters are updated using stochastic gradient descent (SGD). The values for the batch size, learning rate, momentum, weight decay, and number of training epochs are optimized as 6, $\:1.0{e}^{-2}$, 0.95, $\:5.0{e}^{-2}$, and 300 correspondingly. The proposed model is tested utilizing the public ADNI dataset. Here, an 80 − 20 dataset split was considered, meaning that 20% of the data is utilized for validation and 80% for training. To guarantee that the proposed findings are not influenced by a specific split, we carried out this procedure five times using various random subsets. To deliver a more reliable indication of the generalizability of the model, the model’s ultimate performance is averaged over these five runs.

ROI importance analysis

The suggested model divides the brain into anatomical regions (such as cortical and subcortical ROIs) using the FreeSurfer tool. The top ten ROIs found for MRI and PET data over the 3 different classification tasks are shown in Fig. 6. The top 10 ROIs are identified through registration of the obtained network weights with the automatic-connected anatomical labeling and arranging them in descending order according to the mean values ​​for every ROI. The hippocampus, cingulum, and occipital lobe are vital brain regions that play important roles in a variety of cognitive and affective processes. Their significance in AD research is further highlighted by their relationship with AD disease.

Fig. 6

ROI regions for both MRI and PET images in the classifications tasks (a) NC vs. AD. (b) sMCI vs. pMCI and (c) NC vs. MCI.

Evaluation of classification task

The efficacy of the suggested classification task is shown by implementing various setups for three binary tasks (NC vs. AD, NC vs. MCI, and sMCI vs. pMCI). In Table 2, the effectiveness of the ISI-CSFN is compared with a number of deep learning methods, such as ResNet⁴⁴, ResNext⁴⁵, MobileNet⁴⁶, ShuffleNetv2⁴⁷, EfficientNet⁴⁸, and MMSDL³⁵. With accuracy of up to 97.26% (95% CI: 97.01–97.51), sensitivity of up to 99.27% (95% CI: 97.61–100.93), and specificity of up to 94.72% (95% CI: 92.20–97.24), the proposed model outperformed all other models for NC vs. AD classification. MMSDL also performed competitively with 95.25% (95% CI: 94.62–95.88) accuracy and an AUC of 94.68% (95% CI: 93.93–95.43). Light model types like ShuffleNetv2 and MobileNet, however, had high sensitivity but relatively reduced accuracy. The suggested model performed exceptionally, demonstrating good discrimination between normal controls and mild cognitive impairment, with an accuracy of 88.26% (95% CI: 86.51–90.01) and an AUC of 90.54% (95% CI: 88.79–92.29) for NC vs. MCI classification.

MMSDL performed better than MobileNet with an accuracy of 85.37% (95% CI: 83.62–87.12), while MobileNet performed the worst at 79.65% (95% CI: 77.90–81.40). The discrimination between sMCI and pMCI is inherently more difficult due to the fine distinctions between progressive and stable MCI. The proposed model possessed the highest sensitivity (85.73%; 95% CI: 83.98–87.48), accuracy (81.25%; 95% CI: 79.50–83.00), and AUC (83.47%; 95% CI: 81.72–85.22). MMSDL again fell significantly behind, whereas MobileNet and ShuffleNetv2 exhibited relatively poorer discrimination performance. The suggested model outperformed all the baseline models for all the classification tasks consistently and proved to be robust in tackling both binary and hard progressive classification issues. Also, it succeeded in discriminating between the NC, MCI, and AD classes.

Table 2 Classification performance with 95% confidence intervals (CI) over five runs.

Table 3 Performance of the proposed ISI-CSFN model with Mean ± SD and 95% CI.

Performance of the proposed ISI-CSFN model is presented in Table 3. Its accuracy of 97.26 ± 0.29, 95% CI of [96.90–97.62], and AUC of 95.58 [95.00–96.16] for the NC vs. AD task indicate strong discrimination. The model obtains 88.26 ± 0.32 accuracy for the NC vs. MCI task and 81.25 ± 0.28 accuracy for the sMCI vs. pMCI task with tight confidence intervals. The ISI-CSFN’s consistently high sensitivity (99.27) and specificity (94.72) contribute to the confirmation of its stability in a broad range of categorization conditions.

Table 4 Five-fold cross-validation results of the proposed ISI-CSFN model.

The performance of the suggested ISI-CSFN model in the five-fold cross-validation is presented in Table 4. With fold-wise variations being within ± 0.5% and average accuracy of 97.26% and sensitivity of 99.27%, the model performed repeatedly well on all folds in the NC vs. AD test. In the same way, for NC vs. MCI, the model’s average accuracy was 92.41% and its sensitivity was 88.26%, showing consistent identification despite the tiny class differences. In the most difficult sMCI vs. pMCI challenge, ISI-CSFN had an average accuracy of 81.25% and an F1-score of 82.01% with consistent performance across folds. These consistent results for all partitions confirm the robustness, stability, and generalizability of the proposed framework.

Also, the ROC curves are used to evaluate the effectiveness of various configurations of our technique as in Fig. 7. The ROC curves visually represent the true positive vs. false positive rates for every model over the three tasks. These configurations include: ResNet50, ResNext, MobileNet, ShuffleNetv2, EfficientNet, MMSDL and the proposed model. The proposed model performs better at differentiating between classes since it obtains the best AUC across all classification tasks.

Fig. 7

ROC Analysis (a) NC vs. AD (b) NC vs. MCI (c) sMCI vs. pMCI.

The findings of the proposed model are contrasted with those derived from individual designs of Multi-scale Local and Global Pattern Mining (MS-LGPM), MS-LGPM + ISI, and MS-LGPM + ISI + CSFN. Figure 8 compares the performance of the suggested ISI-CSFN framework and its individual design elements, MS-LGPM, MS-LGPM + ISI, and MS-LGPM + ISI + CSFN, in terms of Accuracy, F1 Score, Precision, and Recall over a range of time step. As demonstrated, the entire model (MS-LGPM + ISI + CSFN) exhibits improved resilience and stability over longer time sequences, consistently outperforming the other configurations across the majority of criteria. The baseline MS-LGPM is greatly enhanced by the addition of intra-scale interaction (ISI), especially in Accuracy and F1 Score. However, the optimal trade-off between precision and recall is achieved by combining ISI and cross-scale fusion (CSFN), which keeps performance high even as temporal complexity increases. This demonstrates how well the suggested ISI-CSFN design captures contextual global relationships as well as fine-grained local dependencies for time-series classification tasks.

Fig. 8

Performance of individual design components (a) Accuracy (b) F1 score (c) Precision (d) Recall.

We performed several sets of classification trials to assess the impact of MRI and PET fusion quantitatively. First, the classification performance is analysed by considering only functional brain imaging (MRI) (as indicated by the red hue in Fig. 9). Next, the classification performance is evaluated using PET (as seen by the blue hue in Fig. 8). At last, combined MRI and PET images were considered (indicated by green line). According to the experimental results, classification accuracy can be increased by 8% to 10% by combining structural and functional brain imaging. Additionally, the results show that combining MRI and PET brain imaging to study AD can increase the ability to predict AD compared to using only one modality.

Fig. 9

Effect of MRI and PET fusion (a) NC vs. AD (b) NC vs. MCI (c) sMCI vs. pMCI.

Cognitive scores prediction

The MAE comparison of the four clinical cognitive tests including FAQ, ADAS, CDR, and MMSE for the suggested MS-LGPM variants such as MS-LGPM, MS-LGPM + ISI, and MS-LGPM + ISI + CSFN is shown in Table 5. The findings show that the addition of intra-scale interaction (ISI) increases prediction accuracy over the baseline MS-LGPM for the majority of measures. Among the four tasks, the whole model, MS-LGPM + ISI + CSFN notably achieves the lowest MAE values: 0.085 ± 0.01 for MMSE, 0.076 ± 0.01 for ADAS, 0.075 ± 0.01 for CDR, and 0.107 ± 0.01 for FAQ. This suggests that the model is better at capturing intricate correlations in multi-scale temporal data. These improvements show how better incorporating intra- and cross-scale interactions across various cognitive measures can reduce prediction error.

Table 5 MAE comparison of the proposed system components.

Table 6 presents the error analysis of MMSE and ADAS cognitive score prediction by MMSE and ADAS cognitive score prediction using various baseline models versus the proposed ISI-CSFN. It is shown that traditional imputation-based recurrent models, i.e., LSTM-M and LSTM-F, produce very high RMSE values, reflecting their poor capability for detecting sophisticated temporal relationships in multimodal data. Despite fewer mistakes, cross-modal interactions also cannot be represented optimally by more sophisticated models like MRNN, PLSTM-Z, and MinimalRNN. The advantages of robust recurrent architectures for sequential prediction are again shown by the DRN method, which lowers RMSE to 2.31 (95% CI: 2.20–2.42) for MMSE and 4.31 (95% CI: 4.16–4.46) for ADAS. Recommended ISI-CSFN does best on average, with RMSEs of 1.28 (95% CI: 1.17–1.39) for MMSE and 3.18 (95% CI: 2.96–3.40) for ADAS.

As the values are much lower than those of all baseline models, they demonstrate how well the intra-scale and cross-scale fusion method takes advantage of complementary information across modalities. The narrow confidence intervals also corroborate the stability and robustness of our approach across multiple experiments. These findings show that ISI-CSFN gives more accurate and solid predictions of cognitive impairment compared to state-of-the-art recurrent and deep learning methods. Through the introduction of multi-scale and modality-specific information, the proposed method not only enhances forecasting performance but also has vast potential for clinical use in tracking the development of Alzheimer’s disease.

Table 6 RMSE for MMSE and ADAS with 95% confidence intervals (CI).

Ablation study

To examine the influences of various model blocks on the prediction results, we compared the performance of ISI-CSFN with its variants. Namely, the model with only MRI image (MRI-ISI-CSFN), the model with only PET (PET- ISI-CSFN), the model with only DSCN (DSCN-ISI-CSFN), the model with only CTransformer (CTransformer-CSFN) and Combined MSLG- ISI-CSFN model. As illustrated in Fig. 10, t-SNE feature visualizations are conducted for various ISI-CSFN technique configurations across three classification tasks: NC vs. AD, NC vs. MCI, and sMCI vs. pMCI. The results demonstrate clear variations in the models’ capacity to distinguish between class clusters. CTransformer-CSFN and DSCN-ISI-CSFN exhibit a considerable degree of overlap and separation, especially when it comes to differentiating between sMCI and pMCI and NC and MCI.

Fig. 10

t-SNE feature visualizations.

The proposed MSLG-ISI-CSFN can capture both fine-grained textures (local) and broader contextual patterns (global) within brain images, making it more effective at identifying subtle differences between normal and diseased brain images. Combining the MRI, PET, DSCN, and CTransformer modules allows the model to take advantage of complimentary data from many modalities and architectures, producing feature representations that are more reliable.

While the suggested ISI-CSFN performs well on the ADNI set, its basis on this benchmark limits its direct generalizability to other sets or clinical environments. Differences in imaging protocols, scanner platforms, demographic distributions, and clinical diagnostic criteria across cohorts can impact the model’s robustness when applied directly in other environments. Also, in contrast to ADNI participants, typically well-curated, real-world clinical data tend to have greater heterogeneity, noise, and missing values.

Table 7 Comparison of the proposed ISI-CSFN model with existing state-of-the-art methods.

Table 7 presents a comparative performance analysis of the proposed ISI-CSFN model against existing multimodal approaches. The proposed model achieved the highest accuracy of 97.26%, with sensitivity and specificity values of 99.27% and 94.72%, respectively. Compared to Multimodal RNN⁴⁹ and LMDP-Net⁵⁰, our method demonstrates a substantial improvement, indicating its robustness in predicting Alzheimer’s disease progression.

Limitations and future work

Although the proposed model is more accurate than baseline models, it has more parameters and higher FLOPs per inference, which adds to computational complexity. Besides its deployment on efficient hardware, future work will investigate optimization methods such as pruning, quantization, and knowledge distillation to cut down computational loads while keeping performance intact. In addition, the supplementary information from MRI and PET methods might not be similarly helpful to all patients or disease phases. Single-modality models with strong performance or adaptive fusion methods can ensure performance while cross-modal complementarity is weak. The comparatively limited dataset size is another drawback, which may limit generalizability and heighten the risk of overfitting. Robustness can be enhanced by validation on bigger, multi-center datasets with data augmentation, transfer learning, and cross-cohort testing. Inclusion of participants with mild preclinical abnormalities within the cognitively normal (CN) control group might bias comparisons against the MCI and AD groups. Subsequent research should take into account stricter screening protocols and cross-validation within different cohorts to ensure that the CN group represents a representative population baseline.

Conclusion

Alzheimer’s disease (AD), a progressive neurodegenerative disorder, poses a major global health challenge because of its medical, societal, and commercial impacts. This study used a disease progression model to define the AD progression prediction problem, which included clinical stage prediction tasks over multiple time periods and prediction of MRI biomarker and cognitive test scores. A Joint multimodal multitask learning framework using the mixture of DSCN and cTransformer is specifically suggested for the purpose of learning AD multiple class categorization and four cognitive score regression concurrently. Furthermore, the inter-scale feature fusion unit aims to combine low-level features that support structural sketching with high-level representations that aid localization, while the intra-scale cross-interaction module is introduced to focus on the cooperation between subtle local patterns and application-driven global benefits. Also, an exhaustive experiment is conducted to prove the model effectiveness by matching with other comparison methodologies in the literature. The suggested framework outperformed opposing methods in numerous quantitative metrics. Therefore, it provides new perspectives in identifying the diverse neural circuits linked to AD. The primary drawback of our model is its excessive resource usage, which is a result of the pre-trained DSCN and CTransformer backbones being introduced in two separate branches. Therefore, future research will focus on lightweight models in more detail.