Pure data correction enhancing remote sensing image classification with a lightweight ensemble model

Abstract

The classification of remote sensing images is inherently challenging due to the complexity, diversity, and sparsity of the data across different image samples. Existing advanced methods often require substantial modifications to model architectures to achieve optimal performance, resulting in complex frameworks that are difficult to adapt. To overcome these limitations, we propose a lightweight ensemble method, enhanced by pure data correction, called the Exceptionally Straightforward Ensemble. This approach eliminates the need for extensive structural modifications to models. A key innovation in our method is the introduction of a novel strategy, quantitative augmentation, implemented through a plug-and-play module. This strategy effectively corrects feature distributions across remote sensing data, significantly improving the performance of Convolutional Neural Networks and Vision Transformers beyond traditional data augmentation techniques. Furthermore, we propose a straightforward algorithm to generate an ensemble network composed of two components, serving as the proposed lightweight classifier. We evaluate our method on three well-known datasets, with results demonstrating that our ensemble models outperform 48 state-of-the-art methods published since 2020, excelling in accuracy, inference speed, and model compactness. Specifically, our models achieve an overall accuracy of up to 96.8%, representing a 1.1% improvement on the challenging NWPU45 dataset. Moreover, the smallest model in our ensemble reduces parameters by up to 90% and inference time by 74%. Notably, our approach significantly enhances the performance of Convolutional Neural Networks and Vision Transformers, even with limited training data, thus alleviating the performance dependence on large-scale datasets. In summary, our data-driven approach offers an efficient, accessible solution for remote sensing image classification, providing an elegant alternative for researchers in geoscience fields who may have limited time or resources for model optimization.

Introduction

Background

Remote Sensing Images (RSIs) are pivotal photogrammetric records in Earth monitoring. The rapid increase in onboard sensors has led to a vast accumulation of data derived from diverse imaging techniques, rendering traditional manual interpretation inefficient. Consequently, deep learning (DL) algorithms have been extensively employed in RSI recognition across various domains¹. Among these algorithms, classification is deemed fundamental and vital due to its substantial impact on the performance of other algorithms^2,3.

Currently, DL-based methodologies are leading the field of RSI classification. Pre-training on large-scale datasets can accelerate the re-training process for subsequent tasks. As a result, most cutting-edge methods utilize pre-trained Convolutional Neural Network (CNN) or Vision Transformer (ViT) models, which were developed on ImageNet-1 K, for RSI recognition. However, most advanced methods typically require a comprehensive reconfiguration of the model’s structure, incorporating sub-modules or sub-models to achieve layered accuracy. Specifically, prior studies have indicated that ViTs may not achieve accuracy as remarkable as CNNs, particularly when training samples are limited, as ViTs have significant performance dependence on large-scale training samples. Consequently, previous CNN- or ViT-based methods often lead to significantly expanded model sizes, complex pipelines, and reduced inference speeds⁴. These conditions present challenges for geoscience researchers, who may lack the necessary expertise or time to utilize these techniques. We propose that this predicament can be partially attributed to the issue of pre-training invalidation.

Pre-training is widely recognized for providing deep models with a stronger starting point in the solution space and addressing statistical challenges when limited data is available for downstream tasks. By pre-training a modified model on large-scale datasets such as ImageNet-1 K, models can leverage prior knowledge, which enhances generalization and overall performance. However, many existing approaches that use modified models often neglect the crucial step of pre-training, resulting in a training process that deviates from standard transfer learning practices. This deviation undermines the effectiveness of fine-tuning, making it less successful in optimizing the modified model. The following empirical evidence supports these concerns, showing that insufficient pre-training significantly hampers the model’s performance during fine-tuning.

For instance, LS-EfficientNet⁵ highlighted that pre-trained CNN models often lead to lower accuracy in classification tasks, even when the bottom layers of the pre-trained model are initialized with random weights. Similarly, our study found that combining or fusing features from two CNN models frequently results in inferior performance compared to using an unmodified model with a standard transfer learning approach, as in LS-EfficientNet. This is further corroborated by the results presented in Table 1, where simple strategies such as concatenating the final layers of two EfficientNet-B0 models yield suboptimal accuracy, as indicated by the training ratio (TR) values. These findings underscore the importance of proper pre-training in achieving optimal model performance.

Table 1 Overall accuracy comparison of different strategies on the AID30 dataset (%).

Remote sensing data display a considerable domain discrepancy when compared to natural images. While neural architecture search holds the potential to identify superior models for classifying RSIs, previous approaches have generally focused on relatively small RSI datasets containing tens of thousands of samples^6,7,8, which presents a statistical challenge when compared to large-scale datasets like ImageNet-1 K, which consists of one million samples. Additionally, RSI samples typically feature more complex scenes, with multiple objects and noisier backgrounds, further complicating the classification task. As a result, training augmentation strategies developed for ImageNet-1 K may be less effective when directly applied to RSIs without appropriate modifications. These factors highlight the need for tailored approaches that account for the distinct characteristics of RSIs, rather than simply replicating techniques developed for natural image datasets.

Fig. 1

Differences in background complexity between RSIs and natural images (Source: Authors own work by Python version 3.9).

As shown in Fig. 1, the three natural images on the left clearly distinguish between semantic targets and their backgrounds. As a result, regularization techniques, such as CutMix⁹, typically have a low probability of misclassifying a target, such as transforming the ‘elephant’ category into ‘grassland.’ In contrast, the three remote sensing samples on the right feature more complex targets and noisier backgrounds. For instance, the cut-and-mixed sample at the rear resembles a church class, even though the majority of the scene corresponds to a park. This illustrates that RSIs often exhibit greater overlap between categories than natural images, which complicates the classification task. Consequently, regularization strategies like CutMix should be applied with reduced strength during training to account for the more challenging characteristics of RSIs.

Fig. 2

Challenges in qualitative augmentation (Source: Authors own work by Python version 3.9).

As illustrated in Fig. 2, the intensity of sunlight plays a crucial role in determining the imaging quality of remote sensing data. Additionally, fluctuations in other imaging conditions, such as direction and blur, can significantly affect the quality of the samples. To ensure the majority of RSIs meet quality standards, we typically use a checking algorithm to filter out low-quality samples. In this process, low-quality samples are represented by red, blue, or black dots, while normal samples are marked in green. A classic RSI dataset in a real-world task is depicted by subset A in Fig. 2, which consists of mostly high-quality samples, with some abnormal ones. However, RSI datasets often contain only around ten thousand samples, leading to a training subset that may suffer from statistical challenges when compared to the full dataset’s distribution. For example, subset B in Fig. 2 demonstrates how a randomly selected subset, designed to prevent overfitting, may fail to capture the full variability of the data. To address this under-regularization issue, previous studies typically apply qualitative augmentation strategies. However, as shown at the bottom of Fig. 2, such qualitative pipelines often convert all data points into a single, abnormal type, as the transformation functions remain active throughout. As a result, traditional qualitative strategies frequently lead to biased samples and over-regularization.

Motivation

To overcome the challenges identified in previous approaches, we introduce a novel training augmentation strategy called quantitative augmentation (QA). The primary objective of QA is to address data distribution imbalances within remote sensing data, thereby improving the classification accuracy of CNN and ViT models. In addition to QA, we enhance two existing regularization techniques—CutMix and Online Label Smoothing (OLS)¹⁰—to offer a more adaptable method for developing diverse classifiers. Building on these improvements, we also propose the Exceptionally Straightforward Ensemble (ESE), a method that combines two lightweight models—either a CNN and a ViT or two CNNs—into an ensemble network. The key contributions of our approach can be categorized into three main areas, each of which plays a pivotal role in enhancing model performance and flexibility.

Firstly, the QA and its implementation, the QA Module (QAM), are more effective than traditional data augmentation in RSI classification. The QAM, a plug-and-play unit, is exclusively active during the processing of training samples. As a result, the QA can function in conjunction with any CNN or ViT to enhance performance. Secondly, ESE introduces a more straightforward algorithm to generate more robust ensemble classifiers. It is more convenient compared to traditional, complex methods of generating ensemble models, such as repetitively manipulating training samples or combining too many models. Lastly, our method provides comprehensive algorithms for leveraging open-source models while eliminating the need for extensive experience in re-modifying models. It makes the method both time-efficient and easily reproducible, a notable advantage over existing methods.

We used three benchmark datasets for our evaluation. The results show that our ESE method outperforms 48 advanced techniques introduced between 2020 and 2024, significantly improving accuracy while reducing both the number of parameters and inference time. Additionally, our QA-based CNN and ViT models consistently exceed the performance of other methods using the same model, even if training samples are limited. We summarize the key contributions of our work as follows:

• (1) We propose a novel augmentation strategy, called QA, along with its implementation, the QAM, to improve the performance of CNNs and ViTs in RSI classification. The QA method effectively aligns the data distribution between training and testing subsets, enabling the models to outperform existing state-of-the-art techniques, particularly in scenarios with limited training samples. As a result, our strategy significantly reduces the reliance on large-scale training datasets, which is a common challenge when applying ViTs to RSI classification tasks.

• (2) We introduce a lightweight ensemble method, termed ESE, to further boost the performance of CNNs or ViTs in the RSI classification. The ESE model, with its compact size and concise generating algorithm, outperforms other advanced methods by a significant margin. For instance, on the most challenging NWPU45 dataset, our ESE models achieved an OA of up to 96.8%, approximately 1.1% higher than the top-ranked method in existing literature. Notably, the improvement within our smallest ESE is accompanied by a reduction of up to 90% in parameters and 74% in inference time, respectively.

• (3) Our method presents an efficient and straightforward way to boost the performance of existing models. Importantly, it sidesteps the complexities associated with model reconfiguration. As a result, it significantly reduces the barriers to developing DL methods for RSI classification tasks. This accessibility is particularly advantageous for geoscience researchers who may have limited experience with model modifications.

The structure of this paper is as follows: section “Related works” presents a review of relevant literature. Section “Methodologies” outlines the methodologies employed in this study, including the QA strategy, QAM, method framework, mathematical foundations, and other key parameters. Section “Experiments and results” provides a comprehensive analysis of the experimental results. Finally, section “Conclusions” concludes the paper and summarizes the findings of the study.

Related works

Single model approach

Researchers often rely on transfer learning for RSI classification, using deep models pre-trained on ImageNet-1 K, as seen in models like LS-EfficientNet⁵. However, some studies have explored alternative approaches, such as handcrafted designs or neural architecture searches, to identify more effective solutions distinct from traditional transfer learning. For example, TPENAS⁶ proposes a two-phase evolutionary neural architecture search framework to optimize deep learning models for RSI classification, with the first phase determining the optimal model depth and the second phase searching for the best overall structure. Similarly, LCNN-HWCF¹¹ addresses RSI classification challenges through dimension-wise convolution and hierarchical-wise convolution fusion, reducing model parameters and computational complexity while maintaining a balance between speed and performance. Although these methods offer smaller model sizes than established CNNs pre-trained on ImageNet-1 K, they rely solely on RSI datasets for training and validation, lacking the benefit of prior knowledge from large-scale datasets, which results in comparatively lower performance than transfer learning-based methods like LS-EfficientNet.

With the emergence of ViT, this new architecture has been quickly adopted and proven effective for RSI classification. For example, ET-GSNet¹² uses ViT as a teacher to guide a smaller ResNet18 student model through knowledge distillation. The method improves classification performance without increasing computational complexity by smoothly transferring dark knowledge from the teacher model to the student model. Compared to CNN, a ViT usually requires more training samples to reach convergence. Consequently, RSP-ViT¹³ uses the MillionAID dataset with a million samples to train their models from scratch. The results show that RSP-ViT significantly improves performance but still faces challenges in addressing task discrepancies across different downstream tasks.

Among various single-model approaches, a common strategy involves incorporating self-designed modules into pre-trained models. For instance, EAM-Net¹⁴ introduces an enhanced attention module to improve the feature extraction and generalization capabilities of CNNs, while SCViT¹⁵ combines both spatial and channel-wise information through a multi-head self-attention mechanism, a lightweight channel attention module to prioritize important channels, and a progressive patch aggregation technique to preserve local structural features. However, these methods often train the modified models directly on RSI datasets without first retraining them on large-scale datasets. As a result, despite theoretical differences in architecture that might suggest improved accuracy, the experimental outcomes show that these methods exhibit similar performance. This indicates that the lack of large-scale pretraining hinders their ability to achieve significant gains.

Feature fusion approach

Since 2022, many feature fusion-based methods have been proposed, achieving notable performance improvements. For instance, GCSA-Net¹⁶ introduces a novel approach that combines global context spatial attention with densely connected convolutional networks to effectively capture multiscale global features. Similarly, ACGLNet¹⁷ employs a dual-model deep feature fusion strategy, integrating two CNNs to extract complementary multiscale features, using spatial attention to filter out redundant background information, and applying bilinear fusion to combine global and local features. Additionally, SF-MSFormer¹⁸ presents a spatial-frequency multiscale Transformer framework, which integrates spatial-domain and frequency-domain branches to capture comprehensive global multiscale features. However, a common drawback of these methods is the absence of retraining on large-scale datasets, which limits their competitive performance and generalization across diverse datasets.

Furthermore, researchers have extensively investigated techniques for combining multiple models, demonstrating competitive performance but sacrificing their model size and inference speed. For instance, GRMA-Net¹⁹ introduces a gated recurrent multi-attention neural network that leverages multilevel attention modules to focus on informative regions and capture discriminative features from both shallow and deep layers. It further incorporates a deep-gated recurrent unit to model long-range dependencies and contextual relationships. By comparison, P2FEViT²⁰ introduces a hybrid architecture combining CNNs and ViTs to enhance classification performance by capturing both global context and local multimodal information. This approach leverages CNN features embedded into the ViT framework, improving classification performance, reducing reliance on large-scale pre-training data, and accelerating model convergence with minimal training data.

Local features are critical for pattern recognition in RSI samples, especially when handling low-resolution images. To address the limitations of ViTs in capturing these features, TSTNet²¹ introduces a two-stream Swin Transformer network that combines original image features with edge information through a differentiable edge Sobel operator module in the edge stream. Additionally, IBSwin-CR²² enhances the Swin Transformer by incorporating an inductive biased shifted window multi-head self-attention module, a random dense sampler, and cyclic regression loss. These modifications effectively capture long-range dependencies and spatial information in RSI samples. Despite these advancements, a common drawback of ViT-based methods is their large model sizes, which can limit efficiency and scalability.

Ensemble approach

Ensemble-based methods have become increasingly prevalent in satellite data analysis, with applications ranging from hyperspectral image processing^23,24 to land cover change detection^25,26. Since 2020, several ensemble approaches have been proposed for RSI classification. For instance, the CNN–HMM model²⁷ combines a modified multi-scale CNN with a hidden Markov model, effectively extracting multi-scale structural features while minimizing computational complexity. This method leverages CNNs for feature fusion and HMMs for context information mining. In comparison, MGML-FENet²⁸ introduces a multigranularity multilevel feature ensemble network by combining a multigranularity feature fusion branch and a multigranularity feature ensemble module to effectively extract and combine diverse features across different network levels. This approach leverages channel-separate and full-channel feature generators to enhance feature extraction and minimize the impact of confusing information. Similarly, ESD-MBENet²⁹ employs a compact multibranch ensemble network that enhances feature representation by fusing output logits and intermediate feature maps. It addresses the challenges of model complexity and inference efficiency by sharing weights across branches and incorporating feature augmentation. Furthermore, the integration of a self-distillation technique enables the main branch to approximate the performance of the full ensemble, significantly reducing computational overhead during inference. Despite these advancements, a common drawback of these methods is their large model sizes, which can limit efficiency and scalability.

Currently, CNN-Transformer hybrid models have demonstrated the potential to leverage multi-modal advantages^30,31. Furthermore, efficient classifiers are crucial for real-world remote sensing applications, such as onboard satellite image classification³² or few-shot satellite image classification^33,34. However, to the best of the authors’ knowledge, efficient yet accurate methods using a hybrid ensemble approach are quite rare in RSI classification.

Methodologies

Structural components of QAM

Fig. 3

Training framework and structural components of QAM (Source: Authors own work by Python version 3.9).

The training framework for CNN or ViT models is depicted at the top of Fig. 3. As indicated by the red solid arrows, the original RSI sample is first processed by the QAM. The outputs from the QAM are then fed into the models for training. As shown by the red dotted arrow, the m-OLS updates its soft labels at each training epoch.

As illustrated at the bottom of Fig. 3, the QAM comprises a series of Gated Regularization Operators (GROs) arranged in sequence. Notably, each GRO has a shortcut that outputs the original inputs. Let ${x}_{i}$ and ${x}_{i}^{{\prime }}$ represent the original RSI samples and the outputs of the GROs, respectively. Let ${P}_{n}$ denote the threshold value of probability to activate the transformation, and let ${p}_{n}$ denote the result of the probability calculator in the GROs. The regularization algorithm in the GROs, denoted as ${f}_{GRO}$, can be described as follows:

$${x}_{i}^{{\prime }}=\left\{\begin{array}{ll}{x}_{i},&\quad {p}_{n}<{P}_{n}\\ {{f}_{GRO}(x}_{i}), &\quad {p}_{n}\ge {P}_{n}\end{array}\right. .$$

(1)

At present, we utilize a maximum of eight GROs within the QAM. These regularization functions include color jitter, horizontal and vertical flips, rotation, grayscale, auto contrast, Gaussian blur, and CutMix, arranged sequentially. We have chosen these functions only from the PyTorch library, which facilitates the replication of our methods by readers.

Importantly, we set a threshold for the activation of the GROs to ensure a quantitative application of these regularization functions. The thresholds for the grayscale and auto contrast functions are specifically set at 0.3, while those for the remaining functions are set at 0.5 according to the ablation experiment results proposed in subsequent sections. Furthermore, we have deactivated several GROs for certain ViT models due to their inferior bias-inducing capabilities compared to CNNs. Namely, we can dynamically adapt the architecture of the QAM to accommodate different models or datasets.

Architecture of the ESE

Fig. 4

Architecture of the ESE (Source: Authors own work by Python version 3.9).

The architecture of the ESE is illustrated in Fig. 4. In essence, the ESE architecture consists of two individual classifiers chosen from the five candidate models within the table in Fig. 4, each with fixed weights. These classifiers are independently trained on RSI datasets. During inference, the prediction scores from both classifiers are weighted by a hyperparameter, denoted as α. These weighted outputs are subsequently fused to generate the ensemble’s final prediction.

Although the accuracy of the ESE can be further enhanced by incorporating additional individual classifiers, experimental results show that performance improvements become marginal as the number of classifiers increases. Therefore, the ESE is configured with two classifiers as the default setting to balance accuracy, model size, and inference speed.

Implementation of the ESE

The ensemble method typically includes bagging, boosting, and stacking³⁵, while ensembles consisting of different classifiers have been widely proven to outperform a single classifier^36,37. Generally, bagging and boosting manipulate training sets or training algorithms to obtain diverse classifiers, while stacking uses different models. Accordingly, previous ensemble methods for RSI classification often consist of complex pipelines, too many components, and huge model sizes. In contrast, our ESE method has only two classifiers, ${C}_{A}$ and ${C}_{B}$. We first weight the outputs of ${C}_{A}$ and ${C}_{B}$ using one hyperparameter, then use the sum of the weighted outputs as the ensemble’s output.

The benefits of the ESE are threefold: First, the ESE is a straightforward ensemble created by stacking two distinct CNN or ViT models. In comparison to prior ensemble methods, the ESE significantly reduces the number of individual classifiers and parameters, eliminating the need for additional remodeling processes. Second, the training process of its individual classifiers only utilizes traditional fine-tuning, thereby excluding any modifications to the open-source models. Finally, the hybrid ESE straightforwardly leverages the strengths of both CNNs and ViTs, resulting in a more robust yet efficient model. Let ${F}_{ESE}$ denote the ESE ensemble model. Then, ${F}_{ESE}$ can be expressed as follows:

$${F}_{ESE}=\alpha\times{C}_{A}+{\left(1-\alpha\right)\times\:C}_{B},$$

(2)

where $\alpha$ is a hyperparameter set to [0, 1].

In accordance with the definition provided in Eq. (2), we proceed to choose 99 classifiers from the ESE’s candidate hypothesis set, utilizing a minimum α change of 0.01. Specifically, the first selected classifier has an α value of 0.01, while the 99th classifier has an α value of 0.99. Following this, we validate these 99 classifiers and select the one with the highest OA as the optimal solution for ESE.

Model’s architecture

To assess our method, we utilized three CNNs: EfficientNet-B0, EfficientNet-B3, and ResNet-50, in addition to two ViTs: Swin-T-Tiny and N-ViT-Small (N-ViT-S). Detailed descriptions of these five models are available in the Refs.^38,39 and⁴⁰.

In contrast to ResNet-50, both EfficientNet-B0 and -B3 incorporate an internal channel attention structure. ViTs, on the other hand, exhibit self-attention structures combined with position embedding parameters. Furthermore, N-ViT retains convolutional layers as its feature meta-extractors, while Swin-T employs fixed windows for local feature splitting. Consequently, N-ViT demonstrates superior capability in inducing feature biases compared to Swin-T.

In terms of classification performance on ImageNet-1 K, N-ViT-S achieves the highest ranking, followed by Swin-T-Tiny, EfficientNet-B3, -B0, and finally ResNet-50.

Algorithms

Algorithm 1

Training procedure of CNN or ViT models (pseudocode).

The training algorithm for all CNNs and N-ViT-S is shown in Algorithm 1. The training process spans 300 epochs, with a fixed batch size of 32. However, we introduce a bifurcation in the training process specifically for EfficientNet-B0 to obtain diverse classifiers. Initially, we designate the training epochs from 1 to 60 as Step 1, while the subsequent epochs constitute Step 2. In Step 1, we selectively disable CutMix augmentation or not to yield two distinct groups of EfficientNet-B0 classifiers. In contrast, CutMix remains consistently active during Step 2. The initial learning rate for all CNNs is set to 1E−04, while for all ViTs, it is 5E−05. We employ cosine decay as the learning rate scheduler, utilize the Adam-W optimizer, and apply the cross-entropy loss function.

Our experiments indicate that CNNs and N-ViT-S exhibit robust performance due to their ability to induce feature biases. As a result, we disabled the grayscale and auto-contrast functions when training these models. However, we observed that Swin-T converges more slowly than CNNs due to its lesser ability to induce bias. Consequently, we enabled all eight GROs in the QAM when training Swin-T-Tiny, along with modifications to the batch size (increased to 64) and the total training epochs (extended to 600). Additionally, we also implement this enhanced strategy for N-ViT-S on the low-resolution AFGR50 dataset.

Algorithm 2

Ranking selection for m-CutMix samples (pseudocode).

The traditional label-smoothing technique typically employs fixed values as soft labels for dataset categories. As a self-adaptive technique, Zhang et al.¹⁰ proposed OLS to reflect the differences in similarity among categories. Initially, the OLS algorithm sets the soft labels to zero and then dynamically updates these labels based on the correctly classified results on training sets. Consequently, it introduces a hyperparameter, denoted as $\:\delta\:$, to balance the ground truth (hard) and soft labels before outputting the final labels. Let $\:{\:Label}_{H}$and $\:{Label}_{S}$ denote the hard and soft labels, respectively. The final output labels of OLS, denoted as $\:{Label}_{OLS}$, can be described as follows:

$${Label}_{OLS}=\delta \times {Label}_{S}+\left(1-\delta \right)\times {Label}_{H}.$$

(3)

We employ an empirical value of 0.9 for $\delta$ instead of the 0.5 used on ImageNet-1 K, as the similarity among RSIs is higher than that of natural images.

The CutMix algorithm randomly cuts a subarea in a class-A image and then overlaps this patch onto another class-B image. Therefore, if CutMix is used, the cut-and-mixed samples of RSI will contain two category scenes within one image. To guide the model in learning a processed sample, the original CutMix algorithm transforms the sample’s label. Let ${Label}_{A}$ denote a sample’s label from class A, with ${Label}_{B}$ representing a sample’s label from class B. Let $\beta$ be the ratio of the cut patch versus its original image. Then, a cut-and-mixed sample’s label, ${Label}_{CM}$, can be described as follows:

$${Label}_{CM}={\beta \times Label}_{A}+\left(1-\beta \right){\times Label}_{B}.$$

(4)

Moreover, we modified the CutMix algorithm to obtain diverse classifiers for EfficientNet-B3 by using a ranking selection for the cut-and-mixed training samples. As shown in lines 1 to 5 of Algorithm 2, we initially obtain the similarity among a batch of training samples using the similarity matrix from m-OLS. Subsequently, we sort those samples in the batch according to their similarity and employ a ranking selection number, denoted as $N$, to choose the sample list for cutting patches. Specifically, we set $N$ in m-CutMix equal to 15, or the category number of datasets, to obtain two groups of EfficientNet-B3 classifiers. In other words, we continue to use the original CutMix for training the other models.

Algorithm 3

Testing procedure of the ESE (pseudocode).

Algorithm 3 provides the process to obtain the final ESE classifier. Specially, we verify the performance of the 99 candidate ensembles with an increase of 0.01 to α per epoch. It is much simpler than other stacking strategies that use extreme gradient boosting to generate the final prediction.

Experiments and results

Dataset introduction

This study utilizes three RSI datasets, specifically AID30, NWPU45, and AFGR50, as benchmarks for assessing the performance of the proposed method. Representative samples from each category within these sets are illustrated in Figs. A1–A3 (located in the Appendix section). As indicated in Table 2, all datasets, with the exception of AID30, exhibit balance across categories. In addition, AFGR50 comprises 50 categories of fine-grained aircraft images, while the remaining datasets consist of classic RSIs depicting ground scenes. Consistent with previous studies, this study also randomly divides all the RSI sets into training and testing subsets.

Table 2 Summary of datasets and TRs.

Evaluation metrics

For performance evaluation, we used two metrics: OA and the confusion matrix. Let ${N}_{c}$ and ${N}_{t}$ represent the total number of correctly classified samples and the sum of tested samples, respectively. Then, OA is calculated as follows:

$$OA= \frac{{N}_{c}}{{N}_{t}}.$$

(5)

The confusion matrix is a table that contains all category names of a set and provides detailed results per category, including correctly classified and misclassified amounts.

Implementation details

Hyperparameter settings

The models were trained using the Adam-W optimizer with a weight decay of 10⁻⁶, and all datasets were processed with an input resolution of 256 × 256. The initial learning rate was gradually adjusted throughout the training using a cosine decay schedule. To ensure robustness and reliability, the outcomes of all experiments are presented as averages, calculated from a minimum of three trials.

Experimental environment

The experiments were conducted on three distinct RTX 4070Ti graphics cards, utilizing PyTorch version 2.10.0 in the Ubuntu 20.04 environment. The memory requirement for the Ubuntu 20.04 environment does not exceed 16 GB.

OA results of individual classifiers

In this study, we construct the ESE by utilizing four variations of Algorithm 1 to train EfficientNet-B0 and B3. All modifications to Algorithm 1 originate from setting changes for m-CutMix. For EfficientNet-B0, we derive two classifiers, termed Dual-CutMix-B0 (DC-B0) or Single-CutMix-B0 (SC-B0), by activating the original CutMix in Steps 1 and 2, or only in Step 2, respectively. For EfficientNet-B3, m-CutMix employs two variants: random selection and sorted selection (i.e., $N$ in Algorithm 2 equals 15), with the cut-and-mix operation consistently active. This results in two additional classifiers, termed Random-CutMix-B3 (RC-B3) and Sorted-CutMix-B3 (SC-B3). Additionally, we procured the N-ViT-S and Swin-T-Tiny classifiers using Algorithm 1, with varying training batch sizes and training epochs as introduced in Sect. 3.5.

The OA results of the six models are displayed in Table 3, with the value in bold indicating superior accuracy within a certain column. The results reveal that the two ViTs outperform the other four CNNs on the AID30 dataset, but Efficient-B3 still presents competitive accuracy on the other datasets in comparison with its two ViT competitors. Furthermore, Efficient-B3 even surpasses Swin-T-Tiny on the AFGR50 dataset. AFGR50 consists of low-quality samples at a resolution of 128², while the sample resolution of AID30 is 600². By comparison, the target in the AFGR50 samples is only one aircraft, but samples in AID30 always have multiple targets in a scene. Therefore, Efficient-B3 excels on AFGR50 due to its proficiency at inducing feature biases. On the other hand, ViTs excel on AID30 due to their advantage in capturing global feature dependencies. Consequently, N-ViT-S presents superior and robust accuracy across different datasets due to its CNN-ViT hybrid structure. Similarly, Swin-T-Tiny shows lower OAs on the NWPU45 or AFGR50 due to its limited capability of inducing bias.

Table 3 OAs (%) of different classifiers on the three RSI datasets.

Additionally, the OA results from different models on the AID30 (TR-50%) and AFGR50 (TR-30%) are similar. This suggests that models reach saturation when using these larger TRs as indicators. Therefore, a performance comparison using OAs on a smaller TR or on the challenging NWPU45 dataset is more objective.

OA results of ESE

We assessed the ESE method by applying it to enhance five classifiers listed in Table 3. Initially, we introduced three ESEs to augment the performance of RC-B3. These include: RD (RC-B3 with DC-B0), RS (RC-B3 with SC-B0), and DualB3 (RC-B3 with SC-B3). Subsequently, we introduced two ESEs to augment SC-B3, namely SD (SC-B3 with DC-B0) and SS (SC-B3 with SC-B0). In the third step, we introduced DualB0, an ESE to enhance DC-B0, which consists of DC-B0 with DC-B0. In the fourth step, we introduced two ESEs to augment Swin-T-Tiny, namely DNV (DC-B0 with N-ViT-S) and RNV (RC-B3 with N-ViT-S). Finally, we introduced two more ESEs to enhance Swin-T-Tiny, namely DST (DC-B0 with Swin-T-Tiny) and RST (RC-B3 with Swin-T-Tiny).

As depicted in Table 4, we organized the OA values of the proposed ESE classifiers into five sections, separated by bold black lines. The bold values signify the highest OA within a column of these sections. In the first section, it is evident that RC-B3 can achieve substantial OA enhancements by collaborating with other EfficientNe-B3 or -B0 models. However, ESEs comprising different architectures, such as RD-ESE or RS-ESE, exhibit superior OA values. This observation establishes the first principle of our ESE method: incorporating two classifiers with varied architectures in ESE results in better performance. In the second section, it is clear that the ESE method can also augment the accuracy of SC-B3. Yet, SC-B3 displays lower OAs on the AFGR50 compared to RC-B3. As a result, SD-ESE and SS-ESE also demonstrate lower OAs on the AFGR50. This observation establishes the second principle of our ESE method: the inclusion of a more robust classifier in ESE leads to better performance.

Table 4 OAs (%) of different ESE classifiers on the three RSI datasets.

In Tables 3, 4 and 5, the OA results validate the aforementioned principles once again. Comparatively, the ESE classifiers comprising N-ViT-S exhibit superior OAs than those comprising Swin-T-Tiny. However, our ESE method amplifies the OAs of Swin-T-Tiny more significantly than those of N-ViT-S, though Swin-T-Tiny has a limited capability for inducing feature biases. Therefore, this observation validates that our ESE method is an effective and efficient strategy to amalgamate the benefits of CNNs and ViTs.

Table 5 OAs (%) of different ESE-9 C classifiers on the three RSI datasets.

Sensitivity analysis

The process of generating ESEs entails two pivotal decisions that can significantly influence the performance of the method. Initially, we implement an optimal search among 99 potential ESEs, a step that corresponds to substantial time expenditure during training. Subsequently, we restricted our use to only the EfficientNet-B0 and -B3 models as CNN classifiers in previous experiments. This restriction, however, limits the generality of the ESE method when other CNNs are excluded from the method’s framework. To address these limitations, we conducted two supplementary experiments with two primary objectives: firstly, to curtail the time expenditure associated with generating ESEs, and secondly, to verify the applicability of generating ESEs using another CNN.

First, we select only nine candidate ESEs (denoted by a suffix: 9 C) by setting α in Eq. (2) to increase from 0.1 to 0.9 at a 0.1 interval. We present the corresponding OA results in Table 5. In the table, we omit the values of the TRs where models reach saturation, while the values in bold denote a decreased OA value. As Table 5 reveals, the accuracy losses of all these ESE classifiers obtained by our nine-candidate strategy are minimal, ranging from 0.0 to 0.05%. This observation demonstrates the robustness of the ESE method against variations in α. Consequently, we can adopt the nine-candidate strategy to reduce time costs by 91% during the verification of ESE classifiers, with a negligible sacrifice in accuracy of up to 0.05%.

In addition, we replaced the EfficientNet-B0 or -B3 model with a Resnet50 model in various ESEs such as RD-ESE, DuanlB0-ESE, RST-ESE, or RNV-ESE. Table 6 illustrates the OA comparison for these ESE classifiers, now based on Resnet50 (denoted by a suffix: R50), divided into four unique sections. We excluded the TRs where models reached saturation, and the values highlighted in bold or italics represent a decrease or increase in OA, respectively.

Table 6 OAs (%) of different ESE-R50 classifiers on the three RSI datasets.

The first half of the sections show that substituting Efficient-B0 with Resnet50 results in a slight OA improvement for DuanlB0-ESE on three datasets, except for the AFGR50, while similar OA enhancements for RD-ESE are negligible. On the contrary, substituting Efficient-B3 with Resnet50 within RST-ESE or RNV-ESE leads to more noticeable reductions in OA, as depicted in the last two sections of Table 6. Most importantly, this substitution using Resnet50 leads all four ESEs to show significant OA decreases on the low-resolution AFGR50 dataset, especially when Swin-T-Tiny, with its limited capability for inducing feature biases, is included in the ESE model.

Despite this, all these Resnet50-based ESEs still exhibit improved OAs when compared to the single-CNN or -ViT classifiers in Table 3. Therefore, this observation indicates that our ESE method retains its effectiveness when different CNNs are incorporated into the ESE model. However, the ESE method can achieve better performance, particularly when it leverages the benefits of superior CNNs to enhance its ViT partner.

Performance comparison with previous methods

This study conducts an exhaustive performance comparison of the proposed models with 48 cutting-edge methods published between 2020 and 2024. The comparative results are encapsulated in Tables 7, 8 and 9. Within these tables, the term ‘None’ signifies the absence of detailed information in the corresponding literature, while ‘>’ denotes values estimated by the base model. Furthermore, values highlighted in bold indicate superior performance within a specific column.

Table 7 OA results across various methods on the AID30 dataset.

Table 8 OA results across various methods on the NWPU45 dataset.

Table 9 OA results across various methods on the AFGR50 dataset.

Results on AID30

As Table 7 illustrates, two-thirds of the previous methods achieved OAs below 96.0% when using the TR of 20% on the AID30 dataset as a test bed. In contrast, our ESE, CNN, and ViT models demonstrate significantly higher accuracy, reaching up to 97.9%, 97.2%, and 97.6%, respectively. Only two previous methods that employ dual Swin-T models, specifically TSTNet and IBSwin-CR, have achieved accuracies closely matched with our CNN or ViT models. However, these two methods both have an excessively large model size, which hinders inference speeds. Therefore, our ViT or ESE models can perform better with an 80% reduction in parameters compared to their top competitors.

Our methods maintain strong performance at TR-50%, although several previous methods achieve similar OA values. However, deep learning models tend to reach saturation when sufficient training samples are available, making TR-20% a more objective measure of performance. Therefore, using the TR-20% as a performance evaluator is more objective. Nonetheless, the OA values reveal that most previous methods still present OAs below 98.0% at TR-50%, even when employing multiple models. Thus, the comparison on the AID30 dataset also underscores the advantage of our methods, particularly when the availability of training samples is constrained.

Results on NWPU45

As Table 8 illustrates, all of our models achieve an accuracy exceeding 94.4% when using a TR of 10% on the NWPU45 as a benchmark. Specifically, the OA of the ESE models reaches up to 95.3%, surpassing N-ViT-S and Swin-T-Tiny with 0.5% and 0.9% improvements, respectively. In contrast, 34 out of 39 previous methods yield an OA value below 93.5%, while only P2FEViT achieves an OA value of 94.9%. Moreover, TSTNet and IBSwin-CR, which perform well on the AID30, both yield inferior OAs below 94.1%.

Similarly, our models both achieve an accuracy exceeding 96.0% when using a TR of 20% as a performance benchmark. Notably, our RST-ESE model yields OAs up to 96.8%, surpassing the top-ranked P2FEViT or TSTNet with approximately a 1.1% OA improvement. In comparison, most previous methods yield OAs below 95%. This phenomenon substantiates that our models consistently exhibit robust generalization capabilities when utilizing different TRs on either the AID30 or NWPU45 datasets as benchmarks.

P2FEViT employs a CNN-ViT hybrid architecture with a substantial model size of approximately 112 million parameters. Consequently, P2FEViT presents a slight OA improvement over our ViT models at a TR of 10% due to its hybrid structure. However, our hybrid ESE models not only exhibit significant OA improvements of up to 1.1% but also demonstrate a 70% reduction in parameters when compared to P2FEViT. Therefore, this comparison underscores that our ESE method is a more efficient strategy to leverage the strengths of both CNNs and ViTs.

Results on AFGR50

Table 9 presents the comparison of OA on the AFGR50 dataset, demonstrating that MBC-Net holds the highest OA among previous methods. However, MBC-Net exhibits gaps in OA of 1.6–2.3% when compared to our ViT and ESE methods at a TR of 10%. As the TR increases to 20%, MBC-Net narrows the OA gaps to between 0.8% and 1.6%. When models reach saturation at TR-30%, P2FEViT also exhibits OAs comparable to our single-model methods, yet still presents significant OA gaps in comparison to our ESE methods.

In conclusion, based on the OA results depicted in Tables 7, 8 and 9, it is clear that our QA strategy is highly competitive for enhancing CNN or ViT in classifying RSIs. Generally, in comparison to previous state-of-the-art methods in the literature, our single-CNN or single-ViT models provide advantages in terms of accuracy and efficiency. Furthermore, our ESE method takes a step further to offer a lightweight and beginner-friendly approach for integrating the benefits of CNNs and ViTs, which demonstrates further improvements in accuracy while maintaining efficiency and simplicity.

Confusion matrixes

In this section, we introduce the confusion matrix figures (CMFs) of RNV-ESE and RST-ESE to assess performance across various categories. We apply the CMFs at a TR of 20% on the AID30, NWPU45, and AFGR50 datasets. Figures 5, 6 and 7 display the CMFs for RNV-ESE, while we present those of RST-ESE in Figs. A5–A7 in the Appendix. In these CMFs, we interpret a value of 1.0 as an OA of 100%. Category names highlighted in yellow or values in bold indicate the most confusing class (referring to categories with low accuracy scores) in a given dataset. We use blue values to signify a misclassified ratio (the percentage of all class-A samples misclassified as class-B) that exceeds 3.0%. For enhanced readability, we exclude categories that achieve a 100% OA on the AID30 or an OA above 98% on the NUPU45 and AFGR50 from the CMFs.

Fig. 5

The confusion matrix of RNV-ESE on the AID30 dataset (TR = 20%) (Source: Authors own work by Python version 3.9).

As Fig. 5 illustrates, 23 categories within AID30 have an OA below 100%, but only 8 of these 23 classes have an OA below 98.0%. More specifically, only three categories, including the most confusing ones of resort, school, and square, present OAs lower than 93.0%. Additionally, only the park category has a misclassification ratio larger than 3.0%. This observation reveals that less than one-third of categories exhibit high inter-class similarity, and the most confusing ones pose the key challenge within AID30. The sample numbers of the resort and school classes are 290 and 300, respectively, both below the average value of 333 within all categories. Therefore, this CMF of AID30 further substantiates that using the TR of 20% on AID30 as a test bed is more convincing than the TR of 50%.

Fig. 6

The confusion matrix of RNV-ESE on the NWPU45 dataset (TR = 20%) (Source: Authors own work by Python version 3.9).

As depicted in Fig. 6, 25 categories within NWPU45 have OAs below 98.0%, with 7 and 5 of these 25 classes presenting OAs below 94% and 93.0%, respectively. Additionally, 8 categories in this CMF have a misclassification ratio larger than 3.0%, with three of them having a high misclassification value, reaching up to 7.5%. These observations indicate that the inter-class similarity and intra-class dissimilarity within NWPU45 are more pronounced compared to AID30.

The categories within NWPU45 that pose significant challenges include the church, dense residential, medium residential, palace, and rectangular farmland. Additionally, the mountain and wetland classes register an OA below 94.0%. In comparison, the challenging classes within AID30 do not include nature scenes. This difference indicates that the inherent features of RSIs include not only noise backgrounds associated with human settlements but also a state of low-quality imaging. As a result, the NWPU45 dataset serves as a more credible benchmark for performance evaluation.

Fig. 7

The confusion matrix of RNV-ESE on the AFGR50 dataset (TR = 20%) (Source: Authors own work by Python version 3.9).

Figure 7 shows that 22 categories within AFGR50 present OAs below 98.0%, and six categories have OAs under 93.0%. Additionally, nine categories exhibit a misclassification ratio above 3%, as the blue indicators denote. AFGR50 is a fine-grained dataset that includes various types of aircraft at a resolution of 128². However, each sample in AFGR50 features a plain background. As a result, adverse imaging conditions pose the main challenge for accurate recognition within AFGR50. Therefore, the CMF for AFGR50 highlights the significant impact of imaging conditions on model performance.

Comparing our RNV-ESE with earlier methods, similar categories of confusion are evident across the AID30, NWPU45, and AFGR50 datasets. However, RNV-ESE exhibits enhancements in OA across all categories, not just a specific class. Some prior methods have excelled in certain AID30 categories but fell short in the more challenging NWPU45 categories. For example, the square category of AID30 shows a high OA with the TRS method. This suggests that some previous methods lack consistent performance across various RSI datasets.

From all CMFs and comparisons, it’s clear that the main source of RNV-ESE’s performance improvements is the superior quality of the training data. Our QA strategy effectively aligns the data distribution features between training and testing subsets, leading to an overall enhancement in model performance across all categories.

Visualization and analysis

We use two visualization methods to highlight two critical aspects of the proposed models: activation mappings and feature effectiveness. We initially use the gradient-weighted class activation mapping (Grad-CAM)⁷¹ to visualize the essential information within the model predictions. We then use the t-Distributed Stochastic Neighbor Embedding, commonly abbreviated as t-SNE, to demonstrate the effectiveness of the model features.

Grad-CAM results

Fig. 8

Grad-CAM visualizations of models on three datasets (Source: Authors own work by Python version 3.9).

Figure 8 presents the Grad-CAM figures (GCFs) across ten distinct RSI categories, with the first through ninth categories corresponding to three particularly challenging classes from the AFGR50, AID30, and NWPU45 datasets, respectively. The final category, representing the river, is a typical scene dominated by long-range watercourses. These GCFs offer a detailed view of how different models process and highlight key areas within the scenes, providing insights into the models’ varying strategies for feature extraction.

The second-to-last row in the figure displays the GCFs for the RC-B3 model. These GCFs reveal that RC-B3 focuses on local features of ground targets, such as the wings of aircraft or the rounded roofs of buildings. These localized features are highlighted with brighter colors, indicating higher activation levels. While RC-B3 is effective at identifying these key visual patterns within specific RSIs, its reliance on convolutional operations limits its ability to capture long-range dependencies, which could be crucial for understanding more complex scenes. This limitation is particularly evident in scenarios where the relationships between distant features are critical for accurate recognition.

In contrast, the third-row GCFs for the N-ViT-S model demonstrate a more sophisticated approach. While N-ViT-S still identifies local features as key activation areas, it goes a step further by capturing long-range dependencies between these features. This is especially noticeable in the resort and river categories, where N-ViT-S exhibits clearer distinctions compared to RC-B3. The model’s ability to integrate global context through its vision transformer architecture enables it to recognize more complex relationships within the scene, improving its performance over traditional convolution-based models like RC-B3.

The final row of GCFs showcases the Swin-T-Tiny model, which introduces a unique feature extraction strategy. Instead of relying on convolutional operations like RC-B3 or the transformer mechanisms of N-ViT-S, Swin-T-Tiny utilizes window-based splitting operations. This allows the model to first capture local features and then process dependencies between them. Swin-T-Tiny places significant emphasis on the central regions of ground targets, which can be crucial for accurate scene recognition. Its ability to handle both local and global dependencies through this window-based mechanism enables it to achieve a more comprehensive understanding of complex RSIs, particularly in large or intricate scenes.

Overall, the GCFs vividly illustrate the architectural differences in feature extraction strategies across the models. RC-B3 excels at local feature recognition but struggles with long-range dependencies. N-ViT-S effectively integrates the strengths of both CNNs and ViTs, balancing the need for local feature extraction with the ability to capture global relationships. Swin-T-Tiny, with its window-based approach, focuses on central features and dependencies, offering a more holistic understanding of complex scenes. These differences underscore that N-ViT-S, by leveraging both convolutional and transformer architectures, demonstrates superior performance in recognizing both high- and low-resolution RSIs, positioning it as the most robust model among the three for diverse input data.

t-SNE results

Fig. 9

t-SNE Visualization of RNV-ESE on the AID30 Dataset (TR = 20%) (Source: Authors own work by Python version 3.9).

The t-SNE clustering figure (TCF), a bi-dimensional projection of post-principal component analysis processed model features, is illustrated in Fig. 9. It shows that all AID30 categories are distinctly spaced. Yet, two areas (marked by a red rectangle) contain samples that overlap from four classes. Rectangle A includes the commercial and school classes, while rectangle B pertains to the resort with parking. Earlier confusion matrices have suggested a higher similarity between these four categories. Therefore, this observation aligns consistently with the TCF representation.

Fig. 10

t-SNE Visualization of RNV-ESE on the NWPU45 Dataset (TR = 20%) (Source: Authors own work by Python version 3.9).

As illustrated in Fig. 10, all categories within NWPU45 are distinctly separated, but six classes within three red rectangles contain overlapping samples. Rectangle A encompasses the dense and medium residential categories; rectangle B includes the church and palace; and rectangle C comprises the railway and the railway station. This observation also aligns with the previously identified challenging categories presented by confusion matrices.

Fig. 11

t-SNE visualization of RNV-ESE on the AFGR50 Dataset (TR = 20%) (Source: Authors own work by Python version 3.9).

Figure 11 illustrates that all categories within AFGR50 are distinctly separated. Yet, the TCF uncovers that six categories have overlapping samples, as indicated by the red rectangles. The categories posing challenges include classes A25 and A41 in rectangle A, and classes A9, A14, A34, and A37 in rectangle B. This TCF mirrors the difficult categories within AFGR50 that were previously identified by confusion matrices. Furthermore, it substantiates that the misclassification within AFGR50 is more pronounced among all classes compared to those in NWPU45. Essentially, this observation affirms that the TCF for AFGR50 is capable of consistently revealing patterns of misclassification that align with those identified in the confusion matrices. Therefore, we have excluded the TCFs of RST-ESE from our discussion, as their t-SNE results exhibit a similar level of consistency with their confusion matrices.

Contrary to previous studies^{12,46,58,59,60,62,27,28,29}, our TCFs display superior clustering results across various datasets, thereby maintaining an impressive performance consistent with the OA results. Consequently, the effectiveness of the RNV-ESE’s feature structure is confirmed by all the TCFs.

Computational efficiency

We have compared the computational efficiency of our method with several previous methods, including ViT-Base16⁴⁴, P2FEViT²⁰, and TSTNet²¹. We replicated these methods using only their base models, without any functional modules proposed in the corresponding literature. This implies that our evaluated inference speeds for these methods are faster than their actual speeds. Given that the number of activations can significantly influence a model’s inference speed on GPUs⁷⁰, we used computational efficiency in practical inference as a metric. We conducted experiments with all models by predicting the category label of 25,200 RSI samples at a resolution of 224². The test environment was consistent with the previously introduced one, running on a single RTX-4070Ti GPU. The number of floating-point operations (FLOPs) is measured in giga (G).

As shown in Table 10, our single-model methods demonstrate significant OA improvements when using the TR-20% on NWPU45 as a test bed. Furthermore, our single-model methods have superior inference speeds and fewer parameters. For example, N-ViT-S shows superior OA than the top-ranked TSTNet in the literature, with an 81% and 72% reduction in parameters and inference time, respectively. Moreover, this comparison also reveals that Swin-T is more efficient than ViT-Base16, but only our QA strategy has significantly enhanced the former’s capability when compared to TSTNet, which employs dual Swin-T-Base models.

Table 10 Comparative analysis of computational efficiency across different methods.

Furthermore, our ESEs have achieved enhanced performance with little sacrifice in model size or inference speed when compared to the top-ranked TSTNet or P2FEViT. For instance, RST-ESE shows superior OA than TSTNet, with a 77% and 64% reduction in parameters and inference time, respectively. Similarly, RNV-ESE significantly outperforms TSTNet, with a 71% and 58% reduction in parameters and inference time, respectively. In particular, RD-ESE outperforms TSTNet, with remarkable reductions of 90% and 76% in parameters and inference time, respectively.

Ablation experiments

To underscore the superiority of our QA method over traditional data augmentation, we performed ablation studies on the AID30, NWPU45, and AFGR50 datasets, respectively. The OA results are presented in Tables 11 and 12. Our method has demonstrated significant advantages, especially when training samples are limited. As such, we employed three challenging TRs as test beds (TR-20% for AID30, TR-10% for NWPU45, and TR-10% for AFGR50).

Table 11 OAs (%) from ablation experiment A on the AID30, NWPU45, and AFGR50 datasets.

Table 12 OAs (%) from ablation experiment B on the AID30, NWPU45, and AFGR50 datasets.

In Table 11, there are three partitions divided by a bold black line, presenting the ablation results of N-ViT-S, Swin-T-Tiny, or RC-B3, respectively. ‘Baselines’ refer to the original settings of the proposed QAM during training, while the second to seventh lines of the ‘Test Item’ column correspond to a specific operator being tested. The key difference between our QA and usual data augmentation lies in the fact that the latter persistently activates these operators during model training. As a result, each operator in the first partition has an operation probability (OP) of 100%. In particular, ‘Overdone’ signifies a situation where all operators in the QAM are perpetually active (all OPs = 1.0), which aligns more closely with the strategy of qualitative augmentation for model training. In addition, bold or italics values indicate a decrease or increase in OA compared to their baselines, respectively. We have highlighted the crucial information in bolditalics for improved readability. In contrast, Table 13 maintains the same settings as Table 11, except that all OP values are set to 0. These experiments expose the performance degradation across three models when the augmentation strategy is insufficiently robust.

As indicated in Table 11, setting the OP of most QAM operators at 1.0 sequentially reduces the OAs of N-ViT-S on the AFGR50 and NWPU45 datasets. However, the reductions on AID30 are minor. Conversely, setting all QAM operators with a 1.0 OP simultaneously results in substantial reductions for N-ViT-S on all datasets. In comparison, the decreased OAs on NWPU45 and AID30 are smaller and smallest, respectively. Furthermore, we also observed that both Swin-T-Tiny and RC-B3 experience declines in OA, with the OA decline on AFGR50 being the most severe. This observation suggests that RSIs with a higher resolution can mitigate the impact of qualitative regularization on the performance of all models. However, both the CNN and ViT models will still be suboptimal when traditional qualitative regularization is active during training.

As demonstrated in Table 12, disabling a certain GRO within the QAM results in noticeable decreases in OAs for the N-ViT-S model. In particular, the results further reveal that all three models encounter significant OA declines from 1.3 to 7.4% on three datasets when the six GROs in the QAM are all deactivated. Similar to the results in Table 12, the degradation on AID30 is moderate, while that on NWPU45 or AFGR50 is more severe. This deterioration worsens as the resolution of the RSI decreases.

In conclusion, the ablation studies presented in Tables 11 and 12 substantiate that our QA strategy markedly surpasses conventional data augmentation in RSI classification. Our QA approach offers a direct method to unlock the potential of open-source models for efficient and accurate RSI classification.

Conclusions

This study presents a straightforward ensemble method, named ESE, for classifying RSIs with notable accuracy. The proposed method has two unique features. First, we introduce the QA method as a replacement for conventional data augmentation. This approach aims to better incorporate the inherent characteristics of RSIs, thereby significantly enhancing the performance of a single CNN or ViT model. Second, we propose a simple algorithm for generating a lightweight ensemble of dual classifiers, which can efficiently leverage the advantages of both CNNs and ViTs to further improve accuracy.

An evaluation of our method was conducted on three separate RSI datasets. The findings illustrate that our QA-enhanced CNNs and ViTs consistently yield robust and superior accuracy across all datasets. Furthermore, the results confirm that our ESE technique can augment accuracy beyond that of single models. Comprehensive experiments reveal that the exceptional performance of the ESE method primarily stems from the diverse architectures and high precision of its dual classifier components. Sensitivity analyses suggest that the time costs of generating the ESE models can be reduced substantially by 91%, with only a marginal decrease in accuracy of up to 0.05%.

When compared to the other 48 studies from recent literature, our CNN or ViT models exhibit superior OAs, especially when the number of training samples is scarce. Furthermore, our ESE models display more significant improvements in OA, with a minor sacrifice in model size and inference speed. Notably, when compared against the top-ranked method in existing literature, our ESEs not only display an improvement in OA of up to 1.1% on the challenging NWPU45 dataset but also demonstrate a reduction of up to 90% and 74% in parameters and inference time, respectively. In addition, our methods rely solely on open-source models and avoid any intricate alterations to a model structure. Thus, our research offers a user-friendly yet efficient and precise method for RSI classification, which is particularly advantageous for geoscience researchers with limited experience or time for model reconfiguration.

This study is still in its early stages, and several limitations must be addressed moving forward. First, the proposed method may exhibit sensitivity to hyperparameter settings, which could affect its performance and generalizability. Second, the effectiveness of the approach in other RSI tasks, such as object detection⁷² or knowledge distillation^73,74, has yet to be thoroughly evaluated, representing a promising area for future research. Additionally, there is potential for further optimization of the proposed ESE models by reducing redundancy, which could be achieved through a more detailed investigation of various CNN and ViT architectures. The authors plan to expand upon this research in future publications, exploring these avenues and addressing the identified limitations.