Visual feature-based multi-scale hybrid attention network for fine-grained Hawthorn varieties identification

Abstract

Hawthorn is a well-known economic crop widely recognized for its efficacy in cardiovascular protection and blood pressure reduction. However, accurately identifying Hawthorn varieties, which arise from diverse cultivation conditions, poses a significant challenge in species authentication. To address this challenge, we introduce a visual feature-based method for Hawthorn identification. Specifically, we propose a multi-scale hybrid deep learning model to capture and merge both local and global features of Hawthorn images. Our model incorporates shallow prior and high-level semantic information, thereby enhancing classification precision. Furthermore, to improve the model ability to recognize local details in fine-grained images, we propose a novel spatial local attention mechanism. The loss functions are designed to reduce the low-frequency features in the fine-grained image. Extensive experiments conducted on our Hawthorn dataset, as well as two public datasets, demonstrate that our model outperforms state-of-the-art methods.

Introduction

Hawthorn, a member of the Rosaceae family¹, encompasses fruit-bearing trees and shrubs. Renowned for its protective effects on cardiovascular, anti-oxidation, anti-cancer, and anti-inflammatory properties, as well as its ability to lower blood pressure and cholesterol^2,3. Hawthorn has over 10 varieties⁴, serving different purposes, the confusion surrounding these varieties has raised concerns about their impact on quality and commercial value, prompting increased public awareness⁵. Consequently, there is an urgent need for practical authentication methods to distinguish between Hawthorn varieties in real-world applications.

Testing the active ingredients, such as organic acids and flavonoids⁶, can aid in identifying the different varieties of Hawthorn. However, relying solely on physicochemical identification and biological evaluation^7,8,9,10 may be difficult to reflect the intrinsic quality, as these methods focus on limited compounds and can be time-consuming^11,12. An effective alternative gaining prominence is the use of sensory technology, including an electronic nose, electronic tongue, and electronic eyes, coupled with chemometric methods^13,14,15. While these methods have shown promise, they typically require professional equipment. A growing emphasis is on developing non-destructive and accurate identification methods for Hawthorn.

With the development of deep learning, computer vision technology is widely used in image identification^16,17,18, leveraging its capacity to discern images solely through visual information^19,20. Convolutional Neural Network (CNN) has gained widespread interest in computer vision due to its efficient feature expression capabilities. For example, Wang et al.²¹ proposed a CNN-based model to fuse features of leaves from different parts of soybean plants for cultivar recognition. Pérez et al.²² used CNN to classify and detect the five types of potential adulteration. However, a notable limitation of CNN-based models is their focus solely on local relationships^23,24. To overcome this limitation, transformer-based models are proposed to extract global information. For instance, Chang et al.¹⁹ proposed to use a vision transformer model to extract edge features for plant disease identification. Pacal²⁵ introduced an advanced transformer module to extract features from maize leaves for detecting diseases. However, one drawback of those transformer-based models is their high computational cost and poor generalization^26,27.

Moreover, some recent references^28,29,30 propose different similarity-preserving metrics and quantization methods for fine-grained image retrieval. In contrast, the OTQ method³¹ leverages cross-X semantic hypergraph learning to mine discriminative features through synergistic interactions across scales, layers, and images, thereby enhancing retrieval robustness against intra-class variations. Inspired by our observations above, we propose a rapid and effective visual feature-based model for identifying the different varieties of Hawthorn. Based on the characteristic of small inter-class differences of the fine-grained image, our model obtains and merges local details and global features from the fine-grained images by combining MBConv³² and Swin Transformer³³. Specifically, MBConv is used to extract multi-scale subtle features, while Swin Transformer is employed to extract the global information features. Then, the shallow-level feature map and high-level feature are fused by Patch-Merge layers using global and local features at different scales. Through an element-wise add operation, we combine shallow-level and high-level features. After that, we use a separated CNN layer and a fully connected layer to output the Hawthorn identification results. The framework of our model is presented in Fig. 4, and the details of our proposed method are presented in Sect. 3. On the other hand, a comprehensive visual multi-varieties Hawthorn images dataset is constructed, where high-resolution images are captured using a self-developed acquisition device, which is detailed in Sect. 2. The experimental results and analysis are shown in Sect. 4. The conclusion is drawn in Sect. 5.

Our contributions are highlighted as follows:

• A novel multi-scale hybrid model is proposed to capture and merge local details and global features of fine-grained images. Through the infusion of shallow prior features to guide high-level semantic information, our method demonstrates superior accuracy in identifying varieties of Hawthorn. Remarkably, to the best of our knowledge, our approach stands as the pioneering application of deep learning technology in Hawthorn identification.

• A newly spatial local attention module is designed to enhance the awareness of detailed local features within fine-grained images and improve the learning capacity. Softmax normalizes the features of a linear transformation. The loss of local information in maximum pooling is carefully reduced by highlighting the low-frequency features of the fine-grained image.

• To address the absence of publicly available image datasets for different varieties of Hawthorn, a visual Hawthorn image dataset from various varieties is first constructed.

• Extensive experiments are performed on our dataset as well as public datasets. The experimental results show that our model achieves the highest accuracy among state-of-the-art models. Moreover, the practical utility of our proposed model extends beyond Hawthorn identification, showcasing excellent performance in the broader context of plant identification.

Dataset collection

We collect 8 different varieties of Hawthorn, namely (A) Xiaojinxing, (B) Xinglongshisheng, (C) Hongkongqi, (D) Baiquan, (E) Damianqiu, (F) Yubeihong, (G) Dajinxing and (H) Dawuling. The collected samples illustrated in Fig. 1, are sourced from the Lotus Pond Chinese herbal medicine market in Chengdu. These samples are certified by experts from the Hospital of Chengdu University of Traditional Chinese Medicine (China), and samples encompass various slices obtained from intact specimens. Our comprehensive Hawthorn dataset comprises 3960 images, with the distribution of classes presented in Fig. 2.

The image acquisition process utilizes a wood-made device equipped with lighting and image acquisition systems. The lighting system incorporates PHILIPS Graphica TL-D light with a temperature of 5000 K, comprising four light tubes and scattering plates to eliminate shadows during the image capture. The wood-made device has a reflective grey coating with a reflectivity of 18%, as depicted in Fig. 3(A).

Fig. 1

The collected Hawthorn dataset illustration. Our dataset contains 8 varieties, namely (A) Xiaojinxing, (B) Xinglongshisheng, (C) Hongkongqi, (D) Baiquan, (E) Damianqiu, (F) Yubeihong, Dajinxing, and (H) Dawuling. Different varieties of Hawthorn have different colors and textures, as well as different tastes and flavors.

Fig. 2

The distribution of the number of images within each Hawthorn variety in our dataset.

In the image acquisition system, a camera (Canon EOS 60D) is employed to capture high-resolution images of 5120 × 3840 × 3 pixels, as shown in Fig. 3B. YoloV4³⁴ is used to obtain the 2D bounding boxes of images. Moreover, subsequent cropping is performed based on the detected bounding box, as demonstrated in Fig. 3C. Additionally, we manually remove any incomplete, blurry, or inappropriate images. Then, data augmentation strategies can be applied to mitigate the effects of data imbalance by generating additional samples, enhancing model generalization, and improving classification performance through preprocessing transformations such as rotation, flipping, cropping, and duplicating minority class samples.

Fig. 3

The image acquisition process for different varieties of Hawthorn.

Methodology

Overview

Our model was designed to extract local and global information from input images for Hawthorn identification, as illustrated in Fig. 4. The architecture incorporates spatial local attention (SLA) and MBCovn layers to capture multi-level features deemed as local characteristics. Additionally, Swin-Transformer Blocks were employed to acquire hierarchical features through the merging and reduction of spatial resolution, serving as global features.

Fig. 4

The overview of Hawthorn identification architecture. Our model first uses the SLA and MBCovn layers to extract the subtle multi-level features of images. Subsequently, we fuse features extracted from the Swin Transformer block by using PatchMege. Then we obtain a fused shallow-level feature map F_s, and a fused high-level feature map F_h. After a separated CNN layer, the shallow-level feature map and the high-level feature map are combined by element-wise add. The combined features are through another CNN layer and a fully connected layer to output the identification results.

Subsequently, PatchMerge layers were used to fuse the local and global features by doubling the channels and halving the width and height. Two fused feature maps: F_s for shallow-level features and F_h for high-level features were obtained. Finally, the identification results were computed through fully connected layers. This approach enables adeptly handling fine-grained analysis by leveraging the flexibility of various scale features.

Local feature extraction

Spatial local attention

Our model included two sub-modules in the local feature extraction. Due to the similarity of visual features of images, SLA was proposed to focus more precisely on the color and surface details of different varieties of Hawthorn and learn more about the subtle features of images. The structure of SLA is displayed in Fig. 5.

Fig. 5

The spatial local attention module. (A) is our proposed SLA. The input feature map X was processed to obtain the feature vector of $\:1\times\:1\:{C}^{{\prime\:}}$by the MaxPool layer and AvgPool layer. Meanwhile, X was compressed by $\:1\times\:1$ Conv and then normalized by the Softmax layer. This can obtain the self-attention weight of spatial dimension. It can highlight the low-frequency regions of the image and reduce the loss of local features in the Maximum pooling. (B) is the Spatial Attention Module (SPA) in the Convolutional Block Attention Module (CBAM).

In SLA, the maximum pooling layer is first used to contain the highest values in the local area and effectively retain the most significant features. However, other important fine-grained information (such as texture and global context) is ignored, which may lead to a decrease in the expressiveness of features. To address this issue, an average pooling layer was incorporated to produce a flat vector, enabling the extraction of global spatial context information from the feature maps.

$$\:{F}_{max}\left(x\right)=\underset{i=\text{1,2}\dots\:C}{\text{max}}X,\:\:\:{F}_{max}\left(x\right)\in\:{\mathbb{R}}^{h\times\:w\times\:1}$$

(1)

$$\:{F}_{avg}\left(x\right)=\frac{1}{C}\sum\:_{i=1}^{C}X,\:{F}_{avg}\left(x\right)\in\:{\mathbb{R}}^{h\times\:w\times\:1}$$

(2)

where X is the input feature. Then the fusion feature $\:{F}_{con}$ by concatenating the outputs of average pooling and maximum pooling according to the channel dimension.

$$\:{F}_{con}=\text{C}oncat({F}_{avg}\left(x\right),{F}_{max}\left(x\right))$$

(3)

Furthermore, compared with SPA, the SLA compresses the feature map using a 1 × 1 linear mapping, which reduces high-dimensional features into more compact low-dimensional representations while aggregating cross-channel information. It is expressed in formula 4.

$$\:C\left({X}_{i}\right)=\frac{{e}^{{Z}_{i}}}{{\sum\:}_{j=1}{e}^{{Z}_{j}}}$$

(4)

Then, Softmax is subsequently applied to generate normalized attention weights, emphasizing important channels and enhancing fine-grained features. In this way, self-attention weight can be obtained to focus more on fine-grained features. Mathematically,

$$\:{Z}^{{\prime\:}}=\sum\:_{i=1}\frac{{\:W}_{n}\times\:{X}_{i}}{C\left({X}_{i}\right)}{\times\:W}_{m}$$

(5)

Combining with the maximum pooling feature vector, the average pooling feature vector, and self-attention weight, the local feature Z’ was obtained. Where W_m is a linear transformation matrix of 1 × 1. C(X_i) is the normalization factor, $\:{Z}_{i}$is the output of the i node, and j is the number of output nodes. The self-attention weight of [0–1] is calculated by the Softmax function.

Then, the 7 × 7 Conv W_v was used to extract features, and then the spatial attention weight with channel number 1 was obtained by the Sigmoid activation function. The larger kernels have larger receptive fields and can capture local dependencies of images. The attentions were divided by smoothing the obtained attention weight. The low-frequency regions of the image were highlighted and the loss of local features in the maximum pooling was reduced. Then the weight of attention was multiplied by the connected feature vectors:

$$\:{Z}^{{\prime\:}{\prime\:}}=\sigma\:\left({Z}^{{\prime\:}}\times\:{\:W}_{v}\right)$$

(6)

where σ is the non-linear Sigmod function, Z ” represents the output weight. Finally, the sigmoid activation function was used to generate the feature attention. By multiplying the attention weight value Z” and the input X, the attention-weighted X’ can be obtained. As shown in formula 7.

$$X^{\prime } = Z^{{\prime \prime }} \times X$$

(7)

The experimental results have shown that SLA can enhance the perception of local fine-grained features while suppressing significant sensitive areas in images. Subsequently, this module was integrated into the MBConv module to enable a more accurate focus on the specific features and the correspondence.

Improved MBConv module

MBConv combined depth-wise convolution and attention mechanisms to reduce the number of parameters and improve learning ability. Inspired by the CBAM³⁵, the SLA module was added after the Squeeze and Excitation (SE) layer. In this way, it was an effective and simple attention module that stacked channel attention and spatial attention modules. The comparison of our newly designed MBConv with the original one is presented in Fig. 6.

Fig. 6

MBConv modules structure comparison. (A) was the network structure of the original MBConv. (B) was the proposed MBConv module, and the new SLA was added after SE. The SE and SLA modules are used to obtain more detailed information. The attention maps were multiplied by the input feature map for adaptive feature refinement.

Depth-wise convolution is first applied to each input channel by the $\:K\times\:K$ convolution kernel, expressed as follows in formula 8:

$$\:{Y}_{depth}\left(i,j,c\right)=\sum\:_{m=1}^{K}\sum\:_{n=1}^{K}X\left(i+m,j+n,c\right).{W}_{depth}(m,n,c)$$

(8)

where $\:{Y}_{depth}\in\:{\mathbb{R}}^{h\times\:w\times\:c}$ is the output feature map, $\:X\in\:{\mathbb{R}}^{h\times\:w\times\:c}$is the input feature map. $\:{W}_{depth}\in\:{\mathbb{R}}^{h\times\:w\times\:c}$ is the depth convolution kernel, $\:K$ is 3. Then, the SE and SLA modules are used to obtain more detailed information. The attention maps were multiplied by the input feature map for adaptive feature refinement. Depth-wise convolution operates on each channel independently, preserving local spatial details of the input features. Finally, the $\:1\times\:1$ convolution kernel is used to fuse the output channels of the depth-wise convolution.

$$\:Y\left(i,j,c\right)=\sum\:_{c=1}^{C}{Y}_{depth}\left(i,j,c\right).W(c,k)$$

(9)

where $\:W(c,k)\in\:{\mathbb{R}}^{1\times\:1\times\:c\times\:c}$ is the point-wise convolution kernel. Our model can learn abundant details and features to deal with the similarity of visual features. These processes were repeated from low level to high level, so that could obtain multiple local features.

Global feature extraction

In this paper, the Swin Transformer was employed to obtain global features. To enhance a greater generalization property of our model, Multi-head Self-Attention (MSA) was replaced with window-based long-head self-attention (W-MSA) and the shifted window-based long head self-attention (SW-MSA) in Swin Transformer. The input $\:X\in\:{\mathbb{R}}^{h\times\:w\times\:c}$ is divided into fixed-size windows $\:W\times\:W$. Then, it is computed by self-attention, the scaled dot-product attention is to obtain $\:Z\in\:{\mathbb{R}}^{d\times\:d}$.

$$\:Z=Attn\left(z\right)=Softmax(Q{K}^{\text{{\rm\:T}}}/\sqrt{w})V$$

(10)

the $\:Softmax$ attention $\:Attn(\bullet\:)$ with a global receptive field works as the following nonlinear mapping:

$$\:y{\prime\:}=WMSA\left(LN\left({Z}^{l-1}\right)\right)+{Z}^{l-1}$$

(11)

where $\:LN\left(\bullet\:\right)$ is the Layer Normalization that essentially is a learnable column scaling with a shift, and $\:FFN\left(\bullet\:\right)$ is a standard two-layer feedforward neural network applied to the embedding of each patch. The scaled dot-product attention (8) of $\:Z$, the jth element of its ith row $\:{Z}_{i}^{j}$ is obtained in Formula 12.

$$\:{Z}_{i}^{j}=\frac{{e}^{{(Q{K}^{\text{{\rm\:T}}}/\sqrt{w})}_{i}}}{\sum\:_{j=1}^{h}{e}^{{(Q{K}^{\text{{\rm\:T}}}/\sqrt{w})}_{ij}}}.\:V=\:Softmax({q}_{i}{K}^{\text{{\rm\:T}}}/\sqrt{w})V$$

(12)

Perform multi-head attention fusion on the attention results of each layer of windows to obtain hierarchical fusion features and enhance attention to fine-grained features. The calculation process is shown in formula 14:

$$\:{z}^{l+1}=SWMSA\left(LN\left({Z}^{l}\right)\right)+{Z}^{l}$$

(13)

The multi-scale structure, built upon the enhanced MBConv module, effectively captures features ranging from shallow details to deep semantic information and enhances classification precision.

Multi-features fusion

The cropped images, $\:{x}^{i}$ were as input to MBConv and obtain the appearance features $\:{O}_{s}$. All patches were calculated to double channels and halve the width and height by the PatchMerge layer. The extracted features were transferred into Swin Transformer, and then the shallow feature maps $\:{F}_{s}$ were generated, similarly, the higher-level features $\:{F}_{h}$ were obtained:

$$\:{F}_{h}=Swin\left(PatchMerge\left({O}_{s}\right)\right)$$

(14)

$$\:y=Conv\left({{F}_{s})+Conv(F}_{h}\right)$$

(15)

Subsequently, the features were superimposed by skipping the connection. The shallow prior features were injected to guide high-level semantic information. The local and global features of images were extracted to identify the fine-grained images.

Results and discussions

Evaluation

Training parameters

The model is optimized by the AdamW³⁶ algorithm. The initial learning rate is 1.5e-3, and the learning rate decay strategy is StepLR³⁶. The batch size is set to 16, and the final model is obtained when it reaches 400 epochs. Due to the imbalanced data, the Focalloss is used as the loss function. The code is built by using Pytorch = 1.8.1 and Python = 3.9. The model is trained on a PC (equipped with an Intel i7 processor) with a graphics processing unit card (NVIDIA 2080TI, 11G memory).

Metrics

We consider four metrics to evaluate our model identification performance, including Accuracy, Precision, Recall, Specificity, and F 1Sore:

$$\:\text{P}\text{r}\text{e}\text{c}\text{i}\text{s}\text{i}\text{o}\text{n}=\frac{TP}{TP+FP}$$

(16)

$$\:\text{R}\text{e}\text{c}\text{a}\text{l}\text{l}=\frac{TP}{TP+FN}$$

(17)

$$\:\text{S}\text{p}\text{e}\text{c}\text{i}\text{f}\text{i}\text{c}\text{i}\text{t}\text{y}=\frac{TN}{FP+TN}$$

(18)

$$\:\text{F}1\:\text{S}\text{c}\text{o}\text{r}\text{e}=\frac{2\text{*}\text{P}\text{r}\text{e}\text{c}\text{i}\text{s}\text{i}\text{o}\text{n}\text{*}\text{R}\text{e}\text{c}\text{a}\text{l}\text{l}}{\text{P}\text{r}\text{e}\text{c}\text{i}\text{s}\text{i}\text{o}\text{n}+\text{R}\text{e}\text{c}\text{a}\text{l}\text{l}}$$

(19)

where $\:TN$ represents the number of True Negative, and $\:TP$ denotes the number of True Positive. $\:FN$ indicates the number of False Negative, and $\:FP$ represents the number of False Positive³⁷.

Hawthorn identification

Results of our model

We split our dataset into a training set and a testing set. Specifically, 80% of the data are used for training, and the remaining 20% is used for testing. The change curves of loss and accuracy are used to reflect the training performance, and the testing loss and accuracy results of our model are shown in Fig. 7. During the training process, the loss gradually decreased, the model converged and the accuracy gradually increased. Until 200 epochs, the loss becomes stable. And the model achieves the highest accuracy of 90.96% in 384 epochs. The results of the testing set show that the model has good identification performance.

Fig. 7

The loss and accuracy results of our model on the testing set.

The experimental results are detailed in Table 1, where we can see that the recall is higher than precision, which indicates the predicted positive is much higher than the actual positive.

Table 1 The experimental results of different varieties of Hawthorn by our model.

The identification performance for (A) Xiaojinxing (C) Hongkongqi (D) Baiquan (G) Dajinxing (H) Dawuling have better performance, benefiting from their distinct appearance information and diverse places of origin. However, for (B)Xinglongshisheng, (E) Damianqiu, and (F) Yubeihong, where visual similarities lead to confusion. This difficulty is compounded by the fact that (E) Damianqiu and (F) Yubeihong are different varieties but from the same origin. To further evaluate our model performance, we employ the confusion matrix, and the area under the Receiver Operating Characteristic Curve (ROC)³⁸ curve is calculated. The experiential results are depicted in Fig. 8a, b, respectively.

In Fig. 8a, the columns represent the predicted labels, and the rows represent the true labels. The values corresponding to rows and columns indicated the number of correct classes predicted from true data. The distribution of the actual and predicted numbers of each class is shown. Consistent with the results in Table 1, (B) Xinglongshisheng has confusion numbers with (E) Damianqiu and (G) Dajinxing, especially with (G) Dajinxing. For (E) Damianqiu, it is incorrectly predicted as (A) Xiaojinxing. And (F) Yubeihong is also incorrectly predicted as (G) Dajinxing. Our dataset has the characteristics of imbalance, (G) Dajinxing has the largest number.

Fig. 8

(a) The experimental results of the confusion matrix. (b) The experimental results of ROC, the range of the ROC curve is between 0 and 1, (the larger the better). The numbers from 0 to 7 correspond to different classes from A to H.

Therefore, the similar visual features of different Hawthorns are analogous, the model has a stronger learning ability and learns more features for the classes with the largest classes. Similarly, based on the confusion matrix, ROC is computed to reflect the difference between the True Positive Rate and the False Positive Rate. It measures the ability of our model to correctly identify certain categories in Fig. 8b. Among them, (B)Xinglongshisheng, (E) Damianqiu, and (F) Yubeihong have the lower accuracy. It also reaffirms the credibility of the identification results generated by our proposed model.

Manual identification

To evaluate the speed and accuracy of our method, we compare our model with the manual identification results from five experts. These experts, affiliated with the pharmacy department of the Hospital of Chengdu University of Traditional Chinese Medicine, perform manual identification solely by visual features. The accuracy for each class is calculated from the identification of 346 images, taking the average of the results from the 5 experts.

Table 2 The results of manual identification by five experts.

Table 2 displays the outcomes, revealing an overall accuracy of 65.24% for manual identification. Notably, our model surpasses this performance, demonstrating higher accuracy for each class. In contrast to manual identification, our model showcases superior speed and remarkable accuracy, emphasizing its substantial value and potential for practical applications.

Ablation study

Different attention modules

To verify the effectiveness of the SLA module, the SPA³⁵ in CBAM and the Global Context Attention Module (GC)³⁹ are compared. In fairness, the baseline is based on the improved model, and the remaining parameters remain unchanged. The classification accuracy is shown in Table 3. The comparative loss and accuracy results of multiple attention modules on the testing set are shown in Fig. 9.

Table 3 Ablation study results for attention mechanism modules.

From Fig. 9, we can see SLA has the best advantages in improving performance. From Table 3, SLA has the best accuracy of 90.96%, which is superior to the others. Compared with SPA, SLA has an increase of 1.64%. Simultaneously, compared with no additional attention module, it can be confirmed that the performance of the model can be significantly improved. Compared with GC, although the computational complexity of this model is lower, the identification results are limited, and our SLA still has the best performance. Moreover, the parameters have not changed much, and SLA has good practicability. To better verify the effect of the two attentions, further, the visualized heat maps are depicted in Fig. 10.

Fig. 9

The loss and accuracy results of our model. SLA is the proposed attention module in this paper, SPA is the CBAM attention module, and NO is no other attention module.

The difference in regions of interest (ROIs) under different attentions can be observed via the visualized heatmaps. The heatmaps illustrate the regions of the image that the model emphasizes when making classification decisions. Bright regions highlight areas of greater significance, where the model has identified critical features. All can focus on the ROIs of images, which demonstrate the ability of models to focus on key regions of interest, such as color variation, texture, or structural anomalies.

Fig. 10

The visualization of the different attention modules for each class. The heat maps of each class are randomly selected. The first is the original image, the second is the heatmap without attention, the third is the heatmap with SPA, and the last is ours.

Compared with SPA and GC, SLA has a deeper and wider range of ROIs, such as (A) Xiaojinxing, (D) Baiquan, and (E) Damianqiu. For (A) Xiaojinxing, the model pays more attention to the texture details in pulp. Surface color features are more critical for the identification of (D) Baiquan. For (E) Damianqiu, the model pays more attention to the texture and color details, which can better improve the classification performance. For B (Xinglongshisheng) and F (Yubeihong) with lower precision, the different ROIs are focused on by different models, and the comparative advantage is not obvious. Besides, for (H) Dawuling, compared with SLA, GC pays more attention to blank redundant areas, which affect the model performance and lead to lower accuracy. And SLA concentrates on the fine-grained features of the target and enhances the classification performance. In short, SLA can learn more effective features and improve recognition accuracy. It can also be used to recognize fine-grained images by focusing on the detailed features of local areas.

Fig. 11

The heat maps of the failure results are presented in these experiments. The heat maps of each class are randomly selected.

Moreover, some failure results are also presented in Fig. 11. The confusion matrix indicates that (A) Xiaojinxing and (E) Damianqiu are frequently confused in Fig. 8. Corresponding heatmaps reveal that the model focuses on similar regions for these two classes. Misidentified (A) Xiaojinxing images appear to place greater emphasis on color details. For (B) Xinglongshisheng, common misclassifications are (F) Yubeihong and (G) Dajinxing, with heatmaps showing the model focusing more on blank areas during these errors. When processing (C) Hongkongqi, the model primarily attends to the core region and is also prone to confusion with (A) Xiaojinxing. The highest degree of confusion occurs between (F) Yubeihong and (G) Dajinxing, which exhibit significant visual similarity as evidenced in Fig. 11. Through analysis, we can know that the SLA can enhance local feature awareness within single images (e.g., flesh texture details), the isolated learning paradigm fails to leverage inter-sample semantic relationships. This limitation leads to confusion between visually similar classes, such as (A) Xiaojinxing and (E) Damianqiu in Fig. 8. To address these issues, we propose integrating a hypergraph attention layer⁴⁰ after SLA. Specifically, local features extracted from multiple hawthorn samples would serve as hypergraph nodes, with semantic classes defining hyperedges. This structure would explicitly model discriminative cross-sample associations, enabling feature refinement through high-order interaction propagation. Such synergy would enhance robustness against intra-class variance while preserving fine-grained locality.

Visualization SLA module

The visualization of the convolutional layer feature maps is crucial for a deeper understanding of the SLA’s effectiveness in extracting fine-grained features for different hawthorn varieties. Thus, the compared experiment that visualizes the feature maps at various layers with the SLA is designed. The detailed Grad-CAM visualizations are included to further illustrate how the SLA influences the decision-making process of models and highlights the key visual features the model relies on when distinguishing between different hawthorn varieties. The visualization results are shown in Fig. 12.

Fig. 12

The visualization of the different convolutional layers focuses on when classifying the model based on the attention mechanism for each class. The heat maps of each class are randomly selected. The decision-making process of the model is analyzed through a visualized heat map.

The brighter areas highlight the regions where the model’s attention is focused during classification. As shown in Fig. 11, Different convolutional layers focus on distinct regions in the heatmaps for various hawthorn species, revealing the key visual features the model considers when distinguishing between different hawthorn varieties. For instance, for (A) Xiaojinxing, the texture details in pulp are key visual features. B (Xinglongshisheng) and F (Yubeihong) have similar visual features. For (E) Damianqiu, the model pays more attention to the texture and color details, which can better improve the classification performance. The spatial localized attention (SLA) mechanism proposed in this paper demonstrates its ability to precisely capture color and surface details, enabling the learning of richer fine-grained features. This effectiveness is evident in the refined area distinctions presented in the heatmaps, underscoring the role of SLA in improving classification performance.

Different CNN modules

To verify the effect of the MBConv, ablation experiments are performed, wherein the MBConv is replaced with other CNN modules. To ensure fairness, we maintain the network structure. The accuracy comparison with various CNN modules is presented in Table 4.

Table 4 The comparative experimental results based on different CNNs.

Notably, our original model stands out with optimal parameters and computational complexity and attains the highest accuracy. Our original model surpasses the one replaced by ResNet at least 3.69%. MBConv combines depth-wise convolution and an attention mechanism, which improves the learning ability by residual connections. Compared with the Depthwise Separable block, MBConv has a higher accuracy by 1.14%. The experimental results also verify the validity and rationality of the MBconv module.

Horizontal comparison and performance trade-offs

Comparative experimental results

For horizontal comparison, state-of-the-art ConvNet models and multiple different Transformer models are compared with ours. And FocalLoss is chosen as the loss function for all models. The experimental results for the multiple ConvNet models are shown in Table 5; Fig. 3.

Table 5 The results of the comparison for the different ConvNets.

Fig. 13

The experimental results of a confusion matrix for different ConvNets models. (A) VGG⁴¹, (B) ResNet⁴², (C) DenseNet⁴³, (D) EfficientNet⁴⁴, (E) MobileNet⁴⁵, (F) CoAtNet³².

Simultaneously, the value of Parameter (Params) and Floating-point operations per second (FLOPs) are used to reflect the computational complexity and number of parameters of the model. And the Frames Per Second (FPS) are computed to evaluate the model speed. Our model achieves the highest accuracy among the compared models. It is 3.69% higher than DenseNet and 8.95% higher than EfficientNet. From Fig. 12, we can see (F) yubeihong and (G) dajinxing are easily confused. The lower the accuracy of the model, the greater the amount of confusion for different classes. In addition, from the parameters and complexity, the basic convolution has many parameters, high computational complexity, and limited classification accuracy. Compared with the lightweight network, the parameters of models are greatly reduced, and the accuracy is also improved. It also provided a theoretical foundation for the selection of the MBConv module. Simultaneously, multiple state-of-the-art Transformer algorithms are compared, including ViT⁴⁶, FocalNet⁴⁷, Swin Transformer³³, CMT⁴⁸, CvT⁴⁹, PVT⁵⁰, MaxViT⁵¹, EfficientViT⁵², and SwinFG⁵³.

The experimental results are shown in Table 6; Fig. 14, where we can find that our model has the best identification accuracy among state-of-the-art Transformer models. Interestingly, Transformer-based models have lower accuracy than based-ConvNets models. Transformers require a large amount of data to improve their learning ability, and the ConvNets-based models have a strong priority by inductive bias.

Table 6 The experimental results for multiple different Transformers.

Fig. 14

The experimental results of the confusion matrix for different Transformers models. (A) ViT⁴⁶ (B) FocalNet⁴⁷ (C) Swin Transformer³³ (D) CMT (E) CvT (F) PVT (G) PVT (H) PVT (I) PVT.

Hence, our model has good generalization and high accuracy in multi-scale structures. It is 20.66% higher than ViT, 5.29% higher than Swin Transformer, and 17.2% higher than CvT. Our model has fewer parameters than Transformer-based models such as FocalNet, MaxViT, and SwinFG, yet it achieves the highest accuracy. This demonstrates that our model effectively balances computational efficiency and classification performance. Although the FPS of our model is not the fastest, the experimental results show that the hierarchical network is more advantageous for classifying fine-grained images. By fusing multi-level features of images, it can learn more effective subtle details and achieve visual fine classification for the different varieties of Hawthorn.

Visualization comparison results

To better reflect the classification performance of our model, the visualization of different models is intuitively shown in Fig. 15.

Fig. 15

The visualizations of different models for each class show heat maps randomly selected from each category. These heat maps reveal that different models concentrate on distinct areas of the images. Notably, our model demonstrates a more precise focus on fine-grained features, highlighting its superior ability to capture critical details for classification.

According to the comparative experimental results of Tables 5 and 6, we randomly selected the visualization results of DenseNet, MobileNet, and Swin Transformer. From Fig. 14, traditional models focus more on more scattered areas, indicating that the model has limited ability to capture key features. Traditional models tend to focus on irrelevant regions, such as the background or secondary areas, indicating a limited ability to capture key features. In contrast, the heat maps generated by the model proposed in this paper show concentrated attention on critical areas, such as the color, surface texture, and flesh morphology of hawthorn. This demonstrates the heightened sensitivity of the model to specific, meaningful features essential for classification. In addition, similar attention distribution is shown in the heatmaps of different varieties for traditional models. The focus on differential features enhances the interpretability of our model. Notably, our model demonstrates a more precise focus on fine-grained features, highlighting its superior ability to capture critical details for classification.

Blossoms and leaf and fruit identification

To validate the generalization of our method, additional experiments are conducted on two publicly available herb datasets, with comparisons made against other methods. The two herb datasets are Chinese Medicinal Blossom⁵⁴, Medicinal Leaf⁵⁵, and Fruit dataset⁵⁶, and excerpts of these datasets are illustrated in Figs. 16 and 17, and Fig. 18. The Chinese Medicinal Blossom dataset comprises 12 different types of Chinese medicinal flowers, comprising a total of 11,500 images. The Medicinal Leaf dataset contains 30 types of healthy herbal leaves, comprising a total of 6,500 images. The Fruit dataset includes 45 different types of fruit, comprising a total of 21,668 images.

Table 7 Comparative analysis of classification results on Chinese medicinal blossom, Chinese medicinal leaf, and fruit datasets.

Chinese medicinal blossom We compare our method with other state-of-the-art methods on the Chinese medicinal blossom dataset, the comparison results are shown in Table 7. From the table, we can see that our model exhibits the highest accuracy among other mainstream methods. Our proposed model focuses more on the fine-grained features, while the fusion of local features and global information enhances model performance and robustness.

Fig. 16

Chinese medicinal blossom dataset. It comprises 12 different types of medicinal flowers, with all images enhanced through random rotation, flipping, cropping, and scaling ratio adjustment. The dataset of blossom images for traditional Chinese medicinal flowers are twelve categories: (1) Syringa, (2) Bombax malabarica, (3) Michelia alba, (4) Armeniaca mume, (5) Albizia julibrissin, (6) Pinus massoniana, (7) Eriobotrya japonica, (8) Styphnolobium japonicum, (9) Prunus persica, (10) Firmiana simplex, (11) Ficus religiosa, and (12) Areca catechu.

Chinese medicinal leaf The comparison results of different mainstream methods and our method on the public Chinese Medicinal Leaf dataset are shown in Table 7. The medicinal Leaf dataset comprises 30 medicinal plants that exhibit subtle visual differences. From Table 7, the conventional CNN showcases limited classification effectiveness. Conversely, our demonstrates conspicuous advantages. Its commendable performance across various datasets underscores its robust generalization capability.

Fig. 17

Chinese Medicinal Leaf dataset. All images are captured under a consistent light source, encompassing a diverse collection of 30 types of healthy herbal leaves. (1) Rasna, (2) Arive-Dantu, (3) Jackfruit, (4) Neem, (5) Basale, (6) Indian Mustard, (7) Karanda, (8) Lemon, (9) Roxburgh fig, (10) Peepal Tree, (11)Hibiscus Rosa-sinensis, (12)Jasmine, (13) Mango, (14) Mint, (15) Drumstick, (16) Jamaica Cherry -Gasagase, (17) MurrayaKoenigii, (18) Oleander, (19) Parijata, (20) Tulsi, (21) Betel, (22) MexicanMint, (23) IndianBeech, (24) Guava, (25) Pomegranate, (26) Sandalwood, (27) Jamun, (28) RoseApple, (29) CrapeJasmine, and (30) Fenugreek.

Fruit dataset The comparison results of different mainstream methods and our method on the public Fruit dataset are shown in Table 7. The fruit dataset comprises 45 different fruits. As shown in Table 7, CNNs exhibit limited classification performance, and Transformers have lower accuracy. In contrast, our approach demonstrates significant advantages, with consistently strong results across diverse datasets, highlighting its superior generalization capability.

Fig. 18

Fruit dataset. All images are captured under a consistent light source, encompassing a diverse collection of 45 types of healthy herbal leaves. (1) Apricot (2) Beetroot (3) Blueberry (4) Cabbage white (5) Cactus fruit (6) Carambula (7) Carrot (8) Cauliflower (9) Chestnut (10) Clementine (11) Cocos (12) Dates (13) Fig. (14) Ginger Root (15) Granadilla (16) Guava (17) Hazelnut (18) Huckleberry (19) Kaki (20) Kiwi (21) Kohlrabi (22) Kumquats (23) Limes (24) Lychee (25) Mandarine (26) Mangostan (27) Maracuja (28) Melon Piel de Sapo (29) Mulberry (30) Orange (31) Papaya (32) Passion Fruit (33) Pepino (34) Pitahaya Red (35) Pomegranate (36) Pomelo Sweetie (37) Quince (38) Rambutan (39) Raspberry (40) Redcurrant (41) Salak (42) Tamarillo (43) Tangelo (44) Walnut (45) Watermelon 1.

Conclusion

Considering the impact of different cultivation conditions and areas on the varieties of Hawthorn, hence, was an indispensable need for species identification. With the development of deep learning, a novel supervised learning hybrid model was designed to better capture the local and global information of fine-grained images via injecting shallow prior knowledge to guide high-level semantic features. On this basis, a novel SLA was proposed to highlight the low-frequency features of the image by calculating the self-attention weight of spatial dimension. It can enhance the awareness of detailed local features and improve the learning capacity. Extensive experiments were performed on our dataset as well as the public dataset. Compared with different state-of-the-art ConvNets and Transformer models, ours have the best accuracy of 90.96% and significantly outperforms others. Furthermore, the effectiveness and rationality of the model have also been proved by the different ablation experiments. Hence, it has been confirmed that our model was preferable to identify the varieties of Hawthorn. It has also been proven to be an exceptional practice for the high efficiency of agriculture and plant technology.

However, some limitations to the identification still exist. Due to the small and unbalanced dataset, moreover, the similarity of visual features of images, the learning ability of the model for subtle features has still been lacking, such as the low accuracy of (B) xinglongshisheng and (F) yubeihong. Thus, while the amount of dataset was gradually increased in the follow-up, the identification of the subtle differences between similar classes was researched to improve the identification accuracy of Hawthorn. Moreover, we will explore generative data augmentation methods to address the issue of imbalanced data distribution, such as Denoising Diffusion Probabilistic Models (DDPM) and data resampling, to optimize data distribution.