Introduction

The Leshan Giant Buddha, located in Leshan City, Sichuan Province, was carved in 713 AD, during the early years of Emperor Xuanzong’s reign in the Tang Dynasty. It is one of the world's largest stone-carved seated Maitreya Buddha statues [1]. In February 1982, the State Council of China designated the Buddha as a National Key Cultural Relic, and in December 1996, UNESCO included it in the World Heritage List for both natural and cultural heritage. The ushnisha atop the Buddha's head is composed of 1021 individual sandstone blocks, bonded with an inner layer of lime and covered by two layers of plaster. This design not only holds symbolic significance but also serves a practical purpose. The 18-layer structure of the ushnisha forms a drainage system, with transverse drainage channels located at the 4th, 9th, and 18th layers, decorated with lime plaster. This drainage system works in conjunction with the Buddha’s collar, robe folds, chest drainage channels, and the ventilation tunnels behind the ears (which connect to the cliffs), creating a complete network of drainage and ventilation. This network helps to slow down erosion and weathering.

However, prolonged exposure to a harsh geological conditions has led to widespread issues within the ushnisha, including cracking (Fig. 1c), loss of lime plaster (Fig. 1d), and biological damage from plant growth (Fig. 1e). These issues have diminished the drainage system's effectiveness, allowing rainwater to infiltrate the Buddha's body, altering the physical and mechanical properties of the bedrock, and thereby compromising the overall stability of the statue. This poses a potential threat to its long-term preservation. Given the critical role that the ushnisha plays in the Buddha’s structure, its deterioration has significantly impacted the head’s drainage efficiency. Consequently, the timely protection and reinforcement of the ushnisha have become a priority in current cultural heritage conservation efforts.

Fig. 1
figure 1

Current state of the ushnisha on the Leshan Giant Buddha’s head. a Geographic location of the Leshan Giant Buddha; b Side view of the Leshan Giant Buddha; c Crack damage; d Missing plaster layers; e Biological damage from plant growth

Full size image

Since 1914, the Leshan Giant Buddha has undergone seven major restoration efforts by relevant departments and organizations. However, inspecting the 1021 ushnisha elements has always been a daunting task, as manual detection is time-consuming, labor-intensive, and inefficient. Moreover, due to the Buddha’s massive size—standing 71 m tall, with a head height of 14.7 m and an ear length of 6.72 m—manual inspection poses significant safety risks. Thus, exploring more efficient, safe, and cost-effective inspection methods is crucial for improving the Buddha’s preservation and ensuring sustainable conservation [2, 3].

As deep learning advances, the development of automated digital systems, tools, and techniques has become a focal point of the Industry 4.0 revolution [4]. Utilizing algorithms such as convolutional neural networks (CNNs) based on deep learning to detect defects in cultural heritage (CH) images can help overcome the time-consuming and labor-intensive nature of manual inspections [5]. In practical applications, deep learning technology has been widely used in the inspection of bridges [6,7,8], roads [9, 10], the construction industry [11, 12], and modern concrete buildings and surfaces [13,14,15]. In the CH domain, one of the latest digital technologies is AI-based automation [16]. Research on applying deep learning-based computer vision in CH is expanding and can be categorized into three main areas [17].

The first category involves direct classification and assessment of cultural heritage defects and damage in images. For example, Karimi et al. [18] utilized the Inception-ResNet-v2 model to detect cracks, spalling, erosion, and salt weathering defects in the bridges of Isfahan, achieving accuracy, precision, and recall rates of 96.58%, 96.96%, and 96.24%, respectively. Meklati et al. [19] used the MobileNetV2 model to detect wall damage, achieving a weighted average precision of 0.868 in identifying common surface damage types in wall photographs. Wang et al. [20] used a deep learning framework to detect multiple types of damage, such as spalling, cracks, and weathering, in the walls of Beijing's Forbidden City, achieving an average accuracy of 94.3%. They then applied a Faster R-CNN model based on the ResNet101 framework for real-time damage detection in masonry buildings [2]. Ugail et al. [21] employed deep transfer learning for visual analysis and attribution of paintings by Raphael, demonstrating an accuracy of 98% in image-based classification tasks during validation on a test set of well-known and authentic Raphael paintings.

The second category involves the use of object detection algorithms to locate defects and damage in cultural heritage images, mainly including the YOLO [22] series and R-CNN [23, 24] series of algorithms. For instance, Yang et al. [25] used the YOLOv4 machine learning model to automatically detect five types of damage in historical gray brick buildings within the buffer zone of the World Cultural Heritage site in Macau. Mishra et al. [26] applied the YOLOv5 framework to detect four types of defects—discoloration, exposed bricks, cracks, and spalling—in monuments in New Delhi, India. Mansuri et al. [27] utilized the Faster R-CNN framework to detect three types of defects on a monument in Surat, India, and summarized various AI techniques and algorithms used in CH conservation in related literature [28], highlighting significant advances in AI for heritage conservation. Pathak et al. [29] used the Faster R-CNN model to detect damage in 3D point cloud data collected from cultural heritage sites and to locate detected cracks and spalling on the 3D model.

The third category involves the detection and segmentation of cultural heritage defects and damage using semantic segmentation models. This task is typically more complex than image classification and object detection but has seen some successful applications in the CH field. For example, Yuan et al. [30] used the Res-UNet semantic segmentation method to identify cracks and painting losses in the Forbidden City in Beijing, achieving accuracy and F1-score values of 98.19% and 93.42%, respectively. Wu et al. [31] proposed a new U-Net convolutional neural network for mural crack segmentation tasks, demonstrating precise segmentation performance with an mIoU value of 0.7731. Garrido et al. [32] combined infrared thermography technology and the Mask R-CNN algorithm, using thermal image sequences as input for the Mask R-CNN learning process to automatically detect and segment surface and subsurface defects. Liu et al. [33] used the DeepLabV3 series of semantic segmentation models to monitor the semantic segmentation of crowdsourced images of historical facades, achieving an IoU value of 66.9%.

The aforementioned studies demonstrate the widespread and successful application of deep learning algorithms in the CH domain, showing the feasibility of replacing manual defect detection with these techniques. However, in the numerous restorations and conservation efforts of the Leshan Giant Buddha over the years, the ushnisha damage has been manually detected, with no deep learning methods employed thus far. To fill this research gap, this study conducted field surveys to obtain images of the ushnisha and employed two deep learning models, PSPNet and K-Net, to perform pixel-level segmentation experiments on the damage. To improve the accuracy of crack damage segmentation, a weighted strategy was proposed, which adjusts the model’s loss function according to the proportion of the damaged area. Additionally, the study conducted secondary damage detection based on the model’s generalization ability, identifying cracks missed during manual annotation, and completed statistical and quantitative evaluations of all ushnisha in the dataset. Finally, to address the challenge of model interpretability in deep learning, this study utilized the Grad-CAM algorithm to perform interpretability analysis on the damage identification results, visually displaying the areas of interest during model detection through heatmaps, thereby gaining a deeper understanding of the model’s decision-making process.

Data collection and workflow

Due to the loss of lime plaster on the ushnisha, rainwater can directly infiltrate these areas during rainfall, causing damage to the Buddha's structure. The cyclical growth of plants on the ushnisha leads to dehydration, shrinkage, flaking, and spalling of the rock surface, exposing new rock faces and exacerbating deterioration. Cracking is an early stage of the damage process, where plant roots grow along the microcracks in the ushnisha, further damaging it through root wedging. Additionally, the organic acids secreted by the roots contribute to the corrosion of the ushnisha [34]. Consequently, this study focuses on three types of damage: loss of lime plaster, biological damage from plant growth, and cracks. To analyze these forms of deterioration, we collected images of ushnishas exhibiting these specific damage patterns. The image data was acquired in May 2021 using a Canon EOS 200D II camera with a resolution of 2976 × 1984. To ensure image quality and consistency, all photographs were taken under clear weather conditions. This approach minimized the impact of light variations and shadows on image quality, thereby enhancing the accuracy and reliability of subsequent analysis.

As shown in Fig. 2, the complete workflow of this study is outlined as follows. First, 149 images of the ushnisha on the head of the Leshan Giant Buddha were collected in the field. These images underwent preprocessing, including noise reduction, contrast enhancement, and brightness adjustment. Next, the Labelme tool was used to annotate the damage in the images, constructing a dataset for training the semantic segmentation neural network. A total of 120 images were designated as the training set, while 29 images were used as the test set. The training set was then augmented through operations such as flipping, translation, sharpening, HSV transformation, and noise addition, resulting in a final training set of 360 images. It should be noted that in this study, the data augmentation strategies were combined in a probabilistic manner, meaning that each image may have undergone one or more augmentation techniques (for example, an image may have both noise addition and sharpening applied at the same time), with the goal of obtaining more diverse training samples. These images were used to train the deep learning models, which were subsequently evaluated on the test set to assess their performance. Since the images in the test set were entirely new to the models, they provided an effective test of the models’ generalization capabilities. Finally, based on the damage segmentation results from the models, a quantitative assessment of all ushnisha damage was completed.

Fig. 2
figure 2

Workflow for damage segmentation and quantitative assessment

Full size image

Methodology

Semantic segmentation models

The objective of semantic segmentation is to classify each pixel in an image into a specific category. This task requires not only identifying objects within the image but also precisely localizing them. Compared to image classification and object detection, semantic segmentation is significantly more challenging due to its pixel-level precision requirements. In this study, we employed two models, the Pyramid Scene Parsing Network (PSPNet) and K-Net, to segment the damage on the ushnisha of the Leshan Giant Buddha’s head. Additionally, we proposed a weighted strategy based on the proportion of damage area to optimize the training process.

PSPNet

PSPNet [35] enhances the model’s understanding of complex scenes by capturing global contextual information through the multi-scale pooling of feature maps. The architecture of PSPNet is typically based on a pre-trained convolutional neural network (e.g., ResNet18, ResNet50, or ResNet101) as the backbone, responsible for extracting fundamental image features. These features are then processed through a pyramid pooling module that integrates multi-scale information, and finally, a convolutional layer generates the refined segmentation predictions.

Figure 3 illustrates the structure of the PSPNet model. Initially, the input image is resized to 512 × 512 pixels, and feature extraction is performed using ResNet50 [36], resulting in a feature map that is 1/8th the size of the original input. The pyramid pooling module then processes these features, generating a feature map that incorporates global information. This is concatenated with the pre-pooled features and passed through a convolutional layer to produce the segmentation output. The pyramid pooling module divides the feature map into grids of 6 × 6, 3 × 3, 2 × 2, and 1 × 1, performing multi-scale pooling to capture contextual information at various scales. PSPNet excels in handling complex scenes and object boundaries, making it particularly suitable for multi-scale damage detection tasks in this study, including the identification of loss of lime plaster, biological damage, and cracks of varying sizes.

Fig. 3
figure 3

Diagram of the PSPNet model structure

Full size image

K-Net

K-Net [37] is a neural network architecture designed for image segmentation that enhances segmentation performance through the learning and updating of dynamic convolutional kernels. The core concept of K-Net is to dynamically adjust the convolutional kernel parameters during the feature extraction and segmentation process, thereby improving the model's generalization ability and accuracy. Figure 4 illustrates the structure of the K-Net model. The entire network process can be divided into the following five steps:

  1. 1)

    Input and Feature Extraction: In this study, the Swin Transformer was selected as the backbone network to extract features from the input image, which are then processed through a neck network to output the feature map F.

  2. 2)

    Kernel Initialization and Mask Prediction: Semantic kernels and instance kernels are generated through a learning mechanism, which are then convolved with the feature map F to produce initial mask predictions.

  3. 3)

    Kernel Updating and Dynamic Adjustment: The convolutional update head utilizes the dynamic kernels from the previous stage to convolve with the feature map, updating and outputting new dynamic kernels. This process is repeated multiple times to progressively optimize the segmentation results.

  4. 4)

    Category and Mask Prediction: After each kernel update, category prediction is performed, and the updated dynamic kernels are used to predict the mask, generating the final segmentation results.

  5. 5)

    Output: The final segmentation output is produced by combining the category prediction and the mask prediction.

Fig. 4
figure 4

Structure of K-Net model

Full size image

K-Net’s dynamic convolutional kernel updating mechanism and multi-scale feature fusion enable the network to adaptively adjust the convolutional kernel parameters, improving image feature capture and significantly enhancing the accuracy and efficiency of semantic segmentation.

Weighted loss function

During the optimization process, the commonly used standard cross-entropy loss function is formulated as shown in Eq. (1):

$${\text{Loss}} = - \sum\limits_{i = 1}^{C} {y_{i} } \, \cdot \log \left( {\hat{y}_{i} } \right)$$
(1)

Here, \(C\) represents the number of classes, \({y}_{i}\) is the indicator variable for the actual class, which is 1 for the correct class and 0 for the others.\(\hat{y}_{i}\) is the predicted probability of the \({i}^{th}\) class by the model. The closer \(\hat{y}_{i}\) is to 1, the closer \(\log \left( {\hat{y}_{i} } \right)\) approaches 0, thereby reducing the loss value, which aligns with the optimization objective.

However, the standard cross-entropy loss function assigns equal weights to all samples. In detection tasks, if there is an imbalance between positive and negative samples, a large number of easily classified negative samples may dominate, while a small number of difficult samples and positive samples may not contribute effectively, leading to a decrease in model accuracy [38]. Similar issues can arise in semantic segmentation tasks.

In this study, the three types of damage—biological damage, crack damage, and loss of lime plaster—are unevenly distributed in the images. Crack damage appears in 87.2% of the images, biological damage in 92.6%, while loss is relatively rare, occurring in only 45.6% of the images. Although crack damage is more frequent, its area is relatively small, whereas biological damage typically cover a larger area. Moreover, multiple types of damage often coexist on the same ushnisha. This imbalance impacts the recognition of crack damage in semantic segmentation tasks. Therefore, this study proposes a weighted strategy to improve the accuracy of crack detection.

Specifically, the weighted strategy adjusts the loss function by assigning weights to each class, where the weights are inversely proportional to the area ratio of each class. Let \({area}_{i}\)​ represent the area ratio of the \({i}^{th}\) type of damage. The weight \({w}_{i }{\prime}\)​ is calculated using Eq. (2) and Eq. (3), which increases the weight of the loss value for minority classes. The final weighted cross-entropy loss function is shown in Eq. (4):

$${w}_{i}=\frac{1}{{area}_{i}}$$
(2)
$${w}_{i }^{\prime}=\frac{{w}_{i}}{\sum_{i}^{C}{w}_{i}}$$
(3)
$${\text{Loss}} = - \sum\limits_{i = 1}^{C} {w_{i }^{\prime} \cdot y_{i} } \cdot \log \left( {\hat{y}_{i} } \right)$$
(4)

Transfer learning

Transfer learning is a method that applies the knowledge acquired by a model in one task to a related task [39, 40]. The core idea is to transfer pre-trained models or their weight parameters to a new task, reducing the need for large amounts of data, shortening the training time, and improving model performance [41]. In this study, a transfer learning strategy was employed to provide a strong starting point for the model, which reduces the demand for extensive labeled data and speeds up the convergence of the training process. Specifically, the weights of the pre-trained model were loaded into the base model, and all layers were frozen by setting “trainable = False”, which reduces both training time and memory usage.

Grad-CAM

Gradient-weighted Class Activation Mapping (Grad-CAM) [42] is a visualization technique for deep learning models, particularly convolutional neural networks. It generates a Class Activation Map (CAM) to help understand the model's decision-making process by highlighting the regions in the input image that had the greatest influence on its decision. The core idea of Grad-CAM is to use the gradient information to produce a heatmap that highlights the areas with the most significant impact on the model’s output. The process consists of six main steps:

  1. 1)

    Forward Propagation: The input image is passed through the neural network to obtain the final classification score.

  2. 2)

    Gradient Calculation: For the target class, the gradient of the score with respect to the feature map of the last convolutional layer is calculated. These gradients indicate which features most influence the target class score.

  3. 3)

    Global Average Pooling: The gradients for each channel of the feature map are globally averaged to obtain the weight for each channel. These weights reflect the contribution of each feature map to the target class score.

  4. 4)

    Weighted Feature Map Generation: The computed weights are used to perform a weighted sum of the corresponding feature maps.

  5. 5)

    ReLU Activation: ReLU is applied to eliminate negative values, ensuring that only regions with a positive influence on the target class are retained.

  6. 6)

    Heatmap Upsampling: The generated heatmap is upsampled to match the size of the input image, allowing it to be overlaid on the original image for visualization.

Given the output feature map \({A}^{k}\) from a convolutional layer and the score \({y}^{c}\) for the target class, the weight \({\alpha }_{k}^{c}\)​ for each channel is calculated as shown in Eq. (5):

$${\alpha }_{k}^{c}=\frac{1}{Z}\sum_{i} \sum_{j} \frac{\partial {y}^{c}}{\partial {A}_{ij}^{k}}$$
(5)

where \(Z\) is the total number of pixels in the feature map, and \({A}_{ij}^{k}\)​ is the value at position \((i,j)\) in the \({k}^{th}\) feature map. The final Grad-CAM heatmap \({L}^{c}\) is calculated using Eq. (6), which visually illustrates the regions the model focused on during the decision-making process, thereby providing an explanation for the model's reasoning.

$${L}^{c}=\text{ReLU}\left(\sum_{k} {\alpha }_{k}^{c}{A}^{k}\right)$$
(6)

Performance metrics

In semantic segmentation tasks, Mean Accuracy (mAcc) and Mean Intersection over Union (mIoU) are commonly used evaluation metrics. mAcc focuses on the accuracy of predictions within each category, helping to assess the precision of classification, while mIoU is a standard metric for evaluating the overall performance of semantic segmentation, particularly useful in handling datasets with class imbalance. In this study, both metrics were calculated on the test dataset to assess model accuracy, while Frames Per Second (FPS) evaluated the model's speed, providing a comprehensive evaluation of model performance.

Mean accuracy (mAcc)

mAcc measures the proportion of correctly predicted pixels for each class relative to the total number of pixels in that class, focusing on the accuracy of pixel predictions for each category. Given \(C\) classes, where \({n}_{i}\)​ represents the number of correctly predicted pixels for class \(i\), and \({t}_{i}\)​ represents the total number of actual pixels for class \(i\) the accuracy for class \(i\), \({\text{Acc}}_{i}\)​, can be calculated using Eq. (7). The mAcc is the average of the accuracy values across all classes, as shown in Eq. (8).

$${\text{Acc}}_{i}=\frac{{n}_{i}}{{t}_{i}}$$
(7)
$$\text{mAcc}=\frac{1}{C}\sum_{i=1}^{C} {\text{Acc}}_{i}$$
(8)

mAcc assesses classification performance for each class, preventing the overestimation of overall performance due to majority class dominance. However, mAcc is less sensitive to imbalanced datasets, especially when class distributions are highly uneven. Therefore, it is important to use the mIoU metric in conjunction with mAcc to more objectively evaluate the model's performance.

Mean intersection over union (mloU)

In classification tasks, metrics like recall, precision, and F-score are commonly used, whereas mIoU is more commonly used in segmentation tasks. mIoU measures the ratio of the overlap area to the union area between the predicted and the actual results. The IoU for each class \(i\) can be expressed using Eq. (9):

$${IoU}_{i}=\frac{{TP}_{i}}{{TP}_{i}+{FP}_{i}+{FN}_{i}}$$
(9)

where \({\text{TP}}_{i}\) (True Positive) denotes the number of pixels correctly predicted as class \(i\); \({\text{FP}}_{i}\) (False Positive) indicates the number of pixels incorrectly predicted as class \(i\); and \({\text{FN}}_{i}\) (False Negative) represents the number of pixels that belong to class \(i\) but were incorrectly predicted. The mIoU is the average IoU across all classes, as shown in Eq. (10):

$$mIoU=\frac{1}{C}\sum_{i=1}^{C} {IoU}_{i}$$
(10)

The advantage of mIoU is that it accounts for the impact of misclassification when assessing model performance, offering better robustness against class imbalance. However, mIoU can be more sensitive to performance on smaller classes, where the impact of misclassification is relatively more significant. By combining both mIoU and mAcc metrics to evaluate model accuracy, a more comprehensive assessment of model performance can be achieved.

Results and discussion

In the process of training the model, the batch size refers to the number of images input during each training iteration. After all the images in the training set have passed through the model once, an epoch is completed. These hyperparameters typically need to be adjusted based on the hardware and the characteristics of the dataset. The hardware configuration for this study includes an NVIDIA RTX3090Ti GPU with 24 GB of memory and a 6-core Xeon Gold 6142 CPU. Given the high demands of segmentation tasks on image data size, the input image size was set to 512 × 512 to maintain a high level of recognition accuracy. The final configuration set the batch size of 4, with 10,000 epochs of training. The model’s performance was evaluated on the test set every 10 epochs, and the best-performing model on the test set was saved to prevent overfitting.

Training and validation of PSPNet

In this study, the PSPNet model was trained for 10,000 epochs, during which the error between the predicted results and the actual results (Loss) was evaluated. The loss curve during the training process is shown in Fig. 5. As the number of iterations increased, the Loss value steadily decreased, indicating that the model was gradually converging. Statistical analysis showed that the area proportions of the deterioration types were 28.2% for missing parts, 64.7% for plant-related deterioration, and 7.2% for cracks. Using Eq. (2) and Eq. (3), the weight ratios for the three types of deterioration were calculated as 0.187:0.081:0.732. Figure 5 shows that the use of the weighted strategy significantly accelerated the reduction of the Loss value.

Fig. 5
figure 5

Loss reduction curve during PSPNet model training

Full size image

As training progressed, the Loss value stabilized during the later stages. To prevent overfitting—where the model performs well on the training set but underperforms on the test set—validation was conducted every 10 epochs. During validation, the mAcc and mIoU values were calculated on the test set, and only the best-performing model was saved. As shown in Fig. 6, after the 2000th epoch, the improvement in accuracy metrics became gradual. Although there were minor fluctuations, the overall performance remained stable. Without weighting, the highest mAcc on the test set was 69.40%, and the highest mIoU was 61.26%. Using the weighted strategy, mAcc increased to 80.04%, and mIoU to 61.31%. Although the improvement in mIoU was small, the mAcc increased by 10.64%, which is a significant enhancement. Figure 6 also shows that with the weighted strategy, both mAcc and mIoU were notably higher in the early stages of training compared to the unweighted case, with mAcc showing particularly substantial improvement. It is worth noting that these maximum values did not appear simultaneously. For the unweighted case, the maximum mAcc occurred at 8,880 training steps; with the weighted strategy, the maximum mAcc occurred earlier, at 2210 steps, indicating that the weighted strategy improved training efficiency. Similarly, the maximum mIoU occurred at 8610 steps without weighting and at 3460 steps with weighting, further validating the effectiveness of the weighted strategy in enhancing training efficiency.

Fig. 6
figure 6

Changes in various metrics on the PSPNet test set. The blue dashed line represents the mAcc metric without the weighted strategy, with a maximum at Step = 8880; the blue solid line represents mAcc with the weighted strategy, with a maximum at Step = 2210. The green dashed line represents the mIoU metric without the weighted strategy, with a maximum at Step = 8610; the green solid line represents mIoU with the weighted strategy, with a maximum at Step = 3460

Full size image

Table 1 provides a detailed evaluation of the accuracy metrics for each type of deterioration during the training process. Comparative analysis of the Acc metrics reveals that although the accuracy for the background class slightly decreased, the accuracy for all three types of deterioration improved, particularly for cracks, where the Acc increased by 37.02%, showing the most significant improvement. The Acc for missing parts increased by 1.97%, and for plant-related deterioration, it increased by 6.19%. The mean Acc, calculated as the average of the background and the three types of deterioration, increased by 10.64%, consistent with the results in Fig. 6. Although the weighted strategy significantly improved the Acc metric, the IoU improvement was modest. Therefore, this study also explored the K-Net model to further verify the effectiveness of the weighted strategy.

Table 1 Maximum values of various metrics in the test set (PSPNet)
Full size table

Training and validation of K-Net

The loss curve during the training process is shown in Fig. 7. As the number of iterations increased, the Loss value steadily decreased, showing that the model was converging, similar to the PSPNet model’s training process. After applying the weighted strategy to the loss function, the rate of decrease in Loss was faster.

Fig. 7
figure 7

Loss reduction curve during K-Net model training

Full size image

To prevent overfitting, validation was conducted every 10 epochs, calculating the mAcc and mIoU values on the test set. As shown in Fig. 8, the introduction of learning kernels, dynamic kernels, and the convolutional update mechanism in the K-Net model, along with the Swin Transformer as the backbone for feature extraction, led to significant improvements in mAcc and mIoU values compared to the PSPNet model. Without weighting, the highest mAcc in the K-Net model on the test set was 81.61%, and the highest mIoU was 67.15%. Compared to PSPNet, mAcc increased by 12.1% and mIoU by 5.89%. With the weighted strategy, the K-Net model's performance further improved, reaching an mAcc of 90.23% and an mIoU of 69.55%. Compared to the unweighted case, mAcc increased by 8.62% and mIoU by 2.3%. As Fig. 8 shows, the mAcc and mIoU curves with the weighted strategy were higher than those without, indicating superior performance.

Fig. 8
figure 8

Changes in various metrics during the K-Net model training on the test set. The blue dashed line represents the mAcc variation without the weighted strategy, with a maximum at Step = 780; the blue solid line represents the mAcc variation with the weighted strategy, with a maximum at Step = 660. The green dashed line represents the mIoU variation without the weighted strategy, with a maximum at Step = 8380; the green solid line represents the mIoU variation with the weighted strategy, with a maximum at Step = 8290

Full size image

In terms of training efficiency, the K-Net model's performance was consistent with that of the PSPNet model. With the weighted strategy, the maximum mAcc occurred at the 660th iteration, while without weighting, it occurred at the 780th iteration, advancing by 100 steps, indicating higher training efficiency and faster mAcc improvement with the weighted strategy. Similarly, for the mIoU metric, the maximum value without weighting occurred at the 8380th iteration, while with weighting, it advanced to the 8290th iteration, 90 steps earlier, further demonstrating the weighted strategy’s effectiveness in enhancing training efficiency.

Table 2 provides a detailed evaluation of the accuracy metrics for each type of deterioration during the K-Net model training process. A comparative analysis of the Acc metrics shows that while the background class accuracy slightly decreased, recognition accuracy for all three deterioration types improved significantly, particularly for cracks, where the Acc increased by 25.15%, a notable improvement. The Acc for missing parts increased by 3.74%, and for plant-related deterioration, it increased by 7.43%. In terms of IoU metrics, the most significant improvement was in the loss of lime plaster, with a 7.73% increase; IoU for plant-related deterioration and cracks also increased by 0.59% and 1.47%, respectively. The mean values of Acc and IoU for the background and the three types of deterioration showed improvements of 8.62% and 2.4%, respectively, under the weighted strategy, corresponding to the results in Fig. 8, with the maximum mAcc reaching 90.23% and the maximum mIoU reaching 69.55%.

Table 2 Maximum values of various metrics in the test set (K-Net)
Full size table

Comparison of recognition results

As shown in Fig. 9, different grayscale values are used to represent various types of deterioration and the background. The background is depicted in black (grayscale value of 0), missing deterioration in dark gray (value of 80), plant-related deterioration in light gray (value of 160), and cracks in white (value of 240), effectively distinguishing between the different types of deterioration. The figure comprises six columns of images. The first column displays the original images of the ushnisha, presenting the actual scene to be segmented. The second column shows the deterioration information manually annotated by experts, representing the true distribution of the deterioration. The third column presents the deterioration segmentation results using the PSPNet model. The fourth column shows the optimized segmentation results of the PSPNet model after applying the weighted strategy. The fifth column presents the deterioration segmentation results using the K-Net model, and the sixth column shows the optimized segmentation results of the K-Net model after applying the weighted strategy. The figure shows that the segmentation results of the models closely align with the actual deterioration conditions. Whether dealing with a single type of deterioration or multiple types, the models accurately segment the major deterioration areas, meeting the requirements of detection. However, there are differences in the recognition results of certain details between the models. For example, in the ushnisha image numbered 657, the weighted PSPNet model exhibited a more ‘aggressive’ segmentation, displaying thicker cracks. Similarly, in the ushnisha image numbered 720, the weighted K-Net model showed much better recognition of a long crack compared to the unweighted results.

Fig. 9
figure 9

Deterioration segmentation results. The left side shows the ushnisha’s number; the first column is the original image; the second column is the annotation information; the third column shows the segmentation results of the PSPNet model; the fourth column shows the segmentation results of the PSPNet model with the weighted strategy; the fifth column shows the segmentation results of the K-Net model; and the sixth column shows the segmentation results of the K-Net model with the weighted strategy

Full size image

Additionally, it is worth noting that manual annotation of deterioration is time-consuming, and annotators are prone to fatigue, leading to omissions— a common issue in manual deterioration detection. For some fine cracks, manual annotation may result in missed detections. As shown in Fig. 9, the red boxes highlight cracks that were present but were overlooked during manual annotation and were successfully identified by the model. Without zooming in on the image, some cracks are difficult to detect with the naked eye and are easily overlooked. However, semantic segmentation models are pixel-level deep learning models, and after training on the data, they can predict and segment at the pixel level, outperforming manual annotation in terms of efficiency.

Quantitative evaluation based on K-Net

As previously mentioned, the trained model demonstrates excellent generalization ability and can detect fine cracks that were overlooked during manual annotation. Therefore, this study utilized the trained K-Net model to re-assess the entire dataset, supplementing the parts missed during manual annotation and conducting a quantitative analysis of the area proportions of each type of deterioration. This analysis allowed the determination of the specific proportions of missing deterioration, plant-related deterioration, and cracks for each ushnisha.

As shown in Fig. 10a, the proportions of deterioration for each ushnisha were statistically analyzed, where the red line represents the proportion of missing deterioration, the green line indicates the proportion of plant-related deterioration, and the blue line denotes the proportion of cracks. These statistical results allow the identification of the most severely deteriorated ushnisha for each type of deterioration. Figure 10b–d illustrate the results using color overlay methods, where red represents missing deterioration, green indicates plant-related deterioration, and black marks cracks. This method of directly overlaying deterioration information on the original image provides a clear visualization of the distribution and interrelationships of deterioration on the surface of the ushnisha. Figure 10b corresponds to the ushnisha numbered 611, which has the most severe missing deterioration, with a missing deterioration area proportion of 0.278, a plant-related deterioration area proportion of 0.194, and a crack area proportion of 0.00113. Figure 10c corresponds to the ushnisha numbered 615, where plant-related deterioration is the most severe, with a plant-related deterioration area proportion of 0.211, no missing deterioration, and a crack area proportion of 0.002. Figure 10d corresponds to the ushnisha numbered 683, where cracks are the most severe, with a crack area proportion of 0.033, a missing deterioration area proportion of 0.002, and a plant-related deterioration area proportion of 0.131.

Fig. 10
figure 10

Quantitative evaluation results of deterioration; a Statistical results of deterioration area proportions; b Ushnisha with the most severe missing deterioration, with a missing deterioration area proportion of 0.278; c Ushnisha with the most severe plant-related deterioration, with a plant-related deterioration area proportion of 0.211; d Ushnisha with the most severe cracks, with a crack area proportion of 0.033

Full size image

Speed evaluation

In terms of speed evaluation, since the weighted strategy does not inherently affect the model's running speed, this study primarily compared the speed performance of manual annotation, the PSPNet model, and the K-Net model.

The experimental results indicate that deep learning models significantly outperform manual annotation in processing efficiency. The K-Net model’s detection speed is 7.01 images per second, requiring only 22 s to process 149 images, making it 1309 times faster than manual annotation (approximately 8 h). Although the PSPNet model is slightly less accurate than K-Net, it is faster, processing 10.38 images per second, taking only 14.35 s to process the same dataset, which is 2006 times faster than manual annotation. Deep learning models not only significantly improve work efficiency but also effectively avoid the issue of missed detections due to fatigue during manual annotation. Combined with the accuracy evaluation results discussed earlier, the performance comparison of various methods is summarized in Table 3.

Table 3 Speed evaluation results
Full size table

Analysis of model interpretability

Grad-CAM is a visualization technique used in deep learning models to help understand the model’s decision-making process. It generates class activation maps highlighting the regions of the input image that are crucial for a specific decision. The main principle uses the model’s gradient information relative to the input image to generate a heatmap, highlighting the regions with the most significant impact on the model's output. As shown in Fig. 11, when cracks are specified as the target category, Grad-CAM computes a heatmap for the crack category based on Eq. (6). The regions with the most crack-like features are highlighted in the image, with areas closer to red indicating a higher probability of being identified as cracks.

Fig. 11
figure 11

Activation map of crack

Full size image

Additionally, as shown in Fig. 12, there are three images of ushnishas with various types of deterioration. By specifying the type of deterioration to detect, Grad-CAM generates corresponding heatmaps, highlighting areas with specific deterioration characteristics. For instance, when cracks are specified for detection, areas with crack features are highlighted; similarly, when plant-related or missing deterioration is specified, corresponding regions are highlighted. By selecting a specific type of deterioration for detection, the algorithm uses heatmaps to reveal the model’s focus areas, directly showing where the model’s attention lies. This aids in a deeper understanding of the decision-making process and underlying principles of deep learning models.

Fig. 12
figure 12

Interpretability analysis Heatmap (Multiple Deterioration Types Coexisting); The numbers on the left correspond to the IDs of the ushnishas on the head of the Leshan Giant Buddha. The first column shows the original images, the second column displays the heatmap for cracks, the third column shows the heatmap for plant-related deterioration, and the fourth column displays the heatmap for loss of lime plaster

Full size image

However, it must be emphasized that interpreting deep learning “black box” models still faces significant challenges. This study's use of Grad-CAM for model interpretability analysis is an attempt that, while not completely breaking open the “black box” model, has successfully transformed it partially into a “gray box” model. Through this visualization technique, the aim is to enhance our understanding of the model’s decision-making process, “focus points,” and “intentions.” Consider a hypothetical scenario: if, when detecting cracks, the model's focus points were not concentrated on crack areas but randomly distributed across the image, this would contradict human intuitive cognition, making it difficult to understand the basis of the model's decision-making behavior. However, the visualization results from Grad-CAM in this study reassuringly demonstrate that when crack detection is specified, the model's attention areas indeed focus on locations with crack characteristics, rather than being randomly scattered throughout the image. This confirms that the model’s decision-making is based on reasonable grounds and follows certain logic, thereby enhancing our confidence in the model’s decision-making process. It deepens our understanding of the model's internal mechanisms and provides important basis for further optimization and improvement of the model, driving the application of artificial intelligence in cultural heritage protection towards higher accuracy and reliability.

Limitations and future prospects

Balance between computational efficiency and accuracy

While the segmentation models employed in this study demonstrate excellent accuracy, their high computational resource requirements pose challenges for standard computing devices. Currently, the processing speed of these algorithms has not reached the threshold required for real-time video analysis, precluding their direct integration into drone systems for real-time detection. Consequently, they are limited to offline processing of image data. The You Only Look Once (YOLO) series, as one-stage detection models, exhibit outstanding computational efficiency but slightly lower accuracy. Future research could explore improvements based on YOLO models to achieve a more optimal balance between speed and accuracy.

Limitations in model generalization capability

Detecting deterioration in various cultural heritage artifacts typically requires specific datasets to train specialized models, as a single model struggles to generalize across multiple domains. For instance, the model developed in this study is challenging to apply directly to deterioration detection in other cultural heritage contexts. However, it is noteworthy that while this model has limitations in cross-domain applications, it performs excellently in detecting deterioration in the ushnisha of the Leshan Giant Buddha. Considering the enormous scale of the Leshan Giant Buddha (with 1021 ushnisha on the head), each deterioration inspection is a formidable task. The detection capability of this model can be continuously applied to future inspection work, significantly reducing the workload of cultural heritage conservation professionals. This specialized application holds significant importance for the long-term preservation of the Leshan Giant Buddha. Future research directions may include:

  1. 1)

    Exploring methods to enhance model computational efficiency to achieve real-time detection.

  2. 2)

    Developing models with stronger generalization capabilities to adapt to deterioration detection in various types of cultural heritage.

  3. 3)

    Integrating multimodal data, such as hyperspectral imagery or 3D scan data, to improve the comprehensiveness and accuracy of detection.

Limitations of quantitative assessment

Given the unique nature of the Leshan Giant Buddha, there have been no reported studies or literature applying deep learning models for the detection of damage to its ushnisha. Thus, our research represents a relatively recent development. In previous survey work, damage assessment of the ushnisha primarily relied on verbal descriptions, a method that made quantification of damage severity challenging and failed to directly reflect the damage condition of each ushnisha or the urgency of conservation work. In contrast, quantitative assessments based on deep learning models can accurately rank the severity of damage to the ushnisha, allowing for prioritized treatment of the most severely damaged areas within limited time and resource constraints. This approach not only enhances the efficiency of conservation efforts but also maximizes the prevention of rainwater infiltration from the Buddha's head into its body, effectively mitigating further erosion.

It is important to emphasize that the actual damage to the ushnisha is three-dimensional; therefore, the quantitative damage assessment provided by visual models based on two-dimensional images serves only as a semi-quantitative reference and cannot fully represent the actual three-dimensional damage. Furthermore, mosses and herbaceous plants differ significantly in their degree of harm to the Leshan Giant Buddha. In future in-depth studies, mosses and herbaceous plants could be distinguished as two separate categories. This differentiation is based not only on their varying degrees of damage to the Leshan Giant Buddha but also on the specific treatment measures required for each. Therefore, a more refined classification of these two types of plants will contribute to the development of more targeted conservation strategies.

Conclusion

This study employed two semantic segmentation models to perform semantic segmentation and quantitative evaluation of deterioration in the ushnishas of the Leshan Giant Buddha, providing significant support for the detection of deterioration in the statue. The conclusions are as follows:

  1. 1)

    The PSPNet and K-Net models demonstrated excellent performance in detecting deterioration, with the identified deterioration areas closely matching the results of manual detection, while their processing speed was over 1000 times faster than manual methods. By implementing the weighted strategy proposed in this study, the effectiveness of crack detection was further enhanced, achieving a final mean accuracy (mAcc) of 90.23% and a mean intersection over union (mIoU) of 69.55%, meeting the accuracy requirements for deterioration segmentation.

  2. 2)

    The trained models exhibited strong generalization capabilities, enabling precise pixel-level segmentation of deterioration and the detection of fine cracks that were overlooked during manual inspection.

  3. 3)

    Based on the segmentation results from the models, each ushnisha was quantitatively evaluated for deterioration. Ushnisha No. 611 had the most severe missing deterioration, Ushnisha No. 615 exhibited the most severe plant-related deterioration, and Ushnisha No. 683 had the most severe crack deterioration.

  4. 4)

    Grad-CAM was utilized to conduct interpretability analysis of the models, with the resulting heatmaps aiding in the understanding of the model's decision-making process.