Digital restoration of mural paintings from late Tang tomb M1373 in Xi’an based on hyperspectral analysis and image interaction processing

Abstract

In this study, we take the mural of the Late Tang M1373 tomb unearthed in the eastern suburbs of Xi’an, Shaanxi Province in November 2022 as the object, focus on the insufficient consideration of color authenticity in the current use of image processing for mural visual restoration, as well as the lack of systematic mural color and visual restoration method based on hyperspectral images. A improved spectral recognition method of mural pigment based on fusion spectral analysis and mixed spectral modeling was used in real scene; a virtual restoration method for mural pigment colors based on bandpass energy integration, pseudocolor display and color space correction was proposed; in addition, a integrity restoration of painting patterns including draft line enhancement and human-computer interaction image processing was used in the real mural color restoration. The experimental results show that the mural uses traditional mineral pigments such as ochre, earth yellow, mineral green and carbon black for its red, yellow, green and black pigments, and the accuracy rate of this method is higher than 97% in simulation experiments with noise of %4 spectral amplitude. Meanwhile, the richness of painting color of the restoration results is significantly improved compared to the original color photos, especially, the new method with color space correction is significantly better than the method without color space correction in texture.

Introduction

Tomb murals (hereinafter referred to as murals) are an important type of historical and cultural heritage. Due to the rich artistic, historical, and cultural value contained in murals, it is of great significance for understanding ancient society, politics, life, economy, art, technology, and the revitalization and utilization of cultural heritage. Generally, most ancient tomb murals belong to dry murals, usually composed of support layers, ground support layers, white ash layers, and pigment layers. One of the focuses of their protection and research is to protect the pigment layer (painted pigments and painted patterns). However, due to the effects of collapse, flood erosion, fire damage, air lighting, theft and destruction, various diseases usually appear on the surface of murals pigment layer, which seriously damage the value of murals. Therefore, in order to better protect, showcase, and inherit the value of murals, it is very important to sample, conserve, analysis and well founded restore the surface information of pigment layer of murals.

As shown in Fig. 1, which shows the mural of the late Tang tomb M1373 excavated at the end of 2022 in Baiyangzhai, eastern suburbs of Xi’an city, Shaanxi province¹. This mural is a large-scale brick and stone tomb mural, with rich content, bright colors, and obvious characteristics of the era. Archaeological speculation suggests that the owner of Tomb M1373 is Liu Xingshen, a powerful official, or his wife. This tomb is currently the only archaeological discovery of the tomb of the descendants of Liu Honggui, providing important information for studying the layout of family cemeteries in the late Tang dynasty^1,2. The exquisite murals unearthed from this tomb are rare in recent years, especially the content of the murals differs significantly from other late Tang murals discovered, which has important node significance for studying the theme and style changes of murals during the Tang and Song dynasties^1,3.

Fig. 1: Mural of the late Tang tomb M1373 excavated in Baiyangzhai, Xi’an city, at 2022.

The elements in Fig. 1 were obtained from data captured by our team’s camera and high-definition scanner, and no pre-existing external materials were used. The Fig. 1 is integrated using the following software with valid licenses: WPS (Personal Account No.: 385370771, WPS (Individual)).

The overall preservation condition of tomb M1373 is good, regretfully, their murals are also diseases such as cracking, hollowing, surface pollution, and fading. These diseases seriously affect the artistic, historical, and cultural value of murals, so there is an urgent need for disease treatment and mural protection and restoration. So the murals were protected using the method of archaeological site uncovering and later restoration. However, during the process of uncovering and protecting murals, there would be many added materials and human interventions, coupled with the inherent mural diseases, which brought certain interference to the subsequent research of murals. So it is necessary to adopt technical means to comprehensively collect and analyze information on the murals pigment layer before murals are uncovered and restoration, and then evidence-based restoration of surface information of murals would be mainly conduct.

In fact, for murals restoration, arbitrary entity restoration or repainting not only fails to ensure the authenticity of the restored content of the mural, but also disrupt the overall coordination and consistency between the content, color, style, and original appearance of the mural, and even cause secondary damage to the mural. Therefore, it is usually not easy to carry out physical painting restoration, but mainly focus on mural reinforcement and improving mural clarity. However, the above physical mural restoration, which mainly focuses on reinforcement, still faces problems such as insufficient completeness and color authenticity of mural painting, which still impacts on mural value exploration and utilization. With the advancement of cultural heritage protection concepts, the use of digital means to collect, store, analyze, virtually restore, and restore the preserved painted information of murals has become an indirect means of mural protection^4,5. With the help of digital technology, not only can it provide a basis for the physical restoration, but also it can provide complete data for the display and activation^6,7,8,9,10.

While the key to digital virtual restoration of murals is how to record and interpret the value information of murals, and how to ensure the authenticity and integrality of restoration methods and restoration results. In response to the above issues, currently the main methods are using imaging and spectral technology to synchronously study the visual and material properties of murals^11,12,13, and then using image processing to digital restore murals^{14,15,16,17,18,19}. However, the shortcoming of above material analyse method is that the material analysis is still mainly focused on local areas. As a result, for the mural analysis, the above method still lack overall coherence. Moreover, relying solely on image restoration without considering the authenticity of material representation is not sufficient for the authenticity of the restoration results. Hyperspectral imaging and analysis technology is a method for integrated detection and analysis of images and spectra^20,21,22,23, so applying it to the acquisition, analysis, and restoration of mural information is a more feasible and effective technical means.

At this time, for the recognition of mural pigments using hyperspectral image processing technology, spectral matching, machine learning, spectral unmixing algorithms, and other learning methods are usually used to identify pigment components and mixed pigment compositions^24,25. meanwhile, the XRF, Raman spectroscopy, etc, and hyperspectral data can also be combined to achieve precise identification of complex pigments²⁶. In terms of color restoration based on hyperspectral images, the RGB pseudo color display restoration technology based on pigment spectral recognition and spectral reconstruction could be used^27,28. Further, a pigment fading dynamics model can also be constructed based on multi-phase spectra and lab color space, mapping aging spectral features to the original color space to restore degraded color information of murals²⁹. In terms of mural pattern restoration, the visual texture features and generative convolutional networks, dark channel priors, attention mechanisms, etc., can be used to achieve pattern continuity restoration of defect areas^{16,30,31,32,33}.

However, the analytical ability of the above spectral analysis methods for real scenes still needs to be improved for the of pigments degradation and pigments mixing; The research of the overall analysis of vision and materials based on hyperspectral images still needs to be strengthened; Meanwhile, the pure image digital restoration method considers less about the authenticity of color restoration, but more about repairing pattern details, and further research is needed on how to integrally restore more realistic mural patterns based on the authentic color restoration results, rather than purely image-based methods.

Therefore, in this article, we mainly focus on the problems of fading, pollution, and even peeling of pigment layer in the mural of the Late Tang Dynasty tomb M1373 in the eastern suburbs of Xi’an, by using hyperspectral imaging and analysis techniques, combined with hyperspectral image processing and color science, an improved spectral recognition method under mixed modeling and a scheme of mural color and pattern restoration are explored, which provides more suitable pigment recognition methods by simulating real scenes, and avoids the insufficient consideration of restoring the authenticity of mural colors in mural restoration methods that rely solely on image processing. As a result, it will provide the digital analysis and restoration method of the tomb mural, so as to provide the more authentic reference data for the protection, restoration, display and utilization of mural.

Methods

To solve the above problems, we adopt the research method shown in Fig. 2. As can be seen from the above introduction, firstly, we obtained hyperspectral imaging data of the surface pigment layer of the unearthed murals; secondly, we identify the pigment composition by spectral recognition method based on improved fusion modeling, and detect the pigment distribution of mural of tomb M1373; and then based on the identified spectral curve and the pigment distribution, we use bandpass energy integration, pseudo color display and color space correction to build a color restoration sample set and complete the digital virtual restoration of mural colors; At last, image processing is used to enhance the pattern lines of mural patterns, and combined the black line extraction result by spectral detection and the enhanced image, the pattern improvement of mural would be finished.

Fig. 2

Research ideas and design plan of this article.

Obtaining method of mural hyperspectral data

The acquisition of hyperspectral data for murals includes hyperspectral scanning imaging of real murals and spectrum acquisition of mural pigments used for comparison.

On the one hand, the hyperspectral imaging system is the main tool for obtaining hyperspectral images of murals. When it is applied to protect the mural, it is necessary to solve problems such as insufficient imaging resolution, imaging distortion, insufficient stability, and poor specificity (such as low thermal safety light sources). In this study, we used a customized hyperspectral imaging system as shown in Fig. 3, and obtained hyperspectral images of murals in the visible and near-infrared spectrum through frontal scanning. The spectral resolution of the system is about 5 nm, with 128 spectral channels and a coverage range of 400–1000 nm, there are 1380 pixels in spatial dimension. Meanwhile, in actual imaging, to ensure the stability and safety of data acquisition, the track is used for interval translation of the equipment, and the internal push scanning imaging mode of the hyperspectral camera is used to image the local murals at every interval position. At this point, the global spectral image of the mural needs to be obtained by concatenating the local spectral images of the mural.

Fig. 3: Hyperspectral imaging scanning system for tomb murals.

The elements in Fig. 3 were obtained from data captured by our team’s camera, and no pre-existing external materials were used.

What is more, a synchronous lighting scheme by broadband low illumination spectral lamp beads is adopted to ensure the stability of data collection and the non-destructive safety of murals. Figure 4 shows the local hyperspectral image and spectrum of mural obtained by the above hyperspectral imaging scanning system. As can be seen, the overall hyperspectral image of the mural is relatively clear, and the spectral curves of different pigments extracted vary greatly.

Fig. 4: Local hyperspectral image and spectrum of mural pigments obtained by the hyperspectral scanning.

The elements in Fig. 4(a) were obtained from data captured by our team’s hyperspectral imaging scanning system for tomb murals, and no pre-existing external materials were used. The Fig. 4(a) is generated using the following software with valid licenses: ENVI6.0 Trial Version.

On the other hand, before conducting spectral analysis of the mural pigment layer, we first construct a typical pigment spectral data sample set, which is used for recognizing the pigments used in Tang tomb murals. By researching the commonly used pigment and color system in Tang tomb murals, five typical color systems of red, green, yellow, blue and black were selected from the Tang dynasty and before. Then we collect their visible and near-infrared hyperspectral data by a spectrometer named ASD with a spectral resolution of 1 nm and a spectral range of 350–2500 nm, which is used to construct the data set. Specifically, there are 8 types of red pigments, 2 types of green pigments, 3 types of blue pigments, 5 types of yellow pigments, and 4 types of black pigments; Meanwhile, white pigments are selected as auxiliary analytical pigments, as shown in Table 1.

Table 1 Pigment table for common color systems of mural paintings

As can be seen from the above, in order to construct a spectral feature library for the identification of pigment components in real scene murals, in this study, we measured the reflectance spectrum at 350–2500 nm for the above pigments’ powders of different color systems, and plotted their respective spectral curves according to different color systems, as shown in Fig. 5. It is the ratio of pigment spectral radiance to diffuse reflectance radiance of light source, namely, reflectance. It can be seen that there are significant spectral differences at 350–2500 nm among pigments of different color systems, and there are also certain spectral differences at 350–2500 nm among most pigments of the same color systems. Therefore, the above spectral reflectance data can serve as an effective spectral database for identifying different pigments in tomb murals.

Fig. 5

Spectral curves of the pigments of different color systems.

Identification method of mural pigments based on improved fusion spectral analysis and improved multi element mixed simulation

After obtaining spectral data of mural pigments and mastering their spectral characteristics, the identification of pigment components can be completed through spectral analysis recognition methods. However, pigments in real scenes often face situations such as pigment degradation, background and noise interference. Therefore, in this paper we firstly simulated the spectral representation in real scenes through improved multi element mixing, and then used a pigment recognition algorithm based on improved fusion spectral feature analysis to complete pigment recognition in real scenes³³.

Firstly, the spectrum ${{\boldsymbol{f}}}^{(i)},i=\mathrm{1,2},...,t$ of the t kinds of mural pigments possibly used in the actual scenes and mural background’s spectrum ${{\boldsymbol{f}}}^{k}$ are selected to simulate the mixing spectrum by weighted mixing in the actual scenes, which can be expressed as:

$${\boldsymbol{f}}=\mathop{\sum }\limits_{i=1}^{{\rm{t}}}{{a}^{(i)}{\boldsymbol{f}}}^{(i)}+b{{\boldsymbol{f}}}^{k}$$

(1)

In the formula, ${a}^{(i)}$ and $b$ are the weights of the corresponding spectrum of ${{\boldsymbol{f}}}^{({\rm{i}})}$ and ${{\boldsymbol{f}}}^{{\rm{k}}}$. we would select appropriate weights based on the experimental results and actual scenarios.

Then, based on the Spectral Angle Matching (SAM) operator and Spectral Information Divergence (SID) operator, a fusion spectral analysis operator is constructed through weighted fusion to improve the accuracy and adaptability of spectral recognition methods. Here, SAM measures the similarity between the pigment spectrum ${\boldsymbol{f}}$ to be identified and the pigment spectrum ${\boldsymbol{s}}$ of feature library by calculating the angle between the spectral vectors. For the SAM operator, the closer the value is to 0, the higher the matching degree between the two spectrum. Meanwhile, SID is used to treat the spectral vector as a random vector and analyzing the similarity between two random vectors $\left({\boldsymbol{f}},{\boldsymbol{s}}\right)$ based on probability and statistical theory. For the SID operator, the closer the value is to 0, the higher the matching degree between the two spectrum.

At this time, the improved Fusion Mapping (FM) operator could be represented as weighted sum of SAM operator ${\rm{SAM}}\left({\boldsymbol{f}},{\boldsymbol{s}}\right)$ and SID operator ${\rm{SID}}({\boldsymbol{f}},{\boldsymbol{s}})$:

$${{\rm{F}}}_{{\rm{F}}{\rm{M}}}\left({\boldsymbol{f}},{\boldsymbol{s}}\right)={\rm{\alpha }}{\rm{SAM}}\left({\boldsymbol{f}},{\boldsymbol{s}}\right)+{\rm{\beta }}{\rm{SID}}({\boldsymbol{f}},{\boldsymbol{s}})$$

(2)

Where α and β are adjustment coefficients, respectively. To ensure consistency in the dimensions of the terms left and right of the above equation, we will normalize ${\rm{\alpha }}$ and β by introducing new normalization adjustment parameters γ, and set ${\rm{\gamma }}=\frac{{\rm{\alpha }}}{{\rm{\alpha }}+{\rm{\beta }}}$, $1-{\rm{\gamma }}=\frac{{\rm{\alpha }}}{{\rm{\alpha }}+{\rm{\beta }}}$. As a result, the above equation becomes:

$${{\rm{F}}}_{{\rm{F}}{\rm{M}}}\left({\boldsymbol{f}},{\boldsymbol{s}}\right)={\rm{\gamma }}{\rm{SAM}}\left({\boldsymbol{f}},{\boldsymbol{s}}\right)+(1-{\rm{\gamma }}){\rm{\beta }}{\rm{SID}}({\boldsymbol{f}},{\boldsymbol{s}})$$

(3)

Here, the normalized modulation parameters γ is defined as:

$${\rm{\gamma }}=\frac{\frac{1}{M}{\sum }_{m=1}^{M}{{\rm{SAM}}}_{m}}{\frac{1}{M}{\sum }_{m=1}^{M}{{\rm{SAM}}}_{m}+\frac{1}{M}{\sum }_{m=1}^{M}{{\rm{SID}}}_{m}}$$

(4)

In the formula (4), M represents the number of pigments in the feature library, ${{\rm{SID}}}_{m}$ and ${{\rm{SAM}}}_{m}$ refer to the SID and SAM recognition operator value of the pigment to be identified and the m-th pigment in the pigment library separately. Finally, unify the dimensions and perform exponential normalization to rephrase the aforementioned fusion spectral recognition operator, whose new expression is as follows:

$${\widetilde{F}}_{{FM}}=-\log \left(\frac{1}{\sqrt{2{\rm{\pi }}}}* \exp (-0.5* {{{\rm{C}}}_{{\rm{C}}{\rm{M}}}}^{2})\right)$$

(5)

It can be seen that for the new normalized fusion spectral recognition operator ${\widetilde{{\rm{F}}}}_{{\rm{F}}{\rm{M}}}$, the smaller its value, the higher the matching degree.

For the selection of coefficient ${a}^{(i)}$ and $b$ of spectral mixing ${\boldsymbol{f}}$ in formula (1), the correlation coefficient and 2-norm distance were selected as quantitative indicators to evaluate the spectral matching between the collected real pigments and corresponding simulated mixed pigments, as well as the original single pigments. For pigments of different color schemes, multiple mixing coefficients will be simulated and analyzed, and the coefficient with the best matching degree is selected as the mixing coefficient for the real scene. Meanwhile, due to the use fusion of two different principles of operator in this method, in order to analyze the accuracy, robustness of fusion algorithm, and mutual exclusion of the two operators in the fusion process of the algorithm, we will conduct more detailed simulation experiments in subsequent experiments.

Here, it should be noted that the above method mainly identifies the mural pigments with the best matching degree to the database, in order to determine their main pigment components and corresponding pigment spectral curves. For the distribution analysis of mural pigments, it is directly achieved by built-in algorithms in Envi software.

Digital restoration method of mural pigment colors based on bandpass energy integration, pseudo color rendering and color space correction

After obtaining the pigment recognition results of the murals based on above spectral analysis, it is necessary to consider how to use the identified pigment spectral data for digital restoration of mural pigment colors.

Generally, pseudocolor display technology is a method to display color by using spectral data³³. However, directly performing multi-channel synthesis pseudocolor display by single RGB channel may result in insufficient color realism and stability. So we use pseudocolor display technology with bandpass energy integration in RGB color space to achieve virtual display of pigment colors. In other word, by selecting three bandpass spectrum channels corresponding to red, green, and blue, and calculating the integrated energy of the three bandpass spectrum channels of the mural pigment spectrum corresponding to above recognition result, the color of the mural pigment could be displayed by the pseudocolor image with the above three integrated energy of bandpass spectrum channels in the RGB color space.

For the selection of the bandpass for bandpass energy integration used in pseudocolor display technology, the international commission on illumination established specific three primary color standards in 1931: Red-wave (R) =700 nm, Green-wave (G) =546.1 nm, and Blue-wave (B) =435.8 nm. Considering that the quantum efficiency of the camera used in this study is relatively low in the Blue-wave and Red-wave, it is planned to shift the Red-wave to the left by 20 nm, adjust it to 680 nm, and the Blue-wave to the right by 20 nm, adjust it to 455.8 nm; Meanwhile, the new energy values of the RGB three primary colors were calculated using bandpass energy integration under 60 nm spectral width. At this time, the bandpass integrated spectral range of the red, green, and blue channels is 425.8–485.8 nm, 516.1–576.1 nm, and 650–710 nm, respectively. So the three bandpass integrated energy ${({\rm{I}}}_{{\rm{R}}},{{\rm{I}}}_{{\rm{G}}},{{\rm{I}}}_{{\rm{B}}})$ corresponding to the Red-wave (R), Green-wave (G), and Blue-wave (B) channels could be calculated as following.

$${{\rm{I}}}_{{\rm{B}}}=\mathop{\sum }\limits_{{\rm{j}}=425}^{485}{\boldsymbol{f}}({\rm{j}})* {{\rm{I}}}_{{\rm{m}}}/60$$

$${{\rm{I}}}_{{\rm{G}}}=\mathop{\sum }_{j=516}^{576}\,{\boldsymbol{f}}({\rm{j}})* {{\rm{I}}}_{{\rm{m}}}/60$$

(6)

$${{\rm{I}}}_{{\rm{R}}}=\mathop{\sum }\limits_{j=650}^{710}{\boldsymbol{f}}({\rm{j}})* {{\rm{I}}}_{{\rm{m}}}/60$$

In the formula, ${{\rm{I}}}_{{\rm{m}}}$ is the energy peak, taking 8-bit data as an example, it can be taken as 255. It can be seen that in order to achieve a unified standard for the representation of different bandpass integral energies, the number of spectral bands for their respective bandpass integrals is the same, all of which are 60, and their respective bandpass integral energies are averaged.

Based on the new three channel data ${({\rm{I}}}_{{\rm{R}}},{{\rm{I}}}_{{\rm{G}}},{{\rm{I}}}_{{\rm{B}}})$ by bandpass energy integration mentioned above, pseudocolor display technology was used in the RGB color space to obtain the pseudocolor display results of mural pigment colors. As a result, considering the spectral changes caused by the degradation of the pigment layer, compared to the pseudo color rendering under single wavelength representation, the pseudo color rendering under bandpass energy integration representation has a wider adaptation area to the RGB channel, higher tolerance, and a milder response to the spectral changes caused by the degradation of the pigment layer. Therefore, it can more stably represent the color of murals.

However, the above image model, commonly referred to as RGB model, it do not fully depict aspects such as lighting and noise in their color display. Therefore, in the field of image processing, it is generally done by converting RGB model into HSI model. The three attributes of the HSI model are hue (H), saturation (S), and intensity (I). Among them, hue is determined by the spectral wavelength to which the color belongs, which is a qualitative characteristic used to represent the kind and type of color; The saturation depends on the amount of white light mixed in the color, indicating the depth of the color; Intensity depends on the intensity of the color’s light, which is a characteristic of the color in terms of quantity, used to indicate the brightness of the color.

At this time, HSI model can be transformed from RGB model in the following way, which can be represented as $({\rm{H}},{\rm{S}},{\rm{I}})={{\varnothing }}({\rm{R}},{\rm{G}},{\rm{B}})$ as following.

$${\rm{I}}=({\rm{R}}+{\rm{G}}+{\rm{B}})/3$$

$${\rm{S}}=1-\frac{3}{({\rm{R}}+{\rm{G}}+{\rm{B}})}\min ({\rm{R}},{\rm{G}},{\rm{B}})$$

$${\rm{H}}=\left\{\begin{array}{l}\qquad{{\theta }},\qquad{\rm{B}}\le {\rm{G}}\\ {360}^{\circ}-{{\theta }},\quad{\rm{B}} \,>\, {\rm{G}}\end{array}\right.$$

(7)

$${\rm{\theta }}=\arccos \frac{[({\rm{R}}-{\rm{G}})+({\rm{R}}+{\rm{B}})]/2}{{[{({\rm{R}}-{\rm{G}})}^{2}+({\rm{R}}-{\rm{B}})({\rm{G}}-{\rm{B}})]}^{1/2}}$$

Due to the independence of the I component from the color information of the image, the H and S components correspond to the color characteristics perceived by humans. Therefore, in order to quantify the impact of lighting, noise, and other factors on color, in this study, we further utilized the I component in the HSI model of color images and pseudo color images to correct the color restored by the RGB model.

Firstly, the true color photo ${({\rm{F}}}_{{\rm{R}}},{{\rm{F}}}_{{\rm{G}}},{{\rm{F}}}_{{\rm{B}}})$ taken by the camera were converted by the HSI model (7) to obtain the HSI converted data of the true color photos, namely:

$$({{\rm{F}}}_{{\rm{H}}},{{\rm{F}}}_{{\rm{S}}},{{\rm{F}}}_{{\rm{I}}})={{\varnothing }}({{\rm{F}}}_{{\rm{R}}},{{\rm{F}}}_{{\rm{G}}},{{\rm{F}}}_{{\rm{B}}})$$

(8)

At the same time, the pseudo color image ${({\rm{I}}}_{{\rm{R}}},{{\rm{I}}}_{{\rm{G}}},{{\rm{I}}}_{{\rm{B}}})$ simulated by bandpass energy integration using formula (6) for the mural pigment were converted by the HSI model (7) to obtain the HSI converted data of the above pseudo color image, namely:

$$({{\rm{I}}}_{{\rm{H}}},{{\rm{I}}}_{{\rm{S}}},{{\rm{I}}}_{{\rm{I}}})={{\varnothing }}({{\rm{I}}}_{{\rm{R}}},{{\rm{I}}}_{{\rm{G}}},{{\rm{I}}}_{{\rm{B}}})$$

(9)

Then, the I component of the HSI data $({{\rm{F}}}_{{\rm{H}}},{{\rm{F}}}_{{\rm{S}}},{{\rm{F}}}_{{\rm{I}}})$ of the true color photo was used to replace with the I component of HSI data $({{\rm{I}}}_{{\rm{H}}},{{\rm{I}}}_{{\rm{S}}},{{\rm{I}}}_{{\rm{I}}})$ of the aforementioned pseudo color image. Further, we used the inverse transformation of formula (7) to invert the corrected HSI data $({{\rm{I}}}_{{\rm{H}}},{{\rm{I}}}_{{\rm{S}}},{{\rm{F}}}_{{\rm{I}}})$. As a result, the corrected RGB data ${(\bar{{\rm{I}}}}_{{\rm{R}}},{\bar{{\rm{I}}}}_{{\rm{G}}},{\bar{{\rm{I}}}}_{{\rm{B}}})$ could be achieved as following:

$$({\bar{{\rm{I}}}}_{{\rm{R}}},{\bar{{\rm{I}}}}_{{\rm{G}}},{\bar{{\rm{I}}}}_{{\rm{B}}})={{{\varnothing }}}^{-1}({{\rm{I}}}_{{\rm{H}}},{{\rm{I}}}_{{\rm{S}}},{{\rm{F}}}_{{\rm{I}}})$$

(10)

It can be seen that by displaying the corrected RGB data again using pseudocolor display technology, the digital restoration results of pigment colors with color space correction can be obtained.

Further, the aforementioned mural restoration is calculated and corrected based on the colors in color photos, and color photos inevitably encounter difficulties in obtaining information on pigment fading. In this case, pure simulated data ${({\rm{I}}}_{{\rm{R}}},{{\rm{I}}}_{{\rm{G}}},{{\rm{I}}}_{{\rm{B}}})$ are considered for compensation. Therefore, a digital restoration of mural painting colors under simulated color compensation and color space correction is construct:

$$\begin{array}{ll}({\overline{\overline{{\rm{I}}}}}_{{\rm{R}}},{\overline{\overline{{\rm{I}}}}}_{{\rm{G}}},{\overline{\overline{{\rm{I}}}}}_{{\rm{B}}})=({{\rm{I}}}_{{\rm{R}}},{{\rm{I}}}_{{\rm{G}}},{{\rm{I}}}_{{\rm{B}}})+({\bar{{\rm{I}}}}_{{\rm{R}}},{\bar{{\rm{I}}}}_{{\rm{G}}},{\bar{{\rm{I}}}}_{{\rm{B}}})\\\qquad\qquad\quad=\dfrac{1}{2}(({{\rm{I}}}_{{\rm{R}}},{{\rm{I}}}_{{\rm{G}}},{{\rm{I}}}_{{\rm{B}}})+{\varnothing }^{-1}({{\rm{I}}}_{{\rm{H}}},{{\rm{I}}}_{{\rm{S}}},{{\rm{F}}}_{{\rm{I}}}))\end{array}$$

(11)

It should be noted that in actual calibration, every pixel is corrected using the corresponding to each pixel’s I components of HSI data.

In real mural restoration, based on the single pigment color restoration results mentioned above, and pigment distribution and feature enhanced image directly achieved by Envi software, with the help of traditional image processing and human-computer interaction, the overall restoration of the mural can be completed. The relevant detailed explanation is described in the experimental results section.

Results

The design of this paper main includes spectral recognition simulation experiments, real mural information recognition and restoration experiments. Before this, we analyze the spectral characteristics of mural pigments in different states, in order to select the spectral bands and mixing modeling coefficients required for pigment recognition.

Spectral characteristics analysis of mural pigment for presetting the model

Most ancient pigments are mineral pigments, according to the theory of physical optics, their physical coloring reactions are relatively stable, but there are small chemical changes and reflectance changes caused by the aging of pigment colloids and minor degradation. Meanwhile, the spectral characteristics of mural pigments are influenced by factors such as pigment composition, particle size, glue addition, and substrate. At this time, in order to select spectral bands for identifying mural pigments in real scenes, the spectrum of the mural pigment were simulated and analyzed as following.

For selecting key spectral bands,we main analyse spectral characteristics of pigments. On the one hand, the pigment layer in real scenes is a mixed substance with different particle size states, and the influence of particle size on pigment spectrum is mainly reflected in amplitude. It changes the reflection intensity of light by affecting light scattering, but the shape of the spectral curve remains basically unchanged. As shown in Figs. 6 and 7, the reflectance spectra and derivative spectra of stone green and stone blue are presented respectively, which is achieved by using an ASD spectroradiometer. It can be seen that as the particle size of the pigment decreases, the visible and near-infrared reflectance spectra of the pigment generally maintain an increasing trend, especially at the characteristic peak positions. Correspondingly, the derivative spectra of pigments with different particle sizes show similar trends at the characteristic peak positions, but with different rates.

Fig. 6

Reflectance and derivative spectra of stone green with different particle sizes.

Fig. 7

Reflectance and derivative spectra of stone blue with different particle sizes.

On the other hand, the spectral changes of the pigment after adding glue are mainly reflected in the local absorption effect of colloidal organic molecular bonds on the shortwave infrared spectrum. The humidity of the mural substrate mainly affects the water vapor absorption peak in the near-infrared spectrum, while the spectral influence of the mural substrate material on the mural pigment layer is a weighted mixture as a whole. At this time, the spectrum of the mural pigment were mixing simulated and analyzed as following.

Firstly, we added glue to the pigment powder and drew it on a white cardboard, and continued to use an ASD spectroradiometer to measure the reflection spectrum of the drew pigment. Then the representative pigment of different color systems were selected, and the spectrum of powdered pigment and the spectrum of corresponding pigment plotted on white cardboard after gluing are compared in experiment. At this time, we analyses the reflectance spectrum and derivative spectrum of pigment spectrum under two conditions, as shown in Fig. 8.

Fig. 8

Reflectance and derivative spectrum curves of representative pigments of different color systems in different states (dashed lines represent pigments powder, dashed lines represent pigments glued and drawing on white cardboard).

From the derivative spectrum of the visible light bands at 400–800 nm, it can be seen that there is little difference in the characteristics between the glued pigment spectrum plotted on the white cardboard and the original pigment powder spectrum, while the difference in visible near-infrared reflectance spectrum is mainly amplitude differences, indicating that the effect of gluing on the visible light spectrum of the pigment is relatively small. From the near infrared bands of the reflectance spectrum and derivative spectrum, there are significant differences in the characteristics between the two types of spectrum at 1200 nm, 1400 nm, 1900nm, and 2100–2300 nm. Therefore, when not considering the amplitude situation temporarily, the derivative spectral changement are obvious in whole bands in above two states, but the derivative spectral changement are stable in visible light bands in above two states.

Therefore, combining with the previously mentioned, it can be considered to directly use the spectral characteristics of pigments in the visible light bands and a small amount of near infrared light bands for spectral analysis recognition (i.e. 450–800 nm spectral bands) of mural pigments in real scenes, except for the amplitude difference. In addition, in special circumstances, there may be situations where different pigments are mixed and used, which also could be considered by the above mixing model.

The above analysis has provided us with a relatively stable spectral analysis band under the physical fading of pigments. Furthermore, we will continue to simulate and analyses mixed spectrum of mural pigment drawn on white cardboard under smoke and dust pollution in real scenes, and to select the spectral mixing coefficients in formula (1).

Taking the red pigment present in actual murals as an example, we set a pure color pigment and simulate its spectral curves drawn on a white cardboard in different ratios. Here, the ratio weights of red pigment and white cardboard are set to 8:2 and 8.5:1.5, which are defined as simulated mixed pigments 1 and 2, respectively. At the same time, we simulate adding smoke and dust pollution to red pigment drawn on a white cardboard as examples of other mixed pigments states. Here, the ratio weights of red pigment, black pigment (simulates smoke and dust pollution), and white cardboard are 7.5:0.5:2, 8:0.5:1.5 and 8.5:0.5:1, respectively, which are defined as simulated mixed pigments 3, 4 and 5. During experimental verification, conduct experimental analysis using red and yellow research, and red pigment is chosen as ochre, yellow pigment is chosen as orpiment, black smoke is chosen as carbon black, and white substrate is chosen as gypsum. Meanwhile, the spectral curves of the red pigment layer from real murals as control group data were extracted. At this time, the spectral curves of the red and yellow pigments under different states are shown in Fig. 9.

Fig. 9: Spectral curves of red and yellow pigments under different states.

a is a comparison of different spectra of red pigments, and b is a comparison of different spectra of yellow pigments.

Furthermore, in the simulation of the main color pigment, white base, and black pollution, the weights of the main color pigment and white base were dynamically adjusted with a black pollution of 0.5 as the constant value. So the correlation coefficient curve between the simulated spectrum and the real spectrum under dynamic display was obtained for evaluating the rationality of the selected weights in this paper, as shown in Fig. 10.

Fig. 10: Matching degree curves of corresponding simulated spectrum and real pigment spectrum (red, yellow) under different weights of white base.

a shows the matching degree curve corresponding to the red pigment, and b shows the matching degree curve corresponding to the yellow pigment.

At this time, the correlation coefficient and 2-norm distance were selected as quantitative indicators to evaluate the spectral matching between the collected different pigments and corresponding simulated mixed pigments, as well as the original single pigments. The results are shown in Table 2.

Table 2 Spectral matching values between different pigments in different simulation states (including original pigments in database) and the corresponding mural pigments used in real scene

It can be seen that the correlation coefficient between the mural red pigment in real sense and all simulated mixed pigments are greater than the correlation coefficient between the mural red pigment in real sense with the original red pigment in database. Similarly, the 2-norm distance between the mural red pigment in real sense and all simulated mixed pigments are less than the 2-norm distance between the mural red pigment in real sense with the original red pigment in database. For other pigments, the above conclusion is basically the same. Therefore, the proposed scheme of using weighted mixed pigment spectrum to simulate mural spectrum in real scenes is more feasible than directly using the original pure pigment spectrum. Further, from the correlation coefficients, 2-norm distance analysis and matching degree curve in Fig. 10, it was found that the red pigment 5, the yellow pigment 3, the green pigment 3, and the black pigment 3 in white cardboard with smoke and dust pollution had the highest matching degree with the corresponding mural pigments used in real scene. Correspondingly, the selected weighted values of red pigment, yellow pigment, green pigment and the black pigment, to black pollution and white base were selected as 8.5:0.5:1, 7.5:0.5:2, 7.5:0.5:2 and 7.5:0.5:2.

Simulation recognition experiment of mural pigment

In this section, we first discuss the accuracy of the algorithm proposed in this article. Then, the focus is on simulating and analyzing stability and exclusiveness of the fusion recognition algorithm.

The original intention of using weighted fusion of SAM and SID in this paper is: Complementary hypothesis. SAM is robust to changes in illumination but ignores intensity differences, while SID can capture statistical distribution differences, which can measure the longitudinal intensity changes and transverse fine light spectral characteristics of the spectrum. Due to degradation issues such as pollution, pigment particle detachment, oxidation, etc., result in amplitude attenuation of mural spectra, the pigment spectra in actual scenes may exhibit “same substance, different spectra”; Meanwhile, local spectral features also exist in global spectrum. Due to the fact that most of the pigment spectra in this study have similar spectral curves, it can be seen that the role of the SAM operator is dominant in the fusion recognition algorithm. While the combining of SAM and SID is a suitable choice to improve the consistency and adaptability of single SAM algorithm in the spectral intensity variation and local spectral feature analysis. So below is an evaluation of the accuracy of the algorithm.

Taking red pigment as an example, in formula (1), for the simulated real spectra based on standard spectra, adding noise to weights a,b is called multiplicative noise ${e}_{1}$, and adding noise to the entire value is called additive noise ${{\rm{e}}}_{2}$, as shown in following formula:

$${\boldsymbol{f}}=\mathop{\sum}\limits_{i=1}^{{\rm{t}}}{({a}^{(i)}+{e}_{1}){\boldsymbol{f}}}^{(i)}+(b+{e}_{1}){{\boldsymbol{f}}}^{k}+{e}_{2}$$

(12)

Then we use SAM, SID, and the algorithm proposed in this article for spectral recognition. In the specific experiment, 8 red pigments were subjected to 1000 repetitions of random noise, and use the original spectral data as reference data, so the experimental results are shown in Table 3.

Table 3 Accuracy of spectral recognition algorithm under different noise and different algorithm

It can be seen that from Table 3, the hybrid recognition operator in article has better recognition accuracy compared to the SAM or SID operator, so the hybrid recognition operator has slightly better performance. It can be seen that excessive noise has a direct impact on accuracy. For example, the added noise spectrum curves with additive noise and multiplicative noise of 0.07 and 0.05 are shown in Fig. 11(a), whose accuracy is the worst. The added noise spectrum curves with additive noise and multiplicative noise of 0.04 and 0.01 are shown in Fig. 11(b), whose accuracy is the best.

Fig. 11: Added noise spectrum curves for different recognition algorithms with additive noise and multiplicative noise of 0.07 and 0.05, 0.04 and 0.01.

a shows the spectral curves corresponding to additive noise and multiplicative noise of 0.07 and 0.05, respectively. b shows the spectral curves corresponding to additive noise and multiplicative noise of 0.04 and 0.01, respectively.

However, due to the different principles of SAM algorithm and SID algorithm, there may be situations where weighted calculations are mutually exclusive in mixed algorithm. For example, the spectral amplitude scaling of the same substance under different lighting conditions leads to significant differences in SID, but the SAM angles are close. At this point, SAM is the main criterion for determining the category, and it should not be much affected by the SID operator. For another instance, different substances have similar shapes in the most bands, but significant differences in their distribution in the less band. At this point, SAM metrics are relatively close, but SID could capture the distribution differences of small amount of spectrum, it would make SID to play a major role. Therefore, in complex spectral scenarios, by verifying whether the performance of the combination of SID and SAM are better than a single indicator, as well as their mutual exclusion judgment rate, it can be provided the mutual exclusion evaluation of the two algorithms. The specific experiment is as follows.

On the one hand, for the evaluation of the mutual exclusion judgment rate of the fusion algorithm, we adopted the experimental data simulation scheme and experimental fusion recognition scheme in above experiment, and focused on counting the number and proportion of mutually exclusive recognition result samples in SAM algorithm and SID algorithm, and the results is in Table 4.

Table 4 Proportion of mutually exclusive recognition result samples in SAM algorithm and SID algorithm

From the Table 4, it can be seen that the mutual exclusion rate of the recognition results of the SAM operator and SID operator in the fusion algorithm is small, and as the noise increases, the corresponding recognition results deteriorate, and their mutual exclusion rate also increases. So the mutual exclusion between the two algorithms is controllable and mainly affected by noise.

On the other hand, we separately prepared two sets of samples based on the advantages of SAM operator and SID, and conducted comparison experiments between fusion algorithm, SAM algorithm and SID algorithm. Firstly, we set up two experimental groups. Group (1): morphological similarity group, spectra of different pigments in the same color system, with gaussian noise added; Group (2): Intensity variation group, simulated spectra of the same pigment with different light intensities or different particle sizes, with gaussian noise added. Then, we calculate the identification operator values of two sets of data under three methods. Among them, the experimental results of group 1 are shown in above. At this point, we simulate spectrum under different light intensities by using green pigments for group 2, obtain spectral data of different particles of stone green and stone blue pigments through real collection experiments for group 2, and also use the spectral data as reference data to conduct spectral recognition experiments of different algorithms. Here, the corresponding enhanced spectral data for different simulated light intensities are shown in Fig. 12.

Fig. 12

Corresponding enhanced spectral data for different simulated light intensities (#3 > #2 > #1).

Specifically, based on the spectra of green pigments with different particle sizes in Figs. 6 and 7, the spectra of green pigments with different light intensities in Fig. 12 and the noise adding scheme shown in formula (11), by examining whether the recognition accuracy of the fusion algorithm is optimal and most robust in three methods, we will verify the better and more stable recognition ability of the SID operator in the fusion algorithm for changes in light intensity or particle size, as well as local spectral feature. Here, we compute the normalized recognition operator values of different algorithm and our algorithm. For example, the experimental recognition operator results of mineral green,light intensity (#1) for different cases in group 2 are shown in Table 5, the experimental recognition operator results of mineral green,granularity (large) for different cases in group 2 are shown in Table 6. Correspondingly, the curves of the reference spectra and the simulated spectra to be identified for the two cases are shown in the Fig. 13. here, the values of additive noise and multiplicative noise are 0.02 and 0.01.

Table 5 Experimental recognition operator result of mineral green,light intensity (#1) for different cases in group 2

Table 6 Experimental recognition operator result of mineral green, granularity (large) for different cases in group 2

Fig. 13: Curves of the reference spectrum and the simulated spectrum to be identified for the pigment spectrum under different light intensities and pigment spectrum of different particles.

a corresponds to the spectral curves of pigments under different light intensities, and b corresponds to the spectral curves of pigments with different particle sizes.

According to the theory of this study, the smaller the operator value, the higher the matching degree. It can be seen from Tables 5 and 6 that, in the above simulation experiments, our algorithm has smaller values than the operators of SAM algorithm and SID algorithm, so it has better recognition efficiency. Meanwhile, for the spectral recognition of different light intensities or particle sizes of the same pigment, the fusion of SAM and SID operators can achieve consistent and small results, indicating that the fusion algorithm is more accuracy and consistency for recognizing different light intensities and particle sizes of the same pigment.

Of course, if we don’t want to maintain this consistency and want to identify different light intensities and particle sizes, we can increase the weight of the SID operator.

Pigments recognition and construction of color digital restoration database of the murals pigments from late Tang tomb M1373

According to the improved fusion spectral recognition algorithm, pigment recognition result would be achieved based on the visible near-infrared 450-800 nm characteristic spectral data of the pigment layer; Then, based on the spectral curve of the identified pigment mentioned above, we can complete the color restoration of a single pigment according to the digital display scheme of pigment color. So, taking the M1373 mural as an example, real-life experiments will be conducted to verify it.

For the identification of mural pigments and digital restoration sets of pigments colors, firstly, we simulated the different mixed spectrum based on the weighted ratio (8.5:0.5:1) for red pigment, and on the weighted ratio (7.5:0.5:2) for yellow, green and black pigments, and then the improved fusion spectrum matching method were used to identify the mural surface pigments of different color system. Here, the recognized result of each color were the one with the highest matching degree; Then the color restoration display of the mural pigment is performed using spectral bandpass energy integration method and RGB pseudocolor technology; At the same time, the pure pigment restoration results drawn on a white substrate is directly got by pseudocolor image, and the pseudocolor image were corrected as the actual pigment color restoration results using the mixed spectral scheme (main color, background, pollution) and HSI color space correction method.

In the tomb M1373 murals, red pigments are mainly used to draw clothes, blankets, music drums, etc., yellow pigments are used to draw clothes, blankets, tables, etc., green pigments are used to draw clothes, blanket patterns, etc., and black pigments are used to draw tables, hair, shoes, etc. Next, we will conduct experimental analysis one by one for the above pigment.

Firstly, we matched the spectrum at the red position of the tomb mural with the aforementioned mural pigment spectrum database, and it is highly likely that the red pigment spectrum matching result is earth red (ochre, rock ochre), the main components of the three are ferric oxide. The spectrum matching results and spectrum matching degree of red pigment, as well as their pigment color restoration results, are shown in Figs. 14, 15.

Fig. 14: Spectrum matching result and spectrum matching degree of red pigment in the mural.

a shows the spectral matching result of red pigment, and b shows the corresponding spectral matching degree of red pigment.

Fig. 15: Pigment color restoration results of red pigment in the mural of this tomb: pure pigment color restoration by pseudocolor display, pigment color restoration by the mixed spectral scheme and by HSI color space correction method.

a shows the basic pseudo color display results, b shows the corresponding display results under mixed simulation, and c shows the corresponding display results after HSI color space correction.

Secondly, we matched the spectrum at the yellow position of the tomb mural with the aforementioned mural pigment spectrum database, and it is highly likely that the yellow pigment spectrum matching result is mineral yellow (orpiment, yellowish brown), while the main components of mineral yellow and orpiment are the same. The spectrum matching results and spectrum matching degree of yellow pigment, as well as their pigment color restoration results, are shown in Figs. 16, 17.

Fig. 16: Spectrum matching result and spectrum matching degree of yellow pigment in the mural.

a shows the spectral matching result of yellow pigment, and b shows the corresponding spectral matching degree of yellow pigment.

Fig. 17: Pigment color restoration result of yellow pigment in the mural of this tomb: pure pigment color restoration by pseudocolor display, pigment color restoration by the mixed spectral scheme and by HSI color space correction method.

a shows the basic pseudo color display results, b shows the corresponding display results under mixed simulation, and c shows the corresponding display results after HSI color space correction.

Thirdly, we matched the spectrum at the green position of the tomb mural with the aforementioned mural pigment spectrum database, and it is highly likely that the green pigment spectrum matching result is mineral green. The spectrum matching results and spectrum matching degree of green pigment, as well as their pigment color restoration results, are shown in Figs. 18, 19.

Fig. 18: Spectrum matching result and spectrum matching degree of green pigment in the mural.

a shows the spectral matching result of green pigment, and b shows the corresponding spectral matching degree of green pigment.

Fig. 19: Pigment color restoration result of green pigment in the mural of this tomb: pure pigment color restoration by pseudocolor display, pigment color restoration by the mixed spectral scheme and HSI color space correction method.

a shows the basic pseudo color display results, b shows the corresponding display results under mixed simulation, and c shows the corresponding display results after HSI color space correction.

At last, we matched the spectrum at the black position of the tomb mural with the aforementioned mural pigment spectrum database, and it is highly likely that the black pigment spectrum matching result is carbon black. The spectrum matching results and spectrum matching degree of black pigment, as well as their pigment color restoration results, are shown in Figs. 20, 21. It should be noted that the spectral characteristics and spectral reflections of black pigments are not obvious, therefore, the recognition effect of black pigments is not good.

Fig. 20: Spectrum matching result and spectrum matching degree of black pigment in the mural.

a shows the spectral matching result of black pigment, and b shows the corresponding spectral matching degree of black pigment.

Fig. 21: Pigment color restoration result of black pigment in the mural of this tomb: pure pigment color restoration by pseudocolor display, pigment color restoration by the mixed spectral scheme and HSI color space correction method.

a shows the basic pseudo color display results, b shows the corresponding display results under mixed simulation, and c shows the corresponding display results after HSI color space correction.

Through the above experimental analysis, it can be concluded that for the pigments of different color systems in the Tang tomb murals studied in this paper, based on the simulated mixed spectral data of several pigments in real scenes, the fusion spectral feature matching method can be used to provide corresponding recognition results for pigments in the mural such as red, yellow, green and black. Meanwhile, the use of bandpass spectral energy integration, mixing modeling and color space correction methods can be used to obtain virtual restoration results of pigment colors for different pigment system in the mural, including pure color restoration, pigment color restoration by the mixed spectral scheme, and pigment color restoration by the mixed spectrum and color space correction.

Digital restoration of color and patterns of mural painting from late Tang tomb M1373

Due to the long period of time, many diseases have appeared in the murals from late Tang tomb M1373, which seriously affects the authenticity and completeness of the mural painting, and thus affects the artistic and cultural value of the murals. After obtaining the color virtual restoration sets of the aforementioned mural single pigment, it is necessary to perform color and patterns restoration of mural painting in actual sense. For this purpose, in addition to achieving above color restoration of different pigments, it is also necessary to confirm the color distribution of the mural painting in order to carry out targeted color and pattern restoration in next.

Step1: to obtain the pigment distribution of the mural.

Firstly, we performed principal component transformation (by ENVI software) on the mural color photo in Fig. 1, which changed the visual representation of murals. At this time, we can analyses the weak information in mural painting, and find the distribution basis for repairing the color and integrity of mural patterns. As shown in Fig. 22, the three channels composite image from principal component transformation for the mural color photo is presented. From the above figure, it can be seen that the carpet dancer’s line pattern and carpet’s line pattern in the picture do not show a multi-layer overlapping structure, and the other patterns also do not show a multi-layer overlapping structure too. Therefore, it is speculated that it was painted using a single-layer composition and elements in this mural. Based on the above, for the overall color restoration of the mural painting, it main based on intuitive analysis of the color distribution of murals painting and the restoration result of corresponding murals pigment colors, without having to consider multiple overlapping patterns.

Fig. 22: Three channels composite image from principal component transformation for the mural color photo.

The elements in Fig. 22 were obtained from data captured by our team’s camera, and no pre-existing external materials were used. The Fig. 22 is generated using the following software with valid licenses: ENVI6.0 Trial Version.

In specific implementation, due to the use of orbital translation for local imaging in hyperspectral imaging to the mural, in order to understand the global color distribution of the murals, maximum likelihood estimation method by ENVI software was used to obtain the color classification distribution of global color photos, as shown in Fig. 23. At the same time, spectral angle matching by ENVI software was used to classify the color of the mural local hyperspectral images collected through local scanning, as shown in Fig. 24(a). Accordingly, the color classification distribution of local color photos by ENVI software is shown in Fig. 24(b).

Fig. 23: Mural color distribution in global color photos calculated by maximum likelihood estimation method.

The elements in Fig. 23 were obtained from data captured by our team’s camera, and no pre-existing external materials were used. The Fig. 23 is generated using the following software with valid licenses: ENVI6.0 Trial Version.

Fig. 24: Comparison of pigment distribution results based on different local murals.

a shows mural color distribution in local hyperspectral image calculated by spectral angle matching method, b shows corresponding mural color distribution in local color photo calculated by maximum likelihood estimation method. The elements in (a) were obtained from data captured by our team’s hyperspectral imaging scanning system for tomb murals, and no pre-existing external materials were used. The elements in (b) were obtained from data captured by our team’s camera, and no pre-existing external materials were used. The Fig. 24 is generated using the following software with valid licenses: ENVI6.0 Trial Version.

By comparing the color classification map of mural color photos with the color classification map of mural hyperspectral images, it can be concluded that the classification effect of hyperspectral images is basically similar to that of color photos expect some noise. In other words, studying the color distribution of murals by using color photos can achieve an effect similar to that by using hyperspectral images when we don’t considering multiple overlapping patterns. However, the advantage of recognizing color and material properties brought by hyperspectral images is not possessed by color photos. Therefore, mural pigment identification based on hyperspectral images with the methods mentioned above, and mural pigment distribution identification based on color images with the methods mentioned above will be conducted, and then further displayed on color photo substrate use the above color and color distribution results.

Step 2: to digitally restore the overall color of the mural using the color restoration database mentioned above.

Based on the above analysis, for this tomb mural, the colored pigments such as red, yellow, green, as well as auxiliary pigments such as black and white are used. Based on the color classification distribution map in Figs. 23, 24, global virtual restoration will be performed on the colors of different colored pigments of mural painting in sequence.

In order to achieve overall restoration of the color of the mural painting, the pixel values of the corresponding recognized pigments in the color photos are replaced by the bandpass energy integration pseudocolor pixel values of the pigments simulated on a white substrate with pollution, which is also corrected by the aforementioned color space. In other word, in order to make the restored color by color space correction more dynamic and realistic, different pixel values of specific pigments are corrected using the corresponding brightness/intensity components in the HSI model of the mural color photo shown in Fig. 25.

Fig. 25: Corresponding brightness/intensity components in the HSI model of the mural color photo.

The elements in Fig. 25 were obtained from data captured by our team’s camera, and no pre-existing external materials were used. The Fig. 25 is generated using the following software with valid licenses: MATLAB R2024b (License No.: 40827055, MATLAB (Individual)).

Before performing the overall color restoration of the mural, the detached part of the mural in the global color photo is first repaired. The image value of the detached part of the mural is directly replaced with the image value of the non-detached white base in the color photo, so we would obtain the new color image of the mural after the background restoration, as shown in Fig. 26(a).

Fig. 26: Restoration results of different process nodes during mural color restoration.

a shows color image of the mural after the background restoration, b shows color restoration results of red pigment without RGB-HSI color space correction, c shows color restoration results of red pigment with RGB-HSI color space correction, and d shows color restoration results of red, green, yellow pigments. The elements in Fig. 26 were obtained from data captured by our team’s camera, and nd the pre-existing elements in Figs. 15, 17, 19 were used. The Fig. 26 is generated using the following software with valid licenses: MATLAB R2024b (License No.: 40827055, MATLAB (Individual)).

Then, based on the color image of the mural after the background restoration in Fig. 26(a), the pigment recognition results obtained from the previous experiment and the pigment color restoration display method under color space correction were used to globally restore the pigments of different color in this tomb mural. At this time, we do a comparative experiment as following: taking red pigment as an example, the color restoration results of pigment without RGB-HSI color space correction and with RGB-HSI color space correction are presented, as shown in Fig. 26(b), (c).

By comparing the experimental results, it can be concluded that a simple RGB pseudo color display can restore the color of the pigment, but due to the lack of spatial dynamic changes in color, the rendering of color intensity does not exist for same pigment; While the color restoration results of the pigment with RGB-HSI color space correction in this article not only achieve single color expression, but also are more realistic in the spatial dynamic changes. Based on the above good results, color restoration was also carried out on the yellow and green pigments of the mural. So the repaired color image of the red, yellow, and green pigment of mural painting is shown in Fig. 26(d).

It can be seen that the color restoration of red, yellow, and green pigments in mural painting has a good overall restoration effect, with some fading pigments significantly restored. Moreover, the restoration is not simply replacing the color of the same pigment with the pseudo color pixel values obtained from the previous experiment, but using the brightness/intensity component in the HSI color space for dynamic correction of pixel values, which preserve as much as possible the different color depths of the same pigment in the painted pattern.

Step 3: to perform digital restoration of the black baseline and pattern of the mural as a whole.

From the above analysis, although the colored pigment color of the mural have been well restored through the previous restoration steps, the black pigment, the bottom line and texture of the mural have still been overlooked. Therefore, further improved restoration will be carried out on the aforementioned mural color restoration image, like black enhancement, texture enhancement, and interactive processing is needed for repairing the patterns in mural painting. Here, image texture enhancement and black manuscript extrication are directly achieved through Envi software. After completing the above work, the mural painting color and patterns would be digitally restored.

Specifically, on the one hand, when the black pigment color, black mural baseline were been repairing, black pigment recognition result is selected as the recognition and reparation basis, And perform color restoration operations similar to other pigments mentioned above. Here, the black pigment and black bottom line of the mural is recognized and restored according to the aforementioned pigment color recognition and correction methods, the result is as shown in Fig. 20. On the other hand, according to the differentiated textures and lines in the contour enhanced image, further line enhancement and supplementary drawing were performed on the above repaired image through human-computer interaction. Here, for obtaining texture enhanced images, the mural painting picture were processed by decorrelation stretchline with Envi software, the first and second channels images after decorrelation stretchline are as shown in Figs. 27 and 28.

Fig. 27: First channel of images after decorrelation stretchline to mural image.

The elements in Fig. 27 were obtained from data captured by our team’s camera, and no pre-existing external materials were used. The Fig. 27 is generated using the following software with valid licenses: ENVI6.0 Trial Version.

Fig. 28: Second channel of image after decorrelation stretchline to mural image.

The elements in Fig. 28 were obtained from data captured by our team’s camera, and no pre-existing external materials were used. The Fig. 28 is generated using the following software with valid licenses: ENVI6.0 Trial Version.

Based on the black pigment color representation results obtained from the previous recognition, and contour enhanced image, the black color and bottom line of the mural on the aforementioned mural color restoration image was improved, as shown in Figs. 29(a), (b), respectively. From the Fig. 29, it can be seen that using black pigment color restoration and black baseline restoration based on black pigment recognition, as well as texture contour processing based on painting texture enhancement, the degraded information in the black patterns, bottom lines and textures of mural painting images have been repaired to a certain extent, improving the beauty and integrity of murals. The restoration results provide important data resources for the physical restoration, research, and activation utilization of mural painting in the future.

Fig. 29: Restoration results of different process nodes during baseline lines and patterns restoration.

a shows restoration image of mural painting black pigment, and b shows restoration image of mural painting pattern and bottom line. The elements in Fig. 29 were obtained from data captured by our team’s camera, and the pre-existing elements in Figs. 21, 27, 28 were used. The Fig. 29 is generated using the following software with valid licenses: • MATLAB R2024b (License No.: 40827055, MATLAB (Individual)). • Microsoft drawing board

Here, it should be noted that, unlike existing digital restoration methods based on visual images, this method is based on the material properties of hyperspectral images and takes into account visual information, resulting in better authenticity of color, and the restoration of its pattern texture also has a certain effect. However, its ability to restore visual integrity is insufficient at present. In the future, image filling technology and artificial intelligence methods will continue to be used to research and improve the integrity and rationality of visual texture restoration.

Discussion

In this study, we take the mural of late Tang dynasty tomb M1373 in Xi’an, Shaanxi Province, as the research object, based on mural hyperspectral image, studied the non-destructive identification of mural pigments, the virtual restoration of mural pigment colors, and digital restoration of mural painting patterns, and then digitally restored the “music and dance image” in the tomb murals. By Extensive spectral analysis simulation experiments and mural restoration experiments, we can achieve significant pigment identification and visual restoration of the murals with basic preserved patterns but color degradation or pollution. The research method provides more practical analytical methods and more real data reference for the physical restoration and activation display utilization of murals.

Specifically, an mural pigment spectral recognition method based on improved fusion spectral analysis was used in real scenes. Its feature is to optimize the suitable spectral mixing modeling coefficients through matching degree analysis under mixed spectral modeling, so to simulate the actual degradation and pollution of pigment as good as possible. Meanwhile, a improved virtual restoration method by using color space correction, based on spectral mixing modeling, bandpass energy integration pseudocolor display for mural pigment was proposed. Its advantage is to use the above recognized standard pigment’s spectrum to digital restore the actual pigment color by utilizing bandpass energy integration and pseudo color display technology, based on the three primary color standards developed by the International Commission on Illumination, and then perform color correction based on the brightness changes of the pigment to ensure its authenticity. In addition, the draft line enhancement and human-computer interaction image processing was used to further restore pattern in the real mural restoration.

The experimental results show that the mural uses traditional mineral pigments such as ochre, earth yellow, mineral green and carbon black for its red, yellow, green and black pigments, and the fusion recognition method has better recognition accuracy and robustness. Meanwhile, the richness of painting color of the restoration results is significantly improved compared to the original color photos, especially, the new method with color space correction is significantly better than the method without color space correction. At last, the patterns of the murals have also been improved to a certain extent.

However, there are still some shortcomings in current research. Firstly, the hyperspectral images of murals are mainly obtained through local scanning rather than one-time scanning, and the data collection mode of their stitching has an impact on the subsequent consistency of global restoration. Then, considering the high computational complexity of using hyperspectral images for overall processing, this study focuses on both spectral analysis and hyperspectral image transformations, and the overall data processing and analysis efficiency and capability still need to be improved. Meanwhile, because this method restores and corrects mural colors based on the brightness of color images in real scenes, when the light source in the scene changes, it may alter the relative energy distribution of RGB wide channels, thereby affecting the converted brightness representation. At this time, it is necessary to recalibrate the restoration results of mural colors. In addition, current restoration mainly relies on color restoration, supplemented by texture, its ability to restore visual integrity still needs to be improved. What is more, it need to establish a comprehensive comparison index system and comparative experimental scheme between visual restoration based on pure image and comprehensive restoration based on hyperspectral images. Finally, relevant research also needs to be strengthened in the simulation and dynamic recognition of mural pigment degradation.