Introduction

Tangible cultural heritage represents the wealth remains of cultural value created by human beings in their historical practice1,2, and the information age has brought new opportunities and challenges to protecting tangible cultural heritage3,4,5. The London Charter, published in February 2009, proposed a theoretical framework for computer-based visualization of cultural heritage; and the Seville Principles6, approved in December 2017, proposed guidelines for the field of virtual archeology based on the London Charter. In this regard, there are more authoritative guidelines for virtual restoration in the field of tangible cultural heritage. Guided by the principles of the London Charter and the Seville Principles, an increasing number of scholars are conducting virtual restoration on different types and scales of tangible cultural heritage from the fields of archeological sites7,8, architecture9, cultural relics10,11,12,13, etc.; and these virtual restoration methods have been widely discussed. The digital technology methods as adopted have also become widely explored by scholars. For example, Pacina et al. used a method based on archival aerial photos and old maps to virtually reconstruct the geographical terrain of the dam in the Predam Valley on the Vltava River14. Bartzis et al., when performing virtual restoration of the broken Doric columns, helped the fragments to fall into place by searching the overlap between the original point-cloud and the ideal point-cloud15. Liu et al. used a method that combined digital acquisition, virtual matching, virtual reconstruction and 3D printing to perform transparent and reversible groove restoration on a black tripod bowl as decorated with gold foil16. Wei et al. proposed a multi-dimensional digital replenishment framework for missing bronze fragments because most bronze artifacts unearthed in China suffered from missing fragments, and thus designed a geometric morphology editing technology for these missing fragments17. Comes et al. proposed a complete framework to address the accuracy and precision of 3D reconstruction by virtue of 3D scanning and reverse engineering techniques, and performed a virtual restoration of the Dacia relief discs18.

During virtual restoration, data acquisition is one of the most important and critical steps. So, Fazio et al. proposed to divide modeling data into main data sources and auxiliary data sources19. Among them, main data sources are obtained through 3D digital measurement, and can directly participate in the later modeling process. 3D laser scanning, oblique photogrammetry and their combination are the most important and effective methods for directly acquiring data in the field of cultural heritage these days. Optimization methods for data acquisition have been discussed and proposed against different types, scales, endogenous environments and site conditions. For example, for large monuments and sites, Fabio et al. believed that a combination of 3D digital technologies was needed to achieve complete and detailed virtual restoration20; for 3D mapping of rock carvings, Pena-Villasenín et al. believed that simple SfM photogrammetry was more applicable21; and for pottery relics, Kaneda et al. proposed an efficient SfM/MVS photogrammetry method22. Auxiliary data sources mainly refer to obtaining model reconstruction data from images (drawings, photos, paintings, etc.) and 3D models similar to virtual restoration objects. So far, image-based virtual restoration modeling is widely used and applied due to its completeness, economy and flexibility23,24. Scholars have explored various aspects such as rapid reconstruction of a single image25,26, computer-aided technology27,28, camera calibration29,30, and photo accuracy31. Among them, the lack of image information is a common situation and major difficulty as encountered in this method. In response, scholars choose to apply the perspective reversion method to obtain entity data of the research object in historical photos, so as to facilitate virtual restoration. Considering that perspective images have linear geometric rules with real objects, the process can be reversed, that is, the object shape and position can be reconstructed based on perspective images. Such a perspective method was initiated by J. H. Lambert in the eighteenth century, and has been continuously optimized with the development of image drawing and acquisition technology32. In the research on architectural heritage, historical buildings are widely used because of their clear geometric rules, construction rules, boundary contours and symmetry. For example, Ramón et al. conducted perspective reversion on historical photographic data, and introduced the information about demolished palaces into AutoCAD to make drawings of palace reconstruction models33; Garcia-Gago et al. used a single historical photo to perform dimensional analysis and 3D reconstruction of demolished historical buildings, and then developed a modeling method based on a single image34; Dzwierzynska et al. addressed the problem in establishing basic perspective elements of photo images when camera calibration could not be done, and meanwhile inherited a formula for finding perspective feature points from single-view images, and proposed various image perspective technologies for virtual restoration of buildings32,35,36.

Compared with other heritage types, historic gardens have complex elements, and their appearance reflects seasonal cycles, natural changes and garden art. The Florence Charter put forward in 1982 is an international convention with guiding significance for the protection and management of garden heritage, setting forth basic guidelines for the protection of historic gardens37. Chinese gardens are one of the three major garden systems in the world, and royal gardens in Beijing inherited from the Qing Dynasty are the representative types of existing Chinese historic gardens. Since there are many historical documents and typical cases about the royal historic gardens in Beijing inherited from the Qing Dynasty, scholars have carried out virtual restoration by starting from styles and archives38, photos and drawings39,40, and archeological sites41,42 to virtually restore the overall layout of gardens that no longer exist or from a specific historical period. For example, the team of Guo Daiheng43, that of Wang Qiheng and Zhang Fengwu44, and that of Meng Xiangbin45 virtually reconstructed some scenic spots in Yuanmingyuan; the team of Wu Xiaomin46 virtually restored and reconstructed the scenic spots in the Mountain Resort; the team of Zhang Long restored the Qiwangxuan architectural site from the Qingyi Garden period47; the team of Fu Fan virtually restored some scenic spots in Jingyi Garden in Fragrant Hills Park48, etc. Through the above literature, it is not difficult to find that the virtual restoration work in the field of garden heritage is mainly for the object of the garden environment and garden architecture.

There are a large number of damaged stone setting in and their images about the garden heritage, especially the royal stone setting of the Qing Dynasty, which are the epitome of Chinese stone setting49, as accompanied by the most literature, archives, paintings, historical images and existing cases related to arranged stones. From the material point of view, stone setting are natural stones with ornamental value as produced in different regions. After artificial chiseling and trimming, they are placed in gardens for ornamental purposes. The basic external form and texture of arranged stones are mainly determined by their material, artificial trimming and placement methods. During the appreciation of stone setting, garden owners and appreciators often engrave their names and poems on these stones. Generally speaking, these arranged stones with names or poems had a higher ornamental value at that time, because of unique form, large volume, rich texture, outstanding color, etc. Therefore, based on the amount and presence of historical and cultural information on arranged stones, these stones can be divided into four categories, i.e., stones with both names and poems, stones with only names, stones with only poems, and stones without names and poems. Among them, the first three categories are quite common in Chinese classical gardens and their quantity is huge, while these with both names and poems are outstanding among stone setting, but their quantity is relatively small. Taking these royal gardens in Beijing inherited from the Qing Dynasty as an example, there are only six stones with names and poems on them, namely Qingzhixiu, Qinglianduo, Qingyunpian, Lingfeng, Wenfeng and Yunqifeng. Except for Qinglianduo, which is a Taihu stone (limestone) from the southern China, all the other five are all Taihu stones (limestone) from Dafang Mountain near Beijing. And their names and images are also well known to the world (Fig. 1).

Fig. 1: Part of the setting stone preservation status.
figure 1

a Relics in Beihai Park—Yunqifeng; b Relics in Yuanmingyuan—Qinglianduo; c Relics in the Summer Palace—Qingzhixiu.

Full size image

For the virtual restoration of stone setting there is still a large gap in the virtual restoration of the work, the Qing Dynasty Beijing Royal Garden in the stone there are a large number of cases urgently need to carry out research to explore how to achieve the virtual restoration of high-precision stone setting is a prerequisite for the in-depth implementation of the work. In this paper, it proposes a virtual restoration method suitable for stone setting, and takes the virtual restoration of Lingfeng stone in Yuanmingyuan as an example for empirical research, aiming to provide a new path and reference for the virtual restoration, protection, display and application of stone setting elements in Chinese classical gardens.

Limited by the complexity and diversity of garden elements, in previous research and projects on virtual restoration or reproduction of Chinese historical gardens, the virtual restoration of garden environments is usually holistic, or based on the current understanding of re-understanding or re-creation. Historic Chinese gardens can be divided into mountains, water, architecture and plants according to their elements, of which the architectural elements are fully artificial, i.e., they can be approximated through historical images, design drawings and archeological excavations. The elements of mountains are mainly rockeries and stone setting, which can be interpreted as semi-artificial elements, especially the beautifully shaped stone setting, which are inscribed with names and inscriptions by garden owners and other connoisseurs. The virtual restoration of stone setting in Chinese historical gardens is a difficult task, and there is no targeted restoration research, including the study of historical data and information of the stones themselves and the method of virtual restoration. The innovation of this paper lies in: (1) The uniqueness of the research object: focusing on the virtual restoration of the stone elements in the garden heritage, especially the stone in the royal garden of Beijing in the Qing Dynasty, which is still a research gap in this field. (2) The innovation of the methodology: proposing a virtual restoration method applicable to the stone in the garden, which provides a new scheme for solving the technical difficulties in the virtual restoration of the stone in the garden. (3) Typicality of the case: the case of Lingfeng Stone in Yuanmingyuan is typical and representative, providing a model for the research and practice of virtual restoration of stones in Chinese classical gardens.

Methods

Due to material deterioration, artificial damage and human destruction, stone setting usually suffer from weathering, breakage, fracture, loss and other damage, or wither away due to war. During virtual restoration, it is particularly important to collect data, carry out relevant surveys, and conduct thorough research. Text archives can prove the placement location, function, form, scale and cultural connotation of the object stones. Image information such as drawings, paintings and photos of these stones can provide more accurate or high-precision external form and texture. Unlike artificially constructed material forms showing obvious geometric rules, the external form and texture of stone setting are more complex. Therefore, image information is more direct and important in virtual restoration. In addition, similar origins, specifications, forms and stones that are considered to be a pair by the garden owner can also provide reference in terms of form, texture and color. Because of their clear geometric form, the arranged stones with names, poems/inscriptions can provide a more accurate location, scale and proportion reference, and thus can provide an important basis for age judgment, damage determination and specific data of the object stones. All of the above are auxiliary data sources for virtual restoration. A detailed survey can help comprehensively assess the current occurrence of the object stones and compare them with historical data in a comprehensive and detailed manner. On this basis, the data about the existing object stones, i.e., main data sources, are obtained through 3D digital surveying and mapping.

Also on this basis, we, in line with the spirit of relevant charters and principles, as well as relevant conventions and laws on cultural heritage protection, propose working methods for virtual restoration of stone setting. And the specific workflow contains five parts: firstly, collect information on literature, image photos, similar stones, etc., and conduct on-site inspections and research on the virtual restoration site to comprehensively verify the virtual restoration object in the early stage of analysis; then, classify the existing and missing parts of the virtual restoration object to obtain the data of different accuracies based on previous analysis; next, create a 3D model based on virtual restoration data; thereafter, classify and evaluate the accuracy of virtual restoration achievements of stone setting; and finally, activate and display the achievements of virtual restoration. The specific working methods are as follows (Fig. 2):

Fig. 2
figure 2

Working methods for virtual restoration of stone setting.

Full size image

Case information

In this paper, Lingfeng stone in Wenyuan Pavilion of Yuanmingyuan is selected as a case to practice the working methods and processes, so as to verify whether they are feasible and whether virtual restoration achievements are accurate and scientific. Lingfeng stone is the largest peak-standing stone in the royal gardens inherited from the Qing Dynasty. It was mined in Dafang Mountain, western suburbs of Beijing, and was placed in a pool in front of Wenyuan Pavilion of Yuanmingyuan, the royal library where the Complete Library of the Four Treasuries was stored, in the 40th year of Emperor Qianlong’s reign (1775). These arranged stones are located on the central axis of Wenyuan Pavilion, and their location is very important and critical. Emperor Qianlong and his ministers carried out an appreciation on Lingfeng stone, and engraved their names and poems on the south of the stone, bringing the largest quantity of inscriptions in royal gardens in the Qing Dynasty. Lingfeng stone not only has ornamental and esthetic value as a natural wonderful peak, but also has important cultural value in the fields of history, literature, calligraphy and book collection. Lingfeng stone was blown up around 1924, and its remaining fragments have largely maintained their original positions, making it the most severely damaged among the Mingshi in the Qing Dynasty royal garden. No matter from the importance and representativeness of stone setting, or from the representativeness and complexity of the occurrence of these stones, Lingfeng stone is typical as a virtual restoration case.

Conducting evidence verification of virtual restoration

In the preliminary analysis stage of the virtual restoration of stone settings, the work is mainly carried out from two aspects: collecting and organizing historical materials, and exploring on-site evidence. The comprehensive acquisition of virtual restoration evidence such as historical documents, photos, drawings, similar stones and current conditions provides a basis and guarantee for subsequent work.

Collection and organization of historical materials. Three types of historical materials about Lingfeng stone are sorted out, among which the first type is textual materials about Lingfeng stone, including textual descriptions such as overall height, form description, main contents of engraved poems and inscriptions, and its damage and protection50. The second type is drawings about Lingfeng stone, including 5 drawings about Qing Dynasty style houses, all of which clearly depict the overall environment of Wenyuan Pavilion, the location of Lingfeng stone in the garden, and the form of the base of Lingfeng stone. The third type is 3 historical photos of Lingfeng stone before it was damaged, which were taken from the front, right and left of the south side of Lingfeng stone, i.e., the direction of the stone inscription side. Historical photo c from Old Shadows of Three Mountains and Five Gardens, historical photo d from History of Qing Dynasty Royal Gardens, historical photo e from Qing Lian Duo Duo. The above historical photos clearly show the real form and texture in the west, south and east of Lingfeng stone, especially the position and outline of the 10 inscriptions on the south side, thereby providing real and clear evidence for virtual restoration of Lingfeng stone (Fig. 3).

Fig. 3: Historical Images of Lingfeng stone.
figure 3

a, b Lingfeng Stone in the style and image archives of the National Library of China; and c–e three existing historical photos of Lingfeng Stone50.

Full size image

On-site evidence exploration. Since 2017, a team of people including the author has been conducting plenty of fields surveys and investigations on the remains of Wenyuan Pavilion of Yuanmingyuan, especially the remains of Lingfeng stone. It was found that the fragments of Lingfeng stone have largely maintained their positions when they were destroyed, 4 stone inscriptions were known to be identifiable at the site, and 2 inscription fragments of Lingfeng sone were preserved in the warehouse of Yuanmingyuan due to their small size. On this basis, the team discovered 4 new stone inscriptions at the site, all of which were defective except 1. Combined with historical documents, the makers and location orders of all 10 stone inscriptions on Lingfeng stone, as well as the specific contents of 8 stone inscriptions were determined (Fig. 4), and their contents were numbered and managed50 (Table 1).

Fig. 4
figure 4

Location and preservation status of ten inscriptions.

Full size image
Table 1 Inscription number and details
Full size table

Obtaining virtual restoration data

High-precision virtual restoration data of stone setting are obtained mainly through the existing and missing parts. The existing parts involve surveying and mapping the remaining fragments on the site as well as those that have not been preserved on site. After data processing, a digital model of the existing stone fragments is obtained and compared with the parts that come with historical data, especially image data; but it can be divided into two situations for the missing parts. If there are historical data, especially image data, that can be visually presented, the missing parts can be analyzed by comparing the overall form and the geometric form of the arranged stones with obvious features and details, and then relevant form data can be obtained and virtual restoration can be carried out; but if there are no historical data to support the missing parts, the virtual restoration data can be further inferred and supplemented by comparing and referencing the same or similar origins, materials, scales, textures, and arranged stones at different historical periods.

Obtaining virtual restoration data of the existing stone setting. Based on evidence verification of virtual restoration, 3D digital surveying and mapping was conducted on the Lingfeng stone site and 2 fragments that were not preserved on site. The data about the fragments of Lingfeng stone were identified and extracted from such surveying and mapping data, and special data processing was performed on such data. Specifically, it was divided into the following three steps:

The first step was to collect the point-cloud data of the existing parts of Lingfeng stone. A 3D digital surveying and mapping exercise was carried against Lingfeng stone and the site surroundings as a whole in order to document and map as completely and comprehensively as possible. The surveying and mapping was conducted on cloudy days in winter to ensure color consistency and avoid interference from plants in the course. The surveying and mapping range was about 34 meters long and 12 meters wide. A RIEGL VZ-400i scanner was used to complete high-precision 3D scanning and photo acquisition in a free standing mode. The scanner had an accuracy of 3 mm, a maximum measurement range of 800 m, a vertical scan of 100° and a horizontal scan of 360°; and the point-cloud data was saved as a PTS format file. The scanner was equipped with a Nikon D800 camera and a 20 mm/2.8 F lens, which automatically captured images along with the scanning progress to provide true color point clouds, enabling subsequent point-cloud achievements to contain color information51. We chose to supplement surveying and mapping data with camera shots of the details of the stones at the site that were obscured by the folding of the stones over each other and the details of the stones at the crevices. And Sony Alpha 6000 with an effective pixel of approximately 24.3 million and a focal speed of approximately 0.06 s was used (Fig. 5).

Fig. 5
figure 5

Surveying and mapping of the site and results.

Full size image

After obtaining official permission from the Management Office of Yuanmingyuan, we collected data from 2 fragments of Lingfeng Stone as stored in the warehouse through a high-precision handheld laser scanning equipment CREAFORM’s Metra SACN 70 that had a scanning area of 275*250 mm, a measurement rate of 480,000 times per second and an average resolution of 0.03 mm. In the meantime, NIKON D810 as equipped with a 50 mm/1.8 D lens was used to photograph these 2 fragments and record their materials and colors (Fig. 6).

Fig. 6
figure 6

Surveying and mapping of fragments stored in the warehouse and results.

Full size image

The second step was to process the collected point-cloud data and convert it into a triangular mesh model52. A total of 160 stones in the surveying and mapping range were numbered, and the point-cloud file data corresponding to the numbered stones were extracted separately. Due to the interference from gravel and weeds at the site as being with complex surface texture of the fragments, the density of the obtained point-cloud data was different, so we automatically and manually removed the noise around the point-cloud many times after coloring the point-cloud to ensure data accuracy. The point-cloud file for each stone was imported into Geomagic Wrap for point-cloud encapsulation to create a triangular mesh21.

The third step was to identify the fragments of Lingfeng stone among 160 stones. The identification of the fragments of Lingfeng stone could be divided into two situations, i.e., fragments with inscriptions and fragments without inscriptions. Through sorting out historical data and conducting field surveys, 6 inscribed fragments of Lingfeng stone were extracted, where the two inscriptions on Fragment 48 corresponded to Inscriptions 8 and 10, the two inscriptions on Fragment 49 corresponded to Inscriptions 3 and 4, the contents of the three inscriptions on Fragment 2 partially preserved the words “Lingfeng” as written by Emperor Qianlong on Inscription I, and Inscriptions 2 and 5; the contents of one of the inscriptions on Fragment 1 partially preserved Inscription 9; the contents of Fragment 159 corresponded to Inscription 6; and the contents of Fragment 160 corresponded to Inscription 7. The uninscribed fragments were identified by comparing with the materials, textures and colors of these 4 inscribed ones. The material of Lingfeng stone was significantly different from other rockery stones and base stones. Nineteen uninscribed fragments of the ontic Lingfeng stone were identified and screened out, namely: No.3, No.4, No.5, No.7, No.8, No.24, No.44, No.45, No.50, No.70, No.71, No.72, No.85, No.99, No.100, No.146, No.150, No.151 and No.152. Finally, we identified a total of 25 fragments of Lingfeng stone (Fig. 7).

Fig. 7
figure 7

25 fragments of Lingfeng Stone preserved at the site and warehouse.

Full size image

Obtaining virtual restoration data of geometric forms. Inscriptions with geometric forms are important references for improving the accuracy of virtual restoration, especially for the missing parts. In the virtual restoration of architectural heritage, where there is a lot of image information missing and there are no camera parameters, the photo perspective reversion method is commonly used to obtain the actual physical size of the target building. Such a method has reference value for the virtual restoration of stone setting with geometric inscriptions. By measuring the dimensions of the inscriptions with geometric forms in the remains of stone setting, and taking these data as a reference, the overall actual dimensions of the inscriptions on stone setting can be deduced through the photo perspective reversion method, thereby obtaining virtual restoration data of these stones. For the case where there is no geometric inscription in stone setting, some characteristic points in these stones can be found as reference for positioning and measurement. Perspective reversion is just the reverse process of perspective drawing, which means that given the perspective view of an object, the true width, vanishing point, apparent horizon and point of sight of any horizontal line segment in the perspective view, any plane point of the object can be reversely calculated39. Generally, there are two types of images that can be virtually restored through perspective reversion, wherein the first type is photos of existing objects taken for the purpose of this research, and the second type is old historical photos in the historical materials of the research object53.

During virtual restoration of Lingfeng stone, 3 historical photos had no information such as camera parameters, so we took the perspective reverse method to analyze these old historical photos to obtain the length, width, height and existing inscription dimensions of the ontic Lingfeng stone, and fitted and compared them with the measured existing inscription dimensions, so as to improve the virtual restoration accuracy of the ontic Lingfeng stone as a whole and in detail.

The steps of perspective reversion are as follows:

The first step was to impose geometric constraints on the ontic Lingfeng stone in historical photos. The reversion of the perspective view of an object is based on the corresponding photographic image of the object. Before reversely calculating the perspective view of historical photos, we performed overall geometric abstraction on Lingfeng stone and inscriptions. Combining with historical photos, we selected lines with obvious feature points and clear boundaries as a reference for geometric constraints, and clarified the range of values for the overall length, width and height of Lingfeng stone (Fig. 8).

Fig. 8: Extraction of geometric wireframe of the Ontic Lingfeng stone.
figure 8

a–c Extraction of geometric wireframe of the Ontic Lingfeng stone in historical photos.

Full size image

The second step was to follow the principle of perspective reversion and established the basic elements for perspective reversion. As shown in Fig. 9a, the one-point perspective, which also becomes parallel perspective, is a perspective view in which the main inside of the form is parallel to the picture. Given the relative positions of the projection plane V and the point of sight S, the main surface ABFE of the cuboid ABCD-EFGH is parallel to the picture V, while the perspective view of the cuboid on the projection plane V is abcd-efgh, representing a typical one-point perspective view54. Assuming that the height AE of the cuboid is known and the length of the edge ae in the perspective view can be measured, it can be concluded that AE/ae is a constant. After letting μ = AE/ae, then the dimensions of any edge corresponding to the edge of a real space object can be calculated from the perspective view. The two-point perspective is also called angular perspective, at which time, there is an angle between the picture and a certain elevation of the form, so two vanishing points are formed on the picture, both of which are at the apparent horizon level. As shown in Fig. 9b, two sets of parallel principal horizontal lines are extended to obtain vanishing points F1 and F2, respectively. Such vanishing points constitute the apparent horizon h. A perpendicular line h is drawn from D to intersect the semicircle with F1 and F2 as diameters at P. And arcs are drawn with F1 and F2 as centers and F1P and F2P as radii to intersect the apparent horizon h at M1 and M2, respectively. From measuring points M1 and M2 while passing through ABCDEF, a perspective line is drawn to reflect onto the picture, resulting in the actual shape of the facade55.

Fig. 9: Schematic diagram of the principle of perspective reversion.
figure 9

a Principle of one-point perspective reversion; and b principle of two-point perspective reversion.

Full size image

The inscriptions in the garden stones are artificially carved, and their boundaries present obvious geometric forms. The boundary of the inscriptions on Lingfeng stone shows a rectangle. Considering the current placement of the fragments of Lingfeng stone and the preservation of the inscriptions, the actual measurement of the length and width of these inscriptions is based on the two sets of side lengths AB and AC. Through the collation and analysis of surveying and mapping data, we manually measured the dimensions of some inscriptions on Lingfeng stone with the help of infrared rangefinder and a tape measure, but the point-cloud data failed to be obtained. To ensure data accuracy, the average value was taken after three measurements, resulting in 10 sets of supplementary measurement data (Table 2).

Table 2 Side Lengths of on-site measured inscriptions
Full size table

The third step was to calculate the length, width and height of the ontic Lingfeng stone and the actual dimensions of the inscription through perspective reversion. According to the above principle of perspective reversion drawing, the perspective reversion is carried out on 3 historical photos, respectively. For Historical Photo 1, the one-point perspective was adopted to obtain the true dimensions of Inscriptions 1, 4 and 8 through point-cloud surveying and mapping, as well as the measurement values in CAD. Then the constant μ = 10.21 was calculated, and the dimensions of remaining inscriptions, as well as the length and width of the ontic Lingfeng stone, were determined with the constant μ. For Historical Photo 2 and Historical Photo 3, the basic elements of two-point perspective were established to perform perspective reversion on these two photos, so as to obtain dimensions of the inscriptions and the length, width and height of the ontic Lingfeng stone (Fig. 10).

Fig. 10: Perspective reversion based on historical photograph.
figure 10

a Perspective reversion based on historical photograph 2; and b perspective reversion based on historical photograph 3.

Full size image

The fourth step was to synthesize the perspective reversion data of three historical photos. The dimensions of the remaining 10 sets of inscriptions of Lingfeng stone, as well as the overall length, width and height of Lingfeng stone were obtained. The sample standard deviation was used to verify the data aggregation degree of inscription dimensions, and overall length, width and height in each historical photo to verify the stability of each group of data, and then the average value of the sample was calculated. The sample mean formula was defined as Eq. (1):

$$\bar{{\rm{{\rm X}}}}=\frac{{\chi }_{1}+{\chi }_{2}+\ldots {+\chi }_{n}}{n}$$
(1)

and the sample standard deviation formula was defined as Eq. (2):

$$S=\sqrt{\frac{\sum {\left({{\rm{{\rm X}}}}_{i}-\bar{{\rm{{\rm X}}}}\right)}^{2}}{n-1}}$$
(2)

where S represented the sample standard deviation, and \(\mathop{{\rm{{\rm X}}}}\limits^{\bar{} }\) represented the sample mean. Through analysis, the data dispersion degree of the dimensions of the remaining ten sets of inscriptions (Table 3) and the length, width and height of the ontic Lingfeng stone (Table 4) as obtained through the perspective reversion method are all within three times the standard deviation, and the average values of the obtained data are stable and accurate.

Table 3 Side length of the inscription under perspective reversion based on historical photos
Full size table
Table 4 Length, width and height of Lingfeng Stone under perspective reversion based on historical photos
Full size table

Obtaining virtual restoration data of irregular forms. Based on the characteristics of diverse shapes and numerous holes of stone setting, the grid mapping method was used to determine the shapes and holes of stone setting in the image data so as to further obtain accurate virtual restoration data, thereby enhancing the acquisition of detailed data of the missing part during virtual restoration of stone setting.

In terms of Lingfeng stone, based on three historical photos, the grid mapping method was used to accurately locate the outer contour of the ontic Lingfeng stone, as well as the forms and positions of the inscriptions and holes. Since the inscription 10 is preserved intact and its precise size data can be obtained through 3D digitizing technology, we take point C in the lower left corner as the reference point for the historical photo a, a 50 cm*50 cm one-point perspective large grid was constructed, and then the minimum accuracy of subdivided grid was controlled at 10 cm*10 cm, and then according to the inscription 10, point C as the datum combined with the perspective relationship to locate the historical photographs b and c. of the square grid. By locating the three historical photographs on a grid of squares, we obtained data on the length and width of the ten inscriptions, the specific morphology of the right side face of fragments 1 and 2; and the specific morphology of the left side face of fragments 48, 5, 146, 148, 151 and 152 (Fig. 11).

Fig. 11
figure 11

Outer contour of the Ontic Lingfeng Stone, and forms of inscriptions and holes in historical photos as located by grid mapping method.

Full size image

Reasonably speculating to supplement virtual restoration data. After making full use of the existing historical data to obtain the virtual restoration data of stone setting, there are still some missing parts that have not been determined. In this case, it is necessary to supplement the missing data through reasonable speculations.

The specific form of the north side of Lingfeng stone cannot be obtained from historical documents, drawings and photos. Subject to the actual situations, three methods are used to reasonably speculate the missing parts and supplement the data. Therefore, the first case is that the fragments have a clear texture and are well preserved, with relatively few missing parts. In this case, the missing parts can be supplemented by referring to the texture on the existing fragments, such as Fragments 1, 2 and 49. The second case is to make corresponding supplementation and texture adjustment to the fractures of the fragments, such as Fragments 4 and 5. The third case is to draw lessons from the arranges stones with the same place of origin, and similar texture, scale and standing time. Wenfeng Stone, the sister of Lingfeng stone, is currently housed in the Palace Museum in Beijing. The missing parts of Lingfeng stone that do not conform to the first two cases can be supplemented by referring to Wenfeng Stone.

Results

Based on the data obtained for the virtual restoration of the stone setting, both the existing and missing parts are used to improve and present the final virtual restoration model.

Virtual restoration modeling of the existing parts

The specific location of the fragments was comprehensively determined based on specific characteristics, historical photos and perspective reversion results. The south side of Lingfeng stone was used as a reference surface, and the data of the 6 fragments with inscriptions were imported into the Zbrush software in turn for virtual model splicing. In the meantime, 3D printing was carried out on the fragment models with more inscriptions, such as Fragments 48, 49, 1 and 2, to assist virtual restoration in the form of solid models.

Nineteen fragments of the ontic Lingfeng stone without inscriptions were spliced. Through historical photos, texture, fracture form and other evidence, fragments were imported into the Zbrush software for sequential positioning (Fig. 12). The two images a and b show the specific morphology of the north and south sides of the Lingfeng stone in the two phases of virtual restoration respectively.

Fig. 12: Virtual restoration models of Lingfeng stone.
figure 12

a Virtual restoration models of 6 fragments with inscriptions; b virtual restoration models for 6 fragments with inscriptions and 19 fragments without inscriptions.

Full size image

Complete virtual restoration modeling of the missing parts

Firstly, correct and supplement the dimensions of rectangular inscriptions on Lingfeng stone. The dimension data of rectangular inscriptions as obtained through field measurement and historical photo perspective reversion method were used to supplement the missing inscriptions (Table 5).

Table 5 Virtual restoration modeling data of inscriptions
Full size table

Secondly, supplement the overall form of Lingfeng stone and adjust the details of virtual restoration model. (1) According to the length, width and height of Lingfeng stone as a whole obtained through perspective reversion method, the existing model was supplemented, and the overall dimensions of the model were adjusted (Fig. 13).

Fig. 13
figure 13

Virtual restoration model after supplementing the overall form.

Full size image

(2) The overall shape and outer contour of the model, and the dimensions and forms of the holes are accurately filled and dug based on the data of the outer contour of the ontic Lingfeng stone, the forms and positions of the inscriptions and holes were obtained through the grid mapping method. We control the size of the overall form of the Lingfeng Stone and the location of the surviving parts based on the grid coordinate system established in the historical photographs. On the basis of this, we compare the historical photographs and add the size and shape of the missing parts to the corresponding grid coordinates, so as to form a more accurate virtual restoration. (3) And with reference to the virtual restoration data obtained by reasonable speculation, the texture of the fragments on the north side of Lingfeng stone, the shape of the fracture, and the appearance of Wenfeng Stone were supplemented and finely restored (Fig. 14).

Fig. 14: High-precision virtual restoration model of Lingfeng stone.
figure 14

a Completion of missing parts of the debris using the grid mapping method and b accurate virtual restoration of outer contour of the model, and forms of inscriptions and holes through grid mapping method.

Full size image

Presentation of virtual restoration model of Lingfeng Stone

It is very important to adjust the texture of the whole virtual restoration model of Lingfeng stone to improve fidelity of the model, which is a very important link in realizing virtual restoration. We use Alpha tools in combination with UV layers to create the mapping of inscriptions and texture details at the inscriptions; but use the ‘material’ function to map the texture onto the model in other details (Fig. 15).

Fig. 15
figure 15

Virtual restoration model.

Full size image

Discussion

With the help of historical documents, photos, drawings and digital technology, a virtual restoration model of the ontic Lingfeng stone was established. In order to make the model not only a perfect display, but also a scientific and transparent research achievement, it is necessary to discuss the accuracy of virtual restoration achievements56. Methods on how to improve transparency and scientificity of the virtual resilience process have been explored, but there are already recognized international documents in this direction that have formulated relevant guidelines. For example, the London Charter puts forward the concept of “paradata”, which is defined as information about the process of human understanding and interpretation of data objects, including how evidence can be used to interpret artefacts stored centrally in structured data, or comments on premise methods in publications. It is closely related to “contextual metadata”, but its focus tends to convey the process by which one or more artefacts are processed or interpreted. Article 7 of the Seville Principles also states that the combination of metadata and paradata is crucial for ensuring scientific transparency in any virtual archeological project. Data and metadata should be clear, concise and easily accessible. In addition, it should provide as much information as possible. The scientific community should contribute to the international standardization of metadata and paradata57.

Since the accuracy of the virtual recovery data obtained in this research case varies, we validate the virtual recovery results by two methods, namely, quantitative analysis using the error analysis method and qualitative analysis using the reliability matrix modeling method, respectively: i. The results of the geometric forms in the virtual recovery model are quantitatively analyzed using the error analysis method, and we can reflect the deviations between the data points and the mean value by calculating the standard deviations deviation between the data points and the mean value, the mean square error can measure the accuracy of the prediction model, and the confidence interval can estimate the confidence range of the experimental data. These indicators can visually reflect the actual size and relative importance of the error, and provide reliable information support for virtual recovery, scientific research and decision-making. Secondly, by establishing a ‘reliability matrix’ to qualitatively analyze the results of the remaining part of the virtual restoration, the method can not only share the results of the reconstruction process, but also intuitively share the reliability level of the reconstruction of the project parts and elements, as well as the relevant possibilities of enriching the model due to the new discoveries and interpretations. This enables the virtual recovery results to be considered as an architectural layer of analysis rather than as individual architectural elements.

The relative error results were used to evaluate the virtual restoration accuracy of rectangular inscriptions on Lingfeng stone. Relative error refers to the ratio of absolute error to true value, and its function is defined as Eq. (3):

$$\delta =\frac{\Delta }{L}\times 100 \%$$
(3)

where \(\Delta =|X-L|\) represents absolute error, X represents measured value, and L represents true value. The dimensions of the inscriptions on the ontic Lingfeng stone in this case are virtual restoration measured values. After correcting historical photos with known inscriptions, the dimensions of these inscriptions as obtained are the true values58. By taking the dimensions of Inscriptions IX and X as manually measured and well preserved as the standard values, the true dimension values of 10 inscriptions were obtained by two photo corrections in Auto CAD, respectively (Table 6).

Table 6 True values of inscriptions
Full size table

According to the above relative error function formula, the error calculation of virtual restoration achievements of inscriptions is carried out, and the results are as follows (Table 7).

Table 7 Error calculation of virtual restoration model for inscriptions
Full size table

Through verification of the relative errors of 20 groups of data, AB and AC, in 10 inscriptions, 19 groups had a relative error less than ±5%, while only AB in Inscription 7 had a relatively large relative error between ±5% and ±10%. By comparing the AB values of Inscription 7 as obtained through the perspective reversion method against 3 historical photos, it was found that the data from the perspective reversion of Historical Photo 2 was significantly larger than those of the other two photos, resulting in the virtual restoration measured value being relatively large after averaging. The reason for large relative error might be that the edge of Lingfeng stone in Historical Photo 2 was very blurred and deformed, and the boundary of Inscription 7 was relatively blurred, thereby affecting the accuracy in extracting the geometric shape of Inscription VII during perspective reversion. We used jpgHD software to perform lossless high-definition repair against Historical Photo 2 with the help of artificial intelligence in a hope to correct this set of data, but the final processing result still had a large error, so this set of obviously wrong data was eventually removed to improve data accuracy (Table 8).

Table 8 Correction of virtual restoration model data for inscription VII
Full size table

We obtained the true value of the dimensions of the ten inscriptions through computer correction, which has high reliability, so we believe that it is relatively scientific to take this group of values as the standard of error measurement, and also combined with the results of the error distribution, we finally determined that the threshold of acceptable error is within ±5%.

After revision, the errors of the side length data of 20 groups of inscriptions are all within ±5%, which further improves the accuracy of virtual restoration model data. And the correction results are as follows (Table 9).

Table 9 Error calculation of revised virtual restoration model of inscriptions
Full size table

The error calculation results show that the historical photo perspective reversion method can be applied to obtain virtual restoration data of the stone setting in geometric form, and also can help obtain the accurate virtual restoration data of such stones.

In the process of virtual restoration in this paper, we firstly considered relevant literature, drawings, archives and other historical evidence, and secondly designed different types of virtual restoration data sources. It is essential to link the transparency and scientificity of the virtual restoration process with the sustainability of the virtual restoration achievements under relevant guidelines. In this regard, we establish an evaluation matrix for the virtual restoration model of Lingfeng stone to represent the reliability levels of different types of sources involved in virtual restoration. Such a matrix can assign a reliability level from 0 to 5, as also known as the Level of Reliability (LOR). The five elements for assessment include: The evaluation score for each element is 0 (missing) or 1 (existing), and the evaluation value for incomplete elements is 0.5, based on whether they are present, whether there are texts, whether there are pictures, whether the point-cloud data are complete, and whether the restoration position is determined59. This method allows for an intuitive evaluation of where the case lacks information in the virtual restoration process, and where it is more useful to focus on new information and applications that can increase the accuracy of the final results of virtual restoration (Table 10).

Table 10 Reliability assessment of virtual restoration achievements
Full size table

The prospect of applying virtual restoration achievements of Lingfeng stone is guided by two visualization mode schemes: one is to establish a database about the fragments of Lingfeng stone for digital management, and the other is to visually display the virtual restoration achievements, and then create a historical evidence scale and make it a universal, understandable and applicable standard for experts and the public, so that the public can better understand archeological relics and specific scientific work as carried out, thereby correctly explaining and spreading the value of virtual restoration and heritage virtual restoration, and placing the determined model in a virtual environment for display and experience.

In the first scheme, we expect to create an interactive public database for each fragment of Lingfeng stone, make each of them have their own high-resolution color photos, shareable 3D models and detailed semantic information. This database easily combines traditional low-tech documentation methods with digitized smart devices to ensure systematic collection of complete model digital information51. Moreover, a special numbering system for fragments in the database is essential to guide the numbering, naming and storage of each fragment; and the database also allows simplified retrieval and cross-reference to ensure no missing details60. In the second scheme, we expect to develop a historical evidence scale suitable for the virtual restoration of stone setting based on the historical evidence scale from Byzantine’s 1200 cases, and evaluate the virtual restoration of the ontic Lingfeng stone. The accuracy of the work performed is shown by the applied color scale in the historical evidence scale, with warm colors representing more historical archeological evidence while cool colors representing less evidence. The level of evidence and the choice of color are key to what the public can easily and clearly understand61. As for the presentation of 3D models, we expect to find a tool and platform that can accept various file formats, support cross-platform deployment and visualization, enable online sharing of our virtual restoration achievements, and support the public to appreciate the original appearance of Lingfeng stone in VR/AR and even MR, thereby exploring the cultural significance of this heritage62 (Fig. 16).

Fig. 16
figure 16

Activation and utilization of virtual restoration achievements.

Full size image

In this research, it proposed a complete set of high-precision virtual restoration method for stone setting, and applied it in practice with reference to the highly representative Lingfeng stone. Firstly, the evidence of virtual restoration of stone setting was verified, including the collection of historical materials such as documentary records, drawings, archives and photos; and the restoration evidence was explored through on-site surveys and investigations, and separate consultations. Secondly, 3D digital surveying and mapping was applied to obtain relevant data for the existing parts of stone setting, and subsequent data processing was carried out with the help of relevant software. For the missing parts, they were mainly divided into geometric and irregular forms according to the characteristics of these stone setting with inscriptions. The geometric form of the virtual restoration data was obtained through the perspective reversion method against historical photos to calculate the precise dimensions of the geometric form, while the irregular form of the virtual restoration data was obtained mainly through the grid mapping method to accurately locate the texture and holes of the stone setting against historical photos to improve the restoration accuracy of the missing parts; and additionally, the existing models and similar samples were used to reasonably speculate and supplement the virtual restoration data. Thirdly, an accurate 3D model was established on the basis of the acquired virtual restoration data, and the achievements were subjected to error analysis and grade evaluation. Finally, the virtual restoration achievements were activated, utilized and displayed.

In the virtual restoration of Lingfeng stone, based on on-site research and relevant virtual restoration evidence, we refer to the above proposed high-precision virtual restoration workflow to carry out virtual restoration of its existing parts and missing parts of the different situations. This case basically includes most of the cases of virtual restoration of stone setting, and the accuracy of the virtual restoration results fully proves the feasibility of the virtual restoration work method. It can be seen that our proposed working method has the potential to carry out in-depth research on stone setting, and with the gradual attention paid to the virtual restoration of garden heritage, the efficiency and accuracy of virtual restoration of stone setting can be greatly improved by using this working method.

To sum up, the working method proposed in this paper for high-precision virtual restoration of garden stones can not only restore the historical appearance of stone setting in a more comprehensive way, but also achieve a high degree of accuracy and precision of the virtual restoration model, and more intuitively present the virtual restoration results of the level of evidence and assessment of the situation. This working method is of great value to the virtual restoration of such tangible cultural heritage as stone setting with irregular shapes, inscriptions and other characteristics, and provides a practical new strategy to fill the lack of virtual restoration of stone setting at present. Of course, the shortcomings of this research are that the virtual restoration of Lingfeng stone is restricted to the ontology, but the virtual restoration of the base of Lingfeng stone needs further discussion. As restricted by the protection requirements for the Lingfeng stone site, the parts of the existing fragments that overlap with the ground and other fragments cannot support 3D collection and mapping because of too small gaps, which brings difficulties and challenges to virtual restoration, and then affects the virtual accuracy. The grade evaluation of virtual restoration achievements needs to be combined with the specific achievements, and a special grading and display of evidence should be carried out. Due to cost constraints, the virtual restoration accuracy can be further improved if all existing parts of Lingfeng stone are digitally collected with sub-millimeter equipment such as Metra SACN 70. It is expected that the follow-up work can supplement relevant contents, further optimize the workflow of virtual restoration of stone setting, explore a more accurate and efficient perspective reversion method under the condition of sufficient historical data, especially historical camera information, and ultimately achieve a greater breakthrough in virtual restoration of stone setting.