Introduction

3D reconstruction is considered a key procedure in architectural heritage research, conservation and interpretation. The accuracy of 3D reconstruction relies on synthesising multi-source evidence1,2,3, from quantitative data like photogrammetry4,5, remote sensing6, to qualitative documentation like archaeological reports7, photographs8, archival documentation9, oral histories10, etc. Among these, the iconographic evidence from historical images, such as architectural drawings11,12,13 and ancient paintings14, is invaluable, especially when physical remains are scarce3,15,16.

To enhance the efficiency of 3D modelling, parametric tools have proven effective from modelling architectural forms17,18, to designing artefacts19 and patterns20,21,22,23. By encoding architectural logic, such as shape grammars, into mathematical formulas or algorithms, these tools facilitate rapid generative modelling24,25,26,27. Platforms like Rhino-Grasshopper offer interfaces for graphical programming, and can be easily integrated with systems like HBIM (Heritage Building Information Modelling) and HGIS (Historical Geographic Information System), performing optimisation functions28,29,30,31.

Parametric 3D reconstruction based on iconographic evidence has been explored in many studies recently15,32. Despite some advancements, integrating iconographic data sources systematically into heritage conservation requires expertise in iconography, architecture theory and digital modelling. Such an interdisciplinary task faces challenges in knowledge communication, collaboration, and time constraints19,33. Meanwhile, inaccuracies in each step can affect the authenticity of modelling results. For example, images from incorrect periods or imitation copies might be misused when lacking the capacity to evaluate iconographic evidence34. As suggested in the London Charter for the Computer-Based Visualisation of Cultural Heritage, computer visualisation should systematically identify and evaluate sources, and its dissemination should be accessible to a maximum range of users for cultural heritage learning and protection purposes35.

This study focuses on the application of parametric 3D reconstruction on iconographic evidence, taking the Song Dynasty architectural massing typologies as a case study. In Chinese traditional architectural conservation, the surviving architecture and archaeological remains, often missing information of previous upper structures, could be insufficient for a comprehensive 3D reconstruction36. Historical images complement these data sources by offering contemporaneous visual documentation of architectural forms37,38,39, and can serve as references for 3D reconstruction and interpretation if properly analysed15. As a period with typical architectural form and standardisation achievements, exemplified by the publication of Yingzao Fashi (“Building standards”, “营造法式” in Chinese), the Song Dynasty offers insights into ancient architectural design principles40,41. One of these impactful design principles that was handed down by the following dynasties was the “architectural massing typologies”39,42,43, referring to the correlative relationships between the main volume (正屋) and its affiliation volumes44 (Fig. 1). Different typologies can be found in a small number of existing buildings and archaeological sites, while hardly mentioned in Yingzao Fashi (e.g. “Xie Wu 挟屋”, “Lang 廊”, “Guitou Wu 龟头屋”) and other Song Dynasty texts (e.g. “Bo Shui 泊水”). “Xie Wu (挟屋)”, “Bo Shui (泊水)” and “Lang (廊)” are three typical massing typologies.

Fig. 1: Illustration of Song architectural massing typologies.
figure 1

This figure shows the massing typology of Xie Wu, growing from the main building volume, and combing with Lang, adapted from ref. 77. a Main building volume; b Main building volume + Xie Wu; c Main building volume + Xie Wu + Lang

Full size image

Recent iconographic studies of Chinese traditional architecture were mainly historical studies45,46,47, with minimal attention to bridging iconographic evidence with parametric heritage documentation, 3D reconstruction, and interpretation. Some other research for 3D reconstructing Chinese traditional architecture mainly relies on the Yingzao Fashi principles, which don’t always explain the case-specific facts. Existing studies have also explored parametric modelling for Chinese traditional architecture, such as relying on textual frameworks from Yingzao Fashi48,49,50,51, and Gongcheng Zuofa Zeli (工程做法则例)51,52. Recent works also investigated the composition and design principles of Qing courtyard buildings (四合院)53, Hutongs (胡同)54, roof tile components55, and even modern architecture56. For extracting, documenting, and parametric modelling the design principles of Song Dynasty architectural massing typologies revealed from historical images, there is still a gap. Therefore, this study develops a workflow and algorithms that can support heritage professionals and even non-experts to generate efficient and authentic 3D reconstruction models under different research, learning, or design scenarios.

Methods

The selection of multi-source Song Dynasty images based on chronological and appraisal assessment

Multi-source Song Dynasty images open data that illustrate architectural massing typologies, ranging from paintings, images in local records and inscriptions on stone, to images in architectural decorations are exhausted (Data collection end-date: 18/04/2023) and collected in this study, which forms the main part of the evidence database. Non-image information shown in Yingzao Fashi, other textual documentation, and existing architectural cases and sites are also gathered as dual proof. There are 363 paintings, 38 images from local records and inscriptions from stone, 9 cases from other art forms, as well as textual and existing cases are collected as information resources of Song Dynasty architectural massing typologies (Fig. 2).

Fig. 2: The retrieving and screening of historic information of Song Dynasty architectural massing typologies.
figure 2

This figure illustrate the categories of data sources, the dual proof approach, the selection of three typology cases, and the screening result.

Full size image

The selection of the retrieved images undergoes chronological and painting appraisal assessment, based on theories from scholars as Xu34, Fu38, publications from the Chinese Ancient Calligraphy and Painting Appraisal Group57, and Cahill58. In Table 1 shows the assessment criteria and coding standard.

  • The chronological and appraisal accuracies of the images are categorised into four grades (Very High, High, Medium, and Low).

  • The levels of representation of the images are categorised into four grades (Very High, High, Medium, and Low). The chronological and appraisal assessment ensures that the information from images is authentic and reliable, avoiding the misuse of imitation/fake image versions. Only cases with at least medium accuracy and medium level of representation are used as evidence.

  • The architectural massing typologies and typical characteristics of the structure and form that appeared in the images are also coded.

Table 1 The chronological and appraisal assessment process of historical images.
Full size table

This study takes three important typologies, “Xie Wu”, “Bo Shui” and “Lang” as main modelling cases. The “Xie Wu” refers to smaller parallel volumes connected to the left/right or both sides of the main building volume, following down-grade principles. The “Bo Shui” means a parallel affiliation volume connected to the front or back side of the main building volume. The “Lang” is a type of connecting volume used as corridors or boundaries in the design of courtyards. They represent three main logics in Song Dynasty architecture design, respectively, compositing building volumes from the axes of width and depth directions, and as linear connections. There are 119 cases collected that consist of typologies of “Xie Wu”, 41 cases of “Bo Shui” and 153 cases of “Lang”. After the chronological and appraisal assessment, the remaining evidence cases for the 3 typologies are: 104 cases for “Xie Wu”, 40 cases of “Bo Shui” and 146 cases for “Lang” (Fig. 2).

Massing principles extraction and translation using Rhino-Grasshopper modelling

The Rhino-based Grasshopper plug-in is a graphical programming software that has been widely used in architectural design, urban planning, structural calculation, and heritage documentation, which can automatically generate digital models based on defined parameters and algorithms30,53,59,60,61.

This study extracts and translates the massing principles shown in iconographic evidence. Focusing solely on the scale of building groups and their relationships, rather than on building details, this study only includes parameters that reflect these key characteristics. As some of the characteristics of affiliation volumes follow correlative principles with the main volume, the algorithms in this study are divided into two parts: the basic algorithms and the correlative algorithms.

  • Basic algorithms include the characteristics of a single building volume. These principles have been shown in fruitful studies by Liang40,62, Pan and He41, and Chen63. Previous studies have also shown the potential of translating principles from Yingzao Fashi into algorithms48,49,50,51.

  • Correlative algorithms include the characteristics in the affiliation volumes that might be the same as, be following a downgrade principle, or be independent of the main volumes. The related parameters and principles of “Xie Wu”, “Bo Shui” and “Lang” have not yet been fully studied, so the extraction in this study also contributes to the history and theories of Chinese traditional architecture.

The whole algorithms follow the structure of “input parameters”, “computation - control points and control lines”, and “output - generated 3D building volumes”49,53. Parameter groups range from ground plan & scale, roof shape, to components. We define one parameter for the massing typology selection, and select 24 parameters of basic algorithms and 24 parameters of correlative algorithms to describe the design principles of three massing typologies (Table 2, Fig. 3). Within the correlative algorithms, there are 15 parameters for “Xie Wu”, 12 correlative parameters for “Bo Shui”, and 11 correlative parameters for “Lang” that need independent input. By embedding correlative principles in the algorithm, there are respectively 9, 10, and 6 parameters in the algorithms of “Xie Wu”, “Bo Shui”, and “Lang” that are controlled by the input from the main volume parameters. This means the percentages of reducing independent inputs are respectively 37.5%, 45.5%, and 35.3% in modelling each massing typology, which is expected to improve the modelling efficiency. Meanwhile, the algorithms embed all combination possibilities of different parameters, and allow creating scenarios of non-discovered typologies that follow the principles. In Fig. 3, a more detailed modelling flowchart illustrates the algorithm structure and the mutual influences between parameters.

Table 2 The structure and organisation of input parameters in the algorithms
Full size table
Fig. 3: The Grasshopper algorithm structure.
figure 3

The flow chart illustrates the parameters, options or input range of each parameter, and the relationships and mutual influence between parameters.

Full size image

Main building volume modelling

Referring to previous research on Song Dynasty architecture, we complete the modelling of the main building volume (Fig. 3, grey blocks).

  • A “central control point of the main volume” is linked to the Rhino model.

  • For the ground plan parameters, it can be chosen from “Diantang (殿堂)” to “Tingtang (厅堂)” typologies. Then, more detailed values of the column grid can be selected.

  • For the scale parameters, the main measurement unit, “Cai (材)”, is defined. Then the “width (开间)”, “depth units (进深)” and “eaves pillars height (檐柱高)” of the main building volume, as well as the partitioning parameters such as “width between pillars (柱距)”, “depth per unit (架深)” are defined. These parameters all follow the measurement unit of Cai.

  • For the roof shape parameters, there are three inputs: “roof type” (including 3 main types, “hip-gable (歇山)”, “two-sloped-gable (悬山)” or “hip roof (庑殿)”), “number of floors”, and the “eave layers” (including 2 types, “single eave (单檐)” or “double eaves (重檐)”).

  • For the components parameters, the “Juzhe (举折)” can be influenced by the “building type”. The other 5 parameters, “pillar diameter (柱径)”, “rafter diameter (椽径)”, “purlin diameter (槫径)”, “purlin extension (出际)”, and “bracket sets (斗栱)” follow the measurement unit of Cai.

  • For all parameters mentioned in Yingzao Fashi and previous research, we provide recommended values, but there can be flexible inputs on a case-by-case basis.

The algorithms of “computation - control points and control lines” and “output - generated 3D building volumes” are internal codes that can run automatically. They include the control points/lines of the building “Ceyang (侧样)” based on the input of ground plan & scale, roof shape, and components parameters, along with the bracket sets control point/lines based on the input of bracket set parameters. Finally, the main building volume includes the “walls & openings”, “foundation platform (台基)”, “pillars”, “rafters”, “purlins”, “roof & roof ridges”, and “bracket sets”.

Following the input of the main building volume, architectural massing typologies can be chosen. In this study, we preliminarily provide 5 options, none (single volume), “Xie Wu”, “Bo Shui”, “Xie Wu + Bo Shui”, and an on/off button of “Lang” (Fig. 3, purple block).

“Xie Wu” parametric modelling

According to the previous study of “Xie Wu” correlative principles44 and related parameters defined in this study (Table 2), the algorithms are categorised into three groups (Fig. 3, yellow blocks, Table 3).

  1. 1.

    Parameters controlled by the main volume. These characteristics are directly consistent with the main volume and are not allowed to change on their own initiative. For example, “Xie Wu”’s “building type” of “Diantang” or “Tingtang” follows the input parameter of the main building volume44. According to Yingzao Fashi, the “Cai” of “Xie Wu” is one grade lower than the “Cai” of the main volume, so this parameter is also controlled by the main volume. Similarly, the roof type of the “Xie Wu” usually follows that of the main volume.

  2. 2.

    Independent input parameters influenced by the main volume. The number of “depth units” and the “depth per unit” of the “Xie Wu” are under certain constrains from the main volume. For example, the number of “depth units” should be less than or equal to the main volume’s “depth units”. The value of “depth per unit” also depends on the mutual “structure relationship” and “volume relationship” between the main and affiliation volumes. When there is a shared structure, the values are equal. In many cases, the “eaves pillars height” stays equal between the main and affiliation volumes. The layer of “bracket sets” in the “Xie Wu” usually lower the grade, too, so this value should be less than that in the main volume.

  3. 3.

    Independent input free parameters. For some parameters of the “Xie Wu”, like “structure relationship”, “volume relationship”, “volume side”, and “width between pillars”, the values are not subjected to the main volume, so they can be freely selected by users according to the specific objectives.

Table 3 The relationship between selected parameters, evidence sources, and their integration into the “Xie Wu” algorithm
Full size table

“Bo Shui” parametric modelling

Similar to “Xie Wu”, the correlative principles and related parameters of “Bo Shui” (Table 2) can also be categorised (Fig. 3, green blocks, Table 4).

  1. 1.

    Parameters controlled by the main volume. Due to the affiliation volume in the “Bo Shui” massing typology being adjacent to the front or back side of the main building volume, some parameters of “Bo Shui”, such as the “building type”, “Cai”, “width between pillars, and “eaves pillars height” are usually the same as those from the main volume.

  2. 2.

    Independent input parameters influenced by the main volume. Some parameters like the “number of floors” can be influenced by the main volume, often having the same values.

  3. 3.

    Independent input free parameters. Parameters like “structure relationship”, “volume side”, “depth units”, and “roof types” are independent input parameters.

Table 4 The relationship between selected parameters, evidence sources, and their integration into the “Bo Shui” algorithm
Full size table

“Lang” parametric modelling

The modelling of Lang is based on a central control line, which allows the construction of linear volumes. Compared to “Xie Wu” and “Bo Shui”, the correlative principles of “Lang” with the main building volume are less obvious (Table 2). According to Yingzao Fashi, the “Cai” of “Lang” is two grades lower than the “Cai” of the main volume, which is the main characteristic controlled by the main volume (Fig. 3, blue blocks, Table 5). We also provide several options for modelling the partition walls shown in the cases, including the “wall type” and the “wall location”.

Table 5 The relationship between selected parameters, evidence sources, and their integration into the “Lang” algorithm
Full size table

Validation and application of the massing principles

After building the algorithms, a validation and application process is conducted. Using the Grasshopper algorithms, exercises on 3D reconstruction of historical images are conducted to test the capability of the algorithms. Four students from different backgrounds are asked to use the algorithms to 3D reconstruct the holistic architectural scenes from seven paintings, including building volumes and the surrounding environments. The final 3D reconstruction results they provided should be approved by at least one expert. They are also asked to count the number of minutes spent (including time of parametric modelling and time of manual adjustment), and reflect on the biggest challenge of 3D reconstruction.

For further application, the Biomorpher plugin in Grasshopper is used as an interactive optimisation interface for Song Dynasty architectural massing typologies design. This allows the generation of several alternative optimal solutions instead of inputting manually, and is able to evolve continuously to assist generative design.

Results

Parametric modelling outputs

The parametric modelling outputs show diverse combinations of different parameters. Part of the parameter combinations with 73 outcomes are illustrated (Fig. 4). It should be noted that only some obvious parameters are included in this table, such as the “roof type”, “eave layers”, “volume relationship”, “structure relationship”, and “control line” attributes. Parameters such as ground plan, and component dimensions are not shown in this table to reduce overwhelming information. If bringing the value variations of the non-listed parameters into the combinations, the number of outcomes that this algorithm provides can result in a dramatic increase.

Fig. 4: Examples of 3D reconstruction variations with multiple massing typologies.
figure 4

This figure shows respectively some key examples of parametric “Xie Wu”, “Bo Shui”, “Lang”, and combinations modelling.

Full size image

Validation and effectiveness assessment

The parametric algorithms are tested through a 3D reconstruction process regarding Song Dynasty paintings by four student participants (Fig. 5), with time and challenges recorded. The selected paintings are 1) the western wall murals of YanshanTemple in Fanzhi, Shanxi (山西繁峙岩山寺西壁壁画), 2) the painting Incense Burning Ceremony (焚香祝圣图), 3) the painting Delicate Residence at Songdeng (松磴精庐图), 4) the painting Autumn Scenery of Rivers and Mountains (江山秋色图), 5) the painting Palace banquet (乞巧图), 6) the painting Pictures of Auspicious Omens (中兴瑞应图), and 7) the painting Eighteen Songs of a Nomad Flut (胡笳十八拍) (Table 6).

Fig. 5: The workflow of 3D reconstruction of Song Dynasty painting engaging student participants.
figure 5

The workflow includes identification, parametric modelling, and manual adjustment.

Full size image
Table 6 3D reconstructing Song Dynasty paintings.
Full size table

3D reconstruction modelling efficiency highly depends on the content complexity of the referenced painting, the required accuracy of output, the participant’s academic knowledge and modelling skills. This makes it difficult to quantify the degree of optimisation of this approach compared to traditional 3D reconstruction. However, it can be confirmed that the efficiency has been significantly improved, based on the authors’ previous experience in Chinese Song Dynasty historical painting 3D reconstruction. The 3D reconstruction process usually consists of an identification process, including collecting references and background information, identifying typologies, speculating possible values of parameters, modelling, and adjustment and iteration. Usually, a painting including a series of building volumes takes days to weeks to reach essentially accurate outcomes. A worse situation is mentioned by the traditional modelling workflows of other types of architecture based on iconographic evidence64.

Through the application of this algorithm, the modelling of each building volume can be completed around 15 min, and that for modelling all elements in one painting can be reduced to around 2.5 h. Particularly in the modelling of some regular and symmetrical courtyard spaces, the time can be as short as 15 min for completing both parametric modelling and manual adjustment (Table 7).

Table 7 Time spent by each 3D reconstruction case
Full size table

It is recognised by the participants that parametric modelling didn’t consume much time for only choosing from the pre-defined options for each input parameter, but more time was spent on adjusting the relative positions of different building volumes and comparing different output versions, to gradually increase the accuracy with the referenced 2D painting. Besides, the most time-consuming process is still the manual modelling of building volumes and surrounding environments that don’t have pre-defined algorithms. For example, the modelling of undefined building volumes intersection features is extremely difficult, as the current algorithm is not yet sufficient to address all variations of intersection details (Table 8).

Table 8 Challenges in the 3D reconstruction process mentioned by each participant
Full size table

Scenarios comparison using Biomorpher

As an extended application of the algorithms, the Biomorpher plugin in Grasshopper helps generate combinations of parameters, which can be used for design comparisons. Biomorpher can compute a set of 12 scenarios each time (Fig. 6). According to the actual needs of the design, evolution directions can be set, such as 1) the need to achieve the maximum use of space (largest volumes), 2) the need to achieve the most complex form (largest surface area), 3) the main and affiliation volumes have the same value of pillar heights, etc. As the evolutionary direction is determined, it will evolve to the best design as the number of generations increases. The use of Biomorpher extends the function of the algorithms to fulfil both research and design purposes.

Fig. 6: Interactive design through Biomorpher.
figure 6

This figure shows the multi-generation evolution using Biomorpher to compare random generated scenarios and select based on preference to approach the ideal solution. a Zero generation; b First generation of evolution - multiple scenarios comparison; c After multi-generation evolution - close to the ideal solution.

Full size image

Discussion

This paper presents a novel and preliminary study of developing parametric modelling workflows using Rhino-Grasshopper to document, 3D reconstruct and interpret Song architectural massing typologies, integrating iconographic evidence. First, multi-source historical images are screened based on chronological and appraisal accuracies, and levels of representation. The extracted and translated principles result in algorithms that can realise a great amount of massing typology modelling by changing the combinations of values among 24 basic parameters and 24 correlative parameters. Second, a validation and application process is conducted to test the algorithm’s effectiveness and robustness. There are 7 paintings successfully 3D reconstructed by student participants, meanwhile reflecting on the potential challenges. The Biomorpher plugin is also used to explore the application of this algorithm to serve for other architectural design and scenario-comparison tasks.

The current algorithms also have the following implementations and limitations:

  • The massing typology principles are embedded in the Grasshopper algorithm, so that users need not input manually, which can also avoid false inputs. This can expand the scope of 3D reconstruction researchers, for example, from architectural scholars to public.

  • The typologies of “Xie Wu”, “Bo Shui” and “Lang” typologies have been preliminarily researched and documented in this study. If more modules of architectural single volume algorithms and massing typologies algorithms are added to the existing algorithm, more elements from paintings can be automatically generated and reducing time. Future studies can explore other typologies and add to this database.

  • This study only focuses on specific parameters in the scale of architectural massing typology, but not all parameters of Song Dynasty architecture. More basic and correlative parameters can be explored following the same workflow of extracting, translating, and parametric modelling, to provide more detailed models (e.g. spatial partitions inside the volumes, chronological changes of the characteristics within different periods of Song Dynasty, integrating with Song Dynasty courtyard layout and urban planning principles, etc.). It is also found more diverse form, structural, and spatial parameters are represented in different paintings, especially regarding volume conjunction point. Future optimisation of algorithms is expected.

  • Currently, there is still a gap in heritage documentation, integrating both parametric and reality-based modelling65. To better empower research and 3D reconstruction of real sites, the algorithms in this study are welcome to be tested and optimised through implementation in Song Dynasty (and can be extended to Sui and Yuan Dynasty) archaeological sites. As the 3D reconstruction of 2D paintings also required more time to predict the volume relationships and enhance similarity with the paintings, 3D reconstruction conducted based on textual or archaeological evidence might take less time since they can tolerate relatively higher levels of speculation and have a clearer information of relative positions of different building volumes. Other challenges may occur related to specific types of reconstruction basis, for example, developing automatic modelling processes linking real-site data with theoretical principles, e.g. smart point clouds66.

  • It is also needed to discuss validation tools for ensuring the authenticity of output forms, which is to clarify the relationship between the output model and its evidence source, therefore enhancing articulation35. More advanced technologies can be used in chronological and appraisal assessment67,68,69. On one hand, since manual identification of architectural massing typologies is still a timely step in the current workflow, it is still needed to organise collaboration between professionals of art history and architectural conservation, to facilitate multi-disciplinary knowledge exchange and its implementation in real practice. On the other hand, artificial intelligence, e.g. machine learning70,71, can be introduced to all steps of research, documentation, conservation and interpretation, such as massing typology identification, principles extraction, automatic documentation, and volume-intersection adjustment.

  • This easy-input and real-time interface of Grasshopper also sheds light on other implementation directions, such as interactive heritage education72, serious game scene design73,74, cultural dissemination75,76, etc, which expect future exploration.

In conclusion, this study bridges the gap between iconography of art history and architectural conservation in current practices, and has the potential to be implemented in real archaeological research, efficient 3D reconstruction and scenarios comparison. We provide a tool for heritage professionals to research, 3D reconstruct and interpret Song Dynasty architectural massing typologies. This interdisciplinary tool not only enriches our understanding of Song Dynasty architecture, but also sheds light on future innovations in architectural conservation, education, and dissemination.