Introduction

‘Saving the past.’—historic and heritage buildings (HHBs) are essential, representing the legacy and history that shapes culture and contributes to a group’s identity for future generations; meanwhile, HHBs serve as cross-century preservations of a society’s cultural attitudes, architectural design, material technology, and the personal styles of architects and engineers, while also embodying diverse architectural styles, local materials, and construction techniques that define the cultural and historical landscapes of communities and cities1,2,3. Reusing HHBs presents a sustainable strategy for reinvigorating neglected, dilapidated, and underused architectural resources, partially releasing the pressure of social well-being and community development needs4. This method minimises the environmental impact of new construction and demolition by preserving existing structures while offering restored adaptive buildings a renewed purpose and function within contemporary society5,6,7.

Given the varying degrees of damaged HHBs, the restoration process often necessitates the simulation and testing of repair strategies on digital platforms following the reconstruction of 3D models. For architects, achieving precise replication and restoration of damaged sections within digital models presents a considerable challenge8, requiring advanced digital tools to support conservation experts. For instance, the Delft University of Technology employs the Rhino digital operating platform to facilitate digital restoration by identifying and replicating symmetrical elements9. This method has successfully reinstated central symmetrical patterns and reconstructed irregular architectural features, which would be difficult to achieve manually. Applied primarily to small-scale decorative ‘fragments’, this process relies on digital copying and mirroring techniques to enhance restoration precision. However, digital restoration projects for HHBs often face challenges in handling asymmetries and unique localised structures, conditions under which traditional manual restoration methods become increasingly impractical.

The rapid advancement of robotics in architecture, construction, and digitisation is increasingly being explored as a sustainable alternative to traditional manual restoration techniques. Robotics is considered one of the effective solutions for reducing labour resource demands, enhancing construction safety, and achieving high precision in complex real-world architectural projects10. Studies investigating the integration of 3D scanning with robotic repairs have shown promise. However, these approaches often fail to fully account for the diversity of masonry structures and the variability in brick and block compositions. Applying robotic arms using fluid materials, such as concrete and cement, to fill structural gaps establishes a foundation for automated restoration processes. Yet, challenges remain in adapting these methods to heritage conservation, where material compatibility and historical accuracy are paramount. The original Historic Masonry Structures are abandoned as the basic principle of the restoration stage, often through the non-original structure restoration method of additive manufacturing, the destructive restoration of high-efficiency and low-reduction cement mortar, such as the Great Wall, and the low-efficiency and high-precision 3D printing method. More specifically, as seen in the restoration of the Great Wall (Table 1), where a unique historical texture was replaced with a layer of white concrete covering, stripping away its original characteristics and meanings11,12,13,14. In restoring HHBs, 3D printing, while effective in replicating damaged areas, lacks the specific requirements to restore original building techniques and forms fully. Preservation regulations and heritage conservation statutes emphasise using original materials and traditional construction methods to maintain historical authenticity. As a result, 3D printing remains limited in its ability to accurately reconstruct masonry structures, particularly when reassembling components that require craftsmanship, material compatibility, and structural integrity beyond mere geometric replication.

Table 1 Historic and heritage restoration review
Full size table

Restoring HHBs, particularly masonry screen walls, requires a balance between historical authenticity, structural integrity, and technological feasibility15. Traditionally, restoration relied on archival research, craftsmanship, and material replication, which are time-intensive and prone to human error16. Modern methods integrate computer-aided design (CAD), parametric modelling, and automation technologies to enhance restoration precision17. CAD tools, such as Grasshopper for Rhino, facilitate high-precision geometric reconstruction and digital scanning, allowing for accurate analysis and recreation of missing elements18. However, CAD-based restoration struggles with material variability and non-standard masonry geometries, making it difficult to replicate the irregularities of deteriorated structures. Automation and robotics, including 3D printing, CNC milling, and robotic fabrication, have been explored for masonry restoration, providing efficiency and precision in brick placement, material application, and modular reconstruction19,20,21. While additive manufacturing enables the production of missing architectural elements, limitations persist in replicating aged textures, adapting to unstable masonry, and preserving historical authenticity, often resulting in overly uniform replacements22,23. Additionally, AI-driven reconstruction methods require extensive datasets, yet many HHBs feature rare or unique architectural details that are underrepresented in training datasets, affecting accuracy. The reliance on archival records, which may be incomplete or inaccurate, further complicates digital retrieval.

The extended reality (XR) has facilitated the implementation of teleoperation from both the physical and virtual realms, enabling collaborative tasks involving robotic arms24,25. Integrating XR and robotics offers a promising solution by enabling real-time remote implementation and teleoperation for experts who cannot be physically present on-site26,27. XR facilitates seamless interaction between the physical and virtual realms, allowing remote specialists to guide robotic arms in complex restoration tasks with gesture-based calibration and locomotion. Through mixed reality (MR) interfaces and 3D volumetric visualisation, experts can assess structural conditions, manipulate digital reconstructions, and execute precise restoration processes from a distance. Additionally, real-time 3D volumetric telepresence enhances spatial awareness, enabling a more intuitive and interactive approach to remote restoration28. Under internet-controlled teleoperation, robotic systems can execute restoration processes with high accuracy and minimal delay, ensuring that conservation efforts maintain historical integrity while improving efficiency.

This study aims to develop a scalable and repeatable 3D restoration framework for masonry HHBs, integrating volumetric-based restoration thinking, digital retrieval, XR, and automation technologies to enhance precision and efficiency. The key objectives include: a. Developing a Historic-Led Restorative System (HLRS): a restoration approach informed by neighbouring masonry structures, traditional craftsmanship, and historic building behaviours; b. Extending Automation in Restoration (Extended Automation System (EAS)): implementing hybrid additive manufacturing techniques to integrate new materials with existing structures, even when faced with non-standard geometries or unstable masonry elements; c. Integrating a Volumetric-Based Restoration System: establishing a digital retrieval pipeline that seamlessly connects 3D scanning, automated design workflows, an XR-assisted platform, and robotic-assisted physical reconstruction for architects and engineers; d. Evaluating the framework through a case study: using a historic shadow wall to assess the system’s accuracy, efficiency, and applicability in real-world restoration projects.

This study seeks to address the following research questions:

  • How can digital retrieval methods be improved to accommodate the unique and often undocumented structural characteristics of HHBs?

  • What are the limitations of existing ML-based reconstruction techniques in capturing rare historic masonry features?

  • How can automation and robotics be adapted to handle historic masonry structures’ irregularity and material variability?

  • What are the challenges in integrating hybrid additive manufacturing methods with existing HHBs while maintaining historical and structural integrity?

  • Is it possible to bypass distance restrictions for teleoperated restoration robots that perform tasks in both virtual and real environments?

The HLRS is a computational framework designed to restore and reconstruct historic masonry structures digitally, integrating computer vision, voxel-based modelling, and structural analysis (Fig. 1). The workflow has three main stages: virtual restoration, virtual reconstruction, and structural integrity assessment.

Fig. 1
Fig. 1
Full size image

Workflow of the Historic-Led Restorative System (HLRS).

Virtual Restoration: the process begins with image and model acquisition using photogrammetry and Kinect scanning. While photogrammetry captures high-resolution textures and geometric details, Kinect scanning provides real-time, fast point cloud collection, enabling efficient digital documentation of heritage structures.

Computer Vision Detection: the edges of masonry elements are identified using the Canny edge detector, ensuring precise segmentation.

Voxelization and Masonry Block Packing: the scanned data undergoes voxelisation, where divided masonry block packing is applied to optimise the geometric configuration of the structure.

Virtual Reconstruction: once the raw data is processed, the 3D volumetric segmentation step organises the masonry blocks into coherent digital forms. The Wave Function Collapse (WFC) algorithm is then used to generate optimised 3D reconstructions that respect the original structure’s design and constraints. The reconstructed model is further refined through 3D model reconstruction and layered prototyping, preparing the structure for physical implementation.

Structural Integration and Data Transfer: to ensure the feasibility of the reconstructed masonry, a structural assessment is conducted using Finite Element Analysis (FEA) to verify load distribution, stress points, and structural integrity. Finally, the processed data is output to the Robot Operating System (ROS), where the precise position of each brick is determined for automated or semi-automated assembly using robotic systems.

Following the digital reconstruction and structural analysis performed by the HLRS, the EAS translates the generated data into physical action. This system is designed to manage the on-site assembly and manufacturing process, bridging the gap between the digital model and the physical restoration. The EAS (Fig. 2) incorporates a dual-mode architecture enabling seamless transitions between autonomous robotic operation and MR-mediated manual control. The EAS workflow is structured into four primary stages: data simulation, automated assembly, MR-assisted manufacturing, and structural integrity verification.

Fig. 2
Fig. 2
Full size image

The architecture of the Extended Automation System (EAS).

Data Simulation: the workflow begins by receiving the positional data for each masonry component from the HLRS via the ROS. These data are used to generate a complete, real-time simulation of the assembly sequence, allowing for collision detection and path planning before any physical action is taken.

Automated Assembly: in its primary mode, the system autonomously executes the pre-planned assembly sequence. A robotic arm precisely places each masonry element according to the optimised plan generated by the HLRS, ensuring high precision and efficient construction.

MR-Assisted Additive Manufacturing: the EAS also incorporates a manual override mode, enabling human experts to guide the restoration process using an MR interface. This allows for real-time adjustments, expert intervention for complex tasks, and teleoperation, where specialists can control the robotic systems remotely.

Structural Integrity Verification: once the physical assembly is complete, the final structure undergoes a final verification. This involves both digital scans and physical assessments to confirm that the restoration meets the required standards for load distribution and historical accuracy, ensuring the long-term stability of the restored structure.

Methods

Principle of restoration as ‘initial’: volumetric design method: from 2D pixel to 3D voxel

The restoration process in this study follows a volumetric design methodology, transitioning from 2D pixel-based representation to 3D voxel-based modelling to facilitate high-precision reconstruction of historic masonry structures. The voxel-based system provides a structured approach for segmenting and assembling architectural elements, ensuring accurate digital-to-physical translation. Drawing from soma cube (n × n × n) and pentomino systems (n × n) methodologies29, this approach enables flexible aggregation of masonry components, ensuring adaptability to irregular historic structures.

Kinect scanning

This study employs several Azure Kinect DK devices, equipped with a stable tripod and calibration board, to ensure scanning in a stable lighting environment. In a specific implementation, first perform single-view scanning. Then, fix the device on a tripod and maintain a distance of 0.8–1.2 m to collect depth maps and RGB images of the bricks. Save these as raw data in a bag file and ply format. Subsequently, multi-angle scanning was conducted, with eight scanning points evenly spaced at 45° around the bricks. Each point should ensure an overlap area of at least 30% while recording the approximate position information of the equipment. In the data processing stage, the Kinect SDK is first used to convert the depth map into point cloud data, apply preset denoising parameters, and save the point clouds for each viewpoint. Then, the ICP algorithm is used for automatic registration, and subsequent point clouds are registered sequentially based on the first point cloud. Fixed parameters, such as a maximum iteration of 50 times and a maximum corresponding distance of 0.05 m, are set. Finally, merge all registered point clouds and apply a grid filter with a voxel size of 5 mm to output the complete point cloud30.

FEA constraints

This study employs FEA to quantify, evaluate, and optimise the structural behaviour of architectural assemblies under real-world conditions. As a widely adopted computational tool in structural engineering and digital fabrication, FEA facilitates the identification of critical stress points, structural weaknesses, and load distribution patterns, ensuring a more resilient and stable assembly process. Previous studies have extensively applied FEA to furniture, installations, and building components, enabling the prediction of failure points, structural reliability assessments, and the development of optimised typologies31.

More specifically, FEA provides a computationally driven visualisation of stress concentrations, tension areas, and potential structural weaknesses, represented through colour-mapped digital stress simulations. This approach enables the iterative refinement of structural topology, improving overall restoration accuracy and material efficiency.

Robot Operating System (ROS)

The EAS is implemented through a robust ROS architecture designed for safety, precision, and seamless control transitions. The system operates using a UR10e robotic arm and a Meta Quest 3 MR headset.

A central Mode Manager ROS node governs two distinct operational states to balance algorithmic efficiency with expert human oversight.

Autonomous Robot Control Mode (Fig. 3): in this default state, the robotic arm executes task queues generated by the WFC algorithm without interruption. The Meta Quest 3 functions purely as a monitoring interface, subscribing to the robot’s /current_target topic via MQTT and projecting translucent holograms (50% opacity) of the next three scheduled targets to provide the operator with situational awareness. All gesture recognition is disabled.

Fig. 3
Fig. 3
Full size image

Pwashotogrammetry performed using a high-resolution Sony DSLR camera to obtain a comprehensive image set with a high degree of overlap (actually >70%) around the structure. This resulted in a highly accurate and highly realistic digital model.

Gesture Priority Control Mode (Fig. 4): activated by voice command or a physical button, this mode grants the HoloLens full control authority via a three-tier priority hierarchy. Gestures are published to the /gesture_stream topic at 30 Hz. Four main gestures are recognised: Emergency Stop (Palm Thrust gesture): terminates all robotic motion immediately; Target Override (Single Point gesture): pauses the current queue and inserts a new operator-defined task; Trajectory Adjustment (Two-Finger Drag): modifies the active toolpath in real-time; Queue Navigation (Swipe Left/Right): advances or rewinds the task sequence.

Fig. 4
Fig. 4
Full size image

Determine the appropriate voxel size (90 × 90 × 90 mm) for the input matrix based on the type and size of the bricks. Also, define the percentage of bricks allowed to overflow in each voxel to connect with other voxels.

To ensure safe and reliable operation, the system incorporates critical safeguards compliant with ISO 10218-2. Mode transitions require dual confirmation to prevent accidental activation and are managed by hardware-enforced command isolation, ensuring that only one control stream has authority at any time. The system validates data integrity using CRC-32 checksums and achieves a complete mode switch in 800 ± 150 ms.

Each mode is optimised for its specific function, with performance benchmarks demonstrating this specialisation. Autonomous Mode achieves 1.1 mm placement precision with target updates at 10 Hz and a command latency of 85 ± 12 ms, prioritising efficiency and accuracy for repetitive tasks. Gesture Mode prioritises responsiveness, executing commands at 30 Hz with a latency of 210 ± 38 ms and accommodating human operational variability, resulting in a placement precision of 2.3 mm.

The holographic interface provides immediate operator feedback. In Autonomous Mode, targets are marked with blue bounding boxes. In Gesture Mode, targets pulse with a yellow hue, and collision-avoidance previews are rendered based on the robot’s safety planes. An Emergency Stop activates a flashing red safety sphere that visually envelops the entire robot workspace.

Results

This section presents the empirical results and validation of the integrated digital-to-physical workflow for HHB restoration. The framework was systematically tested using a historic masonry screen wall in Lanxi City, China, as a representative case study.

The selection of a genuine, deteriorated historic structure, rather than a controlled laboratory specimen, represents a critical decision. This choice immediately subjects the proposed workflow to the complex challenges inherent in real-world heritage conservation, including material decay, non-standard component geometries, and structural asymmetries. Consequently, the results validate the robustness and practical applicability of the entire framework when confronted with the unpredictable conditions of an authentic heritage site.

Restoration digitisation

The successful implementation of the HLRS hinges on its ability to accurately capture, analyse, and reconstruct a deteriorated structure in the digital realm, culminating in a high-fidelity, structurally viable model for physical restoration.

Phase 1: 3D Reconstruction Accuracy. Photogrammetry Method: the initial phase involved reconstructing the target structure using photogrammetry. Photogrammetry was conducted using a high-resolution SONY DSLR camera to capture a comprehensive set of images with substantial overlap (actually >70%) around the structure (Fig. 5). The resulting model achieved a high geometric accuracy, with mean deviations from surveyed control points falling within the ±10 mm tolerance generally considered acceptable for professional architectural surveys, and produced high-resolution texture maps. Kinect scanning: in parallel, a Microsoft Kinect sensor was used for real-time 3D scanning. The process involves fast data collection, rapid on-site assessment and real-time point cloud generation. However, the Kinect’s data quality is inherently lower than that of photogrammetry. The random error of its depth measurements is known to increase with distance, ranging from a few millimetres in the near field to approximately 4 cm at its maximum range.

Fig. 5
Fig. 5
Full size image

Bricks voxelization and input definition, and the constrained generation output using WFC.

Phase 2: Brick Detection Following Segmentation. For the precise identification of individual bricks, the YOLACT (You Only Look At CoefficienTs) algorithm was employed. The model was adapted explicitly for masonry structures to accurately segment each brick and its surrounding mortar joints from the 2D images. Post-processing incorporates domain-specific geometric constraints to resolve overlapping brick predictions and refine boundary details, with particular attention paid to maintaining consistent mortar line segmentation. This process yielded high-precision instance segmentation, creating a detailed and accurate map of the brickwork necessary for the subsequent 3D reconstruction and analysis. The successful segmentation provides the foundational data for digitally reassembling the structure.

Phase 3: Modelling Method for 3D Model. Following brick detection, the segmented data is used to reconstruct the masonry structure in a digital environment. This modelling method ensures that the 3D model accurately represents the bricks’ size, shape, and arrangement. The digital model replicates the original masonry structure, allowing for detailed analysis and planning of the restoration process.

Phase 4: Volumetric Design and Bounding Box Voxel Size Definition. This step employs a volumetric design approach to define the bounding box voxel size. The type and scale of the bricks are used to determine the appropriate voxel size (90 × 90 × 90 mm) for the input matrix (Fig. 6). This matrix will serve as the basis for the digital restoration process, ensuring that the reconstructed structure maintains its integrity and coherence.

Fig. 6
Fig. 6
Full size image

Masonry screen wall reconstruction results and the FEA result.

Phase 5: Input Matrix Setting, Rotation, and Connection Mechanisms. The input matrix uses the WFC algorithm, including the bricks’ rotation and connection mechanisms. By propagating local constraints (e.g. ensuring each new block matches the shape of its neighbours and the overall style), the algorithm collapses the solution space to only valid options, yielding a final arrangement consistent with the building’s original design logic. This mechanism is analogous to fitting fragments into a mosaic—only combinations that respect the established pattern are allowed, resulting in a coherent whole. The WFC algorithm ensures that the digital model adheres to realistic structural principles, providing a reliable foundation for the restoration work.

Phase 6: Structure Aggregation for Digital Retrieval. The next phase involves digital retrieval and structure aggregation. The input matrix is processed to produce a cohesive digital model aggregating the individual components into a unified structure. This output serves as a digital blueprint for the restoration process, providing detailed guidance on how the physical restoration should be executed.

Phase 7: FEA for Structural Assessment. Before the actual construction begins, the digital model undergoes FEA to assess its structural integrity31 This analysis evaluates the model’s performance under various conditions, identifying potential weaknesses and areas that require reinforcement. The FEA ensures that the restored structure will be stable and durable, minimising the risk of failure.

With a reliable dataset of segmented masonry units, the core task of digital restoration—filling the damaged voids—was addressed using the WFC algorithm. This approach marks a significant departure from traditional digital restoration techniques, which often rely on simple replication or mirroring of existing elements and fail in cases of asymmetry or complex irregularity. WFC, by contrast, functions as a constrained generative process.

The workflow proceeded as follows:

Voxelization and Input Definition: the segmented brick typologies from the previous stage were converted into a discrete, voxel-based representation. Each unique brick type, including variations in size and orientation, was defined as a module (Fig. 7). The intact portions of the wall were analysed to extract adjacency rules (e.g. ‘a header brick can be placed next to a stretcher brick’, ‘a half-brick is only found at the wall’s edge’). These rules, along with the module library, formed the input constraints for the WFC algorithm.

Fig. 7
Fig. 7
Full size image

The framework of ‘UR-MetaBridge’.

Constrained Generation: the WFC algorithm was initialised on a grid representing the damaged area, where each cell began in a state of superposition, holding all possible brick modules as potential options. The algorithm iteratively ‘collapses’ the state of one cell to a single, definite module, chosen based on weighted probabilities derived from the frequency of modules in the original wall. It then propagates the constraints from this decision to neighbouring cells, eliminating incompatible module options. This process continues until every cell in the damaged area has been assigned a valid module, resulting in a fully populated structure (Fig. 7).

The final output was a complete, aggregated 3D model of the restored wall. Visual comparison between the initially damaged model and the WFC-reconstructed model demonstrated a high degree of stylistic and structural coherence. The algorithm successfully generated a novel masonry arrangement within the damaged section that respected the complex bonding patterns and architectural grammar learned from the surviving original fabric. This result showcases a more intelligent and adaptable form of digital restoration, one that shifts the paradigm from restoration as replication to restoration as constrained, rule-based generation.

While the WFC algorithm produced a visually and stylistically plausible reconstruction, it operates on geometric and probabilistic rules without any simulations of structural mechanics. Therefore, a crucial final step within the HLRS was to verify the mechanical soundness of the generated digital model using FEA.

An FEA model was constructed based on the WFC-generated geometry. The setup adhered to standard practices for structural simulation:

  • Material Properties: masonry units and mortar were assigned appropriate linear elastic material properties, including Young’s Modulus, Poisson’s Ratio, and density, based on typical values for historic brick and lime mortar.

  • Boundary Conditions: the base of the wall was defined with a fixed support constraint, simulating its connection to a foundation.

  • Loading Conditions: two primary load cases were simulated: (1) gravitational self-weight to assess stability under normal conditions, and (2) a uniform lateral pressure representing a standard wind load to evaluate its response to environmental forces.

The FEA solver computed the resulting stress, strain, and displacement distributions throughout the structure. The results were visualised using colour-mapped contour plots (Fig. 8). The analysis confirmed that under both load cases, the maximum von Mises stress concentrations remained well below the established compressive and tensile yield strengths of the masonry materials. Furthermore, the maximum displacement observed at the top of the wall was found to be within acceptable limits for structural serviceability.

Fig. 8
Fig. 8
Full size image

The interface of the Meta Quest 3 for real-time control of the UR10e robot.

This successful FEA result provides the ultimate validation for the WFC-based generative process. It demonstrates that the algorithm did not merely create a visually coherent facade but produced a mechanically stable structure. This retroactively confirms that the ‘architectural grammar’ the WFC algorithm learned from the original historic masonry inherently contained sound structural principles. It suggests a profound, embedded logic within traditional craft techniques that the computational system was able to successfully identify, learn, and replicate, highlighting a powerful synergy between historic craftsmanship and modern computational analysis.

Robot system innovation and communications

In the constantly evolving human-computer interaction environment, seamless integration between immersive technology and industrial robots brings new possibilities for intuitive control and automation. Here, we propose the concept of the UR-MetaBridge framework, which connects the VR function of Meta Quest with the UR (Universal Robots) manipulator through MQTT to achieve real-time remote operation and monitoring (Fig. 9). By utilising ROS for backend communication and MQTT for efficient data transmission, this architecture enables users to control UR robots through natural gestures and interfaces in virtual reality, integrating the digital and physical worlds to achieve smarter.

Fig. 9
Fig. 9
Full size image

The planned, structurally sound, bottom-up, layer-by-layer reconstruction method.

The UR robot bridge component provides the calculation method for the angles of each joint of the UR10 robot. Still, the specific position calculation of the end-effector requires an inverse solution based on the robot’s kinematic model. The following is the position algorithm for calculating the end-effector of the UR robotic arm in experiments.

The transformation matrix for each joint is calculated using D-H parameters, and the pose of the end-effector can be obtained from multiple transformation matrices. Finally, the robot joint angles can be inversely solved based on the specified position of the actuator.

The inverse kinematics solution for the UR10e is derived using a geometric approach. The problem is decomposed into two main stages: positioning the wrist centre and orienting the end-effector. First, the wrist centre position is calculated by subtracting the tool offset from the desired end-effector position. The joint angles for the first three degrees of freedom \(({\theta }_{1},\,{\theta }_{2},\,{\theta }_{3})\) are then solved using trigonometric relationships based on the wrist centre coordinates. The remaining joint angles (\({\theta }_{4},{\theta }_{5},{\theta }_{6}\)) are determined by aligning the end-effector orientation with the desired pose through rotation matrix decomposition. This method yields multiple valid solutions, which are evaluated based on joint limits and task constraints.

\({\theta }_{1}\) can be solved (Eq. 1):

$${\theta }_{1}={\rm{atan}}2\left({p}_{{\rm{wrist}},y},{p}_{{\rm{wrist}},x}\right)$$
(1)

Define the coordinates of the end-effector in the joint coordinate system (Eq. 2):

$$\begin{array}{c}{{\bf{p}}}_{{\rm{wrist}}}^{(2)}=\left[\sqrt{{p}_{{\rm{wrist}},x}^{2}+{p}_{{\rm{wrist}},y}^{2}}-{a}_{1},{p}_{{\rm{wrist}},z}-{d}_{1}\right]\\ \left({a}_{1}=0,{d}_{1}=0.1807\right)\end{array}$$
(2)

In addition, the remaining joint angles can be solved through geometric relationships (Eq. 3):

$$\begin{array}{c}\cos {\theta }_{3}=\frac{{{||}{{\bf{p}}}_{{\rm{wrist}}}^{(2)}{||}}^{2}-{a}_{2}^{2}-{a}_{3}^{2}}{2{a}_{2}{a}_{3}}\\ \left({a}_{2}=-0.6127,{a}_{3}=-0.5716\right)\\ {{{\boldsymbol{\theta }}}_{3}=\pm \arccos \left(\cos {\theta }_{3}\right)}_{5}\end{array}$$
(3)

according to the geometry \({\theta }_{2}\) can be solved (Eq. 4):

$${\theta }_{2}={\rm{atan}}2\left({p}_{{\rm{wrist}},z}^{(2)},{p}_{{\rm{wrist}},x}^{(2)}\right)-{\rm{atan}}2\left({a}_{3}\sin {\theta }_{3},{a}_{2}+{a}_{3}\cos {\theta }_{3}\right)$$
(4)

\({\theta }_{4},{\theta }_{5},{\theta }_{6}\) can be solved:

the rotation matrix of the twist 3 \({{\rm{R}}}_{3}^{6}\) can be calculated (Eq. 5):

$${{\bf{R}}}_{3}^{6}={\left({{\bf{R}}}_{1}^{0}{{\bf{R}}}_{2}^{1}{{\bf{R}}}_{3}^{2}\right)}^{T}\cdot {\bf{R}}$$
(5)

Using the solutions of the first three joints, calculate the rotation matrix from joint 3 to the end.

Solve through Euler angle decomposition (in XYZ order) (Eq. 6):

$$\begin{array}{c}{\theta }_{4}={\rm{atan}}2\left({{\bf{R}}}_{3}^{6}[1,2],{{\bf{R}}}_{3}^{6}[0,2]\right)\\ {\theta }_{5}={\rm{atan}}2\left(\sqrt{{{\bf{R}}}_{3}^{6}{[0,2]}^{2}+{{\bf{R}}}_{3}^{6}{[1,2]}^{2}},{{\bf{R}}}_{3}^{6}[2,2]\right)\\ {\theta }_{6}={\rm{atan}}2\left({{\bf{R}}}_{3}^{6}[1,2],-{{\bf{R}}}_{3}^{6}[2,0]\right)\end{array}$$
(6)

The MR brick repair system achieves an intuitive human-machine collaborative operation experience through innovative MR technology (Fig. 10). The system uses the built-in sensors of Meta Quest 3 combined with the wall collapse algorithm to obtain brick coordinate points, including position point coordinates and elevation angles, which can perfectly align the virtual brick position with the actual wall in real time. The gesture recognition system, using Meta Quest 3 and the interface of the Prefab SDK, can accurately recognise six preset operation gestures, including selecting bricks, removing, applying mortar, placing bricks, adjusting compaction, and emergency pause commands. Even when tools obstruct the hand, it can maintain recognition continuity. The robot control system converts gesture instructions into control signals in real time, uses intelligent path planning algorithms to ensure motion safety, and can automatically avoid obstacles and personnel. The system is equipped with multiple security protection mechanisms, which monitor the distance between humans and machines in real time through electronic fences. When personnel are detected approaching, it will automatically slow down or stop. At the same time, the operator can immediately stop all equipment operation by crossing their arms in an emergency gesture. The MR interface visually displays the location of the bricks to be repaired (highlighted in red), the placement of new bricks (highlighted in green), and the completed area (covered in blue), and provides detailed operation instructions and progress feedback. The system also has a personalised calibration function, and new users only need to perform simple gesture sampling to start using it, significantly reducing the operating threshold. The entire system has implemented a closed-loop operation process from target recognition, gesture control, to robot execution, considerably improving the efficiency and accuracy of brick repair while ensuring safety.

Fig. 10
Fig. 10
Full size image

The architecture of the Autonomous Robot Control system.

Augmented and automated assembly

The efficacy of the workflow was validated through the physical repair of moderately damaged wall specimens, executing computationally generated assembly sequences (Fig. 11). Guided by a planned, structurally sound, bottom-up, layer-by-layer reconstruction, the system first identifies the repair area. This process integrates automated robotics with human oversight via an MR interface. A robotic arm executes the coarse assembly tasks after the headset sensors and a collapse algorithm’s damage mapping, such as applying mortar and positioning bricks. Guided by visual overlays in the MR environment, a human operator makes fine-grained adjustments using intuitive hand gestures for final placement, ensuring precise alignment and maintaining consistent mortar joint thickness. Real-time gesture commands are translated into robot actions with intelligent path planning, while safety is ensured through an electronic fence and an emergency stop gesture. This symbiotic approach leverages robotic precision for repetitive tasks and human cognition for quality control, creating a closed-loop system that significantly enhances the efficiency, accuracy, and safety of masonry repair.

Fig. 11
Fig. 11
Full size image

The architecture of the Gesture Priority Control system.

Evaluation model

The 3D scanning accuracy evaluation model established in this study includes three core indicators: geometric dimension accuracy, surface flatness, and feature fidelity. The geometric dimension accuracy is calculated by measuring the deviation between the scanned values of 12 edges of the brick and the actual values, and calculating the average error (MAE), maximum error (MaxE), and error fluctuation (σ). The improved MSAC algorithm is used for plane fitting of surface flatness, and the residual from the point cloud to the fitting plane is analysed to calculate the waviness and flatness indicators. Feature fidelity evaluates the size reduction rate (FDR) and edge sharpness (ESI) of features such as grooves and holes. The three indicators are comprehensively assessed on a percentage scale with weights of 40%, 35%, and 25%. After experimental verification, the correlation between the evaluation results of this model and professional measurement equipment data reaches 93%, which can effectively quantify scanning accuracy.

To evaluate robotic grasping performance in 3D space, we employ a comprehensive three-tiered metric system. The task success layer utilises Valid Grasping Rate (VGR), calculated as the percentage of successful grasps relative to total attempts (Eq. 7).

Measures the probability of successful object grasping, calculated as:

$${\rm{VGR}}=\frac{{N}_{{\rm{success}}}}{{N}_{{\rm{total}}}}\times 100 \%$$
(7)

\({N}_{\text{success}}\): number of successful grasps

\({N}_{\text{total}}\): total number of grasp attempts

Pose estimation accuracy metrics

For pose estimation accuracy, we implement dual geometric metrics: the Average Distance of Model Points (ADP) measures mean Euclidean distance between predicted and ground truth poses (Eq. 8), while Chamfer Distance (CD) evaluates point cloud alignment through bidirectional nearest-neighbour matching (Eq. 9).

Quantifies the deviation between the predicted grasp pose (rotation \(R^{\prime}\) and translation \(T^{\prime}\)) and the ground truth pose (rotation R and translation T). Given a 3D object model point set \(M=\left\{{x}_{1},{x}_{2},\ldots ,{x}_{n}\right\}\), the average Euclidean distance is computed as:

$${e}_{{\rm{ADP}}}=\frac{1}{n}\sum _{x\in M}{{\rm{||}}({Rx}+T)-\left({R}^{{\prime} }x+{T}^{{\prime} }\right){\rm{||}}}_{2}$$
(8)

x: a point on the 3D model

||⋅||2: Euclidean distance

Measures the matching error between the predicted point cloud P and the ground truth point cloud Q by computing the bidirectional nearest-neighbour distance:

$${\rm{CD}}(P,Q)=\frac{1}{2}\left(\frac{1}{{|P|}}\sum _{p\in P}\mathop{\min }\limits_{q\in Q}{||}p-q{{||}}_{2}+\frac{1}{\left|Q\right|}\sum _{q\in Q}\mathop{\min }\limits_{p\in P}{||q}-p{{||}}_{2}\right)$$
(9)

P: predicted point cloud

Q: ground truth point cloud

Computational efficiency is quantified via Average Time Cost (ATC), representing the mean duration per grasp cycle including perception, planning, and execution phases (Eq. 10).

Evaluates the real-time performance of the algorithm:

$${\rm{ATC}}=\frac{1}{{N}_{{\rm{total}}}}\mathop{\sum }\limits_{i=1}^{{N}_{{\rm{total}}}}{t}_{i}$$
(10)

ti: total time cost for the ith grasp task (including perception, pose estimation, motion planning, etc.)

This tripartite framework—encompassing operational reliability (VGR), spatial precision (ADP/CD), and temporal performance (ATC)—provides a robust assessment methodology for optimising 6-DoF grasping systems in cluttered environments, enabling direct comparison between different algorithmic approaches while addressing both functional success and implementation quality. This evaluation model provides a more detailed assessment of the entire repair process, from the accuracy of 3D scanning to the accuracy of robot positioning. This evaluation model can also be used to optimise the control algorithm.

Discussion

This study presents an integrated framework that combines CAD, robotic automation, and XR to enhance the precision, efficiency, and scalability of HHB restoration. This approach transforms from traditional manual restoration to a data-driven, computationally guided methodology by bridging digital retrieval, automated fabrication, and real-time expert collaboration. The HLRS and EAS were developed to facilitate multi-stage restoration processes, from 3D digital modelling and data simulation to robotic-assisted assembly.

We validated the system’s feasibility through a historic screen wall case study, demonstrating its ability to accurately reconstruct masonry elements, streamline digital-to-physical workflows, and enhance restoration efficiency. The results indicate that integrating robotics with heritage conservation reduces reliance on labour-intensive craftsmanship and enables data-informed decision-making and remote expert supervision through MR interfaces. Furthermore, the introduction of cable-driven parallel robots (CDPRs) represents a scalable alternative to traditional robotic arms, allowing for the precise placement of heavy masonry units over large architectural sites.

Despite these advancements, several challenges remain. Material compatibility, algorithmic refinement, and real-world scalability must be further addressed to ensure seamless adoption in full-scale heritage restoration projects. In particular, the WFC algorithm, utilised for masonry segmentation and voxel-based reconstruction, requires enhanced adaptability to accommodate non-standard historic masonry geometries. Additionally, real-time teleoperation via long-distance signal transmission between restoration teams—such as those in Hong Kong and the UK—necessitates further development to optimise latency, accuracy, and collaborative usability. The next phase of this research will focus on scaling the system for deployment in complex restoration scenarios, with Corfe Castle, UK, identified as a key test site. This study underscores the critical role of automation, digital fabrication, and MR in shaping the future of heritage conservation. By integrating advanced computational methodologies with architectural craftsmanship, this research-innovated system contributes to a sustainable, technologically advanced approach to preserving cultural heritage while maintaining historical authenticity and structural integrity.

Despite promising initial results, the current robotic assembly system shows clear limitations in dexterity and adaptability when confronted with complex historical geometries. In particular, forms like Gothic arches pose challenges due to their intricate profiles and the need for precise alignment of irregular stone blocks. The traditional construction of masonry arches relies on compression-only equilibrium, meaning each voussoir (arch stone) must be supported until the arch is closed. A single robotic arm lacks the adaptability to manage this on its own. Without auxiliary support, placing heavy stones into an open arch can lead to collapse from unbalanced lateral thrusts. Prior research demonstrates that geometrically complex structures such as arches and vaults require extensive scaffolding or centring to counteract out-of-equilibrium forces during assembly32. This requirement indicates the limits of a lone robot’s dexterity: it cannot easily adjust to hold a piece in mid-air while assembling the rest. To address this, scholars have explored multi-robot collaboration. For example, Wu et al. employed a two-arm robotic setup to sequentially assemble a compression arch without scaffolding by having one arm place each block while the other arm temporarily braced the structure, maintaining force equilibrium at every step33. Such collaborative assembly sequencing—essentially robots taking turns as builders and supporters—underscores the importance of coordinated control and planning. Even so, two robots introduce coordination complexity and were found to be only partly sufficient for more elaborate forms. A recent study showed that while two-arm strategies for scaffold-free arch construction are viable, they face challenges in scalability and robustness for larger or more intricate vault geometries. In fact, adding a third robotic participant and optimising the assembly sequence improved stability through all intermediate stages32. These findings highlight that our current system, which is limited to a laboratory robotic arm, would struggle with real Gothic arches or similarly complex elements—it simply does not have the in situ dexterity or collaborative ability to handle the nuanced structural forces at play. The assembly sequencing is critical: placing an arch’s stones in the wrong order or without proper temporary support can cause the entire configuration to collapse under its own weight. Thus, the limitation is twofold: the robot’s physical manoeuvrability (degrees of freedom, reach, grip adaptability) and the system’s ability to intelligently sequence and coordinate tasks (potentially with multiple robots) to maintain structural equilibrium. Recognising these shortcomings informs the next steps in our research and development.

We propose adopting a CDPR system for on-site assembly to overcome the above limitations and move toward full-scale heritage construction. This approach is fundamentally more scalable than the fixed robotic arm used in the lab. Conventional industrial robots are heavy, have a limited reach, and are difficult to transport and deploy19,34. By contrast, CDPRs consist of a mobile platform suspended by cables anchored at the site’s periphery, offering a lightweight yet extensive manipulator. Notably, CDPRs can cover a large workspace and handle high payloads due to efficient force transmission through cables35. A single CDPR can span the volume of an entire construction site and lift substantial masonry units that a standard robot arm might not manage. Furthermore, the system’s configuration is flexible—cables can be attached to temporary masts, existing structures, or cranes, adapting to the irregular layout of a ruin. Our precedent and ongoing projects, ‘CU-brick’ and ‘YES pavilion’ prototype, a low-cost 4-cable on-site assembly robot that demonstrated how reducing the number of cables can minimise interference with surroundings and maximise the usable working envelope and consists of a CDPR, an automated brick delivery system, a human-robot collaborated mortar application and a computational design framework. It allows large-scale brick structures of complex geometry to be constructed due to its high levels of accuracy, efficiency and automation36,37.

Some key advantages of CDPRs in the heritage restoration context include:

  • Expanded Reach: a CDPR can be designed to envelope large structures (tens of metres in span) far beyond the reach of typical robotic arms: a. Expanded Reach: a CDPR can be designed to envelope large structures (tens of metres in span), far beyond the reach of typical robotic arms. This means it could access all parts of a castle wall or a cathedral vault without repositioning equipment; b. High Payload Capacity: through multiple cables sharing the load, CDPRs can potentially lift and precisely place heavy stone elements—on the order of hundreds of kilograms—which is necessary for authentic masonry restoration; c. Deployment Flexibility: the system’s components (winches, cables, and supports) are modular and transportable. Unlike a bulky robot arm that requires a firm foundation, a cable robot can be erected on-site with minimal intrusion. This reduces the risk to fragile remnants of the structure and avoids the inflexibility of stationary machinery34,35; d. Collaborative Operation: multiple cable robots or end-effectors on one cable robot can work in unison within the same frame. This opens possibilities for one unit to stabilise a piece while another places the next—akin to the multi-arm collaboration discussed earlier, but on a building-sized scale19.

  • Restoration Plan and Technological Implementation: with the core robotic approach shifting to a CDPR, the restoration plan centres on the seamless integration of human expertise, remote operation, and MR guidance. A distinctive aspect of our project is the long-distance, real-time operation between Hong Kong and the UK. We plan to establish a control hub in Hong Kong where operators can supervise and direct the on-site cable robot at the UK heritage site. Achieving low-latency, high-reliability signal transfer over such a distance is paramount. Recent advances in teleoperation suggest that next-generation connectivity (e.g. 5G networks) can enable real-time robotic control across continents.

The end goal is an efficient teleconstruction workflow where an expert can effectively ‘be on site’ digitally, directing the restoration as if present, thus pooling global expertise without physical travel. Another cornerstone of the implementation is using MR to facilitate collaboration and decision-making. This tight integration of MR is inspired by emerging applications of AR/MR in cultural heritage, which show that overlaying contextual information directly onto artefacts can greatly aid decision-making and reduce reliance on separate documentation38.

A critical discussion point is the system’s scalability from a small prototype to a full-scale heritage structure. To date, our work has focused on a small-scale screen wall assembly—essentially a portion of a wall constructed under controlled conditions. This served as a proof-of-concept, allowing us to validate the robot’s placement accuracy, the stability of a few assembled courses, and the basic workflow of digital-to-physical construction. However, a screen wall is a simplified scenario: typically planar, of limited height, and using a modest number of components. Transitioning to an actual historic building introduces exponentially greater complexity. Full-scale restoration targets involve irregular layouts, significant heights, variable stone sizes, and unpredictable on-site conditions (from weather to uneven terrain). The robotic system must be robust against these factors. For example, raising the scale means dealing with heavier stones that might push the limits of grip force and cable tension and larger structures that accumulate more positioning error over distance. It also means the control software must handle many more elements being assembled, and the safety systems must cover a larger area around human workers. As a tangible step toward this scale-up, we have identified Corfe Castle in England as the second-stage case study for our system. Corfe Castle, a medieval fortification dating back to the eleventh century, now exists as a substantial ruin—with standing curtain walls, partial towers, and scattered masonry—making it an ideal real-world testbed. This site presents the challenges our approach aims to address: complex geometry (e.g. broken arches, angled stone debris), large scale (the keep and walls span tens of metres), and a delicate historic fabric that must be handled with care39.

The Meta Quest-based UR robotic wall repair collaborative system faces several unverified technical limitations before deployment. The robot’s efficiency needs to be verified experimentally. Furthermore, there are significant uncertainties in the wireless communication link. The Quest’s WiFi 6 connection may be interfered with by other devices in industrial sites, resulting in MQTT message transmission delays fluctuating by up to 50–200 ms. Regarding safety mechanisms, the system lacks native force feedback, and relying solely on visual warnings may increase the risk of misoperation. Furthermore, the 300 ms response delay for gesture emergency stop commands may not fully meet ISO 13849 safety standards. Regarding hardware compatibility, integrating ROS with the Quest requires an additional communication bridge layer, introducing an additional 50–100 ms latency and potentially causing command priority conflicts when using multimodal input.

Regarding environmental adaptability, long-term operation may cause the Quest device to overheat and throttle. Dust and humidity at construction sites may also affect camera tracking performance. It is recommended to gradually verify system reliability through phased testing, focusing on network stability when multiple devices coexist, tracking robustness in extreme environments, and safety response capabilities in emergency situations. At the same time, you can consider adding an external tracking system as redundancy and optimising the communication protocol to reduce latency. These measures will help fully identify and mitigate potential risks before formal deployment. In addition, in terms of control, controllers with more precise feedback mechanisms can be tried, and the introduction of deep learning can make the entire system more portable. This article will make more research and attempt on these two aspects in the future.