Introduction

Genomic data holds extraordinary promise for transforming our understanding of cancer, in the laboratory and the clinic. As the scientific community endeavours to resolve the multitude of remaining questions in cancer, we will rely on layers of information coming from diverse sources and must capitalise on the collective expertise of multidisciplinary teams working at the intersections of disciplines1. In the clinic, interpreting genomic data in its spatial and temporal context is essential for making treatment decisions for cancer patients2,3. Interpreting genomic data with respect to time and space is critical for a deeper understanding of tumour evolution during metastatic dissemination, with the majority of cancer deaths caused by metastases4. Yet, current tools are not equipped to integrate genomic, radiological and therapeutic information5, nor is the human brain effective at holding this information for discussing such complex cases6. Generating data on multiple tumours from an individual is becoming increasingly common in research and the clinic, to face the clinical realities of tumour heterogeneity7,8. Therefore, interpreting genomic data in light of other important system components9, and by multidisciplinary teams, becomes increasingly critical.

A person who had a primary lung neuroendocrine tumour and ninety metastases disseminated through their body came to their oncologist requesting to donate their tumour tissues to research after death. Accepting this invaluable donation set us on a journey, a high-resolution n = 1 study10, where we sought to distil new knowledge about tumour evolution that may be missed in studies of more patients with fewer samples. Research autopsies are emerging internationally as an unmatched resource for studying tumour evolution, the emergence of treatment resistance, immune evasion and optimal drug targeting strategies, driven by selfless patients as partners in research11. We generated a large body of multi-omic data from 42 spatially distinct tumour sites, which presented a significant comprehension and analysis challenge. We aimed to integrate genomic information from multiple sites within metastatic tumours across their body collected at autopsy, complemented by circulating tumour DNA assays, clinical biopsies, radiological information showing tumour growth and shrinkage across a ten-year timeline, and therapeutic information, to produce a detailed understanding of the drivers and processes of tumour evolution in this person’s disease8.

Turning to extended reality (XR) technology for visualising this complex data provided opportunities to rethink and reimagine how cancer data can be presented and interrogated by teams of people. XR is an umbrella term that describes technologies that allow the merging of digital and physical information and encompasses virtual reality (VR), augmented reality (AR) and mixed reality (MR). The latter two are immersive technology that projects 3D digital elements into the physical world, usually through headsets12 or smartphones, that enable both digital and face-to-face interactions primed to enhance collaboration13. Although potential clinical avenues for XR have been explored, including telemedicine14, surgical planning15, psychological treatments16, and medical education17, alongside research-grade bioinformatic tools18,19, integrating multimodal oncology datasets for multidisciplinary investigation hasn’t yet been explored in XR.

Results

Design overview

Our multidisciplinary team of scientists, clinicians, architects, digital spatial design specialists and software developers worked through an iterative design process over two years to produce an advanced extended reality (XR) environment bringing together layers of data. We aimed to produce a tool to integrate genomic, radiology and therapeutic data from across time and space for this single patient with an unusually large research dataset. In doing so, we hoped to better understand the evolution of this patient’s tumours8, and consider the future of novel collaborative and immersive tools beyond this one patient.

Our model takes the form of an immersive extended reality arena, merging physical and digital elements (Fig. 1). It enables the presentation of genomic data, in its anatomical context, alongside tumour scans and other relevant clinical information. Using Microsoft HoloLens II MR headsets, teams of up to ten users may collaboratively interact with the data projected into the physical room through the headsets, including interacting with different parts of the model simultaneously. At its heart, stands an accurate patient body model (skeleton, organs) derived from CT scans, forming the base for other information to be displayed. Overlaid are 3D representations of the tumours derived from the CT scans, enabling users to scroll through the timeline and observe tumours growing, shrinking in response to treatment, and ultimately spreading around the body. The timeline also indicates therapies received (chemotherapy and radiotherapy, including the field of radiotherapy).

Fig. 1: Integrating genomic, radiology and therapeutic cancer data in extended reality—a level playing field for enhanced multidisciplinary communication.
figure 1

a Overview of the information displayed in the model, including interactive phylogenetic tree (left), central body model showing tumours (red) overlaid on organs (blue), with genomic data sites labelled and coloured according to shared genomic information in the phylogenetic tree, and interactive annotated timeline (right). b See tumours changing through time, with respect to other clinical information. Users may slide through a timeline, annotated with therapeutic information, to access radiology-based tumour information displayed on the main body model (in red, shown behind participants). c Recognise genomic tumour heterogeneity. The lung has been isolated from the main body model for detailed inspection. Sampled tumour sites for which genomic data was generated are annotated. Labels relate to sample site codes, with colouring representing the genomic clades of tumour-relatedness. d Workspace for multidisciplinary collaboration. The tool was designed to be used by up to ten participants simultaneously, allowing multidisciplinary teams to discuss different facets of the data and integrate their personal understanding of the layers of data together. For example, major tumours can be ‘pulled’ out of the main skeletal model, as can be seen here, where the participants on the left are examining the pancreatic tumour while the participants on the right are examining the primary lung tumour. A further participant is discerning extra information about tumour genomic data on a large monitor at the back of the room. A phylogenetic tree based on DNA variants shown to the left of the main patient model allows users to view the spatial distribution of tumours that share common genomic changes and indicate relatedness. Users may highlight a clade on the phylogenetic tree and observe these samples highlighted on the main model. Many tasks can be completed simultaneously by small groups of people, or alternatively, one user may guide all other participants through the dataset, with all users seeing the same information through their headsets. b–d shows the view through one participant’s headset, while they are interacting with other participants.

Full size image

Simplistic representations of the genomic data are included at each of the 42 sampled sites8. Genomic relatedness is indicated by colour scheme, where groups of tumours carrying common variants are highlighted in the same colour, allowing the genomic data to be interpreted in the context of tumour changes detected on CT scans and treatment information. An interactive 3D evolutionary tree presents information on the relatedness of tumours based on shared DNA variants, and users may highlight clades (groups of related tumours) on the main body model via the evolutionary tree. Users may reach into the model and ‘pull’ a tumour towards them, to have the tumour enlarged and overlaid with representative genomic information. We also produced a tablet application that provided detailed genomic variant annotations, with the ability to highlight the anatomical position of all tumours carrying that variant in the XR model (viewed on a large monitor in Fig. 1d). The multiplayer XR visualisation allows teams of users to interact with the data together, each member coming from a different speciality and perspective, facilitating the work of a translational multi-disciplinary team. This encourages detailed interdisciplinary discussions and enables a deeper understanding and integration of the datasets than is possible with traditional tools. In addition to designing the XR model, the physical space was carefully designed to facilitate bringing together multiple users to discuss the data, through furniture supporting the use of laptops and other traditional tools, and additional monitors for data display (Fig. 2). A video demonstration of the model can be viewed in Supplementary Video 1.

Fig. 2: Digital and physical elements come together to enable interdisciplinary discussions.
figure 2

a Digital information accessible via Microsoft HoloLens MR headset, including patient model (within yellow circle at centre), model controls including time-slider and evolutionary tree (pink circle), and static overview graphics on the wall (green circle). b Physical elements enable best use of the model, including space to use traditional tools such as laptops and whiteboards, and ample space for members of the team to explore the data individually or together.

Full size image

This XR tool allowed us to better visualise the complex genomic information generated for this patient’s 42 metastatic tumours and the hypothetical evolutionary paths these tumours could have followed. While this information was described in the format of a traditional peer-reviewed journal paper recently8, the static two-dimensional figures that constrain traditional publications provide only limited opportunities to visualise and understand the anatomical and temporal relationships between tumours. However, by using XR, we can better understand the anatomical relationships between genomic clades with respect to tumour growth and spread (Supplementary Fig. 1), and visualise extensive tumour heterogeneity within large tumours where multiple samples were analysed (Supplementary Fig. 2).

Enhanced ‘asymmetric communication’ emerged during development

While our team built this model to meet an immediate need within our research, we realised there might be further applications for this work. We completed a formal qualitative evaluation of an early version of the model, where we invited eighteen cancer clinicians and scientists to interact with the model in pairs while being observed by the research team, and complete an interview on the experience (described in the supplement). We thematically analysed observations and interview responses, coding according to dominant themes (Supplementary Fig. 3). During this testing, clinicians spontaneously described how they would use it in an oncology clinic20. Through presentations at national and international medical conferences, and exhibiting at the international design and technology festival Ars Electronica, Linz, we could informally test hypotheses around how multi-modal cancer data can be integrated and interrogated by teams of people, and share our research with a wide audience, educating them about cancer, tumour evolution and metastasis in the process. The Ars Electronica audience included diverse technical backgrounds spanning engineering, science, medicine, and art, alongside non-specialists, yet their questions were similar, centred around applying this model to other patients and using a tool like this in clinical care. These interactions inspired our team to consider our project as a proof-of-concept for future medical XR tools and investigate potential areas where this collaborative model may add value.

To explore additional views, leaders in oncology, translational research and cancer biology joined the research group and spent two days immersed in the model; their impressions were gathered informally. They were tasked with exploring the model through different lenses, including research, clinical and educational applications. The remainder of this article describes the shared conclusions of this meeting.

The fundamental strength of the XR model was in its role as a communication tool, priming discussion between disciplines. When participants with different areas of expertise can work together on a problem, gains in understanding are made. The XR model provides a universal language and facilitates ‘asymmetric communication’, creating a level playing field when each contributor has a different specialist background.

Cancer data, or indeed any data, holds the most meaning when interpreted in ‘context’21,22,23; in this case, with respect to treatment, disease stage, genomic data from different tumour sites, and more. There are many emerging tools for visualising cancer genomic data24, including other endeavours to visualise cancer data in XR18. However, our approach is distinct as it is designed for use in multidisciplinary environments to achieve a greater understanding at the resolution of a single patient. The XR model provides a fast, powerful and efficient way to transmit complex information and facilitate communication around the results.

Our approach provided users with a holistic global perspective of the patient’s cancer, rather than a reductionist view. This, in turn, offers a visual representation of tumour stage, an important parameter of prognosis25. Users quickly grasp the natural history of these tumours, including sample relatedness and tumour evolution under treatment pressure. By placing genomic findings from single samples in their anatomical context, layered among other clinical data, interpreting tumour heterogeneity comes naturally.

Summarising datasets visually is a balancing act between retaining enough granularity or depth of information (particularly in the genomic data), and interpretability by a range of specialists. Some of our research team reasoned that the simplification of the data limited its use. However, the purpose is not to replace the subspecialist tools but to present the information in a way that can be interpreted by the non-specialist, to facilitate interdisciplinary communication. Therefore access to summarised data and specialised technical data is required in the model. Due to the augmented nature of the technology, XR is easily used in combination with existing subspecialist tools such as genome browsers and bioinformatic tools, as we do in our research group, or radiology browsers as we do in the clinic.

Advancing our understanding of tumour evolution, or improving cancer patient care, requires the coordinated knowledge of multidisciplinary teams, across the intersection of disciplines, each possessing a small piece of the puzzle—i.e., the concept of asymmetric communication. Beyond research, asymmetric communication scenarios that our approach is directly relevant to include multidisciplinary clinical decision-making, clinician-patient communication, and medical education, each explored below.

Multidisciplinary clinical decision-making

Multidisciplinary decision-making in a clinical environment shares commonalities with the research scenario we faced: teams of specialists must gather to integrate knowledge. For example, in a tumour board, such as the molecular tumour board (MTB) at our centre, this information covers treatment history, histopathology, radiology, and genomics (which may include samples from different tumours, different time points, and/or liquid biopsy samples)26. Integrating this data informs genomics-led care for improved patient outcomes27,28. The MTB is an advanced communication exercise at the intersection of many disciplines, serving to integrate domain knowledge held by different specialists. This task, by nature, must occur in teams; therefore, it needs tools optimised for use by teams. Overlaying genomic information at the sites of tumour samples within 3D tumour models encourages users to make connections between previous treatments, tumour growth patterns and genomic profiles. This approach helps to highlight potential genomic tumour heterogeneity and evolution, uncover truncal driver variants suitable for longitudinal monitoring and/or targeted therapy, and shape future treatment decisions29.

An XR tool integrating these layers of information enhances communication between the disciplines participating in an MTB and may act as a 'language bridge', ensuring content is accessible and easily digestible, and promoting a shared conversation where each participant plays an equal role. For example, non-clinician scientists often have little prior experience in interpreting CT scans, and many clinicians are not genomic specialists, so a visual reference to these datasets benefits both parties20. Even for those familiar with CT scans, XR has been shown to improve spatial understanding and time to synthesise information30, and we found that specialists would naturally guide others on information outside their speciality, aided by the model. Better communication produces informed treatment decisions integrating all information27, particularly important in complex cases brought to an MTB. Improving the efficiency of an MTB by bringing complex information together quickly helps to improve accessibility and scalability31, in turn helping to bring the MTB into routine care32. An immersive educational environment33,34, without the distractions present in other learning environments, helps to fulfil the educational mandate of the MTB and provides genomic information at a superficial level suitable for non-specialists. It was the shared inference of our research team that the XR model facilitated the full participation of all members of a multidisciplinary clinical team, helping to ensure that the most appropriate treatment decision would be reached. Participants remarked on the ease of inferring connections between treatment information (e.g., position of radiotherapy beam), tumour spread and genomic heterogeneity, for example (another example demonstrated in Supplementary Fig. 1). However, further studies would be needed to formally measure this. Finally, with XR it is possible to facilitate remote participation in an MTB34,35, particularly in centres lacking international expertise36 (however, we have not included this feature in our prototype). It may enable experts in any location to dial into a treatment meeting via HoloLens and participate in an immersive discussion with all patient data presented. Using XR to augment multidisciplinary clinical communication has the potential to benefit patients from many angles; through better clinical decision-making, upskilled clinicians and the ease of facilitating international expert collaboration.

Elevating clinician-patient communication

When sharing the XR model with our patient’s family, they noted a deep connection to the representation of their deceased family member and requested to spend time alone in the XR model as a family. They noted a greater understanding of their mother’s disease and wished that they could have used the model during her clinical care. This sentiment was shared by members of the public at the Ars Electronica festival. The clinicians in our team echoed this response, recognising that many patients and their families are eager to see, understand and follow their disease data, but noting that current tools (such as 2D computer software for viewing radiological data) are not optimised for easy interpretation by non-specialists. One used a poignant analogy; “the ocean swimmer is more scared of sharks than the scuba diver.” We should give patients the chance to go ‘underwater’ and see with clarity what is going on in their disease, via tools optimised for interpretation by non-specialists. Representing tumours identified in medical imaging in 3D, as opposed to their native grey and black 2D format, may provide information to a patient and their family in a more accessible form. Past studies have highlighted that while patients generally gain a stronger understanding of their disease from seeing their own CT scans37, the use of 3D images/models may aid patients in forming a detailed and accurate understanding of their disease, information recall and trust in their diagnosis38. This may have important flow-on benefits, such as revolutionising consent to surgical procedures through an increased understanding of tumour positioning and surgical consequences39,40,41, and may work towards the democratisation of data, where patients can access their own medical data in a non-specialist format.

While increasing patient understanding may have many benefits, we note that not all patients would want to confront their disease in immersive XR, a point raised during the evaluation of an early prototype with clinicians20. Not all people with cancer have the same psychological coping style, with a spectrum from ‘monitoring’ to ‘blunting’42. Those identifying with ‘monitoring’ typically want all the information about their disease that they can get, whereas those identifying with ‘blunting’ are more likely to avoid this information. It is recognised that patients fare better (both psychologically and physiologically) when the information they receive about their disease is tailored to their psychological coping style42—something clinicians would assess before showing an XR model to their patients.

Medical and scientific education

Whereas the complexities of using XR technologies for patient communication may be solved long-term (via new software development pipelines), we see education as an immediate opportunity for XR technologies. In biomedical science education, grasping the clinical context is difficult for students without direct clinical exposure, and in the clinic, further genomics education is needed. Without any modification, our XR model may facilitate transdisciplinary education to better equip scientists to solve clinical challenges and lift genomics literacy in the clinic, bridging these educational silos. When we conducted a formal qualitative evaluation of an early prototype of our XR tool, many participants (irrespective of career seniority or field of expertise) came away with a greater appreciation for tumour heterogeneity and evolution, the form that metastatic cancer can take, the limitations of assaying single biopsies, and the value of circulating tumour DNA for capturing tumour heterogeneity20. This is critical understanding for interpreting single tumour biopsies in the absence of additional information. Participants also noted a greater appreciation for the patient face of biomedical research, for the selfless person behind the data. The deep individual focus on a single patient in this XR model mirrors the personalised focus during ideal clinical care. Further investigation is needed on the effectiveness of immersive virtual environments in educational scenarios, as some but not all previous studies have found a benefit; however, noting that past studies finding no significant educational benefit to the immersive technology have focussed on virtual rather than augmented technologies43,44.

Pitfalls, perils and possibilities

There are very few examples of new health technology reducing inequity45,46. An XR model, therefore, requires careful implementation in the right clinical scenario to protect against this substantial risk, and development with diverse input, including expertise from Indigenous people. In addition to asserting rights-based (UNDRIP Article 31) Indigenous oversight and stewardship of data belonging to Indigenous patients and whānau incorporated into XR environments, Indigenous knowledge is best placed to shape processes involving such data (including images) in culturally appropriate ways47. We suggest that adopters of this XR concept globally should engage early with Indigenous experts local to the implementation site. Furthermore, implementation must prioritise a clinical scenario where this technology has the greatest potential to benefit patients; where care is multidisciplinary, provided by a network of clinicians, where disease is complex and metastatic, and where genomics might add value. In most clinical cases currently, at our MTB for example, genomic sequencing is available on only one or a small number of tumours and/or blood plasma samples27, however, an XR model still provides an opportunity to integrate this data with serial medical imaging and other clinical information.

Any new clinical tools must be optimised for a time-poor clinical system, noting the learning curve and time to use an XR model in existing clinical workstreams. A major barrier for adoption, particularly in resource-stretched health systems like New Zealand’s, is the cost of the XR technology and its implementation. Currently available XR headsets have several technical limitations, including those related to ergonomics, display resolution and field of view, and processing power. HoloLenses or other XR headsets are not currently used in most hospitals, although this will change as the technology improves and becomes mainstream. The COVID-19 pandemic facilitated the widespread adoption of digital technology in healthcare48—“a crisis provides an opportunity, and this crisis … provides a great opportunity for digital technology”49. Policy supporting the development and adoption of digital healthcare technologies has facilitated the uptake of emerging digital technologies internationally48,50,51. This can already be seen with the emergence of virtual or extended reality tools for surgical planning12,15,52,53 and medical education17,54,55,56,57, but XR tools have not yet been created for multidisciplinary decision-making or cancer patient consultation. Future tools incorporating the learnings of this proof-of-principle project, alongside technical and operational refinements, will allow practical implementation in appropriate clinical and educational environments. Critical will be usability refinements fit for clinical utility. These may include new ways to record discussions and outcomes occurring in XR (including video recording and integrated note-taking), speciality-dependent data views, incorporating domain analytical and decision support tools including AI tools, and integrating other data types such as histopathology58. These will ensure that the benefits of the XR approach are realised in the real world.

Conclusion

Genomics is transforming our understanding of cancer. To ensure these benefits are realised for people with cancer, we require new tools that meet the needs of multidisciplinary clinical teams, people with cancer, and the future genomic workforce. Here, we have shown the opportunities for a novel XR tool bringing together tumour-specific data in the context of medical imaging, creating a level playing field for discussing complex cancer data. We offer a window into future possibilities for meeting the interpretation and communication needs of collaborative cancer teams.

We are indebted to the patient at the heart of this project and her family. Her selfless donation instigated and encouraged this research.

Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.