Abstract
High-dimensional multiplexed imaging can reveal the spatial organization of tumour tissues at the molecular level. However, owing to the scale and information complexity of the imaging data, it is challenging to discover and thoroughly characterize the heterogeneity of tumour microenvironments. Here we show that self-supervised representation learning on data from imaging mass cytometry can be leveraged to distinguish morphological differences in tumour microenvironments and to precisely characterize distinct microenvironment signatures. We used self-supervised masked image modelling to train a vision transformer that directly takes high-dimensional multiplexed mass-cytometry images. In contrast with traditional spatial analyses relying on cellular segmentation, the vision transformer is segmentation-free, uses pixel-level information, and retains information on the local morphology and biomarker distribution. By applying the vision transformer to a lung-tumour dataset, we identified and validated a monocytic signature that is associated with poor prognosis.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$32.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The IMC images, cell-segmentation and patient metadata used in the study were public data available on Zenodo at https://zenodo.org/record/7760826. CANVAS model weights, single-cell-monocyte data, qPCR data, and the cell-type composition from NanoString GeoMx data are available on Zenodo at https://zenodo.org/records/14111081 (ref. 52).
Code availability
The code used for CANVAS training and downstream analysis is available on GitHub at https://github.com/tanjimin/CANVAS (ref. 53).
References
Chen, D. S. & Mellman, I. Elements of cancer immunity and the cancer–immune set point. Nature 541, 321–330 (2017).
Hanahan, D. & Weinberg, R. A. The hallmarks of cancer. Cell 100, 57–70 (2000).
Junttila, M. R. & de Sauvage, F. J. Influence of tumour micro-environment heterogeneity on therapeutic response. Nature 501, 346–354 (2013).
Lomakin, A. et al. Spatial genomics maps the structure, nature and evolution of cancer clones. Nature 611, 594–602 (2022).
Schadendorf, D. et al. Pooled analysis of long-term survival data from phase II and phase III trials of ipilimumab in unresectable or metastatic melanoma. J. Clin. Oncol. 33, 1889–1894 (2015).
Reck, M. et al. Pembrolizumab versus chemotherapy for PD-L1-positive non-small-cell lung cancer. N. Engl. J. Med. 375, 1823–1833 (2016).
Hodi, F. S. et al. Combined nivolumab and ipilimumab versus ipilimumab alone in patients with advanced melanoma: 2-year overall survival outcomes in a multicentre, randomised, controlled, phase 2 trial. Lancet Oncol. 17, 1558–1568 (2016).
Hossein, B. et al. Nivolumab versus docetaxel in advanced nonsquamous non-small-cell lung cancer. N. Engl. J. Med. 373, 1627–1639 (2015).
Sharma, P. et al. Immune checkpoint therapy—current perspectives and future directions. Cell 186, 1652–1669 (2023).
Ståhl, P. L. et al. Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science 353, 78–82 (2016).
Chen, K. H., Boettiger, A. N., Moffitt, J. R., Wang, S. & Zhuang, X. Spatially resolved, highly multiplexed RNA profiling in single cells. Science 348, aaa6090 (2015).
Eng, C.-H. L. et al. Transcriptome-scale super-resolved imaging in tissues by RNA seqFISH+. Nature 568, 235–239 (2019).
Stickels, R. R. et al. Highly sensitive spatial transcriptomics at near-cellular resolution with Slide-seqV2. Nat. Biotechnol. 39, 313–319 (2021).
Bandura, D. R. et al. Mass cytometry: technique for real time single cell multitarget immunoassay based on inductively coupled plasma time-of-flight mass spectrometry. Anal. Chem. 81, 6813–6822 (2009).
Giesen, C. et al. Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry. Nat. Methods 11, 417–422 (2014).
Gerdes, M. J. et al. Highly multiplexed single-cell analysis of formalin-fixed, paraffin-embedded cancer tissue. Proc. Natl Acad. Sci. USA 110, 11982–11987 (2013).
Goltsev, Y. et al. Deep profiling of mouse splenic architecture with CODEX multiplexed imaging. Cell 174, 968–981.e15 (2018).
Rao, A., Barkley, D., França, G. S. & Yanai, I. Exploring tissue architecture using spatial transcriptomics. Nature 596, 211–220 (2021).
Fischer, D. S., Schaar, A. C. & Theis, F. J. Modeling intercellular communication in tissues using spatial graphs of cells. Nat. Biotechnol. 41, 332–336 (2023).
Brbić, M. et al. Annotation of spatially resolved single-cell data with STELLAR. Nat. Methods 19, 1411–1418 (2022).
Dong, K. & Zhang, S. Deciphering spatial domains from spatially resolved transcriptomics with an adaptive graph attention auto-encoder. Nat. Commun. 13, 1739 (2022).
Varrone, M., Tavernari, D., Santamaria-Martínez, A. & Ciriello, G. CellCharter reveals spatial cell niches associated with tissue remodeling and cell plasticity. Nat. Genet. 56, 74–84 (2024).
Kim, J. et al. Unsupervised discovery of tissue architecture in multiplexed imaging. Nat. Methods 19, 1653–1661 (2022).
Wu, Z. et al. Graph deep learning for the characterization of tumour microenvironments from spatial protein profiles in tissue specimens. Nat. Biomed. Eng 6, 1435–1448 (2022).
Calderaro, J. et al. Histological subtypes of hepatocellular carcinoma are related to gene mutations and molecular tumour classification. J. Hepatol. 67, 727–738 (2017).
Wu, H.-J. et al. Spatial intra-tumor heterogeneity is associated with survival of lung adenocarcinoma patients. Cell Genom. 2, 100165 (2022).
Sorin, M. et al. Single-cell spatial landscapes of the lung tumour immune microenvironment. Nature 614, 548–554 (2023).
Vaswani, A. Attention is all you need. In Advances in Neural Information Processing Systems 30 (NIPS, 2017).
Dosovitskiy, A. et al. An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations (ICLR, 2021).
He, Kaiming, et al. Masked autoencoders are scalable vision learners. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 15979–15988 (IEEE, 2022).
Schumacher, T. N. & Thommen, D. S. Tertiary lymphoid structures in cancer. Science 375, eabf9419 (2022).
Sautès-Fridman, C., Petitprez, F., Calderaro, J. & Fridman, W. H. Tertiary lymphoid structures in the era of cancer immunotherapy. Nat. Rev. Cancer 19, 307–325 (2019).
Ng, K. W. et al. Antibodies against endogenous retroviruses promote lung cancer immunotherapy. Nature 616, 563–573 (2023).
Cabrita, R. et al. Tertiary lymphoid structures improve immunotherapy and survival in melanoma. Nature 577, 561–565 (2020).
Helmink, B. A. et al. B cells and tertiary lymphoid structures promote immunotherapy response. Nature 577, 549–555 (2020).
Hollern, D. P. et al. B cells and T follicular helper cells mediate response to checkpoint inhibitors in high mutation burden mouse models of breast cancer. Cell 179, 1191–1206.e21 (2019).
Anagnostou, V. et al. Evolution of neoantigen landscape during immune checkpoint blockade in non-small cell lung cancer. Cancer Discov. 7, 264–276 (2017).
Garon, E. B. et al. Pembrolizumab for the treatment of non-small-cell lung cancer. N. Engl. J. Med. 372, 2018–2028 (2015).
Tang, K. H. et al. Combined inhibition of SHP2 and CXCR1/2 promotes antitumor T-cell response in NSCLC. Cancer Discov. 12, 47–61 (2021).
Ng, A., Jordan, M. & Weiss, Y. On spectral clustering: analysis and an algorithm. In Advances in Neural Information Processing Systems Vol. 14 (eds Dietterich, T. et al.) 849–856 (MIT Press, 2001).
Chu, T., Wang, Z., Pe’er, D. & Danko, C. G. Cell type and gene expression deconvolution with BayesPrism enables Bayesian integrative analysis across bulk and single-cell RNA sequencing in oncology. Nat. Cancer 3, 505–517 (2022).
Li, F. et al. Pulmonary fibrosis in patients with COVID-19: a retrospective study. Front. Cell. Infect. Microbiol. 12, 1013526 (2022).
Li, X. et al. Pulmonary fibrosis and its related factors in discharged patients with new corona virus pneumonia: a cohort study. Respir. Res. 22, 203 (2021).
Buechler, M. B., Fu, W. & Turley, S. J. Fibroblast-macrophage reciprocal interactions in health, fibrosis, and cancer. Immunity 54, 903–915 (2021).
Gunaydin, G. CAFs interacting with TAMS in tumor microenvironment to enhance tumorigenesis and immune evasion. Front. Oncol. 11, 668349 (2021).
Lakshmanan, L. V. S. et al. Efficient aggregation for graph summarization. In Proc. 2008 ACM SIGMOD International Conference on Management of Data 567–580 (ACM, 2008).
Kim, N. et al. Single-cell RNA sequencing demonstrates the molecular and cellular reprogramming of metastatic lung adenocarcinoma. Nat. Commun. 11, 2285 (2020).
Aran, D. et al. Reference-based analysis of lung single-cell sequencing reveals a transitional profibrotic macrophage. Nat. Immunol. 20, 163–172 (2019).
Stuart, T. et al. Comprehensive integration of single-cell data. Cell 177, 1888–1902.e21 (2019).
Wolf, F. A., Angerer, P. & Theis, F. J. SCANPY: large-scale single-cell gene expression data analysis. Genome Biol. 19, 15 (2018).
Korsunsky, I. et al. Fast, sensitive and accurate integration of single-cell data with Harmony. Nat. Methods 16, 1289–1296 (2019).
Tan, J. Characterization of tumour heterogeneity through segmentation-free representation learning on multiplexed imaging data [Data set]. Zenodo https://doi.org/10.5281/zenodo.14111081 (2024).
Tan, J. tanjimin/CANVAS: First release. Zenodo https://doi.org/10.5281/zenodo.14835248 (2024).
Acknowledgements
This work used computing resources at the NYU School of Medicine High Performance Computing (HPC) Facility. We thank A. W. Lund and T. Y. Papagiannakopoulos for discussion and feedback on LTMEs; A. Heguy and the Genome Technology Center (GTC) for the support on expert library preparation and sequencing; and H. I. Pass for the assistance on lung-tumour data collection. J.T. and D.F disclose support for the research described in this study from P01AG051449 and P01CA288368. A.T. discloses support for the research described in this study from Cancer Center Support Grant P30CA016087 at the Laura and Isaac Perlmutter Cancer Center. Experimental Pathology (RRID:SCR_017928) is partly funded by NIH/NCI 5 P30CA16087.
Author information
Authors and Affiliations
Contributions
J.T. and D.F. conceived the project. J.T., A.T. and D.F. designed the experiments. J.T. designed and implemented the model and framework, and performed downstream analysis with inputs from B.X., Y.B., W.L., J.M.W., M.H. and K.C.; H.I.P., S.R., V.M., C.L., H.L. and N.M. performed GeoMx WTA experiments and analysed the data. Y.B. performed mouse single-cell experiments. Y.H. performed mouse single-cell analysis. Y.B. and Y.L. designed and performed the monocyte experiment. J.D., K.-K.W., Y.B., J.T., A.L.M., B.G.N., A.T. and D.F. interpreted results. J.T. prepared figures with input from H.L., B.X., A.T. and D.F.; J.T., J.D., B.G.N., A.T. and D.F. wrote the manuscript with inputs from all authors. A.T. and D.F. contributed equally.
Corresponding authors
Ethics declarations
Competing interests
A.T. is a scientific advisor to Intelligencia AI. J.T. is co-founder of Morphology Med. D.F. is co-founder of Morphology Med, The Informatics Factory and Bacchus Venture Capital, and he serves on the scientific advisory boards or consults for Spectragen Informatics, Protein Metrics, Proteome Software, Preverna and Protai.
Peer review
Peer review information
Nature Biomedical Engineering thanks Jinman Kim and Faisal Mahmood for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 Validation of the monocytic signature.
a, Schematic of WTA data processing pipeline. b, Cell type composition across biopsies. c, Elbow plot of biopsies ranked by cosine similarity to signature C10, colored by survival. d, Survival analysis of C10-similar patients vs other patients. e, Cell level enrichment score comparison between naïve lung and tumors. Error bars in violin plots indicate minimum, mean, and maximum values within each group. P-value is calculated by Independent Samples T-test. f, Single cell analysis on tumor and normal monocytes. Distribution of monocytes in tumor and naive mouse lung (left). Extracellular Matrix (ECM) pathway gene expression in naive (middle) and tumor (right) lung, colored by enrichment score. g, Schematic of in vitro monocyte co-culture experiments. h, Log2 fold-change of ECM-related gene expression of the treatment group (24 and 48 hours) compared with the control group.
Supplementary information
Supplementary Information
Supplementary figures and tables, and mapping of source data for figures.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Tan, J., Le, H., Deng, J. et al. Characterization of tumour heterogeneity through segmentation-free representation learning on multiplexed imaging data. Nat. Biomed. Eng 9, 405–419 (2025). https://doi.org/10.1038/s41551-025-01348-1
Received:
Accepted:
Published:
Issue date:
DOI: https://doi.org/10.1038/s41551-025-01348-1
