Fig. 1 | Scientific Reports

Fig. 1

From: Using transcripts to refine image based cell segmentation with FastReseg

Fig. 1

FastReseg, an algorithm for improved cell segmentation in spatial transcriptomics. (A) Visualization of typical spatial transcriptomics data showing morphology stains for nucleus and membrane alongside transcript distribution (dots colored by gene identities), providing context for segmentation. (B) Common challenges in image-based cell segmentation, including limited resolution to resolve overlapping cells in the z-axis, weak or incomplete membrane stain in real-world tissue samples (yellow arrows), and lateral spillage of optical signals in densely packed tissues. Example images in (A) & (B) are from a melanoma dataset, where morphology is visualized using antibodies against protein markers (CD298: Blue, PanCK: Green, CD45: Red) and nuclear stain (DAPI: gray). (C) Spatial pattern of transcriptional scores (right) based on each gene’s log-likelihood expression under reference cell types (Epithelial, NK, Mast cells) could help identify the presence of different cell types spatially, while validated by the orthogonal antibody staining (left, CD298: blue, CD45: red, DAPI: gray). Different color hues and intensities of the dots indicate the goodness-of-fit of each transcript (dot) under a given reference cell type as shown in the color bar, ranging from good (blue) to poor (red/yellow). See Methods section for calculation of tLLR scores plotted here. (D) Overview of the FastReseg workflow. Starting with initial cell segmentation and cluster-specific reference expression profiles, FastReseg scores each transcript based on its goodness-of-fit with respect to most probable cell type given the expression profiles of corresponding host cells and then flags cells with high spatial dependency in the spatial pattern of their transcript scores as having putative segmentation errors (i). Within the flagged cells, FastReseg can further identify the transcripts with poor fit, segregate them into spatially distinct groups and flag them as contaminating transcripts from different neighboring cells (ii). In the correction phase (iii), FastReseg evaluates each flagged transcript group’s expression and physical spatial context, deciding refinement actions such as trimming transcripts to extracellular space, merging with neighboring cells, or reassigning them to newly created cells based on a set of heuristic rules, which aims to resolve cell segmentation errors and improve transcript assignment accuracy. See Fig. 4 and Methods section for the detailed process.

Back to article page