Fig. 5

Performance evaluation of FastReseg on an example kidney dataset. (A) Bar plots on the composition of actions taken at cell level throughout the FastReseg workflow. While majority of cells (98.24%) with misassigned transcript groups identified by SVM modeling have received trimming during the transcript refinement stage, the bar plot at the center highlights the different refinement actions applied to the original host cells on top of trimming some transcripts to extracellular space, including merging those misassigned transcripts to neighboring cells (“merged”), designating them as new cells (“split_to_new”), or returning back to the host cell (“none”, n = 2 cells). Color legend of “with_trim” and “no_trim” indicates whether the host cells received more than one refinement actions. (B) Bar plots on the breakdown of operations conducted at the transcript level, illustrating the extent of transcript reassignment across the dataset. (C) Scatter plots of transcript number of each gene for each cell type. Only cells that received transcript trimming are included in this before-vs-removed comparison to show the impact of FastReseg refinement on original host cells under different cell types. The cell number included in each subplot is noted in the subtitle next to the cell type of interest. Dots for each gene are colored based on whether they are on-target or off-target marker genes for the cell type of interest. Genes that are not considered as mutually exclusive markers are colored in gray as “non-marker”. See Methods section (Table 1) for the mutually exclusive marker genes used in this analysis. The solid, dotted and dashed lines indicate trimming at different percentages (0.1%, 1%, 10%). (D) Runtime and (E) peak memory usage of two FastReseg pipelines (flag-only and full-pipeline) along with other transcript-based cell segmentation methods (Proseg and Baysor) on datasets containing different transcript numbers. The statistics reported are based on processing using 75% cores of an Amazon r5b.4xlarge instance (16 vCPU, 128GB memory) unless otherwise noted in Methods section. The input spatial datasets were subsets of the CosMx kidney dataset either with different numbers of fields of view (Table 2) or from different tissue sections (Table 3), each containing varying cell numbers in their original cell segmentation.