Fig. 2
From: Using transcripts to refine image based cell segmentation with FastReseg
Detection of cells with putative segmentation errors based on local transcriptional profiles in both expression and physical space. (A) Flowchart of how FastReseg identifies cells with potential segmentation errors. Cluster-specific reference profiles are utilized to calculate the centered log-likelihood ratio (tLLR score) for each gene under each reference cell cluster/type, where tLLR (i, k) is derived by subtracting the maximum log likelihood of gene i across all clusters from its log likelihood under cluster k. A linear regression model is then trained to simulate the spatial pattern of each cell’s transcriptional score under its best-fit cluster given the overall expression profiles of query cell observed in current cell borders. FastReseg would further evaluate the degree of spatial dependency of the transcript score pattern of each query cell via log-likelihood ratio test between the spatial-dependent model and the invariant null model. Cells with high spatial dependency are flagged as cells with putative segmentation errors. (B) Heatmap of transcript tLLR score for top marker genes (rows) across different reference cell types (columns). The reference cluster-specific profiles (see “example_refProfiles” object inside the package) were derived from an example spatial transcriptomics dataset for melanoma tissue section whose cell types were assigned based on semi-supervised cell typing algorithm with novel clusters using “a” to “f” letters as names. The color scale, matching (C), illustrates the range from good to bad fit, highlighting the variability of score for classic marker genes in different reference cell types. (C) XY scatter plots of transcript tLLR scores within example cells with varying likelihood of segmentation error, as determined by the degree of spatial dependency (defined as –log10(p), and annotated on top of each cell) of their tLLR patterns under their most probable cell types. Cells with spatial dependency scores higher than the provided cutoff (default to 5, shown as the vertical line above) are flagged as improperly segmented cells.
