Table 2 Summary of results for the cell identification challenges

From: Critical assessment of automated flow cytometry data analysis techniques

F-measurea

  
 

GvHD

DLBCL

HSCT

WNV

ND

Mean

Runtime h:mm:ssb

Rank scorec

Challenge 1: completely automated

ADICyt

0.81 (0.72, 0.88)

0.93 (0.91, 0.95)

0.93 (0.90, 0.96)

0.86 (0.84, 0.87)

0.92 (0.92, 0.93)

0.89

4:50:37

52

flowMeans

0.88 (0.82, 0.93)

0.92 (0.89, 0.95)

0.92 (0.90, 0.94)

0.88 (0.86, 0.90)

0.85 (0.76, 0.92)

0.89

0:02:18

49

FLOCK

0.84 (0.76, 0.90)

0.88 (0.85, 0.91)

0.86 (0.83, 0.89)

0.83 (0.80, 0.86)

0.91 (0.89, 0.92)

0.86

0:00:20

45

FLAME

0.85 (0.77, 0.91)

0.91 (0.88, 0.93)

0.94 (0.92, 0.95)

0.80 (0.76, 0.84)

0.90 (0.89, 0.90)

0.88

0:04:20

44

SamSPECTRAL

0.87 (0.81, 0.93)

0.86 (0.82, 0.90)

0.85 (0.82, 0.88)

0.75 (0.60, 0.85)

0.92 (0.92, 0.93)

0.85

0:03:51

39

MMPCA

0.84 (0.74, 0.93)

0.85 (0.82, 0.88)

0.91 (0.88, 0.94)

0.64 (0.51, 0.71)

0.76 (0.75, 0.77)

0.80

0:00:03

29

FlowVB

0.85 (0.79, 0.91)

0.87 (0.85, 0.90)

0.75 (0.70, 0.79)

0.81 (0.78, 0.83)

0.85 (0.84, 0.86)

0.82

0:38:49

28

MM

0.83 (0.74, 0.91)

0.90 (0.87, 0.92)

0.73 (0.66, 0.80)

0.69 (0.60, 0.75)

0.75 (0.74, 0.76)

0.78

0:00:10

28

flowClust/Merge

0.69 (0.55, 0.79)

0.84 (0.81, 0.86)

0.81 (0.77, 0.85)

0.77 (0.74, 0.79)

0.73 (0.58, 0.85)

0.77

2:12:00

24

L2kmeans

0.64 (0.57, 0.72)

0.79 (0.74, 0.83)

0.70 (0.65, 0.75)

0.78 (0.75, 0.81)

0.81 (0.80, 0.82)

0.74

0:08:03

20

CDP

0.52 (0.46, 0.58)

0.87 (0.85, 0.90)

0.50 (0.48, 0.52)

0.71 (0.68, 0.75)

0.88 (0.86, 0.90)

0.70

0:00:57

19

SWIFT

0.63 (0.56, 0.70)

0.67 (0.62, 0.71)

0.59 (0.55, 0.62)

0.69 (0.64, 0.74)

0.87 (0.86, 0.88)

0.69

1:14:50

15

Ensemble clustering

0.88

0.94

0.97

0.88

0.94

0.92

–

64

Challenge 2: manually tuned

ADICyt

0.81 (0.71, 0.89)

0.93 (0.91, 0.95)

0.93 (0.90, 0.96)

0.86 (0.84, 0.87)

0.92 (0.92, 0.93)

0.89

4:50:37

34

SamSPECTRAL

0.87 (0.79, 0.94)

0.92 (0.89, 0.94)

0.90 (0.86, 0.93)

0.85 (0.83, 0.88)

0.91 (0.91, 0.92)

0.89

0:06:47

31

FLOCK

0.84 (0.76, 0.90)

0.88 (0.85, 0.91)

0.86 (0.83, 0.89)

0.84 (0.82, 0.86)

0.89 (0.87, 0.91)

0.86

0:00:15

23

FLAME

0.81 (0.75, 0.87)

0.87 (0.84, 0.90)

0.87 (0.82, 0.90)

0.84 (0.83, 0.85)

0.87 (0.86, 0.87)

0.85

0:04:20

23

SamSPECTRAL-FK

0.87 (0.80, 0.94)

0.85 (0.81, 0.89)

0.90 (0.86, 0.92)

0.76 (0.71, 0.81)

0.92 (0.91, 0.93)

0.86

0:04:25

23

CDP

0.74 (0.67, 0.80)

0.89 (0.86, 0.91)

0.90 (0.88, 0.92)

0.75 (0.71, 0.78)

0.86 (0.85, 0.88)

0.83

0:00:18

19

flowClust/Merge

0.69 (0.53, 0.78)

0.87 (0.85, 0.90)

0.96 (0.94, 0.97)

0.77 (0.75, 0.79)

0.88 (0.81, 0.91)

0.83

2:12:00

18

NMFcurvHDR

0.76 (0.69, 0.82)

0.84 (0.83, 0.86)

0.70 (0.67, 0.74)

0.81 (0.77, 0.84)

0.83 (0.83, 0.84)

0.79

1:39:42

13

Ensemble clustering

0.87

0.94

0.98

0.87

0.92

0.91

–

41

Challenge 3: assignment of cells to populations with predefined number of populations

ADICyt

0.91 (0.84, 0.96)

0.96 (0.94, 0.97)

0.98 (0.97, 0.99)

  

0.95

0:10:49

26.2

SamSPECTRAL

0.85 (0.75, 0.93)

0.93 (0.91, 0.95)

0.97 (0.95, 0.98)

  

0.92

0:02:30

26.2

flowMeans

0.91 (0.84, 0.96)

0.94 (0.91, 0.96)

0.95 (0.93, 0.96)

  

0.93

0:00:01

23.4

TCLUST

0.93 (0.91, 0.96)

0.93 (0.91, 0.95)

0.93 (0.90, 0.95)

  

0.93

0:00:40

23.4

FLOCK

0.86 (0.79, 0.93)

0.92 (0.89, 0.94)

0.97 (0.95, 0.98)

  

0.92

0:00:02

22.2

CDP

0.85 (0.77, 0.92)

0.92 (0.89, 0.94)

0.76 (0.72, 0.81)

  

0.84

0:00:21

16.9

flowClust/Merge

0.88 (0.82, 0.93)

0.90 (0.86, 0.94)

0.83 (0.79, 0.88)

  

0.87

0:49:24

15.9

FLAME

0.85 (0.79, 0.91)

0.90 (0.86, 0.93)

0.86 (0.82, 0.91)

  

0.87

0:03:20

15.9

SWIFT

0.90 (0.84, 0.95)

0.00 (0.00, 0.00)

0.88 (0.84, 0.92)

  

0.59

0:01:37

11.9

flowKoh

0.85 (0.80, 0.90)

0.85 (0.82, 0.88)

0.87 (0.84, 0.91)

  

0.86

0:00:42

9.5

NMF

0.74 (0.69, 0.78)

0.84 (0.80, 0.88)

0.80 (0.76, 0.84)

  

0.79

0:01:00

7.5

Ensemble clustering

0.95

0.97

0.98

  

0.97

–

35

Challenge 4: supervised approaches trained using human-provided gates

RadialSVM

0.89 (0.83, 0.95)

0.84 (0.80, 0.87)

0.98 (0.96, 0.99)

0.96 (0.94, 0.97)

0.93 (0.92, 0.94)

0.92

0:00:18

21

flowClust/Merge

0.92 (0.88, 0.95)

0.92 (0.89, 0.94)

0.95 (0.92, 0.97)

0.84 (0.82, 0.86)

0.89 (0.88, 0.90)

0.90

5:31:50

19

randomForests

0.85 (0.78, 0.91)

0.78 (0.74, 0.83)

0.81 (0.79, 0.83)

0.87 (0.84, 0.90)

0.94 (0.92, 0.95)

0.85

0:02:06

15

FLOCK

0.82 (0.77, 0.87)

0.91 (0.89, 0.93)

0.86 (0.76, 0.93)

0.86 (0.82, 0.89)

0.86 (0.77, 0.92)

0.86

0:00:05

13

CDP

0.78 (0.68, 0.87)

0.95 (0.93, 0.97)

0.75 (0.71, 0.78)

0.86 (0.84, 0.88)

0.83 (0.80, 0.86)

0.83

0:00:15

11

Ensemble clustering

0.91

0.94

0.95

0.92

0.94

0.93

–

26

  1. aIn each data set/challenge, the top algorithm (highest mean F-measure) and the algorithms with overlapping confidence intervals with the top algorithm are boldface (see Online Methods for F-measure calculations).
  2. bRun time was calculated as time per CPU per sample.
  3. cAlgorithms are sorted by rank score within each challenge (see Online Methods for rank score calculations). Data sets: GvHD, graft-versus-host disease; DLBCL, diffuse large B-cell lymphoma; WNV, symptomatic West Nile virus; ND, normal donors; HSCT, hematopoietic stem cell transplant.
Back to article page