Table 2 Test outcomes of standalone and assistive DL systems.

From: Diagnostic performance of deep learning in ultrasound diagnosis of breast cancer: a systematic review

Study

Index test/comparator

AUC (95% CI)

PΔAUC

%Acc (95% CI)

PΔAcc

%Sen (95% CI)

PΔSen

%Spe (95% CI)

PΔSpec

Standalone DL system

Kim 202118

DL

0.575

 

NR

 

30

 

84.9

 

Reader 1

0.545

NR

NR

NR

100

NR

8.9

NR

Reader 2

0.541

NR

NR

NR

100

NR

8.2

NR

Reader 3

0.545

NR

NR

NR

100

NR

8.9

NR

Xiao 201919

DL

0.81 (0.77–0.85)

 

NR

 

85.32 (79.91–89.74)

 

76.96 (70.97–82.24)

 

Less experienced reader

0.7 (0.65–0.74)

<0.0001

NR

NR

92.2 (87.81–95.39)

<0.05

46.96 (40.37–53.63)

<0.05

Experienced reader

0.81(0.77–0.84)

NS

NR

NR

98.62 (96.03–99.72)

<0.05

63.04 (56.45–69.29)

<0.05

Cho 201820

DL

0.815 (0.745–0.885)

 

82.4 (75.5–89.2)

 

72.2 (60.3–84.2)

 

90.8 (83.7–97.8)

 

Less experienced reader

0.901 (0.846–0.956)

0.004

73.1 (65.1–81.1)

0.06

94.4 (88.3–100.0)

<0.001

55.4 (43.3–67.5)

<0.001

Experienced reader

0.887 (0.826–0.947)

0.023

69.8 (61.5–78.0)

0.014

94.4 (88.3–100.0)

<0.001

49.2 (37.1–61.4)

<0.001

Segni 201822

DL

0.82 (0.71–0.91)

 

NR

 

91.1 (78.8– 97.5)

 

70.8 (48.9– 87.4)

 

Less experienced reader

1

0.76 (0.66–0.86)

0

NR

NR

97.7 (88–99.9)

NR

54.2 (32.8–74.4)

NR

2

0.83 (0.73–0.93)

0.831

NR

NR

95.5 (84.5–99.4)

NR

70.8 (48.9–87.4)

NR

3

0.74 (0.63–0.84)

0.151

NR

NR

97.8 (88.2–99.9)

NR

50 (29.1–70.9)

NR

4

0.75 (0.65–0.85)

0.206

NR

NR

100 (92–100)

NR

50 (29.1–70.9)

NR

Experienced reader

0.84 (0.74 –0.94)

0.751

NR

NR

93.2 (81.3– 98.6)

NR

75.0 (53.3–90.2)

NR

Xia 202123

DL

0.948

 

89.6

 

95.8

 

93.8

 

Reader

0.719

NR

43.8

NR

75

NR

68.8

NR

Lee 202224

DL

0.855 (0.825‒0.886)

 

85.4 (82.2‒88.1)

 

86.1 (80.7‒90.1)

 

84.9 (80.6‒88.4)

 

Lee 202224

Reader

0.895 (0.854‒0.936)

0.05

72.4 (69.1‒75.4)

<0.001

95.4 (93.0‒97.0)

<0.001

56.6 (52.2‒60.8)

<0.001

Choi 201925

DL

NR

 

92.1

 

85

 

95.4

 

Less experienced reader

1

NR

NR

79.4

NR

88.8

NR

75.1

NR

2

NR

NR

88.9

NR

81.3

NR

92.5

NR

Experienced reader

1

NR

NR

77.9

NR

88.8

NR

72.8

NR

2

NR

NR

84.2

NR

86.3

NR

83.2

NR

Nicosia 202226

DL

NR

 

NR

 

85.2

 

79.8

 

Less experienced reader

1

NR

NR

NR

NR

75.4

<0.001

68.4

0.001

2

NR

NR

NR

NR

75.4

<0.001

65.8

<0.001

Experienced reader

1

NR

NR

NR

NR

94.4

<0.001

86.8

0.08

2

NR

NR

NR

NR

95.8

<0.001

85.1

0.34

Lai 202227

DL

0.8591

 

94.77

 

96.92

 

55.14

 

Average readers

0.7582 (0.7014–0.8151)

NR

NR

NR

95.77 (90.88– 100.66)

NR

24.07 (15.97–32.17)

NR

Lee 201928

DL

0.73 (0.67–0.78)

 

NR

 

76 (65–86)

 

69 (65–74)

 

Less experienced reader

0.65 (0.58–0.71)

0.013

NR

NR

59 (46–71)

0.007

70 (66–75)

0.71

DL

0.79 (0.74–0.84)

 

NR

 

81 (70–89)

 

77 (73–81)

 

Experienced reader

0.83 (0.8–0.86)

0.101

NR

NR

97 (90–100)

<0.001

70 (65–74)

0.004

Wei 202129

DL

0.874

 

88.3

 

85.5

 

89.3

 

Less experienced reader

1

0.735

<0.001

73.3

<0.001

73.9

0.057

73.1

<0.001

2

0.802

0.014

80.5

0.005

79.7

0.388

80.7

0.009

Experienced reader

1

0.843

0.255

87.2

0.749

78.3

0.227

90.4

0.851

2

0.901

0.113

91

0.146

88.4

0.625

91.9

0.267

Wei 202230

DL

0.906 (0.885–0.924)

 

89.6 (87.4– 91.4)

 

94.2 (91.0– 96.5)

 

87.0 (83.9– 89.6)

 

Wei 202230

Less experienced readersa

0.696 (0.665–0.726)

<0.001

61.6 (58.4–64.7)

<0.001

98.5 (96.5–99.5)

<0.001

40.7 (36.7–44.8)

<0.001

Less experienced readersb

0.874 (0.850–0.895)

0.007

85.9 (83.5–88.0)

0.005

89.6 (85.7–92.7)

0.458

82.1 (78.7–85.1)

0.007

Experienced readersa

0.734 (0.704–0.763)

<0.001

66.5 (63.3–69.5)

<0.001

98.5 (96.5–99.5)

0.001

48.4 (44.2–52.5)

<0.001

Experienced readersb

0.883 (0.860–0.903)

0.057

87.9 (85.6–89.9)

0.21

92.6 (89.2–95.2)

0.014

87.0 (83.9– 89.6)

>0.999

Ciritsis 201931

DL

0.967 (0.86–0.99)

 

NR

 

89.47

 

100

 

Reader 1

0.938 (0.82– 0.99)

NR

NR

NR

100

NR

87.5

NR

Reader 2

0.88 (0.74– 0.96)

NR

NR

NR

84.21

NR

95.83

NR

Gu 202232

DL

0.924 (0.879–0.957)

 

85.57 (79.94–90.12)

 

89.77 (81.47–95.22)

 

82.30 (74.00–88.84)

 

Readers

0.843 (0.819–0.865)

<0.0001

66.27(63.25– 69.19)

<0.0001

96.82 (94.72–98.25)

<0.0001

42.48 (38.36–46.67)

<0.0001

Assistive DL system

Park 201917

Less experienced

Reader 1 + DL

0.828 (0.745–0.912)

 

54

 

97.6

 

23.7

 

Reader 1

0.623 (0.501–0.746)

<0.001

43

0.03

65.9

<0.001

27.1

0.56

Reader 2 + DL

0.823 (0.742–0.904)

 

74

 

85.4

 

66.1

 

Reader 2

0.702 (0.596–0.808)

0.001

61

0.008

75.6

0.1

50.8

0.04

Reader 3 + DL

0.839 (0.762–0.917)

 

58

 

97.6

 

30.5

 

Reader 3

0.759 (0.660–0.859)

0.04

51

0.15

87.8

0.05

27.1

0.59

Experienced

Reader 1 + DL

0.907 (0.848–0.967)

 

74

 

90.2

 

66.1

 

Park 201917

Reader 1

0.856 (0.776–0.936)

0.02

66

0.006

85.4

0.16

52.5

0.02

Reader 2 + DL

0.904 (0.837–0.971)

 

76

 

90.2

 

66.1

 

Reader 2

0.889 (0.821–0.957)

0.16

70

0.05

92.7

0.327

54.2

0.02

Kim 202118

Reader 1 + DL

0.803

 

NR

 

90

 

70.5

 

Reader 1

0.545

<0.001

NR

NR

100

>0.999

8.9

<0.001

Reader 2 + DL

0.658

 

NR

 

100

 

31.5

 

Reader 2

0.541

<0.001

NR

NR

100

NA

8.2

<0.001

Reader 3 + DL

0.758

 

NR

 

90

 

61.6

 

Reader 3

0.545

<0.001

NR

NR

100

>0.999

8.9

<0.001

Cho 201820

Less experienced reader + DL

0.895 (0.835–0.956)

 

86.6 (80.4–92.7)

 

87.0 (78.1–96.0)

 

86.2 (77.8–94.6)

 

Less experienced reader

0.887 (0.826–0.947)

>0.999

69.8 (61.5–78.0)

<0.001

94.4 (88.3–100.0)

0.17

49.2 (37.1–61.4)

<0.001

Experienced reader + DL

0.901(0.844–0.958)

 

85.7 (79.4–92.0)

 

94.4 (88.3–100.0)

 

87.7 (79.7–95.7)

 

Experienced reader

0.901 (0.846–0.956)

>0.999

73.1 (65.1–81.1)

0.015

83.3 (73.4–93.3)

0.04

55.4 (43.3–67.5)

<0.001

Wang 202121

Readers + DLc

0.777 (0.707–0.847)

0.08

75.7 (68.8–81.5)

0.095

97.4 (90.2–99.6)

1

57.9 (47.3–67.8)

0.042

Readers + DLd

0.822 (0.757–0.886)

0.01

80.9 (74.4–86.1)

0.005

94.9 (86.7–98.3)

0.681

69.4 (59.1–78.3)

<0.001

Readers

0.703 (0.626–0.780)

 

67.6 (60.3–74.2)

 

97.4 (90.2–99.6)

 

43.2 (33.2–53.8)

 

Xia 202123

Less experienced reader + DL

0.948

 

89.6

 

95.8

 

93.8

 

Less experienced reader

0.719

NR

43.8

NR

75

NR

68.8

NR

Experienced reader + DL

0.969

 

93.8

 

100

 

93.8

 

Experienced reader

0.802

NR

60.5

NR

79.2

NR

81.3

NR

Lee 202224

Readers + DLe

0.908 (0.876‒0.941)

0.093

75.3 (72.2‒78.2)

<0.001

95.2 (92.4‒97.0)

0.725

61.8 (57.5‒65.8)

<0.001

Lee 202224

Readers + DLf

0.913 (0.886‒0.941)

0.099

79.0 (76.0‒81.6)

0.001

93.8 (90.7‒96.0)

0.087

68.8 (64.7‒72.6)

0.001

Readers

0.895 (0.854‒0.936)

 

72.4 (69.1‒75.4)

 

95.4 (93.0‒97.0)

 

56.6 (52.2‒60.8)

 

Choi 201925

Less experienced reader 1 + DL

0.951

 

86.2

 

95

 

82.1

 

Less experienced reader 1

0.906

NR

79.4

0.045

88.8

0.182

75.1

0.014

Less experienced reader 2 + DL

0.914

 

88.1

 

86.3

 

89

 

Less experienced reader 2

0.895

NR

88.9

0.78

81.3

0.221

92.5

0.211

Experienced reader 1 + DL

0.919

 

90.9

 

86.3

 

93.1

 

Experienced reader 1

0.884

NR

77.9

<0.001

88.8

0.683

72.8

<0.001

Experienced reader 2 + DL

0.942

 

90.1

 

90

 

90.2

 

Experienced reader 2

0.919

NR

84.2

0.046

86.3

0.371

83.2

0.006

Lai 202227

Readers + DL

0.8294 (0.7777– 0.8813)

 

NR

 

98.17 (0.9492– 1.0143)

 

30.67 (21.93–39.40)

 

Readers

0.7582 (0.7014–0.8151)

<0.0001

NR

NR

95.77 (90.88–10.066)

0.2991

24.07 (15.97–32.17)

0.0448

Lee 201928

Less experienced readers + DL

0.71 (0.65–0.77)

 

NR

 

69 (57–80)

 

73 (69–77)

 

Less experienced readers

0.65 (0.58–0.71)

0.001

NR

NR

59 (46–71)

0.008

70 (66–75)

0.033

Experienced readers + DL

0.84 (0.81–0.87)

 

NR

 

96 (88–99)

 

72 (68–77)

 

Experienced readers

0.83 (0.8–0.86)

0.451

NR

NR

97 (90–100)

0.317

70 (65–74)

0.003

Wei 202129

Reader 1 + DL

0.875

 

89.1

 

84.1

 

90.9

 

Reader 1

0.735

<0.001

73.3

<0.001

73.9

0.039

73.1

<0.001

Reader 2 + DL

0.867

 

87.2

 

85.5

 

87.8

 

Wei 202129

Reader 2

0.802

<0.001

80.5

<0.001

79.7

0.125

80.7

0.001

Reader 3 + DL

0.872

 

89.5

 

82.6

 

91.9

 

Reader 3

0.843

0.099

87.2

0.181

78.3

0.375

90.4

0.508

Reader 4 + DL

0.901

 

91

 

88.4

 

91.9

 

Reader 4

0.901

>0.999

91

>0.999

88.4

>0.999

91.9

>0.999

Wei 202230

Less experienced readers + DL

0.87 (0.85–0.89)

 

NR

 

97.24 (96.17–98.31)

 

40.7 (37.49–43.9)

 

Less experienced readers

0.7 (0.66–0.73)

NR

NR

NR

98.47 (97.66–99.27)

NR

77.22 (74.48–79.96)

NR

Experienced readers + DL

0.89 (0.87–0.91)

 

NR

 

96.32 (95.09–97.55)

 

81.39 (78.85–83.93)

 

Experienced readers

0.73 (0.70–0.76)

NR

NR

NR

98.47 (97.66–99.27)

NR

48.35 (45.08–51.61)

NR

Gu 202232

Readers + DLg

0.861 (0.838–0.881)

<0.0001

78.71 (76.04– 81.20)

<0.0001

97.27 (95.28–98.58)

0.8036

64.25 (60.14–68.21)

<0.0001

Readers + DLh

0.908 (0.888–0.925)

<0.0001

80.40 (77.81– 82.81)

<0.0001

97.73 (95.86–98.91)

0.4545

66.90 (62.85–70.77)

<0.0001

Readers

0.843 (0.819–0.865)

 

66.27 (63.25– 69.19)

 

96.82 (94.72–98.25)

 

42.48 (38.36–46.67)

 
  1. Acc accuracy, Sen sensitivity, Spe specificity, NS not significant, NA not applicable.
  2. aCategory 4a as the cut-off value.
  3. bCategory 4b as the cut-off value12.
  4. cIf both the assessments of longitudinal and transverse sections from the DL model were possibly benign, the final BIRADS category would be downgraded.
  5. dIf any of the assessments from DL were possibly benign, the final BIRADS category would be downgraded16.
  6. eSequential reading mode.
  7. fSimultaneous reading mode6.
  8. gIf the DL model assessed the lesion as malignant or benign, the final BIRADS classification would be upgraded or downgraded by one level.
  9. hThe BIRADS assessment was flexibly adjusted by human readers after combining DL’s outcomes.
Back to article page