Table 2 Quantitative evaluation of DLS in distinguishing normal versus abnormal CXRs across 6 datasets. (A) The DLS’s performance with the high-sensitivity operating point. (B) The DLS’s performance with the high-specificity operating point. The AUC is independent of the operating point and is identical to that in (A).

From: Deep learning for distinguishing normal versus abnormal chest radiographs and generalization to two unseen diseases tuberculosis and COVID-19

(A)

Scenario

Dataset (reference label used for evaluation)

High-sensitivity operating point (optimizes for NPV)

AUC (95% CI)

No. predicted negative (%)

NPV (95% CI)

Sensitivity (95% CI)

No. predicted positive (%)

PPV (95% CI)

Specificity (95% CI)

Abnormality detection

DS-1 (normal/abnormal)

2313 (29.9%)

0.98 (0.97–0.99)

0.98 (0.97–0.98)

5434 (70.1%)

0.33 (0.32–0.34)

0.38 (0.37–0.40)

0.87 (0.87–0.88)

CXR-14 (normal/abnormal)

194 (24.0%)

0.85 (0.79–0.89)

0.95 (0.93–0.97)

616 (76.0%)

0.89 (0.86–0.91)

0.71 (0.65–0.76)

0.94 (0.93–0.96)

Unseen disease 1: TB

TB-1 (TB status)

199 (43.1%)

0.88 (0.84–0.93)

0.90 (0.87–0.94)

263 (56.9%)

0.83 0.78–0.87 )

0.80 (0.74–0.85)

0.95 (0.93–0.97)

TB-2 (TB status)

51 (38.3%)

0.98 (0.94–1.0)

0.98 (0.94–1.0)

82 (61.7%)

0.63 (0.51–0.73)

0.63 (0.51–0.73)

0.97 (0.94–0.99)

Unseen disease 2: COVID-19

COV-1 (COVID-19 status)

109 (5.9%)

0.85 (0.78–0.92)

0.97 (0.96–0.98)

1710 (94.0%)

0.33 (0.31–0.35)

0.08 (0.06–0.09)

0.68 (0.66–0.71)

COV-2 (COVID-19 status)

59 (9.8%)

0.56 (0.43–0.68)

0.91 (0.87–0.94)

546 (90.2%)

0.48 (0.44–0.52)

0.10 (0.07–0.14)

0.65 (0.60–0.69)

(B)

Scenario

Dataset (reference label used for evaluation)

High-specificity operating point (optimizes for PPV)

 

No. predicted negative (%)

NPV (95% CI)

Sensitivity (95% CI)

No. predicted positive (%)

PPV (95% CI)

Specificity (95% CI)

Abnormality detection

DS-1 (normal/abnormal)

6027 (77.8%)

0.89 (0.88–0.90)

0.63 (0.61–0.65)

1720 (22.2%)

0.68 (0.65–0.70)

0.91 (0.90–0.91)

 

CXR-14 (normal/abnormal)

715 (88.3%)

0.32 (0.29–0.36)

0.16 (0.13–0.20)

95 (11.7%)

0.99 (0.96–1.0)

1.0 (0.99–1.0)

Unseen disease1: TB

TB-1 (TB status)

260 (56.3%)

0.81 (0.76–0.85)

0.81 (0.74–0.84)

202 (43.7%)

0.95 (0.91–0.98)

0.95 (0.92–0.98)

TB-2 (TB status)

80 (60.2%)

0.94 (0.88–0.99)

0.91 (0.82–0.98)

53 (39.8%)

0.91 (0.81–0.98)

0.94 (0.88–0.99)

Unseen disease 2: COVID-19

COV-1 (COVID-19 status)

1558 (85.7%)

0.72 (0.69–0.74)

0.23 (0.20–0.27)

261 (14.3%)

0.52 (0.46–0.58)

0.90 (0.88–0.92)

COV-2 (COVID-19 status)

537 (88.8%)

0.55 (0.51–0.59)

0.17 (0.12–0.21)

68 (11.2%)

0.71 (0.59–0.81)

0.94 (0.91–0.96)

Back to article page