Table 2 Performance for mortality risk prediction of models in validation cohorts.
From: Machine learning based early warning system enables accurate mortality risk prediction for COVID-19
AUC (95% CI) | Accuracy (95% CI) | Sensitivity (95% CI) | Specificity (95% CI) | PPV (95% CI) | NPV (95% CI) | F1 | Kappa | Brier | |
|---|---|---|---|---|---|---|---|---|---|
Internal validation cohort (SFV) | |||||||||
MRPMC | 0.9621 (0.9464–0.9778) | 92.4% (90.1–94.4%) | 57.3% (46.4–67.7%) | 98.3% (96.8–99.2%) | 85.0% (73.4–92.9%) | 93.2% (90.8–95.2%) | 0.685 | 0.644 | 0.051 |
SVM | 0.9594 (0.9424–0.9764) | 92.4% (90.1–94.4%) | 60.7% (49.8–70.9%) | 97.8% (96.1–98.8%) | 81.8% (70.4–90.2%) | 93.7% (91.4–95.6%) | 0.697 | 0.655 | 0.052 |
GBDT | 0.9454 (0.9246–0.9662) | 91.5% (89.0–93.6%) | 60.7% (49.8–70.9%) | 96.6% (94.7–98.0%) | 75.0% (63.4–84.5%) | 93.6% (91.3–95.5%) | 0.696 | 0.643 | 0.066 |
LR | 0.9614 (0.9456–0.9772) | 92.1% (89.7–94.1%) | 56.2% (45.3–66.7%) | 98.1% (96.6–99.1%) | 83.3% (71.5–91.7%) | 93.1% (90.6–95.0%) | 0.671 | 0.628 | 0.051 |
NN | 0.9615 (0.9456–0.9774) | 92.1% (89.7–94.1%) | 51.7% (40.8–62.4%) | 98.9% (97.6–99.6%) | 88.5% (76.6–95.7%) | 92.5% (90.0–94.5%) | 0.653 | 0.612 | 0.051 |
External validation cohort (OV) | |||||||||
MRPMC | 0.9760 (0.9613–0.9906) | 95.5% (93.8–96.8%) | 45.0% (32.1–58.4%) | 99.6% (98.8–99.9%) | 90.0% (73.5–97.9%) | 95.7% (94.0–97.0%) | 0.600 | 0.579 | 0.029 |
SVM | 0.9774 (0.9640–0.9908) | 95.8% (94.1–97.0%) | 50.0% (36.8–63.2%) | 99.5% (98.6–99.9%) | 88.2% (72.6–96.7%) | 96.1% (94.5–97.4%) | 0.638 | 0.618 | 0.028 |
GBDT | 0.9536 (0.9279–0.9793) | 94.8% (93.0–96.2%) | 48.3% (35.2–61.6%) | 98.5% (97.4–99.3%) | 72.5% (56.1–85.4%) | 95.9% (94.3–97.2%) | 0.580 | 0.553 | 0.039 |
LR | 0.9721 (0.9568–0.9875) | 95.4% (93.7–96.7%) | 45.0% (32.1–58.4%) | 99.5% (98.6–99.9%) | 87.1% (70.2–96.4%) | 95.7% (94.0–97.0%) | 0.593 | 0.572 | 0.031 |
NN | 0.9754 (0.9602–0.9906) | 95.6% (94.0–96.9%) | 46.7% (33.7–60.0%) | 99.6% (98.8–99.9%) | 90.3% (74.3–98.0%) | 95.8% (94.2–97.1%) | 0.615 | 0.595 | 0.028 |
External validation cohort (CHWH) | |||||||||
MRPMC | 0.9246 (0.8763–0.9729) | 87.9% (80.6–93.2%) | 42.1% (20.3–66.5%) | 96.9% (91.2–99.4%) | 72.7% (39.0–94.0%) | 89.5% (82.0–94.7%) | 0.533 | 0.470 | 0.083 |
SVM | 0.9067 (0.8482–0.9652) | 88.8% (81.6–93.9%) | 57.9% (33.5–79.8%) | 94.6% (88.4–98.3%) | 68.8% (41.3–89.0%) | 92.0% (84.8–96.5%) | 0.629 | 0.563 | 0.090 |
GBDT | 0.9021 (0.8347–0.9694) | 87.9% (80.6–93.2%) | 31.6% (12.6–56.6%) | 99.0% (94.4–100.0%) | 85.7% (42.1–99.6%) | 88.1% (80.5–93.5%) | 0.462 | 0.410 | 0.089 |
LR | 0.9213 (0.8710–0.9717) | 87.1% (79.6–92.6%) | 36.8% (16.3–61.6%) | 96.9% (91.2–99.4%) | 70.0% (34.8–93.3%) | 88.7% (81.1–94.0%) | 0.483 | 0.417 | 0.091 |
NN | 0.9202 (0.8700–0.9705) | 88.8% (81.6–93.9%) | 47.4% (24.5–71.1%) | 96.9% (91.2–99.4%) | 75.0% (42.8–94.5%) | 90.4% (83.0–95.3%) | 0.581 | 0.520 | 0.083 |
