Table 2 Performance of ECG-CLIP on internal testing and external validation
From: Diagnosis of cardiac conditions from 12-lead electrocardiogram through natural language supervision
Cardiac condition | Evaluation metrics | Internal testing | External validation |
|---|---|---|---|
Atrial fibrillation/flutter | AUROC | 0.956 (0.954–0.957) | 0.949 (0.947–0.951) |
AUPRC | 0.787 (0.778–0.795) | 0.851 (0.845–0.858) | |
Accuracy | 0.927 (0.925–0.929) | 0.881 (0.879–0.884) | |
Sensitivity | 0.800 (0.792–0.808) | 0.851 (0.844–0.858) | |
Specificity | 0.945 (0.943–0.947) | 0.890 (0.887–0.893) | |
Precision | 0.673 (0.664–0.681) | 0.683 (0.675–0.690) | |
F1 score | 0.731 (0.724–0.738) | 0.758 (0.752–0.763) | |
Supraventricular tachycardia | AUROC | 0.985 (0.969–0.994) | 0.971 (0.969–0.974) |
AUPRC | 0.229 (0.170–0.290) | 0.312 (0.282–0.343) | |
Accuracy | 0.992 (0.991–0.993) | 0.979 (0.978–0.981) | |
Sensitivity | 0.667 (0.582–0.748) | 0.353 (0.321–0.387) | |
Specificity | 0.992 (0.992–0.993) | 0.990 (0.989–0.991) | |
Precision | 0.108 (0.085–0.131) | 0.358 (0.323–0.390) | |
F1 score | 0.186 (0.149–0.220) | 0.356 (0.326–0.385) | |
Sinus tachycardia | AUROC | 0.982 (0.981–0.983) | 0.959 (0.958–0.961) |
AUPRC | 0.764 (0.752–0.776) | 0.742 (0.730–0.754) | |
Accuracy | 0.957 (0.955–0.958) | 0.912 (0.909–0.914) | |
Sensitivity | 0.900 (0.893–0.907) | 0.847 (0.838–0.855) | |
Specificity | 0.962 (0.961–0.963) | 0.924 (0.922–0.927) | |
Precision | 0.690 (0.681–0.700) | 0.681 (0.671–0.690) | |
F1 score | 0.781 (0.774–0.788) | 0.755 (0.748–0.762) | |
Sinus bradycardia | AUROC | 0.981 (0.980–0.982) | 0.985 (0.984–0.986) |
AUPRC | 0.824 (0.814–0.833) | 0.956 (0.952–0.960) | |
Accuracy | 0.962 (0.960–0.963) | 0.965 (0.963–0.966) | |
Sensitivity | 0.956 (0.952–0.960) | 0.973 (0.971–0.976) | |
Specificity | 0.962 (0.961–0.964) | 0.960 (0.958–0.962) | |
Precision | 0.789 (0.781–0.796) | 0.934 (0.930–0.937) | |
F1 score | 0.864 (0.860–0.869) | 0.953 (0.951–0.955) | |
Premature ventricular complex | AUROC | 0.960 (0.957–0.962) | 0.943 (0.931–0.955) |
AUPRC | 0.707 (0.693–0.721) | 0.132 (0.107–0.159) | |
Accuracy | 0.953 (0.952–0.955) | 0.953 (0.951–0.955) | |
Sensitivity | 0.787 (0.775–0.798) | 0.749 (0.701–0.797) | |
Specificity | 0.964 (0.963–0.966) | 0.954 (0.952–0.956) | |
Precision | 0.589 (0.577–0.600) | 0.097 (0.084–0.110) | |
F1 score | 0.673 (0.664–0.683) | 0.172 (0.151–0.192) | |
Premature atrial complex | AUROC | 0.938 (0.935–0.941) | 0.846 (0.837–0.855) |
AUPRC | 0.422 (0.404–0.440) | 0.176 (0.158–0.196) | |
Accuracy | 0.885 (0.883–0.887) | 0.875 (0.872–0.878) | |
Sensitivity | 0.813 (0.800–0.826) | 0.527 (0.500–0.555) | |
Specificity | 0.889 (0.886–0.891) | 0.885 (0.883–0.888) | |
Precision | 0.251 (0.243–0.259) | 0.121 (0.112–0.129) | |
F1 score | 0.383 (0.373–0.394) | 0.197 (0.183–0.208) | |
Pacemaker rhythm | AUROC | 0.968 (0.963–0.971) | Not applicable |
AUPRC | 0.318 (0.296–0.341) | ||
Accuracy | 0.968 (0.966–0.969) | ||
Sensitivity | 0.798 (0.776–0.820) | ||
Specificity | 0.970 (0.969–0.971) | ||
Precision | 0.305 (0.291–0.321) | ||
F1 score | 0.442 (0.424–0.459) | ||
Atrioventricular block | AUROC | 0.887 (0.883–0.891) | 0.822 (0.811–0.833) |
AUPRC | 0.419 (0.404–0.435) | 0.291 (0.266–0.315) | |
Accuracy | 0.806 (0.803–0.808) | 0.701 (0.697–0.705) | |
Sensitivity | 0.784 (0.773–0.795) | 0.779 (0.759–0.799) | |
Specificity | 0.808 (0.805–0.810) | 0.698 (0.694–0.702) | |
Precision | 0.233 (0.227–0.239) | 0.084 (0.079–0.088) | |
F1 score | 0.359 (0.352–0.367) | 0.151 (0.144–0.159) | |
1st degree atrioventricular block | AUROC | 0.898 (0.894–0.902) | 0.868 (0.856–0.879) |
AUPRC | 0.441 (0.426–0.457) | 0.336 (0.305–0.366) | |
Accuracy | 0.827 (0.824–0.830) | 0.754 (0.750–0.758) | |
Sensitivity | 0.786 (0.775–0.797) | 0.804 (0.781–0.827) | |
Specificity | 0.830 (0.827–0.832) | 0.753 (0.749–0.757) | |
Precision | 0.243 (0.236–0.249) | 0.078 (0.073–0.082) | |
F1 score | 0.371 (0.362–0.379) | 0.142 (0.134–0.150) | |
Right bundle branch block | AUROC | 0.971 (0.969–0.972) | 0.969 (0.964–0.974) |
AUPRC | 0.832 (0.823–0.841) | 0.284 (0.258–0.313) | |
Accuracy | 0.963 (0.962–0.964) | 0.921 (0.919–0.924) | |
Sensitivity | 0.800 (0.790–0.810) | 0.927 (0.906–0.947) | |
Specificity | 0.977 (0.976–0.978) | 0.921 (0.919–0.924) | |
Precision | 0.750 (0.739–0.760) | 0.147 (0.136–0.157) | |
F1 score | 0.774 (0.766–0.782) | 0.253 (0.238–0.269) | |
Left bundle branch block | AUROC | 0.974 (0.971–0.977) | 0.940 (0.925–0.952) |
AUPRC | 0.695 (0.675–0.713) | 0.146 (0.113–0.184) | |
Accuracy | 0.961 (0.959–0.962) | 0.982 (0.981–0.983) | |
Sensitivity | 0.864 (0.850–0.877) | 0.511 (0.444–0.575) | |
Specificity | 0.964 (0.963–0.965) | 0.984 (0.983–0.985) | |
Precision | 0.459 (0.446–0.473) | 0.149 (0.125–0.173) | |
F1 score | 0.600 (0.587–0.612) | 0.231 (0.197–0.263) | |
Left ventricular hypertrophy | AUROC | 0.704 (0.698–0.710) | 0.638 (0.618–0.657) |
AUPRC | 0.246 (0.237–0.254) | 0.024 (0.021–0.027) | |
Accuracy | 0.501 (0.498–0.505) | 0.577 (0.573–0.582) | |
Sensitivity | 0.790 (0.782–0.799) | 0.632 (0.595–0.668) | |
Specificity | 0.468 (0.464–0.471) | 0.577 (0.572–0.581) | |
Precision | 0.148 (0.144–0.151) | 0.021 (0.019–0.023) | |
F1 score | 0.249 (0.244–0.253) | 0.041 (0.037–0.045) | |
ST-T change | AUROC | 0.474 (0.469–0.479) | 0.558 (0.543–0.573) |
AUPRC | 0.167 (0.164–0.171) | 0.029 (0.027–0.033) | |
Accuracy | 0.272 (0.269–0.275) | 0.588 (0.583–0.592) | |
Sensitivity | 0.960 (0.957–0.963) | 0.486 (0.459–0.515) | |
Specificity | 0.121 (0.119–0.124) | 0.590 (0.586–0.595) | |
Precision | 0.194 (0.191–0.197) | 0.030 (0.028–0.033) | |
F1 score | 0.322 (0.318–0.326) | 0.057 (0.052–0.062) | |
Prolonged QT interval | AUROC | 0.760 (0.753–0.766) | 0.797 (0.781–0.813) |
AUPRC | 0.116 (0.110–0.123) | 0.025 (0.022–0.030) | |
Accuracy | 0.610 (0.607–0.613) | 0.677 (0.673–0.681) | |
Sensitivity | 0.789 (0.776–0.801) | 0.815 (0.778–0.853) | |
Specificity | 0.601 (0.598–0.605) | 0.676 (0.672–0.680) | |
Precision | 0.089 (0.086–0.092) | 0.022 (0.019–0.024) | |
F1 score | 0.160 (0.155–0.165) | 0.042 (0.038–0.047) | |
Low QRS voltage | AUROC | 0.693 (0.687–0.700) | 0.740 (0.726–0.754) |
AUPRC | 0.206 (0.198–0.215) | 0.071 (0.061–0.082) | |
Accuracy | 0.445 (0.441–0.448) | 0.334 (0.330–0.338) | |
Sensitivity | 0.809 (0.801–0.818) | 0.930 (0.914–0.944) | |
Specificity | 0.410 (0.407–0.414) | 0.320 (0.316–0.324) | |
Precision | 0.114 (0.112–0.117) | 0.031 (0.029–0.033) | |
F1 score | 0.200 (0.196–0.205) | 0.061 (0.057–0.064) | |
Normal ECG | AUROC | 0.872 (0.869–0.875) | Not applicable |
AUPRC | 0.631 (0.623–0.640) | ||
Accuracy | 0.800 (0.797–0.803) | ||
Sensitivity | 0.807 (0.801–0.813) | ||
Specificity | 0.798 (0.794–0.801) | ||
Precision | 0.538 (0.532–0.544) | ||
F1 score | 0.646 (0.640–0.651) | ||
Borderline ECG | AUROC | 0.678 (0.673–0.682) | Not applicable |
AUPRC | 0.384 (0.378–0.392) | ||
Accuracy | 0.551 (0.547–0.555) | ||
Sensitivity | 0.809 (0.803–0.815) | ||
Specificity | 0.464 (0.459–0.468) | ||
Precision | 0.337 (0.332–0.342) | ||
F1 score | 0.476 (0.471–0.481) | ||
Abnormal ECG | AUROC | 0.798 (0.795–0.801) | Not applicable |
AUPRC | 0.806 (0.802–0.810) | ||
Accuracy | 0.718 (0.715–0.722) | ||
Sensitivity | 0.841 (0.837–0.845) | ||
Specificity | 0.584 (0.579–0.590) | ||
Precision | 0.688 (0.684–0.693) | ||
F1 score | 0.757 (0.754–0.760) |
