Table 4 Comparison of multiple machine learning evaluation indexes between training set and validation set.
Data set | Models | AUC(95%CI) | Acuuracy rating(95%CI) | Sensitivity((95%CI) | Specificity(95%CI) | NPV(95%CI) | PPV(95%CI) | F1 score(95%CI) |
|---|---|---|---|---|---|---|---|---|
Validation set | RF + SMOTETomek | 0.819(0.720,0.906) | 0.740(0.628, 0.834) | 0.625(0.438,0.783) | 0.822(0.674,0.915) | 0.755(0.608,,0.862) | 0.714(0.511,0.861) | 0.787(0.691,0.869) |
logistics + SMOTE | 0.808(0.692, 0.903) | 0.753(0.642, 0.844) | 0.625(0.438,0.783) | 0.889(0.408,0.958) | 0.741(0.6,0.8462) | 0.741(0.534,0.921) | 0.816(0.719,0.891) | |
SVM + SMOTE | 0.633(0.423,0.764) | 0.688(0.573, 0.789) | 0.469(0.295,0.650) | 0.844(0.699,0.930) | 0.704(0.582,0.826) | 0.682(0.451,0.853) | 0.753(0.645,0.841) | |
XGBoost + ROSE | 0.738(0.627,0.850) | 0.675(0.559, 0.778) | 0.313(0.168,0.501) | 0.933(0.807,0.983) | 0.615(0.486,0.731) | 0.769(0.460,0.938) | 0.771(0.667,0.852) | |
AdaBoost + SMOTETomek | 0.756(0.646,0.866) | 0.701(0.586, 0.800) | 0.406(0.242,0.592) | 0.911(0.779,0.971) | 0.633(0.498,0.751) | 0.765(0.498,0.922) | 0.781(0.681,0.862) | |
lightGBM + SMOTETomek | 0.753(0.641,0.866) | 0.662(0.546, 0.766) | 0.469(0.295,0.650) | 0.800(0.650,0.899) | 0.7(0.552,0.817) | 0.625(0.408,0.805) | 0.735(0.633,0.822) | |
Training set | RF + SMOTETomek | 0.816(0.754,0.879) | 0.747(0.678, 0.809) | 0.623(0.498,0.735) | 0.823(0.738,0.886) | 0.782(0.695,0.85) | 0.683(0.552,0.791) | 0.802(0.740,0.851) |
logistic + SMOTE | 0.805(0.739,0.869) | 0.742(0.672, 0.804) | 0.667(0.542,0.773) | 0.788(0.699,0.857) | 0.795(0.706,0.863) | 0.657(0.533,0.801) | 0.791(0.730,0.844) | |
SVM + SMOTE | 0.720(0.637,0.800) | 0.720(0.649, 0.784) | 0.623(0.498,0.735) | 0.779(0.689,0.849) | 0.789(0.712,0.866) | 0.632(0.506,0.744) | 0.775(0.714,0.832) | |
XGBoost + ROSE | 0.798(0.732,0.865) | 0.714(0.643, 0.779) | 0.319(0.215,0.443) | 0.956(0.895,0.984) | 0.686(0.606,0.756) | 0.815(0.613,0.930) | 0.806(0.760,0.852) | |
AdaBoost + SMOTETomek | 0.781(0.711,0.852) | 0.731(0.660, 0.794) | 0.377(0.265,0.502) | 0.947(0.883,0.978) | 0.75(0.664.,0.82) | 0.813(0.630,0.921) | 0.814(0.766,0.857) | |
lightGBM + SMOTETomek | 0.713(0.635,0.792) | 0.698(0.626, 0.764) | 0.536(0.413,0.656) | 0.797(0.708,0.864) | 0.754(0.666,0.826) | 0.617(0.482,0.737) | 0.766(0.702,0.825) |
