Table 5 Summary of hyperparameters and hyperparameter optimization (HPO) methods for various machine learning algorithms.
From: Ensemble deep learning and EfficientNet for accurate diagnosis of diabetic retinopathy
ML algorithm | Main HPs | Optional HPs | HPO methods | Libraries |
|---|---|---|---|---|
Ridge & lasso | Alpha | – | BO-GP | Skopt |
Logistic regression | Penalty, c, solver | – | BO-TPE, SMAC | Hyperopt, SMAC |
KNN | n_neighbors | Weights, p, algorithm | BOs, Hyperband | Skopt, Hyperopt, SMAC, Hyperband |
SVM | C, kernel, epsilon (for SVR) | Gamma, coef0, degree | BO-TPE, SMAC, BOHB | Hyperopt, SMAC, BOHB |
NB | Alpha | – | BO-GP | Skopt |
DT | Criterion, max_depth, min_samples_split, min_samples_leaf, max_features, splitter, min_weight_fraction_leaf, max_leaf_nodes | – | GA, PSO, BO-TPE, SMAC, BOHB | TPOT, Optunity, SMAC, BOHB |
RF & ET | n_estimators, max_depth, criterion, min_samples_split, min_samples_leaf, max_features, splitter, min_weight_fraction_leaf, max_leaf_nodes | – | GA, PSO, BO-TPE, SMAC, BOHB | TPOT, Optunity, SMAC, BOHB |
XGBoost | n_estimators, max_depth, learning_rate, subsample, colsample_bytree, min_child_weight, gamma, alpha, lambda | – | GA, PSO, BO-TPE, SMAC, BOHB | TPOT, Optunity, SMAC, BOHB |
Voting | Estimators, voting weights | – | GS | Sklearn |
Bagging | Base_estimator, n_estimators | max_samples, max_features | GS, BOs | Sklearn, Skopt, Hyperopt, SMAC |
AdaBoost | Base_estimator, n_estimators, learning_rate | – | BO-TPE, SMAC | Hyperopt, SMAC |
Deep learning | Number of hidden layers, ‘units’ per layer, loss, optimizer, Activation, learning_rate, dropout rate, epochs, batch_size, early stop patience, number of frozen layers (if transfer learning is used) | – | PSO, BOHB | Optunity, BOHB |
Hierarchical clustering | n_clusters, distance_threshold | Linkage | BOs, Hyperband | Skopt, Hyperopt, SMAC, Hyperband |
DBSCAN | eps, min_samples | – | BO-TPE, SMAC, BOHB | Hyperopt, SMAC, BOHB |
Gaussian mixture | n_components | covariance_type, max_iter, tol | BO-GP | Skopt |
PCA | n_components | svd_solver | BOs, Hyperband | Skopt, Hyperopt, SMAC, Hyperband |
LDA | n_components | solver, shrinkage | BOs, Hyperband | Skopt, Hyperopt, SMAC, Hyperband |
