Table 1 Results (Mean ± 95% confidence interval) for Task Night
From: Benchmark on a large cohort for sleep-wake classification with machine learning techniques
Method | Algorithm evaluation metrics | Sleep quality metrics | |||||||
|---|---|---|---|---|---|---|---|---|---|
Accuracy | Specificity | Precision | Sensitivity | F1 | WASO (min) | MAE WASO | Sleep Eff. (%) | MAE sleep Eff. | |
Ground truth | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 102.1 ± 7.3 | 0.0 | 58.4 ± 1.4 | 0.0 |
Baselines | |||||||||
Manual annotations | 79.8 ± 1.2 | 56.5 ± 2.3 | 75.8 ± 1.5 | 94.8 ± 1.5 | 83.3 ± 1.4 | 45.8 ± 8.6 | 74.7 | 73.0 ± 1.7 | 17.2 |
Device algorithm | 76.2 ± 1.0 | 50.1 ± 1.8 | 72.6 ± 1.3 | 94.3 ± 0.6 | 81.3 ± 1.0 | 54.0 ± 4.2 | 53.1 | 75.7 ± 1.0 | 17.7 |
Always sleep | 58.4 ± 1.4 | 0.0 ± 0.0 | 58.4 ± 1.4 | 100.0 ± 0.0 | 72.8 ± 1.1 | 0.0 ± 0.0 | 102.1 | 100.0 ± 0.0 | 41.6 |
Always wake | 41.6 ± 1.4 | 100.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | 459.2 ± 9.0 | 357.0 | 0.0 ± 0.0 | 58.4 |
Traditional algorithms | |||||||||
Oakley θ= 1032 | 77.5 ± 0.9 | 63.0 ± 1.7 | 76.8 ± 1.3 | 87.2 ± 0.9 | 81.0 ± 1.0 | 95.0± 5.9 | 37.3 | 66.0 ± 1.1 | 10.1 |
Scripps Clinic21 | 76.6 ± 1.1 | 48.8 ± 1.9 | 72.5 ± 1.4 | 95.9 ± 0.5 | 81.8 ± 1.0 | 46.3 ± 4.2 | 58.5 | 77.1 ± 1.0 | 18.9 |
Oakley θ= 4032 | 75.9 ± 1.0 | 49.3 ± 1.8 | 72.2 ± 1.3 | 94.4 ± 0.5 | 81.2 ± 1.0 | 53.1 ± 4.1 | 52.9 | 76.0 ± 1.0 | 17.9 |
Cole-Kripke6 | 75.4 ± 1.1 | 45.0 ± 1.8 | 71.1 ± 1.4 | 96.7 ± 0.4 | 81.2 ± 1.0 | 40.2 ± 3.7 | 63.5 | 79.2 ± 1.0 | 21.0 |
Sazonov9 | 75.2 ± 1.0 | 73.3 ± 1.6 | 79.9 ± 1.3 | 75.5 ± 1.4 | 76.7 ± 1.2 | 149.2 ± 7.7 | 58.7 | 54.9 ± 1.3 | 9.1 |
Oakley θ = 8032 | 73.9 ± 1.1 | 41.2 ± 1.7 | 69.7 ± 1.4 | 96.9 ± 0.4 | 80.3 ± 1.0 | 35.9 ± 3.2 | 67.4 | 80.9 ± 0.9 | 22.7 |
Sadeh5 | 73.4 ± 1.2 | 38.3 ± 1.8 | 69.1 ± 1.4 | 98.3 ± 0.3 | 80.3 ± 1.1 | 26.3 ± 3.1 | 76.5 | 83.0 ± 0.9 | 24.7 |
Webster28 | 73.3 ± 1.2 | 38.2 ± 1.8 | 69.0 ± 1.4 | 98.2 ± 0.3 | 80.3 ± 1.1 | 27.5 ± 3.0 | 75.3 | 83.0 ± 0.9 | 24.7 |
Group average | 75.1 ± 1.3 | 49.6 ± 10.4 | 72.5 ± 3.3 | 92.9 ± 6.6 | 80.4 ± 1.3 | 59.2 ± 35.4 | 61.3 ± 10.6 | 75.0 ± 8.2 | 18.6 ± 5.1 |
Rescoring rules applied to traditional algorithms | |||||||||
Resc. Oakley θ = 40 | 80.3 ± 0.9 | 68.3 ± 1.9 | 79.9 ± 1.3 | 88.1 ± 0.9 | 83.1 ± 1.0 | 93.2 ± 6.6 | 37.7 | 64.4 ± 1.2 | 9.0 |
Resc. Cole-Kripke | 80.2 ± 1.0 | 65.7 ± 2.0 | 78.9 ± 1.3 | 89.9 ±0.8 | 83.3 ± 1.0 | 83.5 ± 6.3 | 40.0 | 66.6 ± 1.2 | 10.2 |
Resc. Scripps Clinic | 80.1 ± 1.0 | 70.4 ± 1.9 | 80.7 ± 1.3 | 86.3 ± 1.1 | 82.6 ± 1.0 | 102.8 ± 7.5 | 41.8 | 62.5 ± 1.3 | 9.1 |
Resc. Oakley θ = 80 | 79.3 ± 1.0 | 59.8 ± 2.0 | 76.6 ± 1.4 | 92.8 ± 0.6 | 83.2 ± 1.0 | 65.0 ± 5.4 | 46.4 | 70.7 ± 1.1 | 13.1 |
Resc. Sadeh | 79.1 ± 1.0 | 59.4 ± 2.0 | 76.5 ± 1.4 | 92.8 ± 0.7 | 83.1 ± 1.0 | 64.1 ± 5.7 | 49.2 | 70.9 ± 1.2 | 13.5 |
Resc. Webster | 79.0 ± 1.0 | 58.9 ± 2.0 | 76.2 ± 1.4 | 93.1 ± 0.7 | 83.0 ± 1.0 | 63.2 ± 5.5 | 48.9 | 71.3 ± 1.2 | 13.8 |
Resc. Oakley θ = 10 | 77.8 ± 1.0 | 81.6 ± 1.6 | 85.5 ± 1.3 | 73.8 ± 1.6 | 78.0 ± 1.3 | 163.9 ± 9.1 | 68.7 | 50.7 ± 1.5 | 10.8 |
Resc. Sazonov | 68.1 ± 1.3 | 90.1 ± 1.3 | 87.8 ± 1.6 | 51.2 ± 2.1 | 62.3 ±2.0 | 258.4 ± 11.0 | 156.7 | 34.0 ± 1.6 | 24.7 |
Group average | 78.0 ± 3.4 | 69.3 ± 9.5 | 80.3 ± 3.6 | 83.5 ± 12.1 | 79.8 ± 6.1 | 111.8 ± 56.8 | 61.2 ± 33.3 | 61.4 ± 10.9 | 13.0 ± 4.3 |
Machine learning algorithms | |||||||||
Extra trees | 81.8 ± 1.0 | 68.1 ± 1.9 | 80.3 ± 1.3 | 90.4 ± 1.2 | 84.3 ± 1.1 | 85.4 ± 7.4 | 42.8 | 65.8 ± 1.4 | 10.3 |
Logistic regression | 81.5 ± 1.0 | 67.2 ± 2.0 | 79.9 ± 1.3 | 90.7 ± 1.2 | 84.1 ± 1.1 | 83.2 ± 7.5 | 45.6 | 66.3 ± 1.4 | 11.1 |
Linear SVM | 81.4 ± 1.1 | 68.0 ± 2.0 | 80.2 ± 1.3 | 89.9 ± 1.3 | 83.8 ± 1.1 | 87.2 ± 7.8 | 45.8 | 65.5 ± 1.5 | 10.8 |
Perceptron | 78.4 ± 1.0 | 69.0 ± 1.8 | 79.4 ± 1.3 | 83.9 ± 1.4 | 80.7 ± 1.2 | 110.3 ± 8.0 | 44.0 | 61.7 ± 1.4 | 9.3 |
Group average | 80.8 ± 2.5 | 68.1 ± 1.2 | 80.0 ± 0.6 | 88.8 ± 5.1 | 83.2 ± 2.7 | 91.5 ± 20.1 | 44.6 ± 2.3 | 64.8 ± 3.4 | 10.4 ± 1.2 |
Rescoring rules applied to machine learning algorithms | |||||||||
Resc. Log. Regression | 78.9 ± 1.2 | 80.7 ± 1.8 | 85.6 ± 1.2 | 75.9 ± 1.9 | 78.8 ± 1.5 | 152.8 ± 10.4 | 64.5 | 52.2 ± 1.7 | 10.6 |
Resc. extra trees | 78.5 ± 1.2 | 82.0 ± 1.7 | 86.1 ± 1.2 | 74.2 ± 1.9 | 78.2 ± 1.5 | 160.4 ± 10.3 | 68.6 | 50.8 ± 1.7 | 11.0 |
Resc. linear SVM | 78.3 ± 1.2 | 81.4 ± 1.7 | 85.8 ± 1.2 | 74.4 ± 2.0 | 77.9 ± 1.6 | 159.4 ± 10.6 | 69.6 | 51.1 ± 1.7 | 11.2 |
Resc. perceptron | 73.4 ± 1.3 | 84.4 ± 1.5 | 85.7 ± 1.4 | 63.8 ± 2.2 | 70.8 ± 1.9 | 202.2 ± 11.3 | 104.4 | 43.7 ± 1.8 | 16.2 |
Group average | 77.3 ± 4.1 | 82.1 ± 2.6 | 86.0 ± 0.3 | 72.1 ± 8.9 | 76.4 ± 6.0 | 168.7 ± 35.9 | 76.8 ± 29.5 | 49.5 ± 6.2 | 12.2 ± 4.2 |
Deep-learning algorithms | |||||||||
LSTM 100 | 83.1 ± 1.0 | 69.9 ± 2.0 | 81.6 ± 1.3 | 91.4 ± 1.1 | 85.5 ± 1.0 | 79.2 ± 7.6 | 43.9 | 65.6 ± 1.4 | 10.0 |
CNN 100 | 82.9 ± 1.0 | 68.8 ± 2.1 | 81.3 ± 1.3 | 91.7 ± 1.2 | 85.3 ± 1.1 | 78.3 ± 7.9 | 46.7 | 66.2 ± 1.5 | 10.8 |
LSTM 50 | 82.7 ± 1.0 | 70.1 ± 1.9 | 81.5 ± 1.3 | 90.5 ± 1.1 | 85.0 ± 1.0 | 85.6 ± 7.6 | 41.3 | 64.9 ± 1.4 | 9.6 |
CNN 50 | 82.5 ± 1.0 | 67.6 ± 2.0 | 80.5 ± 1.3 | 92.0 ± 1.1 | 85.1 ± 1.1 | 75.9 ± 7.4 | 46.6 | 66.9 ± 1.4 | 11.0 |
CNN 20 | 81.4 ± 1.0 | 66.5 ± 1.9 | 79.6 ± 1.3 | 90.9 ± 1.1 | 84.1 ± 1.1 | 81.9 ± 7.1 | 43.2 | 66.7 ± 1.4 | 10.8 |
LSTM 20 | 81.3 ± 1.0 | 65.0 ± 1.9 | 79.0 ± 1.3 | 92.0 ± 1.0 | 84.3 ± 1.0 | 75.3 ± 6.7 | 44.5 | 68.0 ± 1.3 | 11.4 |
Group average | 82.3 ± 0.8 | 68.0 ± 2.1 | 80.6 ± 1.2 | 91.4 ± 0.7 | 84.9 ± 0.6 | 79.4 ± 4.1 | 44.4 ± 2.2 | 66.4 ± 1.1 | 10.6 ± 0.7 |
Rescoring rules applied to deep-learning algorithms | |||||||||
Resc. LSTM 100 | 81.2 ± 1.0 | 77.8 ± 1.8 | 84.8 ± 1.2 | 82.1 ± 1.5 | 82.3 ± 1.2 | 123.4 ± 9.4 | 47.2 | 57.1 ± 1.6 | 8.7 |
Resc. CNN 100 | 80.9 ± 1.0 | 78.3 ± 1.9 | 85.1 ± 1.2 | 81.1 ± 1.7 | 81.7 ± 1.3 | 128.1 ± 9.9 | 50.8 | 56.4 ± 1.7 | 9.3 |
Resc. CNN 50 | 80.6 ± 1.1 | 78.2 ± 1.8 | 84.8 ± 1.3 | 80.6 ± 1.7 | 81.4 ±1.3 | 130.0 ± 9.7 | 51.4 | 56.1 ± 1.6 | 9.3 |
Resc. LSTM 50 | 79.9 ± 1.0 | 80.1 ± 1.7 | 85.6 ± 1.2 | 78.0 ± 1.6 | 80.4 ± 1.3 | 142.9 ± 9.8 | 55.6 | 53.8 ± 1.6 | 9.5 |
Resc. LSTM 20 | 79.5 ± 1.1 | 79.9 ± 1.7 | 85.2 ± 1.2 | 77.5 ± 1.7 | 79.9 ± 1.4 | 145.2 ± 9.8 | 56.9 | 53.6 ± 1.6 | 9.6 |
Resc. CNN 20 | 78.4 ± 1.1 | 81.3 ± 1.7 | 85.7 ± 1.3 | 74.5 ± 1.8 | 78.2 ± 1.5 | 158.5 ± 10.2 | 66.8 | 51.3 ± 1.7 | 10.8 |
Group average | 80.1 ± 1.1 | 79.3 ± 1.5 | 85.2 ± 0.4 | 79.0 ± 3.0 | 80.7 ± 1.6 | 138.0 ± 13.8 | 54.8 ± 7.2 | 54.7 ± 2.3 | 9.6 ± 0.7 |
