Table 3 Top three complexity estimations. Among the 23 theoretical complexity measures tested, only a few exhibit moderate alignment with empirical generalization performance. This highlights a significant concern regarding the reliability of the theoretical estimation.
From: A practical generalization metric for deep networks benchmarking
Complexity measures | ImageNet | CIFAR-100 | |||
|---|---|---|---|---|---|
\(SE_g\) of generalization bounds | \(SE_g\) of 10th percentile | \(SE_g\) of generalization bounds | \(SE_g\) of 10th percentile | ||
ZeroShot% | INVERSE_MARGIN | 0.428571 | 0.321429 | 0.500000 | 0.464286 |
LOG_SUM_OF_FRO | 0.247638 | 0.163783 | 0.642857 | 0.607143 | |
PARAM_NORM | 0.366667 | 0.392857 | 0.714286 | 0.678571 | |
p-value | \(4.718e-10\) | \(8.786e-10\) | |||
SSIM | INVERSE_MARGIN | 0.285714 | 0.142857 | 0.466667 | 0.321429 |
LOG_SUM_OF_FRO | 0.714286 | 0.821429 | 0.166667 | 0.285714 | |
PARAM_NORM | 0.785714 | 0.892857 | 0.10 | 0.357143 | |
p-value | \(1.207e-10\) | \(1.504e-10\) | |||
