Fig. 5: Trade-off between underfitting and overfitting.

a An illustrative low-dimensional example in concrete science, consisting of 40 real concrete mixtures (from the publicly accessible concrete compressive strength dataset^24,112 in Table 2) with one input (water/cement ratio) and one output (compressive strength, MPa). The dataset was divided into a training set and a validation set by an 80/20 split. Three polynomial models with polynomial degrees p = 1, 3, or 10 were fitted to the training data. The polynomial of degree three was identified as the optimal model based on the good prediction performance on the validation set, while those of degrees one and ten were showcased as underfitting (too simple to capture patterns in the data) and overfitting (too complex such that noise was learned), respectively. b Error (root mean square error, MPa) in the prediction of the training and validation sets as a function of model complexity (i.e., polynomial degree herein). Simple statistical models may underfit the data, whereas complex models tend to overfit without extreme caution. The optimal model complexity yields the best generalization performance. As machine learning models are generally much more complex than traditional statistical models, proper implementation and validation are required, especially when dealing with small datasets.