Table 1 Methods from statistical process control (SPC) and their application to monitoring ML algorithms.
Method(s) | What the method(s) detect and assumptions | Example uses |
|---|---|---|
CUSUM, EWMA | Detects a shift in the mean of a single variable, given shift size. Assumes the pre-shift mean and variance are known. Extensions can monitor changes in the variance. | • Monitoring changes in individual input variables |
• Monitoring changes in real-valued performance metrics (e.g. monitoring the prediction error) | ||
MCUSUM, MEWMA, Hotelling’s T2 | Monitor changes in the relationship between multiple variables | • Monitoring changes in the relationship between input variables |
Generalized likelihood ratio test (GLRT), Online change point detection | Detects if a change occurred in a data distribution and when. Can be applied if characteristics of the pre- and/or post-shift distributions are unknown. GLRT methods typically make parametric assumptions. Parametric and nonparametric variants exist for online change point detection methods. | • Detecting distributional shifts for individual or multiple input variables |
• Detecting shifts in the conditional distribution of outcome Y given input variables | ||
• Determining whether parametric model recalibration/revision is needed | ||
Generalized fluctuation monitoring | Monitor changes to the residuals or gradient | • Detect when the average gradient of the training loss for a differentiable ML algorithm (e.g. neural network) differs from zero |
