Table 1 Performance metrics for teams meeting inclusion criteria

From: Evaluation of FluSight influenza forecasting in the 2021–22 and 2022–23 seasons with a new target laboratory-confirmed influenza hospitalizations

Model	Absolute WIS	Relative WIS	MAE	50% Coverage (%)	95% Coverage (%)	% of Forecasts Submitted	Log Absolute WIS	Log Relative WIS
2021-22
CMU-TimeSeries^STAT	12.54	0.74	18.92	47	90	100	0.31	0.79
Flusight-ensemble^ENS	13.86	0.82	20.79	48	86	100	0.33	0.83
PSI-DICE^MECH	14.03	0.83	20.17	43	82	100	0.33	0.84
UMass-trends_ensemble^ENS	14.35	0.85	22.24	71	97	100	0.36	0.91
SGroup-RandomForest^ENS	15.45	0.91	23.87	47	95	100	0.38	0.97
CEID-Walk^STAT	15.63	0.93	22.19	52	82	89	0.39	0.98
Flusight-baseline^STAT	16.99	1.00	24.10	49	83	100	0.40	1.00
MOBS-GLEAM_FLUH^MECH	17.17	1.02	22.25	0.32	0.63	91	0.42	1.07
GT-FluFNP^STAT	17.57	1.03	23.40	0.39	0.69	96	0.38	0.98
SigSci-TSENS^ENS	17.79	1.03	24.86	38	72	96	0.40	1.01
IEM_Health-FluProject^STAT	17.69	1.05	23.98	50	85	100	0.40	1.02
CU-ensemble^ENS	18.32	1.08	25.41	44	77	100	0.39	0.98
LUcompUncertLab-TEVA^ENS,STAT	21.02	1.20	29.99	54	86	89	0.41	1.04
UVAFluX-Ensemble^ENS	21.65	1.27	25.76	38	64	99	0.45	1.14
LUcompUncertLab-VAR2_plusCOVID^STAT	22.03	1.30	28.99	42	74	94	0.42	1.08
LUcompUncertLab-VAR2K_plusCOVID^STAT	24.44	1.39	32.43	0.42	0.74	85.19	0.47	1.19
UT_FluCast-Voltaire^STAT	23.64	1.39	35.19	0.50	0.91	95.13	0.45	1.15
LUcompUncertLab-VAR2^STAT	25.93	1.53	35.05	39	72	94	0.53	1.35
LUcompUncertLab-VAR2K^STAT	26.81	1.54	39.35	42	83	89	0.61	1.54
LosAlamos_NAU-CModel_Flu^{STAT, MECH}	28.69	1.70	36.14	26	59	100	0.63	1.62
SGroup-SIkJalpha^STAT	28.94	1.70	38.59	18	46	100	0.49	1.24
GH-Flusight^ENS	30.93	1.81	31.89	6	13	94	0.74	1.88
SigSci-CREG^STAT	27.36	1.97	31.00	19	44	89	0.80	2.06
2022-23
MOBS-GLEAM_FLUH^MECH	42.20	0.61	57.97	42	78	94	0.37	0.66
CMU-TimeSeries^STAT	44.48	0.67	65.94	49	87	94	0.41	0.70
PSI-DICE^MECH	47.45	0.70	63.17	48	80	100	0.40	0.70
MIGHTE-Nsemble^ENS,AI/ML^, ^STAT	48.99	0.73	67.50	53	82	96	0.41	0.70
Flusight-ensemble^ENS	51.72	0.77	71.04	56	81	100	0.44	0.74
UMass-trends_ensemble^ENS	53.86	0.80	79.40	63	89	100	0.49	0.83
GT-FluFNP^STAT	59.75	0.81	72.88	56	75	89		0.90
SGroup-RandomForest^ENS	54.29	0.82	75.98	53	84	97	0.52	0.87
CU-ensemble^ENS	62.23	0.83	75.57	51	70	84	0.51	0.85
CEPH-Rtrend_fluH^STAT	54.20	0.84	70.47	44	78	86.87	0.58	1.07
UGA_flucast-OKeeffe^STAT	62.13	0.93	77.33	50	72	91	0.61	1.02
VTSanghani-ExogModel^AI/ML	72.30	0.98	92.56	30	61	80	0.63	1.04
Flusight-baseline^STAT	67.69	1.00	80.05	49	74	100	0.59	1.00
SigSci-TSENS^ENS	64.27	1.00	80.02	58	74	93	0.66	1.11
UNC_IDD-InfluPaint^STAT	61.14	1.05	77.90	40	75	76	0.52	0.96
UVAFluX-Ensemble^ENS	78.71	1.11	94.45	22	41	95	0.61	1.02
SigSci-CREG^STAT	79.68	1.33	89.29	38	62	91	0.68	1.16
JHU_IDD-CovidSP^MECH	129.16	1.88	174.98	48	80	81	0.49	0.82

Forecast metrics are across all fifty states, D.C., and Puerto Rico forecast targets. The season is indicated in bold in the model column. The Absolute WIS column refers to the Weighted Interval Score for each model. The Relative WIS compares the WIS value of each model to the Flusight-baseline model. All models with a relative WIS score less than one outperformed the baseline model when evaluated solely upon WIS. 95% and 50% coverage values are provided for the percent of times that reported weekly incidence values were within the 95% or 50% prediction intervals, respectively, across all the forecast targets submitted by each team. The percent of forecasts submitted is determined by the number of forecast targets submitted by each team out of all possible forecast targets occurring within the duration of the evaluation period. See Supplementary Table 1 for additional model details. ^ENSEnsemble, ^STATStatistical, ^MECHMechanistic, ^AI/MLArtificial Intelligence/Machine Learning.

Back to article page

Table 1 Performance metrics for teams meeting inclusion criteria

Search

Quick links