Fig. 4: Observers needed to evaluate subjective tests (ONEST) plots.
From: Augmented reality microscopy to bridge trust between AI and pathologists
Overall percent agreement between pathologists on the y-axis vs the number of observers on the x-axis for manual scoring on day 1 (left panel) and AI-assisted scoring on day 2 (right panel). PD-L1 28-8 CPS ≥ 5 cut-off (solid line) and CIs (dotted lines). ONEST analyses indicate that AI-assisted scoring increased overall interobserver agreement. Agreement among any 2 raters using manual scoring is achieved in 77% of the cases (green box left panel) vs 91% of the cases with AI-assisted scoring (green box right panel) resulting in a 14% improvement. Agreement among 11 pathologists using manual scoring is achieved in 43% of the cases (red box left panel) vs 69% of the cases with AI-assisted scoring (red box right panel) resulting in a 26% improvement.
