Fig. 4: Choice-supportive bias across all models tested and additional datasets (SimpleQA and GSM-MC).
From: Competing Biases underlie Overconfidence and Underconfidence in LLMs
Left: CoM rate in Opposite, Same and Neutral advice conditions, averaged across accuracy of the advice LLM. Right: confidence change from first to second stage in the same conditions. While the choice-supportive bias is evident throughout all advice types, it is most clearly demonstrated in the Neutral Advice condition by: (i) higher CoM rate in Answer Hidden compared with Answer Shown condition, and (ii) positive confidence change in Answer Shown condition (compared with both zero baseline and the Answer Hidden condition). The o1-preview is not included, as no confidence scores were available. SimpleQA is a multiple-choice version of a complex factuality dataset, and GSM-MC is a multiple choice version of the GSM8k math reasoning dataset (Methods). See Supplementary Table 2 for summary data. Bars represent means collapsed across six advice accuracy levels (50%, 60%, 70%, 80%, 90% and 100%), yielding n = 12,000 trials per condition (2,000 unique questions × 6 accuracy levels) for Gemma 12B. For other models, there were n = 3,000 trials per condition. Error bars represent the standard error of the mean across trials.
