Fig. 5: p-curve mixture models can detect and discriminate between differences in effect prevalence and effect size between groups or conditions.

a We simulated EEG data by inserting a simulated evoked response into background noise from a real EEG recording. b On each simulation, we simulated two groups of p-values for within-participant tests of the evoked response, manipulating either the prevalence of the evoked response, on left (1000 simulations), or its magnitude in those who show the effect, on right (1000 simulations). The model was highly sensitive at detecting prevalence or effect size differences between independent groups of participants, on top, or between two within-participant conditions, on bottom. False positives, here, refers to mistaking a prevalence increase for a power/effect size increase or vice versa. c The detection rate at the 5% false positive rate is compared to the sensitivity of a group-level NHST with significance level 0.05. In contrast to the normative interpretation of a significant difference in group mean, NHST was highly sensitive to changes in prevalence but less sensitive to changes in effect size than p-curve mixtures. In the within-group case, there is no apparent sensitivity cost to using p-curve mixtures, which can dissociate between differences in prevalence and within-participant power/effect size.