Extended Data Fig. 10: Behavioral auditory dominance in conflict trials is stable, independent of performance, and depends on relative stimulus timing.
From: Triple dissociation of visual, auditory and motor processing in mouse primary visual cortex
(a) As a proxy for subjective saliency or arousal, we measured pupil dilation over time for saliency-matched auditory and visual trials. Cropped image shows pupil fit. Line and shading indicate mean ± SEM across N = 40 sessions from 9 mice. All data in this figure are from MST mice. (b) Quantification of maximal pupil dilation. The effect of modality on pupil dilation was tested in a linear mixed model with fixed effects of hit/miss, saliency, modality and random effect of mouse ID. Whether it was a hit or miss had the largest effect (F(1,7530) = 1138.95, p = 1.24*10−232), then saliency (F(1,7524) = 33.36, p = 7.97*10−9, with no effect of modality (F(1,7526) = 1.53, p = 0.2164). This supports the idea that visual and auditory conditions were matched in subjective saliency. Center line, box edges and whiskers show 10th, 25th, 50th, 75th and 90th data percentile. N = 931, 933, 926, 916, 1008, 936, 899, 984, trials in conditions from left to right. (c) Behavioral auditory dominance in an example session. Raster plots show for each trial type licks and rewards at the auditory and visual lick spout (red and blue tick marks respectively) aligned to stimulus change (t = 0). Colored zones indicate response window (0 to 1.5 s). Gray: inter-trial interval. Licks before t = 0 were spontaneous. Note how during conflict trials, auditory licks and rewards dominate. (d) Dominance index (DI) heatmap (as in Fig. 6b) for the only animal out of 17 MST mice) displaying visual dominance in its behavior. (e) A heatmap of the behavioral auditory dominance index for conditions binned based on performance (d-prime) on unimodal trials. This is in contrast to the analyses presented in the main text, where conditions were grouped based on the predetermined saliency gauged by psychophysical performance in previous sessions. The current analysis controls for changes in performance by reassigning each bin of the heatmap to d-prime levels within that session. It can be seen that performance-matched conflict trial conditions (along the diagonal) have positive dominance index values, confirming auditory dominance. (f) The saliency-matched dominance index (smDI) for conflict trials that are matched in performance to unimodal trials (conditions along the bottom-left to top-right diagonal of e) is significantly different from zero (Wilcoxon signed rank test, n = 17 mice, p = 0.030). Grey dot is mean ± SEM, *p < 0.05. In (f) to (l), each dot is the smDI of one animal. (g) Auditory dominance was stable across the session, with auditory dominance computed on the first and second half of sessions being similar (Wilcoxon signed rank test, n = 17 mice, p = 0.492). Grey dot is mean ± SEM. (h) Dominance was not correlated with visual performance (d-prime on unimodal trials of maximal visual saliency in the same sessions; r = −0.29, p = 0.26). (i) Dominance was not correlated with auditory performance (d-prime on unimodal trials of maximal auditory saliency in the same sessions; r = 0.24, p = 0.35). (j) Dominance was not correlated with mean reaction time in visual trials (r = 0.38, p = 0.14). (k) Dominance was not correlated with mean reaction time in auditory trials (r = 0.09, p = 0.73). (l) Reaction times on visual, auditory and conflict trials (n = 8.591, 7.758, 2.266 trials respectively). For conflict trials, only saliency-matched conflicts are shown (Vsub + Asub, Vthr + Athr, etc.). Conflict trials were split based on choice. Mean ± SEM. (m) We varied stimulus onset asynchrony between auditory and visual stimulus changes during conflict trials. The plot shows the percentage of auditory choice (red), visual choice (blue) or no lick (black) during saliency-matched threshold-level conflict trials as a function of stimulus onset asynchrony (SOA). A positive SOA value means that the visual change was presented first, followed by the auditory change. Mean ± SEM across trials. (n) Purple error bars show mean and standard deviation of DI as a function of stimulus-onset asynchrony. Black line and gray shading show bootstrapped cumulative Gaussian fit of DI as a function of SOA (median and 95% confidence interval). Top error bar and dotted line indicate crossover point, that is fitted µ parameter (median and 95% confidence interval). Auditory dominance reverses once the visual stimulus change precedes the auditory change by 89.9 ms (95% CI: 47.7–138.7 ms). This is close to the difference in reaction time between saliency matched auditory and visual conditions: 110.5 ms on average. In other words, when the visual change preceded the auditory change by about 90 ms, auditory dominance was halfway to reversing into visual dominance. Further advancing the visual change in time completely reversed the dominance. This may reflect a scenario in which the visual evidence has instructed the decision-making system to an extent that subjects have already committed to a motor plan (namely to lick the visual spout) before the auditory evidence may take control. Similar temporal dominance of audition over vision has been reported in humans95,96,97.
