Neural Encoding and Decoding of Visual Stimuli

Understanding how the brain transforms visual input into neural signals (encoding) and reconstructs perceptual or imagined content from those signals (decoding) lies at the heart of visual neuroscience. Encoding studies examine how features of images such as edges, textures and semantic categories are represented across cortical areas, ranging from primary visual cortex (V1) to higher-order regions engaged in object recognition and scene analysis. Decoding approaches invert this mapping to predict the visual content of stimuli or mental imagery from patterns of brain activity, typically recorded with functional magnetic resonance imaging (fMRI) or electrophysiological techniques. Advances in machine learning, in particular deep neural networks, have provided powerful feature models that approximate hierarchical processing in the visual system. By integrating biologically inspired architectures and generative frameworks, recent work has achieved increasingly accurate reconstructions of perceived and imagined images. These developments hold promise for brain–computer interfaces, clinical diagnostics and a deeper mechanistic understanding of visual perception across species.

Research from Nature Portfolio

Recent studies have demonstrated that deep neural networks optimised to predict human visual cortex activity can spontaneously learn serial and parallel representations, revealing that strict hierarchical architectures are not essential for accurate encoding of early visual areas. A novel generative framework has been applied to fMRI signals to reconstruct complex natural scenes in two stages: first capturing low-level structure with a variational autoencoder, then refining high-level semantics using a latent diffusion model conditioned on multimodal features. Foundational work has also shown that hierarchical visual features derived from convolutional neural networks can be decoded from fMRI patterns to identify both seen and imagined objects, extending decoding beyond trained exemplars and demonstrating correspondence between machine-learned and brain-derived representations at different processing levels.

Research from all publishers

Complementary approaches have enhanced voxel-wise encoding models by integrating representations from multiple deep networks with empirically defined cortical networks, yielding improved predictions of whole-brain responses during naturalistic video perception. Earlier work leveraged pre-trained deep models to translate fMRI activity into hierarchical feature maps and used iterative optimisation to reconstruct pixel-level images that closely resemble viewed stimuli, demonstrating the feasibility of mapping neural signals back to visual content. Studies of semantic decoding have applied hierarchical logistic regression to dynamic movie stimuli, enabling accurate classification of object and action categories and revealing how semantic taxonomies are reflected in brain activity beyond mere low-level feature encoding.

Neural Encoding and Decoding of Visual Stimuli publication trend

The graph below shows the total number of articles in neural encoding and decoding of visual stimuli across all publications each year (not limited to Nature Index journals).

Technical terms

Neural encoding: The process by which sensory stimuli are transformed into patterns of neural activity.

Neural decoding: The inference of presented or imagined stimuli from measured neural signals.

Hierarchical representation: A processing organisation in which visual features are extracted at successive levels from simple to complex.

Voxel-wise encoding model: A computational model that predicts the response of each fMRI voxel to stimulus features.

Latent diffusion model: A generative framework that iteratively denoises latent representations to synthesise complex images.

Deep neural network (DNN): A multilayered computational architecture inspired by biological neural networks, used to learn hierarchical feature representations.

fMRI: Functional magnetic resonance imaging, a method for measuring brain activity by detecting changes in blood oxygenation.

References

1. Brain-optimized deep neural network models of human visual areas learn non-hierarchical representations. Nature Communications (2023).
2. Natural scene reconstruction from fMRI signals using generative latent diffusion. Scientific Reports (2023).
3. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications (2017).
4. Enhancing neural encoding models for naturalistic perception with a multi-level integration of deep neural networks and cortical networks. Science Bulletin (2024).
5. Deep image reconstruction from human brain activity. PLOS Computational Biology (2019).
6. Decoding the Semantic Content of Natural Movies from Human Brain Activity. Frontiers in Systems Neuroscience (2016).