Normalization Techniques in Deep Neural Networks

Normalization techniques have become integral to the training of deep neural networks, serving to stabilise learning dynamics, accelerate convergence and improve generality. At their core, these methods re-centre and re-scale intermediate activations or parameters to mitigate issues such as internal covariate shift. They can broadly be categorised by the dimension over which statistics are computed: batch-level, layer-level, channel- or group-level, and instance-level. Batch normalization, the most widely adopted form, computes the mean and variance across each feature channel over a mini-batch, followed by trainable shift and scale parameters. Layer normalization operates across features within a single sample, making it well suited to recurrent architectures and small-batch regimes. Instance normalization and group normalization further refine this idea by normalising per feature map or per channel group, respectively, offering greater flexibility in tasks such as style transfer and object detection. More recent developments include normalization-free networks, which rely on carefully designed initialisation schemes and architectural tweaks to achieve stable training without explicit normalisation layers. Collectively, these approaches underpin modern models in computer vision, natural language processing and reinforcement learning, enabling faster training, enhanced robustness and a wide range of practical applications from medical imaging to autonomous systems.

Research from Nature Portfolio

No recent Nature Portfolio content available.

Research from all publishers

Contemporary studies have interrogated the relative contributions of each component in batch normalization. An empirical analysis has isolated the role of shift and scale parameters, demonstrating that much of the performance gain arises from improved weight initialisation rather than the normalisation of activations per se. Building on this insight, a new variant termed GhostNorm independently normalises sub-groups within a mini-batch, uncovering a unique regularisation effect that extends beyond conventional batch statistics. Sequential Normalization then applies this group-based normalisation across successive dimensions, yielding consistent performance improvements on benchmarks such as CIFAR-10, CIFAR-100 and ImageNet. In parallel, a biologically inspired method named Habituation Normalization draws on the “negative image” habituation model from olfactory neuroscience to filter redundant information when batch sizes are small. This technique has proven effective in resource-constrained settings, delivering stable accuracy across varied architectures including LeNet-5, VGG16 and ResNet-50, and demonstrating strong robustness in fine-tuning scenarios for EEG signal applications.

Normalization Techniques in Deep Neural Networks publication trend

The graph below shows the total number of articles in normalization techniques in deep neural networks across all publications each year (not limited to Nature Index journals).

Technical terms

Batch Normalization: A layer that normalises the outputs of neurons across a mini-batch to have zero mean and unit variance, followed by learnable scale and shift parameters.

Layer Normalization: A normalisation method that computes statistics across all features of a single sample, independent of batch size.

Instance Normalization: A form of layer normalisation applied separately to each feature map, commonly used in image style transfer.

Group Normalization: A technique that divides channels into groups and normalises within each group, balancing between batch and layer normalisation.

GhostNorm: A batch-normalization variant that partitions a mini-batch into sub-groups and normalises each group independently, introducing enhanced regularisation.

Sequential Normalization: The sequential application of sub-group normalisation across multiple dimensions of the input tensor to improve generalisation.

Habituation Normalization: A biologically inspired method that filters redundant activation patterns via a “negative image” mechanism before computing normalisation statistics, stabilising training on small batches.

References

1. An empirical analysis of the shift and scale parameters in BatchNorm. Information Sciences (2023).
2. Sequential Normalization: Embracing Smaller Sample Sizes for Normalization. Information (2022).
3. Improving Network Training on Resource-Constrained Devices via Habituation Normalization †. Sensors (2022).
4. A Robust Initialization of Residual Blocks for Effective ResNet Training Without Batch Normalization. IEEE Transactions on Neural Networks and Learning Systems (2025).