Delving into Variance Transmission and Normalization: Shift of Average Gradient Makes the Network Collapse

Normalization operations are essential for state-of-the-art neural networks and enable us to train a network from scratch with large learning rate (LR). We attempt explain the real effect of Batch (BN) perspective variance transmission by investigating relationship between BN Weights (WN). In this work, we demonstrate that problem shift average gradient will amplify every convolutional (conv) layer. propose Parametric Standardization (PWS), fast robust mini-batch size module used conv filters, solve gradient. PWS can provide speed-up BN. Besides, it has less computation does not change output enables converge without normalizing outputs. This result enhances persuasiveness explains why works transmission. The code appendix be made available on https://github.com/lyxzzz/PWSConv.