Note on Weight Noise Injection During Training a MLP

Although many analytical works have been done to investigate the change of prediction error of a trained NN if its weights are injected by noise, seldom of them has truly investigated on the dynamical properties (such as objective functions and convergence behavior) of injecting weight noise during training. In this paper, four different online weight noise injection training algorithms for multilayer perceptron (MLP) are analyzed and their objective functions are derived. Most importance, the objective function of injecting multiplicative weight noise during training is shown to be different from the prediction error of a trained MLP if its weights are injected by the same multiplicative weight noise. It provides a firm response to a question being posed for 14 years [8]: Can deterministic penalty terms model the effects of synaptic weight noise on network fault-tolerance?. Besides, we show that the objective function of injecting additive weight noise during training is equivalent to adding a regularizer penalizing the magnitude of the gradient vector of the MLP output with respect to its weight vector. Finally, the issue on their convergence proofs will be discussed.