A. Weight Initialization We derive a weight initialization scheme tailored to the GLU activation function similar to Glorot & Bengio (2010); He et al. (2015b) by focusing on the variance of activations within the network for both forward and backward passes. We also detail how we modify the weight initialization for dropout. A.1. Forward Pass Assuming that the inputs x l of a convolutional laye...

In this paper, we propose a novel weight initialization method for the Random Neural Network. The method relies on approximating the signal-flow equations of the network to obtain a linear system of equations with nonnegativity constraints. For the solution of the formulated linear Nonnegative Least Squares problem we have developed an improved projected gradient algorithm. It is shown that sup...

The architecture and parameter initialization of wavelet neural network are discussed and a novel initialization method is proposed. The new approach can be regarded as a dynamic clustering procedure which will derive the neuron number as well as the initial value of translation and dilation parameters according to the input patterns and the activating wavelets functions. Three simulation examp...

When simulating hybrid systems using switched bond graphs, the initialization of new modes is made by using a generalization of the principle of momentum conservation. Here it is shown how to use causality propagation to get an efficient initialization algorithm. By looking at causal paths, set of variables that have to be initialized simultaneously are found. Furthermore, it is shown how impul...

In this work, a novel approach for the initialization of switching linear dynamic models (SLDMs) as dynamic models for the trajectory of speech features is proposed. Borrowing ideas from the ”k-means++”-algorithm, the goal of this approach is to find distinctly different SLDMs, modelling the complex dynamics of the speech features, already at the initialization stage of a subsequently following...

In this paper, we present the results of an experimental comparison among seven different weight initialization methods in twelve different problems. The comparison is performed by measuring the speed of convergence, the generalization capability and the probability of successful convergence. It is not usual to find an evaluation of the three properties in the papers on weight initialization. T...