Effects of depth, width, and initialization: A convergence analysis of layer-wise training for deep linear neural networks

Deep neural networks have been used in various machine learning applications and achieved tremendous empirical successes. However, training deep is a challenging task. Many alternatives proposed place of end-to-end back-propagation. Layer-wise one them, which trains single layer at time, rather than the whole layers simultaneously. In this paper, we study layer-wise using block coordinate gradient descent (BCGD) for linear networks. We establish general convergence analysis BCGD found optimal rate, results fastest decrease loss. identify effects depth, width, initialization. When orthogonal-like initialization employed, show that width intermediate plays no role gradient-based beyond certain threshold. Besides, use could drastically accelerate when it compared to those depth 1 network, even computational cost considered. Numerical examples are provided justify our theoretical findings demonstrate performance by BCGD.