Tutorial: Perspectives on Learning with RNNs

We present an overview of current lines of research on learning with recurrent neural networks (RNNs). Topics covered are: understanding and unification of algorithms, theoretical foundations, new efforts to circumvent gradient vanishing, new architectures, and fusion with other learning methods and dynamical systems theory. The structuring guideline is to understand many new approaches as different efforts to regularize and thereby improve recurrent learning. Often this is done on two levels: by restricting the learning objective by constraints, for instance derived from stability conditions or weight normalization, and by imposing architectural constraints as for instance local recurrence.