Truncated Backpropagation Through Time (truncated BPTT, Jaeger (2005)) is a widespread method for learning recurrent computational graphs. Truncated BPTT keeps the computational benefits of Backpropagation Through Time (BPTT Werbos (1990)) while relieving the need for a complete backtrack through the whole data sequence at every step. However, truncation favors short-term dependencies: the grad...

On-line Backpropagation has become very popular and it has been the subject of in-depth theoretical analyses and massive experimentation. Yet, after almost three decades from its publication, it is still surprisingly the source of tough theoretical questions and of experimental results that are somewhat shrouded in mystery. Although seriously plagued by local minima, the batch-mode version of t...

The paper investigates the statistical eeects which may need to be exploited in supervised learning. It notes that these eeects can be classiied according to their conditionality and their order and proposes that learning algorithms will typically have some form of bias towards particular classes of eeect. It presents the results of an empirical study of the statistical bias of backpropagation....