We present a neural net architecture that can discover hierarchical and re-cursive structure in symbol strings. To detect structure at multiple levels, the architecture has the capability of reducing symbols substrings to single symbols, and makes use of an external stack memory. In terms of formal languages, the architecture can learn to parse strings in an LR(0) context-free grammar. Given tr...

We present a neural net architecture that can discover hierarchical and recursive structure in symbol strings. To detect structure at multiple levels, the architecture has the capability of reducing symbols substrings to single symbols, and makes use of an external stack memory. In terms of formal languages, the architecture can learn to parse strings in an LR(O) contextfree grammar. Given trai...

Learning to Optimize (Li & Malik, 2016) is a recently proposed framework for learning optimization algorithms using reinforcement learning. In this paper, we explore learning an optimization algorithm for training shallow neural nets. Such high-dimensional stochastic optimization problems present interesting challenges for existing reinforcement learning algorithms. We develop an extension that...

Much recent functional modelling of the central nervous system, beyond traditional “neural net” approaches, focuses on its distributed computational architecture. This paper discusses extensions of our recent work aimed at understanding this architecture from an overall nonlinear stability and convergence point of view, and at constructing artificial devices exploiting similar modularity. Appli...

In this article a new neural-network architecture suitable for learning and generalization is discussed and developed. Although similar to the radial basis function (RBF) net, our computational model called the net with complex weights (CWN) has demonstrated a considerable gain in performance and efficiency in number of applications compared to RBF net. Its better performance in classification ...

We give a rigorous analysis of the statistical behavior of gradients in randomly initialized feed-forward networks with ReLU activations. Our results show that a fully connected depth d ReLU net with hidden layer widths nj will have exploding and vanishing gradients if and only if ∑d j=1 1/nj is large. The point of view of this article is that whether a given neural net will have exploding/vani...