We propose rectified factor networks (RFNs) as generative unsupervised models, which learn robust, very sparse, and non-linear codes with many code units. RFN learning is a variational expectation maximization (EM) algorithm with unknown prior which includes (i) rectified posterior means, (ii) normalized signals of hidden units, and (iii) dropout. Like factor analysis, RFNs explain the data var...

Markov random fields (MRF’s), or undirected graphical models, provide a powerful framework for modeling complex dependencies among random variables. Maximum likelihood learning in MRF’s is hard due to the presence of the global normalizing constant. In this paper we consider a class of stochastic approximation algorithms of the Robbins-Monro type that use Markov chain Monte Carlo to do approxim...

We present a new learning algorithm for Boltzmann machines that contain many layers of hidden variables. Data-dependent expectations are estimated using a variational approximation that tends to focus on a single mode, and dataindependent expectations are approximated using persistent Markov chains. The use of two quite different techniques for estimating the two types of expectation that enter...

Reduction of NO to N2O by denitrifiying bacteria is catalyzed either by a monomeric quinol-nitric oxide reductase (qNor) or by a heterodimeric cytochrome c-dependent nitric oxide reductase (cNor). In ancient thermophilic bacteria belonging to the Thermales and Aquificales phylogenetic groups, the cluster encoding the cNor includes a small third gene (norH), in addition to those encoding homolog...

Unsupervised feature extractors are known to perform an efficient and discriminative representation of data. Insight into the mappings they perform and human ability to understand them, however, remain very limited. This is especially prominent when multilayer deep learning architectures are used. This paper demonstrates how to remove these bottlenecks within the architecture of non-negativity ...

We present a mixture model whose components are Restricted Boltzmann Machines (RBMs). This possibility has not been considered before because computing the partition function of an RBM is intractable, which appears to make learning a mixture of RBMs intractable as well. Surprisingly, when formulated as a third-order Boltzmann machine, such a mixture model can be learned tractably using contrast...

In this article, we propose BNAS-v2 to further improve the efficiency of broad neural architecture search (BNAS), which employs a convolutional network (BCNN) as space. BNAS, single-path sampling-updating strategy an overparameterized BCNN leads terrible unfair training issue, restricts improvement. To mitigate employ continuous relaxation optimize all paths simultaneously. However, performance...