FDA (the Factorized Distribution Algorithm) is an evolutionary algorithm that combines mutation and recombination by using a distribution. The distribution is estimated from a set of selected points. It is then used to generate new points for the next generation. In general a distribution defined forn binary variables has 2n parameters. Therefore it is too expensive to compute. For additively d...

I. INTRODUCTION Boltzmann machines are probability distributions on high dimensional binary vectors which are analogous to Gaussian Markov Random Fields in that they are fully determined by first and second order moments. A key difference however is that augmenting Boltzmann machines with hidden variables enlarges the class of distributions that can be modeled, so that in principle it is possib...

We review Boltzmann machines and energy-based models. A Boltzmann machine defines a probability distribution over binary-valued patterns. One can learn parameters of a Boltzmann machine via gradient based approaches in a way that log likelihood of data is increased. The gradient and Laplacian of a Boltzmann machine admit beautiful mathematical representations, although computing them is in gene...

Probabilistic language modelling has been widely used in information retrieval. It estimates document models under the multinomial distribution assumption, and uses query likelihood to rank documents. In this paper, we aim to generalize this distribution assumption by exploring the use of fully-observable Boltzmann Machines (BMs) for document modelling. BM is a stochastic recurrent network and ...

We consider the relation between the Boltzmann temperature and the Lagrange multipliers associated with energy average in the nonextensive thermostatistics. In Tsallis’ canonical ensemble, the Boltzmann temperature depends on energy through the probability distribution unless q = 1. It is shown that the so-called ’physical temperature’ introduced in [Phys. Lett. A 281 (2001) 126] is nothing but...

FDA (the Factorized Distribution Algorithm) is an evolutionary algorithm that combines mutation and recombination by using a distribution. The distribution is estimated from a set of selected points. It is then used to generate new points for the next generation. FDA uses a factorization to be able to compute the distribution in polynomial time. Previously, we have shown a convergence theorem f...

The Elitist Convergent Estimation of Distribution Algorithm (ECEDA), is a definition of a class of EDA which guarantees convergence to the optimum. This paper introduces the conceptual ECEDA and a practical approach derived from it, called the Boltzmann Univariate Marginal Distribution Algorithm (BUMDA). The BUMDA uses a Gaussian model to approximate the Boltzmann distribution, requiring only o...

The nonnegative Boltzmann machine (NNBM) is a recurrent neural network model that can describe multimodal nonnegative data. Application of maximum likelihood estimation to this model gives a learning rule that is analogous to the binary Boltzmann machine. We examine the utility of the mean field approximation for the NNBM, and describe how Monte Carlo sampling techniques can be used to learn th...

We present a heuristical procedure for eecient estimation of the partition function in the Boltzmann distribution. The resulting speed-up is of immediate relevance for the speed-up of Boltzmann Machine learning rules, especially for networks with a sparse connectivity.