Kullback–Leibler relative-entropy has unique properties in cases involving distributions resulting from relative-entropy minimization. Tsallis relative-entropy is a one-parameter generalization of Kullback–Leibler relative-entropy in the nonextensive thermostatistics. In this paper, we present the properties of Tsallis relative-entropy minimization and present some differences with the classica...

Regularized (stabilized) versions of exponential and linear forgetting in parameter tracking are shown to be dual to each other. Both are derived by solving essentially the same Bayesian decision-making problem where Kullback-Leibler divergence is used to measure (quasi)distance between posterior probability distributions of estimated parameters. The type of forgetting depends solely on the ord...

Exact inference in large and complex graphical models (e.g. Bayesian networks) is computationally intractable. Approximate schemes are therefore of great importance for real world computation. In this paper we consider a general scheme in which the original intractable graphical model is approximated by a model with a tractable structure. The approximating model is optimised by an iterative pro...

This paper offers a gentle introduction to probability for linguists, assuming little or no background beyond what one learns in high school. The most important points that we emphasize are: the conceptual difference between probability and frequency, the use of maximizing probability of an observation by considering different models, and Kullback-Leibler divergence. Nous offrons une introducti...

Recent Automatic Speech Recognition (ASR) studies have shown that Kullback-Leibler diverge based hidden Markov models (KL-HMMs) are very powerful when only small amounts of training data are available. However, since the KL-HMMs use a cost function that is based on the Kullback-Leibler divergence (instead of maximum likelihood), standard ASR algorithms such as the commonly used decision tree cl...

A frequent practice in feature selection is to maximize the Kullback-Leibler (K-L) distance between target classes. In this note we show that this common custom is frequently suboptimal, since it fails to take into account the fact that classification occurs using a finite number of samples. In classification, the variance and higher order moments of the likelihood function should be taken into...