A computationally efficient estimator for mutual information

A computationally efficient estimator for mutual information

Mutual information quantifies the determinism that exists in a relationship between random variables, and thus plays an important role in exploratory data analysis. We investigate a class of non-parametric estimators for mutual information, based on the nearest neighbour structure of observations in both the joint and marginal spaces. Unless both marginal spaces are one-dimensional, we demonstr...

Computationally Efficient Sufficient Dimension Reduction via Squared-Loss Mutual Information

The purpose of sufficient dimension reduction (SDR) is to find a low-dimensional expression of input features that is sufficient for predicting output values. In this paper, we propose a novel distribution-free SDR method called sufficient component analysis (SCA), which is computationally more efficient than existing methods. In our method, a solution is computed by iteratively performing depe...

Statistically and Computationally Efficient Variance Estimator for Kernel Ridge

The Variance of the Mutual Information Estimator

In the case of two signals with independent pairs of observations (xn; yn) a statistic to estimate the variance of the mutual information estimator has been derived earlier. We present such a statistic for dependent pairs. To derive this statistic it is necessary to avail of a reliable statistic to estimate the variance of the sample mean in case of dependent observations. We derive and discuss...

Computationally Efficient Estimation of Squared-Loss Mutual Information with Multiplicative Kernel Models

Squared-loss mutual information (SMI) is a robust measure of the statistical dependence between random variables. The sample-based SMI approximator called least-squares mutual information (LSMI) was demonstrated to be useful in performing various machine learning tasks such as dimension reduction, clustering, and causal inference. The original LSMI approximates the pointwise mutual information ...