High-Order Contrasts for Independent Component Analysis

Given an n × 1 random vector X, independent component analysis (ICA) consists of finding a basis of Rn on which the coefficients of X are as independent as possible (in some appropriate sense). The change of basis can be represented by an n × n matrix B and the new coefficients given by the entries of vector Y = BX. When the observation vector X is modeled as a linear superposition of source signals, matrix B is understood as a separating matrix, and vector Y = BX is a vector of source signals. Two key issues of ICA are the definition of a measure of independence and the design of algorithms to find the change of basis (or separating matrix) B optimizing this measure. Many recent contributions to the ICA problem in the neural network literature describe stochastic gradient algorithms involving as an essential device in their learning rule a nonlinear activation function. Other ideas for ICA, most of them found in the signal processing literature, exploit the algebraic structure of high-order moments of the observations. They are often regarded as being unreliable, inaccurate, slowly convergent, and utterly sensitive to outliers. As a matter of fact, it is fairly easy to devise an ICA method displaying all these flaws and working on only carefully generated synthetic data sets. This may be the reason that cumulant-based algebraic methods are largely ignored by the researchers of the neural network community involved in ICA. This article tries to correct this view by showing how high-order correlations can be efficiently exploited to reveal independent components. This article describes several ICA algorithms that may be called Jacobi algorithms because they seek to maximize measures of independence by a technique akin to the Jacobi method of diagonalization. These measures of independence are based on fourth-order correlations between the entries of Y. As a benefit, these algorithms evades the curse of gradient descent: