Squared-loss Mutual Information Regularization: A Novel Information-theoretic Approach to Semi-supervised Learning
نویسندگان
چکیده
We propose squared-loss mutual information regularization (SMIR) for multi-class probabilistic classification, following the information maximization principle. SMIR is convex under mild conditions and thus improves the nonconvexity of mutual information regularization. It offers all of the following four abilities to semi-supervised algorithms: Analytical solution, out-of-sample/multi-class classification, and probabilistic output. Furthermore, novel generalization error bounds are derived. Experiments show SMIR compares favorably with state-of-the-art methods.
منابع مشابه
A Rate Distortion Approach for Semi-Supervised Conditional Random Fields
We propose a novel information theoretic approach for semi-supervised learning of conditional random fields that defines a training objective to combine the conditional likelihood on labeled data and the mutual information on unlabeled data. In contrast to previous minimum conditional entropy semi-supervised discriminative learning methods, our approach is grounded on a more solid foundation, t...
متن کاملInformation-theoretic Semi-supervised Metric Learning via Entropy Regularization
We propose a general information-theoretic approach to semi-supervised metric learning called SERAPH (SEmi-supervised metRic leArning Paradigm with Hypersparsity) that does not rely on the manifold assumption. Given the probability parameterized by a Mahalanobis distance, we maximize its entropy on labeled data and minimize its entropy on unlabeled data following entropy regularization. For met...
متن کاملSemi-supervised information-maximization clustering
Semi-supervised clustering aims to introduce prior knowledge in the decision process of a clustering algorithm. In this paper, we propose a novel semi-supervised clustering algorithm based on the information-maximization principle. The proposed method is an extension of a previous unsupervised information-maximization clustering algorithm based on squared-loss mutual information to effectively ...
متن کاملEstimating Squared-Loss Mutual Information for Independent Component Analysis
Accurately evaluating statistical independence among random variables is a key component of Independent Component Analysis (ICA). In this paper, we employ a squared-loss variant of mutual information as an independence measure and give its estimation method. Our basic idea is to estimate the ratio of probability densities directly without going through density estimation, by which a hard task o...
متن کاملDependence-Maximization Clustering with Least-Squares Mutual Information
Recently, statistical dependence measures such as mutual information and kernelized covariance have been successfully applied to clustering, called dependencemaximization clustering. In this paper, we propose a novel dependencemaximization clustering method based on an estimator of a squared-loss variant of mutual information called least-squares mutual information. A notable advantage of the p...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2013