A vector space model of semantics using Canonical Correlation Analysis

We present an efficient method that uses canonical correlation analysis (CCA) between words and their contexts (i.e., the neighboring words) to estimate a real-valued vector for each word that characterizes its “hidden state” or “meaning”. The use of CCA allows us to prove theorems characterizing how accurately we can estimate this hidden state. Recently developed algorithms for computing the required singular eigenvectors make it easy to compute models for billions of words of text with vocabularies in the hundreds of thousands. Experiments on the Google-ngram collection show that CCA between words and their contexts provides a mapping from each word to a low dimensional feature vector that captures information about the part of speech and meanings of the word. Unlike latent semantic analysis, which uses PCA, our method takes advantage of the information implicit in the word sequences. 1 The problem of state estimation Many people have clustered words based on their distributionally similarity (see, amongmany articles, (Pereira et al., 1993)), but such clustering ignores the many different dimensions that similarity could be computed on. We instead, characterize words using a vector space model (Turney and Pantel, 2010). We present a method for learning language models that estimates a “state” or latent variable representation for words based on their context. The vector learned for each word captures a wide variety of information about it, allowing us to predict the word’s part of speech, linguist features such as animacy, membership in wide variety of semantic categories (foods, drinks, colors, males, females ...) and direction of sentiment (happy vs. sad). More precisely, we estimate a hidden state associated with words by computing the dominant canonical correlations between target words and the words in their immediate context. The main computation, finding the singular value decomposition of a scaled version of the co-occurrence matrix of counts of words with their contexts, can be done highly efficiently. Use of CCA also allows us to prove theorems about the optimality of our reconstruction of the state. Our CCA-based multi-view learning method can be thought of as a generalization of widely used latent semantic analysis (LSA) methods. LSA (sometimes called LSI, when used for indexing) uses a principle component analysis (PCA) of the cooccurrence matrix between words and the documents they occur in to learn a latent vector representation of each word type (Landauer et al., 2008). We extend this method in two ways: (1) By looking at the correlation between words and the words in their context, we find a latent vector for each word that depends upon the sequence of nearby words, unlike the bag-of-words model standardly used in LSA. (2) We rescale the covariance matrix between words and context to give a correlation matrix (i.e., use CCA), allowing us to prove theorems about how accurately we can estimate the state. Unlike PCA, in CCA the dominant singular vectors are guaranteed to capture the key state information. Importantly, our method scales well, handling gigaword corpora with vocabularies of tens of thousands of words on a small computer. Using CCA to estimate a latent vector for each word based on the contexts it appears in, gives a significantly different (and as we show below, often more useful) representation than, for example, taking the PCA of the same data used to generate the CCA. We show below how to efficiently compute a vector that characterizes each word type by using the right singular values of the above CCA to map from the word space (size v) to the state space (size k). We call this mapping the attribute dictionary for words, as it associates with every word a vector that captures that word’s attributes. As will be made clear below, the attribute dictionary is arbitrary up to a rotation, but captures the information needed for any linear model to predict properties of the words such as part of speech or word sense. Characterizing words by a low dimensional (e.g. 50 dimensional) vector as we do is valuable in that words can be partitioned in many ways. They can be grouped based on part of speech, or by many aspects of their “meaning” including features such as animacy, gender, whether the word describes something that is good to eat or drink, whether it refers to a year or to a small number, etc. Such features can be used, for example, as features for word sense disambiguation or named entity disambiguation, or to help improve parsing algorithms.