Part-of-Speech Tagging from "Small" Data Sets

Probabilistic approaches to part-of-speech (POS) tagging compile statistics from massive corpora such as the Lancaster-Oslo-Bergen (LOB) corpus. Training on a 900,000 token training corpus, the hidden Markov model (HMM) method easily achieves a 95 per cent success rate on a 100,000 token test corpus. However, even such large corpora contain relatively few words and new words are subsequently encountered in test corpora. For example, the million-token LOB contains only about 45,000 diierent words, most of which occur only once or twice. We nd that 3{4 per cent of tokens in a disjoint test corpus are unseen, that is, unknown to the tagger after training, and cause a signiicant proportion of errors. A corpus representative of all possible tag sequences seems implausible enough, let alone a corpus that also represents, even in small numbers, enough of English to make the problem of unseen words insigniicant. Experimental results connrm that this extreme course is not necessary. Variations on the HMM approach, including ending-based approaches, incremental learning strategies, and the use of approximate distributions, result in a tagger that tags unseen works nearly as accurately as seen words. Although probabilistic approaches to linguistic problems were attempted early this century Zipf,1932], early work was hampered by real diiculties of collecting and managing statistics, not to mention challenges to probabilistic methods in principle. Fast inexpensive computer technology and the availability of tagged electronic corpora such as the million-word Lancaster-Oslo-Bergen (LOB) Corpus Johansson,1980, Johansson et al.,1986] changed this situation, and probabilistic approaches to a variety of natural language processing problems have been popular for some time. A striking success of the probabilistic approach has been the use of hidden Markov models (HMMs) to attach part-of-speech (POS) tags to unrestricted text, for example, Kupiec,1992]. Given an actual stream of text, a sequence of tokens (instances of words) w 1 : : : w n , the HMM method computes the word-tag sequence (or simply tag sequence) t 1 : : : t n that most probably generated the sequence. That is, the HMM method nds the tag sequence that maximizes (42:1) Such probabilities are impossible to collect in practice; furthermore their number is exponential in n. However, assuming that 1) the probability of a tag t i directly depends only on the tag immediately preceding it, and that 2) the probability of any word w i depends only upon the tag t i that produced it, the …