A Maximum Entropy Approach to Chinese Word Segmentation

We participated in the Second International Chinese Word Segmentation Bakeoff. Specifically, we evaluated our Chinese word segmenter in the open track, on all four corpora, namely Academia Sinica (AS), City University of Hong Kong (CITYU), Microsoft Research (MSR), and Peking University (PKU). Based on a maximum entropy approach, our word segmenter achieved the highest F measure for AS, CITYU, and PKU, and the second highest for MSR. We found that the use of an external dictionary and additional training corpora of different segmentation standards helped to further improve segmentation accuracy. 1 Chinese Word Segmenter The Chinese word segmenter we built is similar to the maximum entropy word segmenter we employed in our previous work (Ng and Low, 2004). Our word segmenter uses a maximum entropy framework (Ratnaparkhi, 1998; Xue and Shen, 2003) and is trained on manually segmented sentences. It classifies each Chinese character given the features derived from its surrounding context. Each Chinese character can be assigned one of four possible boundary tags: s for a character that occurs as a single-character word, b for a character that begins a multi-character (i.e., two or more characters) word, e for a character that ends a multi-character word, and m for a character that is neither the first nor last in a multi-character word. Our implementation used the opennlp maximum entropy package v2.1.0 from sourceforge.1