A Web-based Approach To Chinese Word Segmentation

Chinese text processing requires the detection of word boundaries. This is a non-trivial step because Chinese does not contain explicit whitespace between words. Existing word segmentation techniques make use of precompiled dictionaries and treebanks. The creation of dictionaries and treebanks is a labor-intensive process and consequently they are updated infrequently. Furthermore, due to their static nature, dictionaries and treebanks lack the latest words that enter the lexicon. This paper proposes a way to leverage content on the Internet to build a bootstrapping Chinese word segmenter. The segmenter can perform automatic updates allowing it to incorporate the latest lexicon. Keywords-text mining; parsing; corpus linguistics; word segmentation; out-of-vocabulary words; automatic language processing