In this article it will be shown that morpheme length in Lakota obeys a special multimodal distribution resulting from a difference equation of second order. This is due to the particular structure of Lakota syllables, which provide the building blocks for morphemes.

In this work, we announce the Morfessor 1.0 software package, which is a program that takes as input a corpus of raw text and produces a segmentation of the word forms observed in the text. The segmentation obtained often resembles a linguistic morpheme segmentation. In addition, we briefly describe the Hutmegs package, also publicly available for research purposes. Hutmegs contains semi-automa...

We propose a novel lexicon acquirer that works in concert with the morphological analyzer and has the ability to run in online mode. Every time a sentence is analyzed, it detects unknown morphemes, enumerates candidates and selects the best candidates by comparing multiple examples kept in the storage. When a morpheme is unambiguously selected, the lexicon acquirer updates the dictionary of the...

This paper describes an unsupervised dynamic graphical model for morphological segmentation and bilingual morpheme alignment for statistical machine translation. The model extends Hidden Semi-Markov chain models by using factored output nodes and special structures for its conditional probability distributions. It relies on morpho-syntactic and lexical source-side information (part-of-speech, m...

Stemming is a process that groups morphologically related words into the same class and is widely used in information retrieval for improving recall rate. Here we study a set of statistical stemmers for Kannada, a resource-poor language with highly inflectional and agglutinative morphology. We compare stemming using simple truncation, clustering and an unsupervised morpheme segmentation algorit...

This paper describes the work that we did at Indian School of Mines, Dhanbad for FIRE 2012. This year we participated in two tasks: Adhoc Retrieval Task and Morpheme Extraction Task (MET). Within the adhoc task, we participated in two monolingual retrieval activities, namely English and Hindi using Lemur and Indri search engine respectively. We submitted a total of 6 runs (3 in English and Hind...

This paper mainly applies the qualia structure theory to the study of compound nouns whose lexical meanings cannot be inferred from their morpheme meanings by taking some undirected disyllabic compound nouns as example from the Chinese Semantic Word-formation Database. The paper concludes some specific ways by which morpheme meanings can be integrated with lexical meanings. It is hoped that the...

Language models are fundamental for many natural language processing applications. The most widely used type of language models are the corpus-based probabilistic ones. These models provide an estimate of the probability of a word sequence W based on training data. Therefore, large amounts of training data are required in order to ensure statistical significance. But even if the training data a...

Morphology is the study of how meaningful components of form are combined to make complex words. Understanding how such complex words can be ‘broken apart’ into their morphological constituents is the problem of morpheme segmentation. While words that have similar meanings tend to share similar forms (e.g., run and running), many morphemes do not have transparently shared meanings. For example,...

This paper describes a morphological analysis method of continuous spoken Korean to solve the integration problem of speech recognition and natural language processing. The method centers on a Viterbi search-based morphological analysis on top of speech signal processing and MLP-based phone recognition. The main contribution of this paper is to introduce a Viterbi search-based morphological ana...