Coping with Data-sparsity in Example-based Machine Translation

Data-driven Machine Translation (MT) systems have been found to require large amounts of data to function well. However, obtaining parallel texts for many languages is time-consuming, expensive and difficult. This thesis aims at improving translation quality for languages that have limited resources by making use of the available data more efficiently. Templates or generalizations of sentence-pairs where sequences of one or more words are replaced by variables are used in the translation model to handle data-sparsity challenges. Templates are built from clusters or equivalence classes that group related terms (words and phrases). As generating such clusters can be time-consuming, clusters are automatically generated by grouping terms based on their semantic-similarity, syntactical-coherence and context. Data-sparsity is also a big challenge in statistical language modeling. In many MT systems, sophisticated tools are developed to make the translation models better but they still rely heavily on a restricted-decoder which uses unreliable language models that may not be well suited for translation tasks especially in sparse-data scenarios. Templates can also be used in Language Modeling. Limited training data also increases the number of out-of-vocabulary words and reduces the quality of the translations. Many of the present MT systems either ignore these unknown words or pass them on as is to the final translation assuming that they could be proper nouns. Presence of out-of-vocabulary words and rare words in the input sentence prevents an MT system from finding longer phrasal matches and produces low quality translations due to less reliable language model estimates. Approaches in the past have suggested using stems and synonyms of OOV words as replacements. This thesis uses an algorithm to find possible replacements which are not necessarily synonyms to replace out-of-vocabulary words as well as rare words based on the context in which these words appear. The effectiveness of each of the template-based approaches both in the translation model and in the language model are demonstrated for English →Chinese and English→French. The algorithm to handle out-of-vocabulary and rare words are tested on English →French, English →Chinese and English→Haitian. A Hybrid approach combining all the techniques is also studied in English→Chinese.