Hierarchical Phrase-Based Statistical Machine Translation System

The aim of this thesis is to express fundamentals and concepts behind one of the emerging techniques in statistical machine translation (SMT) hierarchical phrase based MT by implementing translation from Hindi to English. Basically hierarchical model extends phrase based models by considering subphrases with the aid of context free grammar (CFG). In other models, syntax based models bear a resemblance to hierarchical models since the former requires corpus annotated with linguistic phrases like noun phrase, verb phrase. Hierarchical model overcomes this weakness of syntax based models since it does not require annotated corpora at all. Most Indian languages lack annotated corpus, so hierarchical models can prove to be handy in Indian to English translation. In terms of realtime implementation and translation quality, hierarchical model can coexist and even compete with state of the art MT systems. An accuracy of 0.16 (BLEU score) establishes the effectiveness of this approach for Hindi to English translation. Secondly, we discuss post editing techniques through implementation on the translation pipeline. Post editing techniques have recently emerged as a tool for improving quality of machine translation. In this thesis, we discuss translation for out of vocabulary (OOV) words, transliteration for named entities and grammar correction. OOV words are words that were not present in training data, but were present in test data. We deal with them using two approaches. Firstly, we check whether the word is a named entity and hence can be transliterated. Secondly, if a word is not a named entity, it is sent to the OOV module where it applies statistical technique like canonical correlation analysis (CCA) to translate an unknown Hindi word. The third approach that we discuss is grammar correction. Grammar correction can be considered as a translation problem from incorrect text to correct text. Grammar correction typically follows two approaches: rule based and statistical. Rule based approaches handle each error differently, and no uniform framework seems to be in place. We introduce a novel technique that uses hierarchical phrase-based statistical machine translation (SMT) for grammar correction. SMT systems provide a uniform platform for any sequence transformation task. Over the years, grammar correction data in electronic form has increased dramatically in quality and quantity making SMT systems feasible for grammar correction. Moreover, better translation models like hierarchical phrase-based SMT can handle errors as complicated as reordering or insertion which were difficult to deal with previously. Secondly, this SMT based correction technique is similar in spirit to human correction, because the system extracts grammar rules from the corpus and later uses these rules to translate incorrect sentences to correct sentences. We describe how to use Joshua, a hierarchical phrase-based SMT system for grammar correction. An accuracy of 0.77 (BLEU score) establishes the efficacy of our approach.