Automatic evaluation metrics for Machine Translation (MT) systems, such as BLEU and the related NIST metric, are becoming increasingly important in MT. Yet, their behaviors are not fully understood. In this paper, we analyze some flaws in the BLEU/NIST metrics. With a better understanding of these problems, we can better interpret the reported BLEU/NIST scores. In addition, this paper reports a...

We describe two metrics for automatic evaluation of machine translation quality. These metrics, BLEU and NEE, are compared to human judgment of quality of translation of Arabic, Chinese, French, and Spanish documents into English.

Paraphrasing of reference translations has been shown to improve the correlation with human judgements in automatic evaluation of machine translation (MT) outputs. In this work, we present a new dataset for evaluating English-Czech translation based on automatic paraphrases. We compare this dataset with an existing set of manually created paraphrases and find that even automatic paraphrases can...

There are still a lot of uncertainties about the evaluation of machine translation (MT) systems. In this report, we try to establish a catalog of criteria which we consider to be crucial for a systematic assessment of MT systems; the paper is divided into three parts: technical, linguistic and entrepreneurial criteria. Furthermore, we give hints on how to organize an evaluation project.

In this paper, we report on the results of a full-size evaluation campaign of various MT systems. This campaign is novel compared to the classical DARPA/NIST MT evaluation campaigns in the sense that French is the target language, and that it includes an experiment of meta-evaluation of various metrics claiming to better predict different attributes of translation quality. We first describe the...

this paper intends to analyze the attitudes of translation criticism researchers in order to present a theoretical map for translation criticism studies, extract the investigators' learning resources, infer translation criticism criteria, and determine the kind, quality, and degree of interrelationship between translation criticism variables. the linguists, translatologists, and translator...