We report here on the eighth evaluation campaign organized in 2011 by the IWSLT workshop series. That IWSLT 2011 evaluation focused on the automatic translation of public talks and included tracks for speech recognition, speech translation, text translation, and system combination. Unlike in previous years, all data supplied for the evaluation has been publicly released on the workshop website,...

We describe a large-scale investigation of the correlation between human judgments of machine translation quality and the automated metrics that are increasingly used to drive progress in the field. We compare the results of 124 human evaluations of machine translated sentences to the scores generated by two automatic evaluation metrics (BLEU and NIST). When datasets are held constant or file s...

Intelligibility and fidelity are the two key notions in machine translation system evaluation, but do not always provide enough information for system development. Detailed information about the type and number of errors of each type that a translation system makes is important for diagnosing the system, evaluating the translation approach, and allocating development resources. In this paper, w...

In this paper, I attempt to evaluate the effectiveness of using statistical machine translation to translate an English text into Dutch, using empirical evaluation and the Bleu evaluation metric. I also give a brief overview of the theory behind statistical machine translation and automated translation evaluation metrics. I have translated a sample of the English proceedings of the European Par...

For several years we have been conducting Accuracy Based Evaluations (ABE) of the JANUS speech-to-speech MT system (Gates et al., 1997) which measure quality and fidelity of translation. Recently we have begun to design a Task Based Evaluation for JANUS (Thomas, 1999) which measures goal completion. This paper describes what we have learned by comparing the two types of evaluation. Both evaluat...

The NIST Open Handwriting Recognition and Translation Evaluation 2013 (NIST OpenHaRT’13) is a performance evaluation assessing technologies that transcribe and translate text in document images. This evaluation is focused on recognizing Arabic text images and translating them into English. A Handwriting Recognition and Translation system typically consists of a combination of two systems: a Tex...

Idiom translation is a challenging problem in machine translation because the meaning of idioms is non-compositional, and a literal translation is likely to be wrong. In this paper, we assess the quality of idiom translation of a modern neural MT system. We introduce a new evaluation method based on an idiom-specific blacklist of literal translations, based on the insight that the occurrence of...

Measuring translation quality is a subjective process which relies on human judgements. This paper discusses different frameworks used in the process of translation evaluation with special focus on error classification schemes used both in the translation industry and in translation teaching institutions. Such error-based models, like BlackJack or SAE J 2450, allow human judgements to benefit f...

Much attention has been given to the call-by-value continuation passing style (CBV CPS) translation as a tool in partial evaluation, but the call-by-name (CBN) CPS translation has not been investigated. We undertake a systematic investigation of the eeect of CBN CPS in connection with partial evaluation and deforestation. First, we give an example where CBN CPS translation acts as a binding tim...