A Task-Oriented Evaluation Metric for Machine Translation

Evaluation remains an open and fundamental issue for machine translation (MT). The inherent subjectivity of any judgment about the quality of translation, whether human or machine, and the diversity of end uses and users of translated material, contribute to the difficulty of establishing relevant and efficient evaluation methods. The US Federal Intelligent Document Understanding Laboratory (FIDUL) is developing a new, task-oriented evaluation metric and methodology to measure MT systems in light of the tasks for which their output may be used. This paper describes the development of this methodology for Japanese-to-English MT. It includes a sample inventory of the tasks for which translated material is used, (e.g., filtering, detection, extraction) and describes exercises in which users perform each task with MT output. The methodology correlates the recorded subjective judgments of the raters in the DARPA MT Evaluation with users' performances on the task-based exercises. Analysis of the errors in scored texts determines whether the presence of certain error types in MT affects specific tasks and not others. Source language patterns that produced errors become a test set that can be easily and efficiently scored to evaluate the performance of any new Japanese-to-English MT system in terms of the task inventory.