Can MT Output be Evaluated Through Eye Tracking?
نویسندگان
چکیده
This paper reports on a preliminary study testing the use of eye tracking as a method for evaluating machine translation output. 50 French machine translated sentences, 25 rated as excellent and 25 rated as poor in an earlier human evaluation, were selected. 10 native speakers of French were instructed to read the MT sentences for comprehensibility. Their eye gaze data were recorded noninvasively using a Tobii 1750 eye tracker. They were also asked to record retrospective protocols while watching a replay of their eye gaze reading data. The average gaze time and fixation count were found to be significantly higher for the “bad” sentences, while average fixation duration was not significantly different. Evaluative comments uttered during the retrospective protocols were also found to agree to a satisfactory degree with previous human evaluation. Overall, we found that the eye tracking method correlates reasonably well with human evaluation of MT output.
منابع مشابه
Eye Tracking as an Automatic MT Evaluation Technique
Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc for some time now. Its application as a technique for automatically measuring the reading ease of MT output has not yet, to our knowledge, been tested. We report here on a preliminary study testing the use and validity of an eye tracking methodolog...
متن کاملMethods for human evaluation of machine translation
Evaluation of machine translation (MT) is a difficult task, both for humans, and using automatic metrics. The main difficulty lies in the fact that there is not one single correct translation, but many alternative good translation options. MT systems are often evaluated using automatic metrics, which commonly rely on comparing a translation to only a single human reference translation. An alter...
متن کاملApplication of eye tracking technology for screening and rehabilitation of children with special needs.
Eye tracking is a technology for measuring eye movements when looking at a point. The length of time to look at a particular situation, the sequence of observations, as well as points of attention are processed in the process of eye tracking and eye movements are identified by topics such as fixation, saccade, etc. Information about how the eyes move at a special moment has many potential appli...
متن کاملiAppraise: A Manual Machine Translation Evaluation Environment Supporting Eye-tracking
We present iAppraise: an open-source framework that enables the use of eye-tracking for MT evaluation. It connects Appraise, an opensource toolkit for MT evaluation, to a low-cost eye-tracking device, to make its usage accessible to a broader audience. It also provides a set of tools for extracting and exploiting gaze data, which facilitate eye-tracking analysis. In this paper, we describe diff...
متن کاملEye Tracking as a Tool for Machine Translation Error Analysis
We present a preliminary study where we use eye tracking as a complement to machine translation (MT) error analysis, the task of identifying and classifying MT errors. We performed a user study where subjects read short texts translated by three MT systems and one human translation, while we gathered eye tracking data. The subjects were also asked comprehension questions about the text, and wer...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2009