Question generation (QG) is the problem of automatically generating questions from inputs such as declarative sentences. The Shared Evaluation Task Challenge (QG-STEC) Task B that took place in 2010 evaluated several state-of-the-art QG systems. However, analysis of the evaluation results was affected by low inter-rater reliability. We adapted Nonaka & Takeuchi’s knowledge creation cycle to the...

A task-oriented evaluation of text understanding systems was prepared and conducted. Nine different NLP systems participated in the evaluation. NOSC collected 150 texts to be used as development (i.e. training) and test data and prepared explanatory documentation on them. The performance task--a simulated database update task--and the expected outputs for each text were defined. A scoring syste...

This paper presents the description and evaluation framework of SemEval-2010 Word Sense Induction & Disambiguation task, as well as the evaluation results of 26 participating systems. In this task, participants were required to induce the senses of 100 target words using a training set, and then disambiguate unseen instances of the same words using the induced senses. Systems’ answers were eval...

In 2001, following the development of International Standards in basic medical education, WFME appointed an international Task Force for development of International Guidelines for Postgraduate Specialist Training. Reports of this Task Force were published in September 2001. These Guidelines has been structured in 9 areas and 37 sub-areas. The areas of these guidelines are mission & outcomes, t...

Many-Task Computing (MTC) is a widely used computing paradigm for large-scale task-parallel processing. One of the key issues in MTC is to schedule a large number of independent tasks onto heterogeneous resources. Traditional task-level scheduling heuristics, like Min-Min, Sufferage and MaxStd, cannot readily be applied in this scenario. As most of MTC tasks are usually fine-grained, the resour...

This study examined the effect of reading tasks on the integration of content and source information from multiple texts. Undergraduate students (N = 102) read five newspaper articles about a fictitious incident in either a summarization task condition or an evaluation task condition. Then, they performed an integration test and a source choice test, which assessed their understanding of a situ...

Recent work on evaluating representation learning architectures in NLP has established a need for evaluation protocols based on subconscious cognitive measures rather than manually tailored intrinsic similarity and relatedness tasks. In this work, we propose a novel evaluation framework that enables large-scale evaluation of such architectures in the free word association (WA) task, which is fi...

English. This report describes the EVENTI (EValuation of Events aNd Temporal Information) task organized within the EVALITA 2014 evaluation campaign. The EVENTI task aims at evaluating the performance of Temporal Information Processing systems on a corpus of Italian news articles. Motivations for the task, datasets, evaluation metrics, and results obtained by participating systems are presented...

System architecture, experimental settings and evaluation results of the EIWA in the WAT2014 Japanese to English (jaen) and Chinese to Japanese (zh-ja) tasks are described. Our system is combining rule-based machine translation (RBMT) and statistical post-editing (SPE). Evaluation results for ja-en task show 19.86 BLEU score, 0.7067 RIBES score, and 22.50 human evaluation score. Evaluation resu...