We address the problem of class imbalance in supervised grammatical error detection (GED) for non-native speaker text, which is the result of the low proportion of erroneous examples compared to a large number of error-free examples. Most learning algorithms maximize accuracy which is not a suitable objective for such imbalanced data. For GED, most systems address this issue by tuning hyperpara...

The evaluation of metrics on the data available in change request management (CRM) systems can give valuable information for the management of software development. It can for example be helpful in assessing the current workload, product quality or development process weaknesses. Metrics and charts on change requests are already available in current CRM systems. They provide information about c...

In Information Retrieval (IR), evaluation metrics continuously play an important role. Recently, some risk measures have been proposed to evaluate the downside performance or the performance variance of an assumingly advanced IR method in comparison with a baseline method. In this paper, we propose a novel risk metric, by applying the Value at Risk theory (VaR, which has been widely used in fin...

Evaluation of segment-level machine translation metrics is currently hampered by: (1) low inter-annotator agreement levels in human assessments; (2) lack of an effective mechanism for evaluation of translations of equal quality; and (3) lack of methods of significance testing improvements over a baseline. In this paper, we provide solutions to each of these challenges and outline a new human ev...

OBJECTIVE The article proposes a set of metrics for evaluation of patient performance in physical therapy exercises. METHODS Taxonomy is employed that classifies the metrics into quantitative and qualitative categories, based on the level of abstraction of the captured motion sequences. Further, the quantitative metrics are classified into model-less and model-based metrics, in reference to w...

The evaluation of the quality of ontological classification is an important part of semantic web technology. Because this area is under constant development, it requires improvement and standardisation. This paper discusses existing evaluation metrics, and proposes a new method for evaluating the ontology population task, which is general enough to be used in a variety of situations, yet more p...

Multiple references in machine translation evaluation are usually under-explored: they are ignored by alignment-based metrics and treated as bags of n-grams in string matching evaluation metrics, none of which take full advantage of the recurring information in these references. By exploring information on the n-gram distribution and on divergences in multiple references, we propose a method of...

We describe a dataset containing 16,000 translations produced by four machine translation systems and manually annotated for quality by professional translators. This dataset can be used in a range of tasks assessing machine translation evaluation metrics, from basic correlation analysis to training and test of machine learning-based metrics. By providing a standard dataset for such tasks, we h...

Traditional machine translation evaluation metrics such as BLEU and WER have been widely used, but these metrics have poor correlations with human judgements because they badly represent word similarity and impose strict identity matching. In this paper, we propose some modifications to the traditional measures based on word embeddings for these two metrics. The evaluation results show that our...

Designing meaninful metrics for evaluating MediaEval tasks that are able to capture multiple aspects of system effectiveness and user satisfaction is far from straighforward. A considerable part of the effort in organising such a task must often be devoted to selecting, designing or refining a suitable evaluation metric. We review evaluation metrics from the MediaEval Search and Hyperlinkiing t...