When working in image and video segmentation, the major objective is to design an algorithm producing the appropriate segmentation results for the particular goals of the application addressed. Therefore, the assessment of the segmentation quality assumes a crucial importance to evaluate the likeliness that the application targets are met. Since no well-established methods for objective segment...

We propose and evaluate a family of metrics, named the eXtended Cumulated Gain (XCG) metrics, for the evaluation of content-oriented XML retrieval approaches. Our aim is to provide an evaluation framework that allows to consider the dependency that exists among XML document components and, in particular, incorporate mechanisms to reward the retrieval of so-called nearmisses and to address issue...

According to the requirements and characteristics of software quality comprehensive evaluation, a matter-element extension synthesis evaluation method is introduced, and its shortcomings and problems are also discussed in the practical evaluation process. Especially, the matter-element model, correlation function, matter-element value space and multilevel metrics data transmission in the origin...

The primary topic of this report is ‘metrics.’ More specifically, we discuss quantitative metrics for evaluating Internet services: what should we quantify, monitor and analyze in order to characterize, evaluate and manage services offered over the Internet? We argue that the focus must be on quality-of-experience and, what we call, quality-of-business metrics. These QoE and QoBiz metrics quant...

In this paper we revisit the task of quantitative evaluation of coreference resolution systems. We review the most commonly used metrics (MUC, B, CEAF and BLANC) on the basis of their evaluation of coreference resolution in five texts from the OntoNotes corpus. We examine both the correlation between the metrics and the degree to which our human judgement of coreference resolution agrees with t...

In this paper we present results from an investigation on correlations between subjective and objective evaluation metrics for young people using a conversational agent. The subjective evaluation metrics capture users’ experiences of different aspects of conversations with a virtual agent while the objective evaluation metrics are based on an analysis of the actual conversation between the user...

This work present a novel evaluation framework for topic extraction over user generated contents. The motivation of this work is the development of systems that monitor the evolution of opinionated topics around a certain entity (a person, company or product) in the Web. Currently, due to the effort that would be required to develop a gold standard, topic extraction systems are evaluated qualit...

In this paper, we give a brief introduction of the HTRDP Chinese Information Retrieval Evaluation, which is sponsored by HTRDP (High Technology Research and Development Program of China, namely 863 Program). The web data collection, query design, evaluation metrics, and the evaluation procedures will presents in detail. Like TREC, NTCIR and CLEF, its purpose is to provide the infrastructure nec...

In this paper we give an overview of the evaluation metrics used to measure the performance of backchannel prediction models. Both objective and subjective evaluation metrics are discussed. The survey shows that almost every backchannel prediction model is evaluated with a different evaluation metric. This makes comparison between developed models unreliable, even beside the other variables in ...