In this paper, we give a brief introduction of the HTRDP Chinese Information Retrieval Evaluation, which is sponsored by HTRDP (High Technology Research and Development Program of China, namely 863 Program). The web data collection, query design, evaluation metrics, and the evaluation procedures will presents in detail. Like TREC, NTCIR and CLEF, its purpose is to provide the infrastructure nec...

The evaluation of effectiveness in Information Retrieval systems has been developed in parallel to its evolution, generating a great amount of proposals to achieve this process. This paper focuses on a particular task of Music Information Retrieval: a system for Cover Song Identification. We present a concrete example and then try to elucidate which metrics work best to evaluate such a system. ...

User evaluation in the domain of Music Information Retrieval (MIR) has been very scarce, while algorithms and systems in MIR have been improving rapidly. With the maturity of system-centered evaluation in MIR, time is ripe for MIR evaluation to involve users. In this study, we compare user-centered measures to a system effectiveness measure on the task of retrieving similar songs. To collect us...

ImageCLEF’s Wikipedia Image Retrieval task provides a testbed for the system-oriented evaluation of multimedia and multilingual information retrieval from a collection of Wikipedia images. The aim is to investigate retrieval approaches in the context of a large and heterogeneous collection of images (similar to those encountered on the Web) that are searched for by users with diverse informatio...

Music Information Retrieval may be perceived as part of the larger Multimedia Information Retrieval research area. However, many researchers in Music Information Retrieval are unaware that the problems they deal with have analogous problems in image and video retrieval. Many issues concerning the creation of testbed digital libraries and effective benchmarking of information retrieval systems a...

objective: this research was aimed to measure the number of duplicated items in persian information retrieval from the web, and to compare search engines' abilities in removing duplicated items from the results.methodology: quantitative methods and descriptive statistics and evaluation of information retrieval systems.findings: the findings showed that in subjects such as law, geography, l...

With the emerging needs of creating fairness-aware solutions for search and recommendation systems, a daunting challenge exists evaluating such solutions. While many traditional information retrieval (IR) metrics can capture relevance, diversity, novelty utility with respect to users, they are not suitable inferring whether presented results fair from perspective responsible exposure. On other ...

ImageCLEF’s Wikipedia Retrieval task provides a testbed for the system-oriented evaluation of multimedia information retrieval from a collection of Wikipedia images. The aim is to investigate retrieval approaches in the context of a large and heterogeneous collection of images (similar to those encountered on the Web) that are searched for by users with diverse information needs. This paper pre...