For many emerging applications and environments, information systems designers and implementers must consider the tradeo s between e ciency and the quality of query answers. This exibility, while allowing numerous opportunities for optimization, complicates the development of various system components and their performance evaluation. In this short paper, we rst outline the issues that arise in...

Two methods are used for evaluation of summarization systems: an evaluation of generated summaries against an "ideal" summary and evaluation of how well summaries help a person perform in a task such as informa. tion retrieval. We carried out two large experiments to study the two evaluation methods. Our results show that different parameters of an experiment can (h-amatically affect how well a...

Evaluation techniques are critical to research in all areas of science and engineering. The major evaluation techniques used in information retrieval today were put in place over 20 years ago when Cyril Cleverdon’s Cranfield projects started an evaluation pattern that still largely dominates the field, that is, using recall/precision measures and using one of the standard test collections. Wher...

Although relevance is known to be a multidimensional concept, information retrieval measures mainly consider one dimension of relevance: topicality. In this paper we propose a method to integrate multiple dimensions of relevance in the evaluation of information retrieval systems. This is done within the gain-discount evaluation framework, which underlies measures like rank-biased precision (RBP...

Despite their predominant application in robotics, the utility of methods for information fusion is not limited to sensor-based fusion tasks. The paper presents an information retrieval (IR) system for multimedia weather documents which makes use of linguistic fusion methods and a semantically rich retrieval model based on methods from fuzzy set theory. The computational problem of how to effic...

Because the number of XML documents is dramatically increasing, we need to develop a multimedia information retrieval system which can support both the retrieval based on document structure and the retrieval based on image content. In order to support the structure-based retrieval, we design keyword, structure, element, and attribute index structures by indexing XML documents based on the basic...

Some measures such as mean average precision and recall level precision are considered as good system-oriented measures, because they concern both precision and recall that are two important aspects for effectiveness evaluation of information retrieval systems. However, such good system-oriented measures suffer from some shortcomings when partial relevance judgments are used. In this paper, we ...

This paper presents work on document retrieval for Italian carried out at ITC-irst. Two different approaches to information retrieval were investigated, one based on the Okapi weighting formula and one based on a statistical model. Development experiments were carried out using the Italian sample of the TREC-8 CLIR track. Performance evaluation was done on the Cross Language Evaluation Forum (C...