Evaluating semantic search tools using the SEALS Platform

In common with many state of the art semantic technologies, there is a lack of comprehensive, established evaluation mechanisms for semantic search tools. In this paper, we describe a new evaluation and benchmarking approach for semantic search tools using the infrastructure under development within the SEALS initiative. To our knowledge, it is the first effort to present a comprehensive evaluation methodology for semantic search tools. The paper describes the evaluation methodology including our two-phase approach in which tools are evaluated both in a fully automated fashion as well as within a user-based study. We also present and discuss preliminary results from the first SEALS evaluation campaign together with a discussion of some of the key findings. Posted at the Zurich Open Repository and Archive, University of Zurich ZORA URL: https://doi.org/10.5167/uzh-44862 Originally published at: Wrigley, S N; Elbedweihy, K; Reinhardt, D; Bernstein, A; Ciravegna, F (2010). Evaluating semantic search tools using the SEALS platform. In: International Workshop on Evaluation of Semantic Technologies (IWEST 2010) Workshop, Shanghai, China, 8 November 2010 8 November 2010. Evaluating Semantic Search Tools using the SEALS platform� Stuart N. Wrigley1, Khadija Elbedweihy1, Dorothee Reinhard2, Abraham Bernstein2, and Fabio Ciravegna1 1 University of Sheffield, Regent Court, 211 Portobello, Sheffield, UK {s.wrigley, k.elbedweihy, f.ciravegna}@dcs.shef.ac.uk 2 University of Zürich, Binzmühlestrasse 14, CH-8050 Zürich, Switzerland {dreinhard, bernstein}@ifi.uzh.ch Abstract. In common with many state of the art semantic technologies, there is a lack of comprehensive, established evaluation mechanisms for In common with many state of the art semantic technologies, there is a lack of comprehensive, established evaluation mechanisms for semantic search tools. In this paper, we describe a new evaluation and benchmarking approach for semantic search tools using the infrastructure under development within the SEALS initiative. To our knowledge, it is the first effort to present a comprehensive evaluation methodology for semantic search tools. The paper describes the evaluation methodology including our two-phase approach in which tools are evaluated both in a fully automated fashion as well as within a user-based study. We also present and discuss preliminary results from the first SEALS evaluation campaign together with a discussion of some of the key findings.