The Open UniversitySkip to content
 

Using TREC for cross-comparison between classic IR and ontology-based search models at a Web scale

Fernandez, Miriam; Lopez, Vanessa; Sabou, Marta; Uren, Victoria; Vallet, David; Motta, Enrico and Castells, Pablo (2009). Using TREC for cross-comparison between classic IR and ontology-based search models at a Web scale. In: Semantic Search 2009 Workshop at the 18th International World Wide Web Conference (WWW 2009), 20 Apr 2009, Madrid, Spain.

Full text available as:
[img]
Preview
PDF (Version of Record) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Download (389Kb)
URL: http://km.aifb.kit.edu/ws/semsearch09/semse2009_18...
Google Scholar: Look up in Google Scholar

Abstract

The construction of standard datasets and benchmarks to evaluate ontology-based search approaches and to compare then against baseline IR models is a major open problem in the semantic technologies community. In this paper we propose a novel evaluation benchmark for ontology-based IR models based on an adaptation of the well-known Cranfield paradigm (Cleverdon, 1967) traditionally used by the IR community. The proposed benchmark comprises: 1) a text document collection, 2) a set of queries and their corresponding document relevance judgments and 3) a set of ontologies and Knowledge Bases covering the query topics. The document collection and the set of queries and judgments are taken from one of the most widely used datasets in the IR community, the TREC Web track. As a use case example we apply the proposed benchmark to compare a real ontology-based search model (Fernandez, et al., 2008) against the best IR systems of TREC 9 and TREC 2001 competitions. A deep analysis of the strengths and weaknesses of this benchmark and a discussion of how it can be used to evaluate other ontology-based search systems is also included at the end of the paper.

Item Type: Conference Item
Copyright Holders: 2009 The Authors
Keywords: semantic search; information retrieval; evaluation benchmarks
Academic Unit/Department: Knowledge Media Institute
Interdisciplinary Research Centre: Centre for Research in Computing (CRC)
Item ID: 23481
Depositing User: Kay Dave
Date Deposited: 21 Oct 2010 11:33
Last Modified: 26 Oct 2012 18:07
URI: http://oro.open.ac.uk/id/eprint/23481
Share this page:

Actions (login may be required)

View Item
Report issue / request change

Policies | Disclaimer

© The Open University   + 44 (0)870 333 4340   general-enquiries@open.ac.uk