Information retrieval evaluation in knowledge acquisition tasks

Ghafourian, Y.; Knoth, P. and Hanbury, A. (2021). Information retrieval evaluation in knowledge acquisition tasks. In: WEPIR 2021: The 3rd Workshop on Evaluation of Personalisation in Information Retrieval at CHIIR 2021, 19 Mar 2021, [Online], pp. 88–95.

URL: http://ceur-ws.org/Vol-2863/paper-10.pdf

Abstract

The Cranfield Paradigm is a widely adopted and the de-facto standard approach to the evaluation of IR systems. However, this approach does not inherently support situations in which the user is acquiring knowledge (is learning) during an information seeking session consisting of the submission of a sequence of queries into an information retrieval system. More specifically, during a situation in which the retrieval of a particular document at the beginning of a session can be considered not relevant (due to the user's lack of knowledge), while it can be considered relevant at a later point in the session (once the user acquired all required prerequisite knowledge). In this position paper, we reflect on the limitations of the Cranfield Paradigm in the context of knowledge acquisition tasks and propose several alternatives. These alternatives are based on the notion of evaluating a session consisting of a sequence of individual queries created to address a specific information need as part of a knowledge acquisition task.

Viewing alternatives

Download history

Item Actions

Export

About

Recommendations