The Open UniversitySkip to content
 

Bias-variance decomposition of IR evaluation

Zhang, Peng; Song, Dawei; Wang, Jun and Hou, Yuexian (2013). Bias-variance decomposition of IR evaluation. In: 36th international ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR2013), 28 Jul - 1 Aug 2013, Dublin, Ireland, pp. 1021–1024.

Full text available as:
Full text not publicly available (Version of Record)
Due to publisher licensing restrictions, this file is not available for public download
Click here to request a copy from the OU Author.
URL: http://dl.acm.org/citation.cfm?doid=2484028.248412...
DOI (Digital Object Identifier) Link: https://doi.org/10.1145/2484028.2484127
Google Scholar: Look up in Google Scholar

Abstract

It has been recognized that, when an information retrieval (IR) system achieves improvement in mean retrieval effectiveness (e.g. mean average precision (MAP)) over all the queries, the performance (e.g., average precision (AP)) of some individual queries could be hurt, resulting in retrieval instability. Some stability/robustness metrics have been proposed. However, they are often defined separately from the mean effectiveness metric. Consequently, there is a lack of a unified formulation of effectiveness, stability and overall retrieval quality (considering both). In this paper, we present a unified formulation based on the bias-variance decomposition. Correspondingly, a novel evaluation methodology is developed to evaluate the effectiveness and stability in an integrated manner. A case study applying the proposed methodology to evaluation of query language modeling illustrates the usefulness and analytical power of our approach.

Item Type: Conference or Workshop Item
Copyright Holders: 2013 ACM
Extra Information: SIGIR '13
Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval
ACM New York, NY, 2013
ISBN: 978-1-4503-2034-4
Keywords: bias-variance; decomposition; effectiveness; stability; robustness; evaluation
Academic Unit/School: Faculty of Science, Technology, Engineering and Mathematics (STEM) > Computing and Communications
Faculty of Science, Technology, Engineering and Mathematics (STEM)
Item ID: 38094
Depositing User: Dawei Song
Date Deposited: 02 Aug 2013 09:20
Last Modified: 02 May 2018 13:53
URI: http://oro.open.ac.uk/id/eprint/38094
Share this page:

Metrics

Altmetrics from Altmetric

Citations from Dimensions

Actions (login may be required)

Policies | Disclaimer

© The Open University   contact the OU