Copy the page URI to the clipboard
Zhang, Peng; Song, Dawei; Wang, Jun and Hou, Yuexian
(2013).
DOI: https://doi.org/10.1145/2484028.2484127
URL: http://dl.acm.org/citation.cfm?doid=2484028.248412...
Abstract
It has been recognized that, when an information retrieval (IR) system achieves improvement in mean retrieval effectiveness (e.g. mean average precision (MAP)) over all the queries, the performance (e.g., average precision (AP)) of some individual queries could be hurt, resulting in retrieval instability. Some stability/robustness metrics have been proposed. However, they are often defined separately from the mean effectiveness metric. Consequently, there is a lack of a unified formulation of effectiveness, stability and overall retrieval quality (considering both). In this paper, we present a unified formulation based on the bias-variance decomposition. Correspondingly, a novel evaluation methodology is developed to evaluate the effectiveness and stability in an integrated manner. A case study applying the proposed methodology to evaluation of query language modeling illustrates the usefulness and analytical power of our approach.
Viewing alternatives
Metrics
Public Attention
Altmetrics from AltmetricNumber of Citations
Citations from Dimensions-
Request a copy from the authorVersion of Record (PDF)
This file is not available for public download
Item Actions
Export
About
- Item ORO ID
- 38094
- Item Type
- Conference or Workshop Item
- Extra Information
-
SIGIR '13
Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval
ACM New York, NY, 2013
ISBN: 978-1-4503-2034-4 - Keywords
- bias-variance; decomposition; effectiveness; stability; robustness; evaluation
- Academic Unit or School
-
Faculty of Science, Technology, Engineering and Mathematics (STEM) > Computing and Communications
Faculty of Science, Technology, Engineering and Mathematics (STEM) - Copyright Holders
- © 2013 ACM
- Depositing User
- Dawei Song