The Open UniversitySkip to content
 

Investigating the Dynamic Decision Mechanisms of Users’ Relevance Judgment for Information Retrieval via Log Analysis

Su, Yi; Li, Jingfei; Song, Dawei; Zhang, Pengqing and Zhang, Yazhou (2018). Investigating the Dynamic Decision Mechanisms of Users’ Relevance Judgment for Information Retrieval via Log Analysis. In: Proceedings of PRICAI 2018: Trends in Artificial Intelligence, Part 1, Lecture Notes in Computer Science, pp. 968–979.

DOI (Digital Object Identifier) Link: https://doi.org/10.1007/978-3-319-97304-3_74
Google Scholar: Look up in Google Scholar

Abstract

Measuring relevance of documents with respect to a user’s query is at the heart of information retrieval (IR), where the user’s relevance judgment criteria have been recognized as multi-dimensional. A set of relevance dimensions that are considered as critical factors in document relevance judgment have been investigated, such as topicality, novelty, and reliability. However, most existing work focuses on individual relevance dimensions, yet neglecting how different dimensions would interact with each other to influence the overall relevance judgment in real-world search scenarios. This paper aims at an initial step to fill the gap. Specifically, we divide 7 relevance dimensions in an enriched Multidimensional User Relevance Model (MURM) into three categories according to three main requirements for document relevance, i.e., document content requirement, document quality requirement and personalization requirement. We then exploit the Learning to Rank framework to conduct document ranking experiments on a query log dataset from a prominent search engine. The experimental results indicate the existence of an order effect between different dimensions, and suggest that considering different dimensions across categories in different orders for document relevance judgment could lead to distinct search results. Our findings provide valuable insights to build more intelligent and user-centric information retrieval systems, and potentially benefit other natural language processing tasks that involve decision making from multiple perspectives.

Item Type: Conference or Workshop Item
ISSN: 0302-9743
Academic Unit/School: Faculty of Science, Technology, Engineering and Mathematics (STEM) > Computing and Communications
Faculty of Science, Technology, Engineering and Mathematics (STEM)
Item ID: 66274
Depositing User: ORO Import
Date Deposited: 22 Aug 2019 08:17
Last Modified: 22 Aug 2019 08:32
URI: http://oro.open.ac.uk/id/eprint/66274
Share this page:

Metrics

Altmetrics from Altmetric

Citations from Dimensions

Actions (login may be required)

Policies | Disclaimer

© The Open University   contact the OU