Investigating Order Effects in Multidimensional Relevance Judgment using Query Logs

Uprety, Sagar and Song, Dawei (2018). Investigating Order Effects in Multidimensional Relevance Judgment using Query Logs. In: ICTIR '18: Proceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval, ACM Press, New York pp. 191–194.

DOI: https://doi.org/10.1145/3234944.3234972

Abstract

There is a growing body of research which has investigated relevance judgment in IR being influenced by multiple factors or dimensions. At the same time, the Order Effects in sequential decision making have been quantitatively detected and studied in Mathematical Psychology. Combining the two phenomena, there have been some user studies carried out which investigate the Order Effects and thus incompatibility in different dimensions of relevance. In this work, we propose a methodology for carrying out such an investigation in large scale and real world data using query logs of a web search engine, and device a test to detect the presence of an irrational user behavior in relevance judgment of documents. We further validate this behavior through a Quantum Cognitive explanation of the Order and Context effects.

Viewing alternatives

Metrics

Item Actions

Export

You can export this page using these formats Digital Object Identifier - DOI