Identifying Annotator Bias: A new IRT-based method for bias identification

Amidei, Jacopo; Piwek, Paul and Willis, Alistair (2020). Identifying Annotator Bias: A new IRT-based method for bias identification. In: Proceedings of The 28th International Conference on Computational Linguistics (COLING), 8-13 Dec 2020, Barcelona, Spain, pp. 4787–4797.

DOI: https://doi.org/10.18653/v1/2020.coling-main.421

URL: https://www.aclweb.org/anthology/2020.coling-main....

Abstract

A basic step in any annotation effort is the measurement of the Inter Annotator Agreement (IAA). An important factor that can affect the IAA is the presence of annotator bias. In this paper we introduce a new interpretation and application of the Item Response Theory (IRT) to detect annotators’ bias. Our interpretation of IRT offers an original bias identification method that can be used to compare annotators’ bias and characterise annotation disagreement. Our method can be used to spot outlier annotators, improve annotation guidelines and provide a better picture of the annotation reliability. Additionally, because scales for IAA interpretation are not generally agreed upon, our bias identification method is valuable as a complement to the IAA value which can help with understanding the annotation disagreement.

Viewing alternatives

Look up in Google Scholar

Download history

Metrics

Public Attention

Altmetrics from Altmetric

Number of Citations

Citations from Dimensions

Item Actions

Export

About

CORE (COnnecting REpositories)

Recommendations