Identifying Annotator Bias: A new IRT-based method for bias identification

Amidei, Jacopo; Piwek, Paul and Willis, Alistair (2020). Identifying Annotator Bias: A new IRT-based method for bias identification. In: Proceedings of The 28th International Conference on Computational Linguistics (COLING), 8-13 Dec 2020, Barcelona, Spain, pp. 4787–4797.

DOI: https://doi.org/10.18653/v1/2020.coling-main.421

URL: https://www.aclweb.org/anthology/2020.coling-main....

Abstract

A basic step in any annotation effort is the measurement of the Inter Annotator Agreement (IAA). An important factor that can affect the IAA is the presence of annotator bias. In this paper we introduce a new interpretation and application of the Item Response Theory (IRT) to detect annotators’ bias. Our interpretation of IRT offers an original bias identification method that can be used to compare annotators’ bias and characterise annotation disagreement. Our method can be used to spot outlier annotators, improve annotation guidelines and provide a better picture of the annotation reliability. Additionally, because scales for IAA interpretation are not generally agreed upon, our bias identification method is valuable as a complement to the IAA value which can help with understanding the annotation disagreement.

Viewing alternatives

Metrics

Item Actions

Export

You can export this page using these formats Digital Object Identifier - DOI