Using a New Inter-rater Reliability Statistic

Haley, Debra Trusso; Thomas, Pete; Petre, Marian and De Roeck, Anne (2008). Using a New Inter-rater Reliability Statistic. Technical Report 2008/15; Department of Computing, The Open University.

DOI: https://doi.org/10.21954/ou.ro.0001607a

Abstract

This paper discusses methods to evaluate Computer Assisted Assessment (CAA) systems, including some commonly used metrics as well as unconventional ones. I found that most of the methods to measure automated assessment reported in the literature were not useful for my purposes. After much research, I found a new metric, the Gwet AC1 inter-rater reliability (IRR)statistic (Gwet, 2001), that is a good solution for evaluating CAAs. Section 1.6 discusses AC1,but first I describe other possible metrics to motivate why I think that AC1 is the best available for evaluating an automated assessment system. I focus on two types of metrics that I label external and internal metrics. External metrics can be used for reporting and sharing results. Internal metrics are used for comparing results within a research project. Producers of CAAs need an easily understandable external metric to report results to consumers of CAAs, i.e., those wishing to use a particular system. In addition to reporting results to potential consumers, researchers may wish to share their results with other researchers. Finally, and perhaps most important for this dissertation, producers need an internal- metric to quickly compare the results of selecting different parameters of the assessment algorithm. Many choices need to be made when implementing an LSA-based marking system. The LSA literature frequently leaves many of these choices unspecified, including number of dimensions in the reduced matrix, amount and type of training data, types of pre-processing, and weighting functions. The choice of these parameters is an intrinsic aspect of building an LSA marking system. Therefore, researchers need an adequate way to measure and compare the results of the various selections, as I shall explore in this chapter.