Comparing machine graded diagrams with human markers: some observations

Thomas, Pete (2004). Comparing machine graded diagrams with human markers: some observations. Technical Report 2004/27; Department of Computing, The Open University.



In this paper we examine the performance of an automatic (machine) grading algorithm for entity-relationship (E-R) diagrams by comparing it with human generated marks for a set of student answers to an assignment question. Using a variety of statistical tests it is shown that the performance of the automatic marker is very close to that of the human markers: the Pearson correlation coefficient is 0.964 (significant at the 0,01 level, 2-tailed, N=26) and the Kendall tau-b correlation coefficient is 0.919 (significant at the 0.01 level, 2-tailed, N=26). The investigation revealed deficiencies in both the machine and human markers. There is prima-facie evidence that the orientation (shape) of a diagram may influence humans to award lower marks than they should.

Viewing alternatives

Download history


Public Attention

Altmetrics from Altmetric

Number of Citations

Citations from Dimensions

Item Actions