The Open UniversitySkip to content
 

Seeing the whole picture: evaluating automated assessment systems

Haley, Debra; Thomas, Pete; De Roeck, Anne and Petre, Marian (2007). Seeing the whole picture: evaluating automated assessment systems. Innovation in Teaching And Learning in Information and Computer Sciences, 6(4) pp. 203–224.

URL: http://www.ics.heacademy.ac.uk/italics/vol6iss4.ht...
Google Scholar: Look up in Google Scholar

Abstract

This paper argues that automated assessment systems can be useful for both students and educators provided that its results correspond well with human markers. Thus, evaluating such a system is crucial. We present an evaluation framework and show why it can be useful for both producers and consumers of automated assessment. The framework builds on previous work to analyse Latent Semantic Analysis- (LSA) based systems, a particular type of automated assessment, that produced a research taxonomy that could help developers publish their results in a format that is comprehensive, relatively compact, and useful to other researchers. The paper contends that, in order to see a complete picture of an automated assessment system, certain pieces must be emphasised. It presents the framework as a jigsaw puzzle whose pieces join together to form the whole picture and provides an example of the utility of the framework by presenting some empirical results from our assessment system that marks questions about html. Finally, the paper suggests that the framework is not limited to LSA-based systems. With slight modifications, it can be applied to any automated assessment system.

Item Type: Journal Article
Copyright Holders: 2007 Information and Computer Science Subject Centre of the Higher Education Academy
ISSN: 1473-7507
Project Funding Details:
Funded Project NameProject IDFunding Body
Not SetIST-002205European Community under the Innovation Society Technologies (IST) programme of the 6th Framework Programme for RTD - project ELeGI
Keywords: automated assessment systems; computer aided assessment; CAA; latent semantic systems; LSA systems; teaching programming
Academic Unit/Department: Knowledge Media Institute
Mathematics, Computing and Technology
Mathematics, Computing and Technology > Computing & Communications
Interdisciplinary Research Centre: Centre for Research in Computing (CRC)
Item ID: 24635
Depositing User: Catherine McNulty
Date Deposited: 18 Nov 2010 21:57
Last Modified: 02 Jan 2014 14:48
URI: http://oro.open.ac.uk/id/eprint/24635
Share this page:

Actions (login may be required)

View Item
Report issue / request change

Policies | Disclaimer

© The Open University   + 44 (0)870 333 4340   general-enquiries@open.ac.uk