Seeing the Whole Picture: Comparing Computer Assisted Assessment Systems using LSA-based Systems as an Example

Haley, Debra Trusso; Thomas, Pete; Petre, Marian and De Roeck, Anne (2007). Seeing the Whole Picture: Comparing Computer Assisted Assessment Systems using LSA-based Systems as an Example. Technical Report 2007/07; Department of Computing, The Open University.

DOI: https://doi.org/10.21954/ou.ro.00016068

Abstract

This paper presents a framework for evaluating computer assisted assessment (CAA) systems. It discusses why the framework can be useful for both producers and consumers of these automatic aids to assessing learners. The framework builds on previous work to analyse Latent Semantic Analysis-(LSA) based systems, a particular type of CAA, that produced a research taxonomy that could help LSA CAA developers to publish their results in a format that is comprehensive, relatively compact, and useful to other researchers. The paper contends that, in order to see a complete picture of a CAA system, certain pieces must be emphasised. It presents the framework as a jigsaw puzzle whose pieces join together to form the whole picture and provides an example of the utility of the framework by presenting some empirical results from our LSA-based CAA system that marks questions about html. Finally, the paper suggests that the framework is not limited to LSA-based systems. With slight modifications, it can be applied to any CAA system.