The Open UniversitySkip to content
 

Measuring improvement in latent semantic analysis-based marking systems: using a computer to mark questions about HTML

Haley, Debra; Thomas, Pete; De Roeck, Anne and Petre, Marian (2007). Measuring improvement in latent semantic analysis-based marking systems: using a computer to mark questions about HTML. In: ACM 9th International Australasian Computing Education Conference, 30 Jan - 02 Feb 2007, Ballarat, Victoria, Australia.

Full text available as:
Full text not publicly available
Due to copyright restrictions, this file is not available for public download
URL: http://portal.acm.org/citation.cfm?id=1273677
Google Scholar: Look up in Google Scholar

Abstract

This paper proposes two unconventional metrics as animportant tool for assessment research: the Manhattan(L1) and the Euclidean (L2) distance measures. We used them to evaluate the results of a Latent Semantic Analysis(LSA) system to assess short answers to two questionsabout HTML in an introductory computer science class.This is the only study, as far as we know, that addresses the question of how well an LSA-based system can evaluate answers in the very specific and technical language of HTML. We found that, although there are several ways to measure automatic assessment results in the literature, they were not useful for our purposes. We want to compare the marks given by LSA to marks awarded by a human tutor. We demonstrate how L1 and L2 quantify the results of varying the amount of training data necessary to enable LSA to mark the answers to two HTML questions. Although this paper describes the use of the metrics in one particular case, it has more general applicability. Much fine-tuning of an LSA marking system is required for good results. A researcher needs an easy way to evaluate the results of various modifications to the system. The Manhattan and the Euclidean distance measures provide this functionality.

Item Type: Conference Item
Copyright Holders: 2007 Australian Computer Society, Inc.
Extra Information: ACE '07 Proceedings of the ninth Australasian conference on Computing education - Volume 66
ISBN:1-920-68246-5
Keywords: latent semantic analysis; assessment; metrics; measures
Academic Unit/Department: Knowledge Media Institute
Mathematics, Computing and Technology > Computing & Communications
Mathematics, Computing and Technology
Interdisciplinary Research Centre: Centre for Research in Computing (CRC)
Item ID: 24643
Depositing User: Catherine McNulty
Date Deposited: 04 Mar 2011 10:41
Last Modified: 27 May 2011 16:32
URI: http://oro.open.ac.uk/id/eprint/24643
Share this page:

Actions (login may be required)

View Item
Report issue / request change

Policies | Disclaimer

© The Open University   + 44 (0)870 333 4340   general-enquiries@open.ac.uk