Copy the page URI to the clipboard
Jimenez-Romero, Cristian; Johnson, Jeffrey and De Castro, Ricardo
(2013).
URL: http://academic-conferences.org/ecel/ecel2013/ecel...
Abstract
The motivation of the European Etoile project is to create high quality free open education in complex systems science, including quality assured certification. Universities and colleges around the world are increasingly using online platforms to offer courses open to the public. Massive Open Online Courses or MOOCs give millions of people access to lectures delivered by prestigious universities. However, although some of these courses provide certification of attendance and completion, most do not provide any academic or professional recognition since this would imply a rigorous and complete evaluation of the student’s achievements. Since the number of students enrolled may exceed tens of thousands, it is impractical for a lecturer (or group of lecturers) to evaluate all students using conventional hand marking. Thus in order to be scalable, assessment must be automated. The state-of-the-art in automated assessment includes various methods and computerised tools including multiple choice questions, and intelligent marking techniques (involving complex semantic analysis). However, none of these completely cover the requirements needed for the implementation of an assessment system able to cope with very large populations of students and also able to guarantee the quality of evaluation required for higher education. The goal of this research is to propose, implement and evaluate a computer mediated social interaction system which can be applied to massive online learning communities. This must be a scalable system able to assess fairly and accurately student coursework and examinations. We call this approach “machine and socially intelligent peer assessment”. This paper describes our system and illustrates its application. Our approach combines the concepts of peer assessment and reputation systems to provide an independent computerised system which determines the degree and type of interaction between student peers based on a reputation score which emerges from the marking behaviour of each student and the interaction with other individuals of the community. A simulation experiment will be reported showing how reputation-based social structure can evolve in our peer marking system. A pilot experiment using a population of ninety 16-year old high school students in Colombia measured the marking accuracy of our system by comparing the statistical differences between the scores resulting from teacher marking (the ‘gold standard’), peer assessment using average scores, and our intelligent reputation-based peer assessment. This addresses the research question: to what extent does the proposed approach improve peer marking in terms of marking accuracy and fairness? We report the first results of this experiment, summarise the lessons learned, and describe further work.
CORE (COnnecting REpositories)
CORE (COnnecting REpositories)
