The Open UniversitySkip to content
 

Machine and social intelligent peer-assessment systems for assessing large student populations in massive open online education

Jimenez-Romero, Cristian; Johnson, Jeffrey and De Castro, Ricardo (2013). Machine and social intelligent peer-assessment systems for assessing large student populations in massive open online education. In: Proceedings of the 12th European Conference on e-Learning, SKEMA Business School, Sophia Antipolis, France, 30-31 October 2013. Volume 1, Academic Conferences and Publishing International Limited, pp. 598–607.

Full text available as:
[img]
Preview
PDF (Accepted Manuscript) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Download (513kB) | Preview
URL: http://academic-conferences.org/ecel/ecel2013/ecel...
Google Scholar: Look up in Google Scholar

Abstract

The motivation of the European Etoile project is to create high quality free open education in complex systems science, including quality assured certification. Universities and colleges around the world are increasingly using online platforms to offer courses open to the public. Massive Open Online Courses or MOOCs give millions of people access to lectures delivered by prestigious universities. However, although some of these courses provide certification of attendance and completion, most do not provide any academic or professional recognition since this would imply a rigorous and complete evaluation of the student’s achievements. Since the number of students enrolled may exceed tens of thousands, it is impractical for a lecturer (or group of lecturers) to evaluate all students using conventional hand marking. Thus in order to be scalable, assessment must be automated. The state-of-the-art in automated assessment includes various methods and computerised tools including multiple choice questions, and intelligent marking techniques (involving complex semantic analysis). However, none of these completely cover the requirements needed for the implementation of an assessment system able to cope with very large populations of students and also able to guarantee the quality of evaluation required for higher education. The goal of this research is to propose, implement and evaluate a computer mediated social interaction system which can be applied to massive online learning communities. This must be a scalable system able to assess fairly and accurately student coursework and examinations. We call this approach “machine and socially intelligent peer assessment”. This paper describes our system and illustrates its application. Our approach combines the concepts of peer assessment and reputation systems to provide an independent computerised system which determines the degree and type of interaction between student peers based on a reputation score which emerges from the marking behaviour of each student and the interaction with other individuals of the community. A simulation experiment will be reported showing how reputation-based social structure can evolve in our peer marking system. A pilot experiment using a population of ninety 16-year old high school students in Colombia measured the marking accuracy of our system by comparing the statistical differences between the scores resulting from teacher marking (the ‘gold standard’), peer assessment using average scores, and our intelligent reputation-based peer assessment. This addresses the research question: to what extent does the proposed approach improve peer marking in terms of marking accuracy and fairness? We report the first results of this experiment, summarise the lessons learned, and describe further work.

Item Type: Conference or Workshop Item
Copyright Holders: 2013 The Authors
ISBN: 1-909507-82-2, 978-1-909507-82-1
Project Funding Details:
Funded Project NameProject IDFunding Body
Project EtoileNot SetEuropean Commission
Keywords: MOOCs; automated marking; peer assessment; reputation systems; complex systems education; Etoile
Academic Unit/School: Faculty of Science, Technology, Engineering and Mathematics (STEM) > Engineering and Innovation
Faculty of Science, Technology, Engineering and Mathematics (STEM)
Interdisciplinary Research Centre: Centre for Policing Research and Learning (CPRL)
Design and Innovation
Related URLs:
Item ID: 42087
Depositing User: Cristian Jimenez-Romero
Date Deposited: 25 Feb 2015 10:17
Last Modified: 22 Mar 2017 13:59
URI: http://oro.open.ac.uk/id/eprint/42087
Share this page:

Download history for this item

These details should be considered as only a guide to the number of downloads performed manually. Algorithmic methods have been applied in an attempt to remove automated downloads from the displayed statistics but no guarantee can be made as to the accuracy of the figures.

Actions (login may be required)

Policies | Disclaimer

© The Open University   contact the OU