The Open UniversitySkip to content

Automated Analysis of Reflection in Writing: Validating Machine Learning Approaches

Ullmann, Thomas Daniel (2019). Automated Analysis of Reflection in Writing: Validating Machine Learning Approaches. International Journal of Artificial Intelligence in Education, 29(2) pp. 217–257.

Full text available as:
PDF (Version of Record) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Download (907kB) | Preview
DOI (Digital Object Identifier) Link:
Google Scholar: Look up in Google Scholar


Reflective writing is an important educational practice to train reflective thinking. Currently, researchers must manually analyze these writings, limiting practice and research because the analysis is time and resource consuming. This study evaluates whether machine learning can be used to automate this manual analysis. The study investigates eight categories that are often used in models to assess reflective writing, and the evaluation is based on 76 student essays (5,080 sentences) that are largely from third- and second-year health, business, and engineering students. To test the automated analysis of reflection in writings, machine learning models were built based on a random sample of 80% of the sentences. These models were then tested on the remaining 20% of the sentences. Overall, the standardized evaluation shows that five out of eight categories can be detected automatically with substantial or almost perfect reliability, while the other three categories can be detected with moderate reliability (Cohen's κ ranges between .53 and .85). The accuracies of the automated analysis were on average 10% lower than the accuracies of the manual analysis. These findings enable reflection analytics that is immediate and scalable.

Item Type: Journal Item
Copyright Holders: 2019 The Author
ISSN: 1560-4292
Keywords: Assessment and testing of learning outcomes; Automated content analysis; Educational data mining; Learning analytics; Machine learning; Modeling metacognitive skills; Reflection; Reflective writing; Text analysis
Academic Unit/School: Faculty of Wellbeing, Education and Language Studies (WELS)
Research Group: Centre for Research in Education and Educational Technology (CREET)
Item ID: 59168
Depositing User: Thomas Ullmann
Date Deposited: 18 Feb 2019 13:36
Last Modified: 21 May 2019 08:47
Share this page:


Altmetrics from Altmetric

Citations from Dimensions

Download history for this item

These details should be considered as only a guide to the number of downloads performed manually. Algorithmic methods have been applied in an attempt to remove automated downloads from the displayed statistics but no guarantee can be made as to the accuracy of the figures.

Actions (login may be required)

Policies | Disclaimer

© The Open University   contact the OU