From Hype to Evidence: Exploring Large Language Models for Inter-Group Bias Classification in Higher Education

Albuquerque, Josmario; Rienties, Bart; Holmes, Wayne and Hlosta, Martin (2024). From Hype to Evidence: Exploring Large Language Models for Inter-Group Bias Classification in Higher Education. Interactive Learning Environments (In Press).

DOI: https://doi.org/10.1080/10494820.2024.2408554

Abstract

Inter-group biases can diminish student achievements in several ways. Yet, manually identifying those biases in vast learning texts is challenging because of their subtle nature. In the light of processing nuanced language, approaches based on large language models (LLMs) have emerged as promising mechanisms (e.g., OpenAI's ChatGPT). However, their potential for classifying bias in learning text seems under-explored. This study examines the ability of three LLMs (BERT, GPT, and PaLM) to classify inter-group bias within learning texts. Accordingly, 2024 sentences from 91 Higher Education (HE) courses at The Open University UK were assessed for potential biases by LLM-based approaches. Then, we used a sub-sample of sentences (n = 30) for comparison with student classifications. The models suggested varying degrees of potential biases within the larger sample (BERT = 49.5%; GPT = 10.2%; PaLM = 11.0%). However, varied agreement levels were found within the sub-samples. Specifically, there was no agreement between GPT and both the algorithms and students (Kappa = .0). In contrast, Kappa values for comparisons among other models and between models and humans ranged from .33 to .77. This underscores the complexity of assessing inter-group bias in learning texts where context and culture can play a crucial role. We discuss the relevance of those potential biases for learning settings and advocate for tools that are both context- and culture-aware, ensuring more inclusive learning experiences.

Viewing alternatives

Metrics

Item Actions

Export

You can export this page using these formats Digital Object Identifier - DOI