Copy the page URI to the clipboard
Ullmann, Thomas (2023). Trialling Generative AI for course content production. PVC-Challenges Response; Quality Enhancement and Innovation, The Institute of Educational Technology, The Open University, Milton Keynes, UK.
Abstract
This report outlines our first trial of using AI to produce course content. With the recent AI developments with Large Language Models and chatbot interfaces, such as ChatGPT, Natural Language Processing technologies have become available to a broad audience. These systems can produce plausible responses to a wide range of queries but not necessarily correct ones. The performance of these systems has improved with each new version, improving the quality of responses and allowing more natural interaction.
These systems have knowledge about the properties of language, but they do not have knowledge about the world, which limits their ability. They can be combined with other systems, such as knowledge-based systems, to overcome some of their limitations.
Methodological approach:
Large language models and chatbot interfaces are already used to produce content on a massive scale, making them potentially useful for the purposes of curriculum production. The PVC-RI of The Open University commissioned work to develop a curriculum with AI technology, especially Generative AI, such as ChatGPT. This report is one of three reports for the first stage of this work. It reports the first trial of using Azure AI Chat to generate content for seven common tasks of curriculum production, including:
* Define the central questions of a course and learning outcomes.
* Produce instructions for an assessment.
* Produce course content.
* Produce activity following a specific framework of The Open University (OU Learning Design Framework).
* Assess content according to a specific framework of The Open University (Inclusive Curriculum Design Tool).
* Revise content following feedback.
* Map Knowledge, Skills, and Behaviours to learning outcomes.
We selected those tasks as they are key components of innovative curriculum design processes developed for microcredentials and relevant for the course production of courses of The Open University. For each of the seven tasks, we developed several prompts that worked towards the overall aim of the task. In this report, for each prompt, we show an output of the AI as well as provide our evaluation of the AI response.
Main findings:
From this first trial of the Azure AI Chat technology, we found that across all activities, Azure Chat AI can produce content that could be helpful towards completing the proposed set of tasks. The AI can aid with brainstorming ideas towards solving the task, create a first outline, conduct a first assessment of adherence to specific writing guidance (such as readability and inclusive language), and conduct mappings between teaching frameworks.
In all cases, we noted that the content would need adjustments and checking, ranging from small sections to rewriting generated ideas to make them more suitable for the context of The Open University. We would also need to double-check the correctness of automatically conducted tasks, such as the automated mapping from one framework to another.
The AI chat created responses within seconds. This potentially can reduce the time it takes to complete similar activities and tasks in the future. These time savings should be compared with the time it takes to understand the technology, create and shape prompts, and check and rewrite the responses to the prompts. The last point is not a new requirement for a module team, as content generated by the different module team members needs checking and endorsement by the production team. The extent of manual changes to the content and checks also depends on the prompt, as often small changes to the prompt lead to better responses. We also note that the quality of the responses improved with each version of ChatGPT, and overall, the technological development is rapid, and we expect this to improve further.
We also need to ensure that potentially spending less time on module production is less likely to impact the quality of our offering. Therefore, we need to understand the impact on the quality of the course materials, how students learn from these contents, and how teachers teach them.
The breadth of the AI responses suggests that this technology can work for a broad range of purposes as a generalist. However, we still need to examine to which degree it can adapt to context-specific problems and work as a specialist incorporating subject-specific, pedagogical, and university-specific knowledge.
Finally, we need to understand how the introduction of Generative AI into production teams influences team dynamics and whether Generative AI is useful for all team members or whether some team members benefit more from Generative AI than others. Less experienced module production team members could benefit more from Generative AI than experienced ones.
CORE (COnnecting REpositories)
CORE (COnnecting REpositories)
