Was our pilot study successful? An approach when data are sparse

Edwards, Chris and Whitelock, Denise (2022). Was our pilot study successful? An approach when data are sparse. In: Proceedings of EDEN 2022, 20-22 Jun 2022, Tallinn, Estonia.

URL: https://www.eden-online.org/eden-2022-annual-confe...

Abstract

Developing new ways to support distance learning students using online technologies can be a challenge at every stage, and the transition from pilot study to embedded offering, available to all students is one of the hardest to complete. Universities are often very supportive of experimentation with new tools but rightly very cautious when approaching the decision to move from exciting project to what is often called business as usual. This paper considers some of the issues that are being experienced on one such development to illustrate some of the issues involved, particularly around evidence of impact where data are insufficient.

We consider a development that has succeeded in its technical aims to make an automatic feedback tool to support students in developing their academic writing but for which a pilot study resulted in fewer data than needed for a confident decision. By employing the Technology Acceptance Model, and the concept of plausibility as an alternative view of uncertainty, we show how it is possible to create indicators of impact. Whilst this does not lead to certainty, it does provide a framework that improves confidence in decision making.

The paper concludes that this approach can enable decisions to be made about progressing with a development in educational technology where the data are insufficient for a statistically robust conclusion to be made.

Viewing alternatives

Item Actions

Export

You can export this page using these formats

About

• Item ORO ID
• 83860
• Item Type
• Conference or Workshop Item
• Project Funding Details

• Funded Project Name	Project ID	Funding Body
  Test & Learn - Open Essay Optimiser	Not Set	The Open University (OU)

• Keywords
• Assessment; Feedback; Automated feedback; Educational Technology
• Academic Unit or School
• Institute of Educational Technology (IET)
• Depositing User
• Chris Edwards