Was our pilot study successful? An approach when data are sparse

Edwards, Chris and Whitelock, Denise (2022). Was our pilot study successful? An approach when data are sparse. In: Proceedings of EDEN 2022, 20-22 Jun 2022, Tallinn, Estonia.

URL: https://www.eden-online.org/eden-2022-annual-confe...


Developing new ways to support distance learning students using online technologies can be a challenge at every stage, and the transition from pilot study to embedded offering, available to all students is one of the hardest to complete. Universities are often very supportive of experimentation with new tools but rightly very cautious when approaching the decision to move from exciting project to what is often called business as usual. This paper considers some of the issues that are being experienced on one such development to illustrate some of the issues involved, particularly around evidence of impact where data are insufficient.

We consider a development that has succeeded in its technical aims to make an automatic feedback tool to support students in developing their academic writing but for which a pilot study resulted in fewer data than needed for a confident decision. By employing the Technology Acceptance Model, and the concept of plausibility as an alternative view of uncertainty, we show how it is possible to create indicators of impact. Whilst this does not lead to certainty, it does provide a framework that improves confidence in decision making.

The paper concludes that this approach can enable decisions to be made about progressing with a development in educational technology where the data are insufficient for a statistically robust conclusion to be made.

Viewing alternatives

Download history

Item Actions