The Open UniversitySkip to content

Evaluation methodologies in Automatic Question Generation 2013-2018

Amidei, Jacopo; Piwek, Paul and Willis, Alistair (2018). Evaluation methodologies in Automatic Question Generation 2013-2018. In: Proceedings of The 11th International Natural Language Generation Conference, 5-8 Nov 2018, Tilburg, The Netherlands, pp. 307–317.

Full text available as:
PDF (Version of Record) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Download (267kB) | Preview
Google Scholar: Look up in Google Scholar


In the last few years Automatic Question Generation (AQG) has attracted increasing interest. In this paper we survey the evaluation methodologies used in AQG. Based on a sample of 37 papers, our research shows that the systems’ development has not been accompanied by similar developments in the methodologies used for the systems’ evaluation. Indeed, in the papers we examine here, we find a wide variety of both intrinsic and extrinsic evaluation methodologies. Such diverse evaluation practices make it difficult to reliably compare the quality of different generation systems. Our study suggests that, given the rapidly increasing level of research in the area, a common framework is urgently needed to compare the performance of AQG systems and NLG systems more generally.

Item Type: Conference or Workshop Item
Copyright Holders: 2018 Association for Computational Linguistics
Keywords: Automatic Question Generation; Evaluation; Intrinsic evaluation methodology; Extrinsic evaluation methodology
Academic Unit/School: Faculty of Science, Technology, Engineering and Mathematics (STEM) > Computing and Communications
Faculty of Science, Technology, Engineering and Mathematics (STEM)
Related URLs:
Item ID: 57517
Depositing User: Jacopo Amidei
Date Deposited: 12 Nov 2018 14:45
Last Modified: 03 May 2019 17:32
Share this page:

Download history for this item

These details should be considered as only a guide to the number of downloads performed manually. Algorithmic methods have been applied in an attempt to remove automated downloads from the displayed statistics but no guarantee can be made as to the accuracy of the figures.

Actions (login may be required)

Policies | Disclaimer

© The Open University   contact the OU