The Open UniversitySkip to content
 

The use of rating and Likert scales in Natural Language Generation human evaluation tasks: A review and some recommendations

Amidei, Jacopo; Piwek, Paul and Willis, Alistair (2019). The use of rating and Likert scales in Natural Language Generation human evaluation tasks: A review and some recommendations. In: Proceedings of the 12th International Conference on Natural Language Generation, 29 Oct - 1 Nov 2019, Tokyo, Japan.

Full text available as:
[img]
Preview
PDF (Version of Record) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Download (354kB) | Preview
Google Scholar: Look up in Google Scholar

Abstract

Rating and Likert scales are widely used in evaluation experiments to measure the quality of Natural Language Generation (NLG) systems. We review the use of rating and Likert scales for NLG evaluation tasks published in NLG specialized conferences over the last ten years (135 papers in total). Our analysis brings to light a number of deviations from good practice in their use. We conclude with some recommendations about the use of such scales. Our aim is to encourage the appropriate use of evaluation methodologies in the NLG community.

Item Type: Conference or Workshop Item
Academic Unit/School: Faculty of Science, Technology, Engineering and Mathematics (STEM) > Computing and Communications
Faculty of Science, Technology, Engineering and Mathematics (STEM)
Item ID: 67963
Depositing User: Jacopo Amidei
Date Deposited: 04 Nov 2019 09:15
Last Modified: 24 Mar 2020 15:18
URI: http://oro.open.ac.uk/id/eprint/67963
Share this page:

Download history for this item

These details should be considered as only a guide to the number of downloads performed manually. Algorithmic methods have been applied in an attempt to remove automated downloads from the displayed statistics but no guarantee can be made as to the accuracy of the figures.

Actions (login may be required)

Policies | Disclaimer

© The Open University   contact the OU