The Open UniversitySkip to content
 

Collecting Reliable Human Judgements on Machine-Generated Language: The Case of the QG-STEC Data

Godwin, Keith and Piwek, Paul (2016). Collecting Reliable Human Judgements on Machine-Generated Language: The Case of the QG-STEC Data. In: Proceedings of the 9th International Natural Language Generation Conference (Isard, Amy; Rieser, Verena and Gkatzia, Dimitra eds.), Association for Computational Linguistics, Edinburgh, pp. 212–216.

Full text available as:
[img]
Preview
PDF (Version of Record) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Download (233kB) | Preview
URL: http://www.macs.hw.ac.uk/InteractionLab/INLG2016/#
Google Scholar: Look up in Google Scholar

Abstract

Question generation (QG) is the problem of automatically generating questions from inputs such as declarative sentences. The Shared Evaluation Task Challenge (QG-STEC) Task B that took place in 2010 evaluated several state-of-the-art QG systems. However, analysis of the evaluation results was affected by low inter-rater reliability. We adapted Nonaka & Takeuchi’s knowledge creation cycle to the task of improving the evaluation annotation guidelines with a preliminary test showing clearly improved inter-rater reliability.

Item Type: Conference or Workshop Item
Copyright Holders: 2016 Association for Computational Linguistics
Keywords: natural language generation; corpus annotation; inter-rater reliability; QGSTEC
Academic Unit/School: Faculty of Science, Technology, Engineering and Mathematics (STEM) > Computing and Communications
Faculty of Science, Technology, Engineering and Mathematics (STEM)
Research Group: Centre for Research in Computing (CRC)
Health and Wellbeing PRA (Priority Research Area)
Item ID: 47284
Depositing User: Paul Piwek
Date Deposited: 29 Sep 2016 09:10
Last Modified: 02 May 2018 14:22
URI: http://oro.open.ac.uk/id/eprint/47284
Share this page:

Download history for this item

These details should be considered as only a guide to the number of downloads performed manually. Algorithmic methods have been applied in an attempt to remove automated downloads from the displayed statistics but no guarantee can be made as to the accuracy of the figures.

Actions (login may be required)

Policies | Disclaimer

© The Open University   contact the OU