The Open UniversitySkip to content
 

Citations and Readership are Poor Indicators of Research Excellence: Introducing TrueImpactDataset, a New Dataset for Validating Research Evaluation Metrics

Herrmannova, Drahomira; Patton, Robert; Knoth, Petr and Stahl, Christopher (2017). Citations and Readership are Poor Indicators of Research Excellence: Introducing TrueImpactDataset, a New Dataset for Validating Research Evaluation Metrics. In: 1st Workshop on Scholarly Web Mining, 10 Feb 2017, Cambridge, UK, ACM, pp. 41–48.

Full text available as:
[img]
Preview
PDF (Version of Record) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Download (266kB) | Preview
URL: https://dl.acm.org/citation.cfm?id=3057154
DOI (Digital Object Identifier) Link: https://doi.org/10.1145/3057148.3057154
Google Scholar: Look up in Google Scholar

Abstract

In this paper we show that citation counts and Mendeley readership are poor indicators of research excellence. Our experimental design builds on the assumption that a good evaluation metric should be able to distinguish publications that have changed a research field from those that have not. The experiment has been conducted on a new dataset for bibliometric research which we call TrueImpactDataset. TrueImpactDataset is a collection of research publications of two types -- research papers which are considered seminal work in their area and papers which provide a survey (a literature review) of a research area. The dataset also contains related metadata, which include DOIs, titles, authors and abstracts. We describe how the dataset was built and provide overview statistics of the dataset. We propose to use the dataset for validating research evaluation metrics. By using this data, we show that widely used research metrics only poorly distinguish excellent research.

Item Type: Conference or Workshop Item
Copyright Holders: 2017 Association for Computing Machinery
ISBN: 1-4503-5240-5, 978-1-4503-5240-6
Keywords: information retrieval; scholarly communication; publication datasets; data mining; research evaluation; semantometrics
Academic Unit/School: Faculty of Science, Technology, Engineering and Mathematics (STEM) > Knowledge Media Institute (KMi)
Faculty of Science, Technology, Engineering and Mathematics (STEM)
Research Group: Big Scientific Data and Text Analytics Group (BSDTAG)
Item ID: 59079
Depositing User: Drahomira Herrmannova
Date Deposited: 11 Feb 2019 10:59
Last Modified: 03 May 2019 06:00
URI: http://oro.open.ac.uk/id/eprint/59079
Share this page:

Metrics

Altmetrics from Altmetric

Citations from Dimensions

Download history for this item

These details should be considered as only a guide to the number of downloads performed manually. Algorithmic methods have been applied in an attempt to remove automated downloads from the displayed statistics but no guarantee can be made as to the accuracy of the figures.

Actions (login may be required)

Policies | Disclaimer

© The Open University   contact the OU