The Open UniversitySkip to content

Peer review and citation data in predicting university rankings, a large-scale analysis

Pride, David and Knoth, Petr (2018). Peer review and citation data in predicting university rankings, a large-scale analysis. In: Theory and Practice of Digital Libraries (TPDL) 2018, 10-13 Sep 2018, University of Porto, Portugal.

Full text available as:
PDF (Accepted Manuscript) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Download (816kB) | Preview
DOI (Digital Object Identifier) Link:
Google Scholar: Look up in Google Scholar


Most Performance-based Research Funding Systems (PRFS) draw on peer review and bibliometric indicators, two different method- ologies which are sometimes combined. A common argument against the use of indicators in such research evaluation exercises is their low corre- lation at the article level with peer review judgments. In this study, we analyse 191,000 papers from 154 higher education institutes which were peer reviewed in a national research evaluation exercise. We combine these data with 6.95 million citations to the original papers. We show that when citation-based indicators are applied at the institutional or departmental level, rather than at the level of individual papers, surpris- ingly large correlations with peer review judgments can be observed, up to r <= 0.802, n = 37, p < 0.001 for some disciplines. In our evaluation of ranking prediction performance based on citation data, we show we can reduce the mean rank prediction error by 25% compared to previous work. This suggests that citation-based indicators are sufficiently aligned with peer review results at the institutional level to be used to lessen the overall burden of peer review on national evaluation exercises leading to considerable cost savings.

Item Type: Conference or Workshop Item
Copyright Holders: 2018 Springer Nature
ISBN: 3-030-00066-4, 978-3-030-00066-0
Keywords: REF; citation analysis; bibliometrics; peer review; semantometrics
Academic Unit/School: Faculty of Science, Technology, Engineering and Mathematics (STEM) > Knowledge Media Institute (KMi)
Faculty of Science, Technology, Engineering and Mathematics (STEM)
Research Group: Big Scientific Data and Text Analytics Group (BSDTAG)
Item ID: 55743
Depositing User: David Pride
Date Deposited: 18 Jul 2018 12:28
Last Modified: 31 May 2019 13:43
Share this page:


Altmetrics from Altmetric

Citations from Dimensions

Download history for this item

These details should be considered as only a guide to the number of downloads performed manually. Algorithmic methods have been applied in an attempt to remove automated downloads from the displayed statistics but no guarantee can be made as to the accuracy of the figures.

Actions (login may be required)

Policies | Disclaimer

© The Open University   contact the OU