Automation of Citation Screening for Systematic Literature Reviews using Neural Networks: A Replicability Study

Kusa, Wojciech; Hanbury, Allan and Knoth, Petr (2022). Automation of Citation Screening for Systematic Literature Reviews using Neural Networks: A Replicability Study. In: Advances in Information Retrieval (Hagen, Matthias; Verberne, Suzan; Macdonald, Craig; Seifert, Christin; Balog, Krisztian; Nørvåg, Kjetil and Setty, Vinay eds.), Lecture Notes in Computer Science, Springer, Cham, pp. 584–598.

DOI: https://doi.org/10.1007/978-3-030-99736-6_39

Abstract

In the process of Systematic Literature Review, citation screening is estimated to be one of the most time-consuming steps. Multiple approaches to automate it using various machine learning techniques have been proposed. The first research papers that apply deep neural networks to this problem were published in the last two years. In this work, we conduct a replicability study of the first two deep learning papers for citation screening and evaluate their performance on 23 publicly available datasets. While we succeeded in replicating the results of one of the papers, we were unable to replicate the results of the other. We summarise the challenges involved in the replication, including difficulties in obtaining the datasets to match the experimental setup of the original papers and problems with executing the original source code. Motivated by this experience, we subsequently present a simpler model based on averaging word embeddings that outperforms one of the models on 18 out of 23 datasets and is, on average, 72 times faster than the second replicated approach. Finally, we measure the training time and the invariance of the models when exposed to a variety of input features and random initialisations, demonstrating differences in the robustness of these approaches.

Viewing alternatives

Metrics

Item Actions

Export

You can export this page using these formats Digital Object Identifier - DOI

About

• Item ORO ID
• 81958
• Item Type
• Conference or Workshop Item
• ISBN
• 3-030-99735-9, 978-3-030-99735-9
• ISSN
• 0302-9743
• Project Funding Details

• Funded Project Name	Project ID	Funding Body
  Dossier	860721	European Commission

• Keywords
• Citation Screening; Study Selection; Systematic Literature Review (SLR); Document Retrieval; Replicability
• Academic Unit or School
• Faculty of Science, Technology, Engineering and Mathematics (STEM) > Knowledge Media Institute (KMi)
  Faculty of Science, Technology, Engineering and Mathematics (STEM)
• Research Group
• Big Scientific Data and Text Analytics Group (BSDTAG)
• Copyright Holders
• © 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
• Related URLs
• • https://ecir2022.org/(Other)
  • https://doi.org/10.48550/arXiv.2201.0753...(Publication)
• Depositing User
• Petr Knoth