Copy the page URI to the clipboard
Cancellieri, Matteo; Pontika, Nancy; Pearce, Samuel; Anastasiou, Lucas and Knoth, Petr
(2017).
DOI: https://doi.org/10.1007/978-3-319-70863-8_27
URL: http://www.mtsr-conf.org/
Abstract
CORE, a harvesting service offering access to millions of open access research papers from around the world, has shifted its harvesting process from following a monolithic approach to the adoption of a microservices infrastructure. In this paper, we explain how we rearranged and re-scheduled our old ingestion pipeline, present CORE's move to managing microservices and outline the tools we use in a new and optimised ingestion system. In addition, we discuss the ineffciencies of our old harvesting process, the advantages, and challenges of our new ingestion system and our future plans. We conclude that via the adoption of microservices architecture we managed to achieve a scalable and distributed system that would assist with CORE's future performance
and evolution.
Viewing alternatives
Download history
Metrics
Public Attention
Altmetrics from AltmetricNumber of Citations
Citations from DimensionsItem Actions
Export
About
- Item ORO ID
- 51070
- Item Type
- Conference or Workshop Item
- Keywords
- CORE
- Academic Unit or School
-
Faculty of Science, Technology, Engineering and Mathematics (STEM) > Knowledge Media Institute (KMi)
Faculty of Science, Technology, Engineering and Mathematics (STEM) - Research Group
-
Big Scientific Data and Text Analytics Group (BSDTAG)
Centre for Research in Computing (CRC) - Depositing User
- Lucas Anastasiou