The Open UniversitySkip to content
 

Building scalable digital library ingestion pipelines using microservices

Cancellieri, Matteo; Pontika, Nancy; Pearce, Samuel; Anastasiou, Lucas and Knoth, Petr (2017). Building scalable digital library ingestion pipelines using microservices. In: MSTR 2017: 11th International Conference on Metadata and Semantics Research, 28th November - 1st December 2017, Tallinn, Estonia.

Full text available as:
[img]
Preview
PDF (Accepted Manuscript) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Download (205kB) | Preview
URL: http://www.mtsr-conf.org/
Google Scholar: Look up in Google Scholar

Abstract

CORE, a harvesting service offering access to millions of open access research papers from around the world, has shifted its harvesting process from following a monolithic approach to the adoption of a microservices infrastructure. In this paper, we explain how we rearranged and re-scheduled our old ingestion pipeline, present CORE's move to managing microservices and outline the tools we use in a new and optimised ingestion system. In addition, we discuss the ineffciencies of our old harvesting process, the advantages, and challenges of our new ingestion system and our future plans. We conclude that via the adoption of microservices architecture we managed to achieve a scalable and distributed system that would assist with CORE's future performance
and evolution.

Item Type: Conference or Workshop Item
Academic Unit/School: Faculty of Science, Technology, Engineering and Mathematics (STEM) > Knowledge Media Institute (KMi)
Faculty of Science, Technology, Engineering and Mathematics (STEM)
Interdisciplinary Research Centre: Centre for Research in Computing (CRC)
Item ID: 51070
Depositing User: Nancy Pontika
Date Deposited: 21 Sep 2017 12:03
Last Modified: 21 Sep 2017 12:03
URI: http://oro.open.ac.uk/id/eprint/51070
Share this page:

Download history for this item

These details should be considered as only a guide to the number of downloads performed manually. Algorithmic methods have been applied in an attempt to remove automated downloads from the displayed statistics but no guarantee can be made as to the accuracy of the figures.

Actions (login may be required)

Policies | Disclaimer

© The Open University   contact the OU