Large Language Models for Scientific Question Answering: An Extensive Analysis of the SciQA Benchmark

Lehmann, Jens; Meloni, Antonello; Motta, Enrico; Osborne, Francesco; Recupero, Diego Reforgiato; Salatino, Angelo Antonio and Vahdati, Sahar (2024). Large Language Models for Scientific Question Answering: An Extensive Analysis of the SciQA Benchmark. In: The Semantic Web. ESWC 2024. Lecture Notes in Computer Science, Springer, Cham, 14664 pp. 199–217.

DOI: https://doi.org/10.1007/978-3-031-60626-7_11

Abstract

The SciQA benchmark for scientific question answering aims to represent a challenging task for next-generation question-answering systems on which vanilla large language models fail. In this article, we provide an analysis of the performance of language models on this benchmark including prompting and fine-tuning techniques to adapt them to the SciQA task. We show that both fine-tuning and prompting techniques with intelligent few-shot selection allow us to obtain excellent results on the SciQA benchmark. We discuss the valuable lessons and common error categories, and outline their implications on how to optimise large language models for question answering over knowledge graphs.

Viewing alternatives

Metrics

Public Attention

Altmetrics from Altmetric

Number of Citations

Citations from Dimensions
No digital document available to download for this item

Item Actions

Export

About