Comparative Analysis of Retriever and Reader for Open Domain Questions Answering on BPS Knowledge in Indonesian

Abstract

Enumerators from Badan Pusat Statistik (BPS) still often encounter problems in finding solutions to cases encountered during censuses or surveys. Even though knowledge lists have been created and collected in various systems such as QA and knowledge management systems, enumerators still need to find appropriate answers from long and complex knowledge search results. On the other hand, Open-domain Question Answering (OpenQA) is capable of identifying answers to natural questions based on large-scale documents. OpenQA has main components, namely Retriever and Reader. For Retriever tasks, Dense Retrieval (DR) is proven to outperform traditional sparse retrieval such as TF-IDF or BM25. However, other research actually shows that BM25 is superior to DR in terms of accuracy. In this study, we compared DR and BM25 separately and DR+BM25 as a retriever. Additionally, we combine and evaluate several enhanced language models as Readers. In this way, a model with the best combination of Retriever and Reader can be obtained to be implemented in search systems such as QA and knowledge management systems.