Improving the numerical reasoning skills of question answer systems in finance

• Main Author: Bhatia Nipun
• Other Authors: Shen Zhiqi
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; NLP; AI; Question answer systems; Finance
• Online Access: https://hdl.handle.net/10356/175338
• Institution: Nanyang Technological University

The field of natural language processing (NLP) has witnessed remarkable progress in recent years. Among the many applications, question answering (Q/A) systems have emerged as a crucial tool enabling access to information from various domains. In the financial sector, where complex calculations and in-depth analysis are required, accuracy and correctness are of the utmost importance for a Q/A system. Hence, such a system holds immense potential and value in this domain. In this paper, we first introduce a novel method for improving the Text-to-Text Transfer Transformer (T5) model's performance on question answering tasks based on the Discrete Reasoning Over Paragraphs (DROP) dataset. The key innovation proposed in this study is the introduction of a custom numerical attention layer, which is intended to improve the model's performance in answering numerical queries, an area in which it has traditionally had considerable difficulties. This layer updates the model's representations to increase the focus on the portions of the input text containing numerical values and other references related to those values. Furthermore, it also provides explicit question types as input to the model, guiding it to better understand the task at hand. We then fine-tune this enhanced T5 model with the numerical attention layer, on a wide variety of Q/A datasets, including DROP, to improve its general task knowledge and performance capabilities, after which we compare its performance against other state-of-the-art models, including those employing traditional fine-tuning strategies as well as techniques aimed at improving numerical reasoning. The results demonstrate a notable improvement in T5's numerical Q/A proficiency when incorporating the proposed numerical attention layer with explicit question type information. This work aims to contribute to the field of NLP by demonstrating how tailored model architecture enhancements and dataset augmentation can improve state-of-the-art models on complex domain-specific tasks, enabling them to carry out accurate financial question answering on par with human analysts.