PromptChart: prompting InstructGPT for zero & few-shot chart question answering and summarization

• Main Author: Do, Xuan Long
• Other Authors: Joty Shafiq Rayhan
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Science::Mathematics; Engineering::Computer science and engineering
• Online Access: https://hdl.handle.net/10356/166497
• Institution: Nanyang Technological University

Charts are commonly used in data analysis to summarize key insights and answer complex queries. However, comprehending charts through tasks such as chart question answering (CQA) and chart summarization (CS) can be cognitively intensive. Existing state-of-the-art models rely on fine-tuning strategies and struggle with tasks that involve logical and arithmetic reasoning. Meanwhile, large language models (LLMs) such as InstructGPT have demonstrated strong capabilities to solve target tasks by conditioning on few-shot prompts. Nonetheless, exploring how chart comprehension tasks could benefit from LLMs has received limited attention, especially in the few-shot setting. To bridge this gap, we present PromptChart, the first multimodal framework for few-shot CQA and CS. It consists of three modules: (i) InstructGPT for generating the textual responses, (ii) Visual Data Table Generator (VDTG) aiming to output our proposed visual data tables which integrate data tables and visual colors from charts, and (iii) Prompt Constructor (PC) focusing on creating few-shot prompts. To construct the prompts, we analyze the chart benchmarks carefully, and define a set of attributes for each task to guide the prompt constructions. Compared with existing CQA and CS baselines, PromptChart achieves state-of-the-art performance with significant improvements in automatic metrics and human preferences.