Large language model is not a good few-shot information extractor, but a good reranker for hard samples!

• Main Authors: MA, Yubo, CAO, Yixin, HONG, YongChin, SUN, Aixin
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: LLMs; Information extraction; Databases and Information Systems; Programming Languages and Compilers
• Online Access: https://ink.library.smu.edu.sg/sis_research/8388; https://ink.library.smu.edu.sg/context/sis_research/article/9391/viewcontent/llmIE_emnlp_tbu.pdf
• Institution: Singapore Management University

Large Language Models (LLMs) have made remarkable strides in various tasks. However, whether they are competitive few-shot solvers for information extraction (IE) tasks and surpass fine-tuned small Pre-trained Language Models (SLMs) remains an open problem. This paper aims to provide a thorough answer to this problem, and moreover, to explore an approach towards effective and economical IE systems that combine the strengths of LLMs and SLMs. Through extensive experiments on nine datasets across four IE tasks, we show that LLMs are not effective few-shot information extractors in general, given their unsatisfactory performance in most settings and the high latency and budget requirements. However, we demonstrate that LLMs can well complement SLMs and effectively solve hard samples that SLMs struggle with. Building on these findings, we propose an adaptive \textit{filter-then-rerank} paradigm, in which SLMs act as filters and LLMs act as rerankers. By utilizing LLMs to rerank a small portion of difficult samples identified by SLMs, our preliminary system consistently achieves promising improvements (2.4% F1-gain on average) on various IE tasks, with acceptable cost of time and money.