Examining the Inter-consistency of large language models: An in-depth analysis via debate

• Main Authors: XIONG, Kai, DING, Xiao, CAO, Yixin, LIU, Ting, QIN, Bing
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: Databases and Information Systems; Programming Languages and Compilers
• Online Access: https://ink.library.smu.edu.sg/sis_research/8391; https://ink.library.smu.edu.sg/context/sis_research/article/9394/viewcontent/2305.11595.pdf
• Institution: Singapore Management University

Large Language Models (LLMs) have shown impressive capabilities in various applications, but they still face various inconsistency issues. Existing works primarily focus on the inconsistency issues within a single LLM, while we complementarily explore the inter-consistency among multiple LLMs for collaboration. To examine whether LLMs can collaborate effectively to achieve a consensus for a shared goal, we focus on commonsense reasoning, and introduce a formal debate framework (FORD) to conduct a three-stage debate among LLMs with real-world scenarios alignment: fair debate, mismatched debate, and roundtable debate. Through extensive experiments on various datasets, LLMs can effectively collaborate to reach a consensus despite noticeable inter-inconsistencies, but imbalances in their abilities can lead to domination by superior LLMs. Leveraging a more advanced LLM like GPT-4 as an authoritative judge can boost collaboration performance. Our work contributes to understanding the inter-consistency among LLMs and lays the foundation for developing future collaboration methods. Codes and data are available at https://github.com/WasteWood/FORD.