Inverse factorized soft Q-Learning for cooperative multi-agent imitation learning

• Main Authors: BUI, The Viet, MAI, Tien, NGUYEN, Thanh
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2024
• Subjects: Imitation learning; Multi-agent systems; soft-Q learning; Artificial Intelligence and Robotics; Computer Sciences
• Online Access: https://ink.library.smu.edu.sg/sis_research/9818; https://ink.library.smu.edu.sg/context/sis_research/article/10818/viewcontent/NeurIPS2024___Multi_agent_Inverse_Q_learning_for_imitation_6.pdf
• Institution: Singapore Management University

This paper concerns imitation learning (IL) in cooperative multi-agent systems.The learning problem under consideration poses several challenges, characterized by high-dimensional state and action spaces and intricate inter-agent dependencies. In a single-agent setting, IL was shown to be done efficiently via an inverse soft-Q learning process. However, extending this framework to a multi-agent context introduces the need to simultaneously learn both local value functions to capture local observations and individual actions, and a joint value function for exploiting centralized learning.In this work, we introduce a new multi-agent IL algorithm designed to address these challenges. Our approach enables thecentralized learning by leveraging mixing networks to aggregate decentralized Q functions.We further establish conditions for the mixing networks under which the multi-agent IL objective function exhibits convexity within the Q function space.We present extensive experiments conducted on some challenging multi-agent game environments, including an advanced version of the Star-Craft multi-agent challenge (SMACv2), which demonstrates the effectiveness of our algorithm.