Policy gradient with value function approximation for collective multiagent planning

• Main Authors: NGUYEN, Duc Thien, KUMAR, Akshat, LAU, Hoong Chuin
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2017
• Subjects: Collective behavior; Environment dynamics; Multi-agent planning; Optimization problems; Reinforcement learning method; Sequential decision making; Synthetic benchmark; Value function approximation; Artificial Intelligence and Robotics; Computer Sciences; Operations Research, Systems Engineering and Industrial Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/3871; https://ink.library.smu.edu.sg/context/sis_research/article/4873/viewcontent/7019_policy_gradient_with_value_function_approximation_for_collective_multiagent_planning.pdf
• Institution: Singapore Management University

Decentralized (PO)MDPs provide an expressive framework for sequential decision making in a multiagent system. Given their computational complexity, recent research has focused on tractable yet practical subclasses of Dec-POMDPs. We address such a subclass called CDec-POMDP where the collective behavior of a population of agents affects the joint-reward and environment dynamics. Our main contribution is an actor-critic (AC) reinforcement learning method for optimizing CDec-POMDP policies. Vanilla AC has slow convergence for larger problems. To address this, we show how a particular decomposition of the approximate action-value function over agents leads to effective updates, and also derive a new way to train the critic based on local reward signals. Comparisons on a synthetic benchmark and a real world taxi fleet optimization problem show that our new AC approach provides better quality solutions than previous best approaches.