Constrained reinforcement learning in hard exploration problems

• Main Authors: PATHMANATHAN, Pankayaraj, VARAKANTHAM, Pradeep
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: reinforcement learning; Artificial Intelligence and Robotics
• Online Access: https://ink.library.smu.edu.sg/sis_research/8590; https://ink.library.smu.edu.sg/context/sis_research/article/9593/viewcontent/26757_Article_Text_30820_1_2_20230626.pdf
• Institution: Singapore Management University

One approach to guaranteeing safety in Reinforcement Learning is through cost constraints that are imposed on trajectories. Recent works in constrained RL have developed methods that ensure constraints can be enforced even at learning time while maximizing the overall value of the policy. Unfortunately, as demonstrated in our experimental results, such approaches do not perform well on complex multi-level tasks, with longer episode lengths or sparse rewards. To that end, wepropose a scalable hierarchical approach for constrained RL problems that employs backward cost value functions in the context of task hierarchy and a novel intrinsic reward function in lower levels of the hierarchy to enable cost constraint enforcement. One of our key contributions is in proving that backward value functions are theoretically viable even when there are multiple levels of decision making. We also show that our new approach, referred to as Hierarchically Limited consTraint Enforcement (HiLiTE) significantly improves on state of the art Constrained RL approaches for many benchmark problems from literature. We further demonstrate that this performance (on value and constraint enforcement) clearly outperforms existing best approaches for constrained RL and hierarchical RL.