Hindsight-Combined and Hindsight-Prioritized Experience Replay

• Main Authors: Tan, Renzo Roel P, Ikeda, Kazushi, Vergara, John Paul
• Format: text
• Published: Archīum Ateneo 2020
• Subjects: Experience replay; Deep Q-Network; reinforcement learning; sample efficiency; hybrid algorithm; Logic and Foundations; Mathematics
• Online Access: https://archium.ateneo.edu/mathematics-faculty-pubs/146; https://link.springer.com/chapter/10.1007%2F978-3-030-63833-7_36
• Institution: Ateneo De Manila University

Reinforcement learning has proved to be of great utility; execution, however, may be costly due to sampling inefficiency. An efficient method for training is experience replay, which recalls past experiences. Several experience replay techniques, namely, combined experience replay, hindsight experience replay, and prioritized experience replay, have been crafted while their relative merits are unclear. In the study, one proposes hybrid algorithms – hindsight-combined and hindsight-prioritized experience replay – and evaluates their performance against published baselines. Experimental results demonstrate the superior performance of hindsight-combined experience replay on an OpenAI Gym benchmark. Further, insight into the nonconvergence of hindsightprioritized experience replay is presented towards the improvement of the approach.