Time-inconsistency in reinforcement learning: an equilibrium control paradigm - Lesmana, Nixie Sapphira (Nanyang Technological University, 2024)
Time inconsistency (TIC) describes a situation in which a plan, consisting of current and future actions, that is optimal today may no longer be optimal in the future. In reinforcement learning (RL), TIC often arises as we encode realistic human preferences or specific behaviors into an agent's performance criterion. Such encoding has broad applications in risk-sensitive and human-centric domains such as finance, economics, and assistive robotics. Despite its importance, TIC in RL is difficult to handle as many amenable properties of globally optimal policy under the standard performance criterion fail to extend. One of the most important challenges is the non-applicability of Bellman's Principle of Optimality (BPO), from which many popular RL methods are drawn. In recent years, the subgame perfect equilibrium (SPE) control has risen as an important resolution to TIC in stochastic control, promising both amenable computations (through BPO recovery) and desirable control performance. SPE, as explained in behavioral economics, corresponds to a sophisticated human agent's behavior that handles TIC by taking future deviations as a constraint in their current plan construction, such that the resulting plan becomes time-consistent (TC).