NPE-DRL: enhancing perception constrained obstacle avoidance with non-expert policy guided reinforcement learning

• Main Authors: Zhang, Yuhang, Yan, Chao, Xiao, Jiaping, Feroskhan, Mir
• Other Authors: School of Mechanical and Aerospace Engineering
• Format: Article
• Language: English
• Published: 2024
• Subjects: Engineering; Constrained perception; Deep reinforcement learning
• Online Access: https://hdl.handle.net/10356/181017
• Institution: Nanyang Technological University

Obstacle avoidance under constrained visual perception presents a significant challenge, requiring rapid detection and decision-making within partially observable environments, particularly for unmanned aerial vehicles (UAVs) maneuvering agilely in three-dimensional space. Compared to traditional methods, obstacle avoidance algorithms based on deep reinforcement learning (DRL) offer a better comprehension of the uncertain operational environment in an end-to-end manner, reducing computational complexity and enhancing flexibility and scalability. However, the inherent trial-and-error learning mechanism of DRL necessitates numerous iterations for policy convergence, leading to sample inefficiency issues. Meanwhile, existing sample-efficient obstacle avoidance approaches that leverage imitation learning often heavily rely on offline expert demonstrations, which are not always feasible in hazardous environments. To address these challenges, we propose a novel obstacle avoidance approach based on Non-Expert Policy Enhanced DRL (NPE-DRL). This approach integrates a fundamental DRL framework with prior knowledge derived from a non-expert policy-guided imitation learning. During the training phase, the agent starts by online imitating the actions generated by the non-expert policy during interactions and progressively shifts toward autonomously exploring the environment to generate the optimal policy. Both simulation and physical experiments validate that our approach improves sample efficiency and achieves a better exploration-exploitation balance in both virtual and real-world flights. Additionally, our NPE-DRL-based obstacle avoidance approach shows better adaptability in complex environments characterized by larger scales and denser obstacle configurations, demonstrating a significant improvement in UAVs' obstacle avoidance capability. Code available at https://github.com/zzzzzyh111/NonExpert-Guided-Visual-UAV-Navigation-Gazebo.