Human-machine collaboration for safe and socially compliant trajectory planning of quadrotor

• Main Author: Zhang, Zezhong
• Other Authors: Lyu Chen
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2025
• Subjects: Engineering; Human-machine collaboration; Quadrotor trajectory planning
• Online Access: https://hdl.handle.net/10356/182953
• Institution: Nanyang Technological University

As quadrotors become increasingly utilized across diverse applications, they have to execute rapid, real-time trajectory planning to ensure safe and efficient flight. However, full autonomy in dynamic settings with incomplete mapping and social scenarios requiring human interaction remains a formidable challenge. Moreover, training skilled pilots is resource-intensive and time-consuming. While humans excel in understanding complex environments and making strategic decisions, they often struggle with the precise control required for specific tasks. Conversely, robots are proficient in executing precise actions but lack the intuitive understanding and adaptability characteristic of human cognition. To bridge this gap, human-machine collaboration (HMC) has emerged. The core research goal is to create an effective collaboration mechanism between the high-level guidance from human intelligence (or some intelligence like humans) and the traditional optimization-based trajectory planning for quadrotors. The first contribution is the development of a Human-guided Trajectory Planner (HTP), which integrates high-level human guidance into a local trajectory optimization framework. This guidance encompasses the operator’s judgment of the current situation, utilizing this intelligent decision-making process to complement the advantages of traditional optimization-based methods for obstacle avoidance and trajectory smoothness. Consequently, this approach achieves safe and efficient flight. The proposed method has been validated in the AirSim simulation environment, where results show that the HTP reduces optimization time by 58% compared to a non-human guidance (Non-HG) baseline. Furthermore, the HTP enables quadrotors to reach specified targets at higher speeds and aligns more closely with human preferences than the Non-HG approach. The second contribution addresses the need for social compliance in trajectory planning for quadrotors in pedestrian environments. Current methods prioritize physical safety, but quadrotors in social settings must also ensure pedestrians perceived safety and social compliance. An emotion-aware trajectory planning (EATP) is proposed to utilize the pedestrian’s emotional states as high-level guidance to further dynamically adjust the planned trajectory. By integrating body pose detection and facial expression recognition, a negative emotion detection system has been developed and integrated into the overall local trajectory planning. Validated in several high-fidelity daily life simulation environments, the EATP surpasses the baseline planner by better adhering to social norms and dynamically adjusting the relative distance between the pedestrian and the quadrotor, ensuring safer and more socially aware navigation. Furthermore, large Vision-Language Models (VLMs) have, to some extent, demonstrated human-like understanding and reasoning in various contexts. A VLMguided trajectory planner, FlyVLM, has been proposed to leverage this capability. The VLM’s guidance is integrated as a cost function term in the local trajectory optimization process. This approach addresses two significant challenges: (1) the mismatch between the VLM’s low-frequency outputs, due to latency, and the high-frequency demands of local planning, and (2) the generation of irrelevant guidance by the VLM. Experimental results have shown that FlyVLM can produce smooth trajectories that adhere to social norms in human-interactive scenarios and effectively guide quadrotors in avoiding obstacle-dense environments. Finally, a guidance drone system, equipped with collision avoidance and VLM-based world understanding, leverages human-machine collaboration to provide navigation assistance for people with visual impairments (PVI) in semantically indoor buildings. In summary, this thesis demonstrates the effectiveness of the human (or human-like intelligence)-machine collaboration between high-level guidance and optimization-based trajectory planning for quadrotors. This collaborative framework has the potential to be extended to a broader range of applications, including fixed-wing aerial vehicles, mobile robots, and quadrupedal robots.