Deep learning-based interaction-aware trajectory prediction for autonomous vehicles

• Main Author: Mo, Xiaoyu
• Other Authors: Lyu Chen
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2022
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence; Engineering::Computer science and engineering::Computing methodologies::Pattern recognition
• Online Access: https://hdl.handle.net/10356/163285
• Institution: Nanyang Technological University

Predicting future trajectories of surrounding agents and conducting motion planning based on interaction predictions are of great importance for ensuring the safety and efficiency of autonomous driving in real-world scenarios, especially under critical driving scenarios. However, trajectory prediction is challenging due to its highly interactive and dynamic attributes. The future trajectory of an agent is usually affected by multiple factors, including the agent's dynamics, its interaction with other surrounding agents, and the road structure. The motion patterns of different traffic participants, such as vehicles and pedestrians, are different and need to be considered separately. Therefore, in real-world applications, trajectory predictors should be able to simultaneously predict the motions of heterogeneous traffic participants. Besides, as for a target agent, there are a variable number of possible future trajectories existing, and this inherent multimodality characteristic should also be considered. Further, as trajectory prediction is not the end goal, it should be incorporated into downstream motion planning and control modules to further improve the overall performance of autonomous vehicles. This integration is also challenging and worthwhile investigating. In this thesis, a series of data-driven algorithms are developed using deep neural networks to address the opening challenges in trajectory prediction and predictive motion planning for safe and smart autonomous driving. In order to tackle the trajectory prediction problem and consider those key features (e.g., the target agent's dynamics, interactions, and map features) in a unified way, a deterministic trajectory prediction framework in which the vehicles and the road map are represented using a heterogeneous graph, is proposed. Under this framework, a novel heterogeneous graph social (HGS) pooling method is developed to model the interdependencies among all associated traffic participants and the infrastructures. Besides, to address the heterogeneity of traffic participants for simultaneous prediction of multi-agent trajectories, a novel Heterogeneous Edge-enhanced graph ATtention network (HEAT) with a three-channel architecture is proposed. The inter-agent interactions are represented through an edge-featured heterogeneous graph and processed by the designed HEAT network for interaction modeling. Map features are shared across all agents by introducing a selective gate mechanism. Type-specific trajectory decoders are designed for various categories of target agents. Further, to address the multimodality characteristic of vehicle motion, a map-adaptive multimodal trajectory predictor is proposed. The proposed method can predict a variable number of a target vehicle's possible future trajectories based on the number of candidate centerlines (CCLs). In this approach, the driving scene is first represented using a heterogeneous hierarchical graph, wherein one node represents either an agent or its CCL. Then, a hierarchical graph operator (HGO) with an edge-masking technology is proposed to further model the scene graph. The HGO regulates information flow in the graph operations via edge-masking and outputs the encoded scene features for the downstream map-adaptive trajectory decoder. The map-adaptive predictor, which associates driving modalities with lane options, predicts single-CCL guided, cross-CCL guided, and motion-based predictions in an integrated manner. The motion-based prediction can handle the corner cases where the target vehicle only follows its own dynamics. Finally, how trajectory prediction results can facilitate downstream motion planning is also investigated. A neural network-based predictive planner is designed by integrating an oracle planner and a trajectory predictor to generate the future reference path for the ego vehicle. In addition to vehicles' dynamics, interactions, and map features, during the training stage of the oracle planner, target agents' ground truth future motions are also taken as input. At the implementation stage of the trajectory planning algorithm, the target agents' ground truth future paths are replaced with the predicted trajectories generated by the pre-trained predictor. Results of the experimental validations on a real-world dataset show that both the oracle and predictive planners outperform the non-predictive baselines, demonstrating the effectiveness of the trajectory planning incorporated with the prediction module. Results of the experimental validations on different real-world driving datasets show that the proposed trajectory prediction methods achieve state-of-the-art performance with additional capabilities of multi-agent simultaneous prediction and strong scene adaptability. Besides, the algorithms developed based on the proposed HGS pooling technique and the HEAT network won the championships of two worldwide autonomous vehicle prediction challenges, respectively. These outcomes demonstrate the feasibility and effectiveness of the proposed methods. In addition, the high-level algorithm architectures, methodologies employed, and models developed in this work will expand the current theories of autonomous driving and intelligent transportation systems. They can also be expanded to a wide range of robotics and automation applications.