Modeling perception errors in autonomous vehicles and their impact on behavior

• Main Author: Piazzoni, Andrea
• Other Authors: Lyu Chen
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Simulation and modeling; Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence
• Online Access: https://hdl.handle.net/10356/170164
• Institution: Nanyang Technological University

Autonomous Vehicles (AVs) will majorly impact the transportation sector. The deployment of AVs has the potential to completely reshape the entire transportation industry, from the vehicle market to goods shipments. However, the major obstacle preventing a massive deployment is safety concerns. The scope of this work lies within the context of evaluating AVs' capabilities to operate under specific environmental conditions, which are known to alter the perception quality slightly. The aim is to define methodologies and tools towards answering a practical question: "Is the performance of a perception subsystem sufficient for the decision-making subsystem to make robust, safe decisions" This question lies in the intersection of different research topics (e.g., computer vision, decision models, electronics, and so on), and each of them requires to be adequately considered in order to answer the question. In fact, these fields are rarely tied together with a holistic approach. AVs employ sensors to perceive their surroundings. By analyzing the data (signals), AVs respond by planning the desired trajectory and acting on the steering wheel and the throttle/brake pedals. It is trivial to observe that any error present in the perception of the surroundings will affect the decision taken, eventually leading to undesirable behavior. However, there are only a limited amount of studies aimed at understanding how different kinds of perception errors affect AV decisions and behavior. In Machine Learning, it is common to measure the error (via \textit{loss} functions and evaluation metrics) of classification and regression models. However, those metrics are usually not rich and comprehensive enough to provide a meaningful understanding of the actual error but rather provide a single and summarized numerical value to enable comparisons. Moreover, they do not consider contextual information. The most practical solution is to employ AV simulators and their virtual environment to connect perception quality and AV behavior. I propose Perception Error Models (PEMs), i.e., a model that approximates the function of the sensing and perception module of an AV, as an efficient way to inject perception errors in a virtual environment. PEMs are directly integrated into the simulation pipeline and simulate a perception output by altering the ground truth, i.e., adding errors. I demonstrate their usefulness by both using hand-crafted PEMs, to inject controlled error, and data-driven PEMs, to test an actual sensing and perception system of an AV. This solution solves three problems simultaneously. First, PEM-based simulations are more efficient than a simulation based on synthetic signals and much more meaningful than a simulation based on ground truth (i.e., no perception errors). Second, PEMs-based simulations enable a direct link between perception performances and safety performances. Lastly, PEMs can be tuned to identify specific performance thresholds, issues, and limitations of the decision-making module. Moreover, I propose coPEMs, i.e., an extension to support cooperative perception. Cooperative Perception is a Vehicle-to-Everything (V2X) solution that enables connected vehicles to share information about their surroundings, with the goal of improving the overall perception quality. A synthetic-based simulation could require a dedicated processing unit for each \textit{sensor} and would scale very poorly if multiple vehicles or roadside units were involved. Instead, the efficient approach of PEMs is crucial to afford real-time virtual testing of multiple sources of information (e.g., other AVs). Lastly, I introduce the virtual testing framework I employed to conduct virtual testing with PEMs. This framework includes a diverse set of traffic scenarios, which I analyzed to identify relevant perception challenges and select scenarios where perception plays a critical role in the AV response. In particular, if the AV is not capable of performing safely under \textit{perfect} perception (e.g., lack of features, poor control), it is meaningless to investigate the impact of perception errors in such conditions.