Driver action recognition using artificial intelligence

• Main Author: Lee, Philip Hann Yung
• Other Authors: Yap Kim Hui
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision; Engineering::Electrical and electronic engineering
• Online Access: https://hdl.handle.net/10356/167802
• Institution: Nanyang Technological University

Distracted driving is a significant cause of fatal accidents worldwide. For instance, in the United States, 9% of such accidents are attributed to distracted driving. Car-cabin monitoring solutions can help mitigate this issue by alerting drivers or authorities when instances of distracted driving are detected, potentially saving lives and increasing road safety. To address this problem, video action recognition was investigated as a possible solution for detecting distracted driving in car cabins. A literature survey was conducted to identify relevant models and datasets, followed by hyperparameter tuning and comparison of multiple models to determine the most suitable architecture for real-time driver action recognition in terms of both accuracy and efficiency. Three state-of-the-art models were investigated, with the Temporal Shift Module-based model being recommended based on its balanced accuracy score of 68.47% on the Drive&Act dataset and a latency of 15.0 ms. Class imbalance issues were addressed through various techniques, including a fusion of class weighting and hard sample mining, resulting in a 7% improvement in baseline scores. Additionally, further work was done by fusing data from multiple modalities, resulting in a more robust model with greater prediction accuracy by further increasing scores by 5%. The project concludes that Temporal Shift Module-based models are well-suited for real-time driver action recognition systems, which was further demonstrated by fine-tuning the model on an internally constructed dataset and successfully developing a working prototype for live testing that can run at 30 frames per second. This adds to the significance of the project as it provides a practical implementation of the proposed system.