Multimodal deception detection in videos

• Main Author: Syazwan Bin Jainal
• Other Authors: Alex Chichung Kot
• Format: Final Year Project
• Language: Spanish / Castilian
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence
• Online Access: https://hdl.handle.net/10356/167743
• Institution: Nanyang Technological University

The ability to identify if a person is lying is immensely powerful, to the extent that some might call it a superpower. There have been many approaches to the problem of deception detection, which include psychological, physiological and even machine learning methods. Deception detection has been successful in high-stakes situations, like courtrooms, where subjects are put under a stressful situation and experiments have yielded an accuracy of over 90%. However, in low-stakes situations, the results are often not much better than a random guess with correct results obtained only around 60% of the time. Moreover, existing datasets on deception detection cannot be generalised into the Singapore context, due to the difference in ethnicities of the subjects. This project builds a deception detection dataset and develops a multimodal deception detection detector based on the Convolutional Neural Network (CNN) deep learning architecture, and explores several loss functions to train the CNN model. It takes advantage of the multimodal nature of video data and applies different fusion methods on the visual and audio features. Experimental results have shown that fusion methods such as Multi-Layer Perceptron Mixer and employing loss functions such as Focal Loss can yield a 6% relative increase in accuracy over the simple concatenation fusion method and cross-entropy loss function.