DETECTION OF MICRO EXPRESSIONS AND HEART RATE IN REAL TIME HUMAN FACIAL VIDEOS

• Main Author: Putri Setyaningrum, Anisa
• Format: Theses
• Language: Indonesia
• Online Access: https://digilib.itb.ac.id/gdl/view/85544
• Institution: Institut Teknologi Bandung

This research aims to detect micro-expressions and heart rate in real-time from human facial videos while exploring the relationship between the two. The simultaneous detection of micro-expressions and heart rate is crucial as both provide insights into an individual's emotional and physiological state. Micro­expressions, which are rapid and subtle facial expressions, often reveal emotions that are unconscious or intentionally concealed, making them essential for lie detection and mental health assessments. Heart rate, on the other hand, reflects physiological responses to emotions, such as stress or calmness, and can be monitored through variations in facial skin color caused by blood flow. By analyzing both simultaneously, a more comprehensive and accurate understanding of a person's emotional and physiological state can be obtained, which is valuable in contexts such as security, psychology, and clinical conditions. The methodology employed includes the use of the CASME III dataset and the development of two main models: a Micro-Expression Classification Model using a Transformer architecture with Multi-Head Attention and Micro Attention techniques, and a Heart Rate Detection Model utilizing Eulerian Video Magnification (EVM) and Fast Fourier Transform (FFT). The CASME III dataset is used as the primary data source for training and testing the micro-expression detection model. This dataset contains labeled micro-expression images categorized into six classes: anger, happy, sad, fear, disgust, and surprise. The micro-expression classification model is developed using a Transformer architecture enhanced with Multi-Head Attention and Micro Attention techniques. Multi-Head Attention allows the model to capture various facial features in parallel, while Micro Attention focuses on recognizing the small details critical in micro-expressions. The heart rate detection model employs Eulerian Video Magnification (EVM) to amplify subtle changes in the face related to heart rate. EVM enables the detection of minute movements that may not be visible to the naked eye, such as skin color changes due to blood flow. Once these changes are magnified, the Fast Fourier Transform (FFT) technique is applied to analyze the heart rate signal and calculate its frequency. FFT breaks down the magnified signal into frequency components that can be analyzed to detect heart rhythms. Eulerian Video Magnification (EVM) plays a dual role in this methodology. Firstly, EVM is used as a feature extraction method for the Transformer model developed for micro-expression classification. By amplifying subtle facial movements and changes related to emotions, EVM helps the Transformer model capture important features that may not be visible under normal conditions. This ensures that the model can detect micro-expressions with greater accuracy. The research findings indicate that the EVM method is effective for heart rate detection, with the lowest Mean Absolute Error (MAE) of 4.078 BPM The Multi-Head Attention model achieved the highest accuracy of99 .05% and the best loss value of0.07 , while the Micro Attention model reached an accuracy of98.29% with a loss value of 0.14. The study found a significant correlation between micro­expressions and heart rate. Angry emotions tend to increase heart rate, while sad emotions decrease it. Happy emotions produce more varied physiological responses. These findings underscore the importance of simultaneous analysis of micro-expressions and heart rate for a more complete understanding of an individual's emotional and physiological state. The variation in MAE suggests the need for more advanced models to capture the complexity of physiological responses to emotions. This discovery indicates that micro-expressions not only reveal emotional states but can also influence physiological conditions such as heart rate.