Real-time multimodal affect recognition in laughter episodes

• Main Author: Santos, Jose Miguel
• Format: text
• Language: English
• Published: Animo Repository 2013
• Subjects: Emotion recognition; Laughter
• Online Access: https://animorepository.dlsu.edu.ph/etd_masteral/4383
• Institution: De La Salle University

Emotion recognition is a widely studied subject, due to its importance in human interaction and decision making. The recognition of emotion in laughter is particularly important as laughter can identify non-basic affective states such as distress, anxiety, and boredom. Existing systems are unable to classify the emotion of laughter in real-time, however. This research proposes a real-time multimodal affect recognition system for laughter episodes, using facial expressions and voiced laughter as modalities. The system takes a video stream as input. The video stream can be either a web camera with a microphone attached for audio, or a video file. As laughter take place over a period of time, rather than frame-by-frame, the system will segment the stream into different windows of 1.62 seconds in length. Within the window, image and audio data are extracted, and the AUs in the apex of the window are detected. At the end of each window, the pitch and MFCC values of the audio data collected within the window are computed, and decision-level fusion is applied to the audio and face features. The resulting features are then be passed to the emotion recognition model, which then produces the final valence and arousal values of the window. The emotion recognition model was able to achieve a correlation coefficient of 0.68 for valence and 0.61 for arousal using the Semaine corpus, and 0.75 for valence and 0.83 for arousal using the Pinoy Laughter 2 corpus. The overhead for the whole emotion recognition process is 610.98 ms, however the overhead will be hard to completely eliminate due to the high number of processes required to perform emotion recognition.