Deep learning of facial embeddings and facial landmark points for the detection of academic emotions

• Main Author: FWA, Hua Leong
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2020
• Subjects: datasets; deep learning; emotions; facial emotion recognition; Databases and Information Systems; Educational Assessment, Evaluation, and Research; Graphics and Human Computer Interfaces; Higher Education
• Online Access: https://ink.library.smu.edu.sg/sis_research/6859; https://ink.library.smu.edu.sg/context/sis_research/article/7862/viewcontent/3411681.3411684_pv.pdf
• Institution: Singapore Management University

Automatic emotion recognition is an actively researched area as emotion plays a pivotal role in effective human communications. Equipping a computer to understand and respond to human emotions has potential applications in many fields including education, medicine, transport and hospitality. In a classroom or online learning context, the basic emotions do not occur frequently and do not influence the learning process itself. The academic emotions such as engagement, frustration, confusion and boredom are the ones which are pivotal to sustaining the motivation of learners. In this study, we evaluated the use of deep learning on FaceNet embeddings and facial landmark points for academic emotion detection on a publicly available dataset - DAiSEE that has been annotated with the emotional states of engagement, boredom, frustration and confusion. By modeling both the spatial and temporal dimensions, the results demonstrated that both models are able to detect incidences of boredom and frustration and can be used in the moment-by-moment monitoring of boredom and frustration of learners using a tutoring system either online or in a classroom.