EmotionCues: Emotion-oriented visual summarization of classroom videos

• Main Authors: ZENG, Haipeng, SHU, Xinhuan, WANG, Yanbang, WANG, Yong, ZHANG, Liguo, PONG, Ting-Chuen, QU, Huamin
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2021
• Subjects: Emotion; classroom videos; visual summarization; visual analytics; Graphics and Human Computer Interfaces; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/5362; https://ink.library.smu.edu.sg/context/sis_research/article/6366/viewcontent/zeng_tvcg20_paper.pdf
• Institution: Singapore Management University

Analyzing students' emotions from classroom videos can help both teachers and parents quickly know the engagement of students in class. The availability of high-definition cameras creates opportunities to record class scenes. However, watching videos is time-consuming, and it is challenging to gain a quick overview of the emotion distribution and find abnormal emotions. In this paper, we propose EmotionCues, a visual analytics system to easily analyze classroom videos from the perspective of emotion summary and detailed analysis, which integrates emotion recognition algorithms with visualizations. It consists of three coordinated views: a summary view depicting the overall emotions and their dynamic evolution, a character view presenting the detailed emotion status of an individual, and a video view enhancing the video analysis with further details. Considering the possible inaccuracy of emotion recognition, we also explore several factors affecting the emotion analysis, such as face size and occlusion. They provide hints for inferring the possible inaccuracy and the corresponding reasons. Two use cases and interviews with end users and domain experts are conducted to show that the proposed system could be useful and effective for analyzing emotions in the classroom videos.