Towards aligning slides and video snippets: Mitigating sequence and content mismatches

• Main Authors: LIU, Ziyuan, LAUW, Hady W.
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2022
• Subjects: Content mismatch; Dynamic time warping; Sequence mismatch; Slide to video alignment; Artificial Intelligence and Robotics; Databases and Information Systems; Educational Assessment, Evaluation, and Research
• Online Access: https://ink.library.smu.edu.sg/sis_research/7600; https://ink.library.smu.edu.sg/context/sis_research/article/8603/viewcontent/aied2022.pdf
• Institution: Singapore Management University

Slides are important form of teaching materials used in various courses at academic institutions. Due to their compactness, slides on their own may not stand as complete reference materials. To aid students’ understanding, it would be useful to supplement slides with other materials such as online videos. Given a deck of slides and a related video, we seek to align each slide in the deck to a relevant video snippet, if any. While this problem could be formulated as aligning two time series (each involving a sequence of text contents), we anticipate challenges in generating matches arising from differences in content coverage and sequence of content between slide deck-video pairs. To mitigate these challenges, we propose a two-stage algorithm that builds on time series alignment to filter out irrelevant content and to align out-of-sequence slide deck and video pairs. We experiment with real-world datasets from openly available lectures, which have been manually annotated with start and end times of each slide in the videos to facilitate the evaluation of matches.