The augmented human — seeing sounds

• Main Author: Lim, Nicole Sze Ting
• Other Authors: Cham Tat Jen
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; Assistive technology; Speech separation; Speech recognition; Hearing impariment; Software engineering; Web application
• Online Access: https://hdl.handle.net/10356/175150
• Institution: Nanyang Technological University

“Dinner table syndrome” describes the difficulties faced by those with hearing impairments in engaging in conversations with other speaking participants, which may lead to fatigue and social exclusion. This project designed and implemented an application which provides accessibility to hearing-impaired users in multi-speaker conversations, through automated captioning and speaker identification. Through a user-friendly web interface, a user uploads a video and adjusts settings for its processing. Speech separation and speech recognition models are then applied to produce an annotated video with captions differentiated by speaker. Four video annotation interfaces were proposed, of which floating captions tagged to each speaker’s face were preferred. The user is then able to monitor the processing of the video, and download the annotated video and a transcript with captions divided by speaker identity when processing is complete. Design choices made in the implementation of the final application are evaluated.