Speech-to-sign language virtual avatar translation

• Main Author: Chua, Elvis Xiang En
• Other Authors: Wonkeun Chang
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Electrical and electronic engineering
• Online Access: https://hdl.handle.net/10356/167352
• Institution: Nanyang Technological University

Effective communication is a fundamental human right, and it is essential for individuals who are deaf or hard of hearing to have access to the same information as hearing individuals. The communication barriers faced by hearing-impaired individuals have become increasingly evident, particularly in online settings such as virtual classrooms and video conferences. In response, this project proposes a speech-to-sign language virtual avatar translator that can be integrated into popular virtual communication platforms such as Zoom and Microsoft Teams. The proposed solution utilizes artificial intelligence and natural language processing techniques to convert spoken language into real-time sign language, thus providing hearing-impaired individuals with a means to communicate and comprehend information more effectively in online settings. A user study involving 73 students was conducted to evaluate the effectiveness and usability of the proposed solution and has shown success in demonstrating high levels of accuracy and user satisfaction. The working system offers a promising approach to enhancing the accessibility of virtual communication platforms for the hearing-impaired and has the potential to improve the educational experiences of a marginalized group of students.