Developing an accessible learning application for sign language (B)

• Main Author: Rachman, Steven
• Other Authors: Wesley Tan Chee Wah
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2022
• Subjects: Engineering::Computer science and engineering::Information systems::Information interfaces and presentation; Engineering::Computer science and engineering::Software::Software engineering; Engineering::Computer science and engineering::Computer applications::Arts and humanities
• Online Access: https://hdl.handle.net/10356/158343
• Institution: Nanyang Technological University

Deaf people have a hard time communicating with their family members, and this is mostly due to the lack of experience of their family members in using sign language. Accessible learning resources for sign language in the market are still limited, and this project proposes a new solution for an easy and accessible way to learn sign language by utilizing 3D avatars and a real-time sign recognition feedback system. This report discusses the implementation of the 3D avatar using three.js and React, including with the implementation of the real-time sign recognition system by utilizing MediaPipe and WebSocket. On creating the sign language learning application, this project has received positive feedback on its usability, and it is successful in showcasing the 3D avatar and sign recognition feature of the app. The project is submitted to Microsoft Imagine Cup 2022, and it achieved the World Finalist position and won in the Lifestyle category of the Asia region.