Empowering communication for mute individuals through AI-assisted conversations

• Main Author: Au, Roydon Yew Rong
• Other Authors: Goh Wooi Boon
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; Gen AI; React native; Mobile application
• Online Access: https://hdl.handle.net/10356/181056
• Institution: Nanyang Technological University

With the rising proliferation of generative ai applications it would be useful to explore its application to enhance communication for mute individuals. Being mute has a significant impact on one’s daily life since it affects social interactions, personal relationships and even professional opportunities . This project is focussed on delivering a generative ai mobile application that would serve as a companion for mute individuals to assist them in their conversations with a normal speaker. The application improves the convenience of mute users by generating personalized responses at real time for them to pick to reply to a normal speaker. Overtime, the app grows with the users by learning their habits and past experiences therefore improving the quality of generated responses to become more personal. Lastly, to enhance greater flexibility for the mute user, they are also able to edit the generated responses directly and provide feedback to the ai model through liking or disliking certain responses. Since this project was built from scratch, this report highlights all features implemented, experiments attempted, areas of improvements and next steps. The final application has a fairly high G-Eval score of 0.714 (High relevancy / personalized responses), high context precision score of 0.9 (Low noise of extracted contexts from the vector database) and fairly high context recall score of 0.76 (High relevancy of extracted contexts to the ground truth). All scores are out of 1. Refer to Section 5.2.3 for the explanation of the scores. Participants from the user studies also ranked the mobile application highly useful for mute users.