HAND GESTURE RECOGNITION ON BAHASA ISYARAT INDONESIA (BISINDO) BY USING VISION-BASED MACHINE LEARNING

• Main Author: Nurrahma
• Format: Theses
• Language: Indonesia
• Online Access: https://digilib.itb.ac.id/gdl/view/65105
• Institution: Institut Teknologi Bandung

Sign language, a type of communication based on hand movements, is widely used by deaf people. The two types of sign language used in Indonesia are SIBI (Indonesian Sign Language System) and BISINDO (Indonesian Sign Language). BISINDO is more commonly utilized in everyday life because BISINDO was developed by Deaf People themselves so that it is easier to understand. Communication issues often occur between Deaf people and hearing people. As a result, we need media that can help them communicate more effectively. One of the technologies that can be employed is SLR (Sign Language Recognition). Visionbased SLR is one of the many various approaches used in SLR. The fact that it does not require a specific gadget to be attached to the hand and instead relies on the user making movements in front of the camera with their bare hands provides a number of advantages for vision-based SLR. This research developed a machine learning model that can recognize BISINDO static and dynamic gestures, including the alphabet (A-Z), numbers (1-10), and seven frequent words, using a vision-based SLR technique. The model was created using the LSTM (Long Short-Term Memory) architecture. The LSTM model was trained and tested using the BISINDO gestures dataset. The validation accuracy of the static model was 99.69%, and the validation accuracy of the dynamic model was 99.17 %. For testing using testing dataset, this research yielded the accuracy 99,63% on static gestures and 98,33% on dynamic model. This research also combined the model into a single simple system for realtime testing. The realtime testing was completed 45 times by 15 BISINDO signers, yielding a static model accuracy of 69.42%, a dynamic model accuracy of 56.67%, and an overall model accuracy of 66.45%. User experience measurement was also carried out using User Experience Questionnaire Plus (UEQ+) with the results of Attractiveness (DT) 2.02, Efficiency (EF) 1.32, Reliability Level (TK) 1.53, Novelty (KB) 1.73, Usability (KG) 2.37, Intuitive Use (PI) 1.95, and Quality of Response (KR) 1.67 which can be interpreted as Good for each measured scale. However, the Quality of Response (KR), Usability (KG), and Reliability Level (TK) scales which are ranked 1, 3, and 4 respectively on the importance rating have lower scores when compared to the other scales.