Vision-based tangible maths tutor

• Main Author: Toh, Jun Wei
• Other Authors: Goh Wooi Boon
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2022
• Subjects: Engineering::Computer science and engineering
• Online Access: https://hdl.handle.net/10356/162843
• Institution: Nanyang Technological University

Object detection techniques have been improving over the past several years and they have seen many applications in various domains. However, the early childhood education domain is one that has been relatively unexplored. This project contributes by exploring the use of object detection in early childhood education. A tangible user interface-based learning system was developed for young children to learn mathematics from. The system, called VTMT, uses a set of tangible manipulatives for interacting with the system. A YOLOv4-tiny object detection model was trained to detect these tangible manipulatives and deployed in a mobile application. The model was able to achieve a very high accuracy of more than 95%, thus ensuring the reliability of the system for learning. The mobile application serves as a tutor, recognising the manipulatives children had placed in a work area and providing relevant auditory feedback. Exploratory-based, practice-based, and assessment-based activities were developed as part of the learning activities of the system. Multiple ways of interacting with the system using the tangible manipulatives were implemented. Such methods differed in (1) the use of special manipulatives for controlling the system and (2) the main tangible manipulatives used for learning. An informal user study was conducted with the primary objective of determining the usability of each method. Results from the user study showed no clear preference in the different usages of the special manipulatives, while participants generally favoured using numbers as the main tangible manipulatives.