Tangible figurine story composer and narrator

• Main Author: Choo, Jin He
• Other Authors: Goh Wooi Boon
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering
• Online Access: https://hdl.handle.net/10356/171850
• Institution: Nanyang Technological University

Tangible User Interfaces (TUIs) hold promise for enhancing learning, particularly among children, including those with Autism Spectrum Conditions (ASC). This report addresses challenges in ASC-focused TUIs, including cost and standardization issues. It reviews two ASC TUI projects, highlighting innovations and limitations. A novel project is introduced, aiming to develop affordable TUIs designed for teaching social interactions to children with autism. It leverages machine learning, specifically Convolutional Neural Networks, to facilitate real-time object detection of LEGO Minifigures. The integration of this technology into a mobile application, known as TFSCAN, opens doors for generating interactive experiences. A comprehensive user study was conducted, involving a demonstration phase followed by story creation using two application versions. The results show that participants with no prior stop-motion background found both versions, designed for storytelling and story composition, more engaging than users with prior experience. Furthermore, the findings underline a strong preference for Version 2, which involves camera-based composition, in terms of ease of use, engagement levels, and intuitiveness when compared to Version 1, which relies on user interfaces. The study not only underscores the potential of this project but also its significant role in teaching social skills to children with ASC.