Digital Theremin with computer vision

• Main Author: Chua, Ryuichi
• Other Authors: Goh Wooi Boon
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; Computer vision; Human computer interaction; Mediapipe; Theremin
• Online Access: https://hdl.handle.net/10356/175253
• Institution: Nanyang Technological University

This study presents the methodology and development of a digital Theremin using MediaPipe to enable gesture-based controls for pitch, volume, and vibrato. Through a comprehensive user study, the project evaluated the intuitiveness of the gesture-based controls and the produced sound quality with PyAudio. The digital Theremin was generally well-received, with participants demonstrating high controllability and showing a strong preference for a particular sound version noted for its warmth and expressiveness. Challenges identified include weak detection at screen edges and precise note-playing. Future work includes enhancing responsiveness and gesture recognition accuracy to expand the digital Theremin's appeal. Additionally, future work can investigate leveraging hand gestures in Unity development