Robotic gesture-speech synchronization

• Main Author: Lim, Keck Soon.
• Other Authors: Xiao Zhongmin
• Format: Final Year Project
• Language: English
• Published: 2011
• Subjects: DRNTU::Engineering::Mechanical engineering
• Online Access: http://hdl.handle.net/10356/44964
• Institution: Nanyang Technological University

This report provides a detailed description of the steps taken to build a text to speech to gesture engine. The building of the engine is separated into three main segments. The first segment involves gesture planning. This includes generating a blending motion between the first and subsequent motions. The blending motion is created to prevent jerking motions from occurring during the translation of motion. Two methods were introduced to remove the need of having a huge library of pre-recorded gestures. The first method is to introduce coherent noises into existing gesture file to produce gesture of similar type yet different movement which can be perceived. The second method is to generate beat gestures during run time which can totally remove the need of a gesture library. The next segment of the project is to create and link various engines. The method used for communication was via transmission control protocol/internet protocol. A text engine was created to prepare the modified text format to be processed by the text to speech engine. Timings are obtained from the engine and sent to the gesture engine for synchronization. The gestures files are created from the gesture engine and the simulation can be seen. To make it more user friendly for non programmers, a graphical user interface was created. In addition, an auto tagging function was included to automatically determine the start and end time of gesture. The last segment of the project is to implement the engine into an actual robot, Olivia 2.1. The challenges faced include keeping the speech and gesture in sync while respecting the robot’s physical constraints. Also, to determine the coefficient that affects the sampling rate of the robot.