Novice chef : a robotic system that learns recipes from natural language conversations

• Main Author: Yang, Zhutian
• Other Authors: Shum Ping
• Format: Final Year Project
• Language: English
• Published: 2019
• Subjects: DRNTU::Engineering::Electrical and electronic engineering::Control and instrumentation::Robotics
• Online Access: http://hdl.handle.net/10356/78532
• Institution: Nanyang Technological University

If we want our future robots to be adaptable and interactive collaborators, they need to understand how to execute our natural language instructions, learn new task procedures through two-way interactions, and ask for the information missing from initial plans. However, previous robotic systems understood mostly simple imperatives and cannot generate purposeful English for conversing with humans. In this project, I developed a robotic system, Novice Chef, that learns cooking recipes from natural language conversations and asks questions during problem solv- ing. Novice Chef robot has successfully made three cuisines at Robotic Living Studio – fruit salad, breakfast cereals, and instant pasta. The core program for learning, Novice, was evaluated by 15 human testers to learn cooking recipes. It has 73% success rate in generating an executable recipe that the tester intended to teach. It took on average 1.45 sentences to learn one step during conversations. By integrating previous systems for natural language understanding, computer vi- sion, problem solving, and speech recognition, this interdisciplinary project brought all of them to the next level of real-world applications.