Annapurna: Building a real-world smartwatch-based automated food journal

• Main Authors: SEN, Sougata, SUBBARAJU, Vigneshwaran, MISRA, Archan, BALAN, Rajesh Krishna, LEE, Youngki
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2018
• Subjects: cameras; mouth; image capture; gesture recognition; gears; sensors; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/4256; https://ink.library.smu.edu.sg/context/sis_research/article/5259/viewcontent/Annapurna_pv.pdf
• Institution: Singapore Management University

We describe the design and implementation of a smartwatch-based, completely unobtrusive, food journaling system, where the smartwatch helps to intelligently capture useful images of food that an individual consumes throughout the day. The overall system, called Annapurna, is based on three key components: (a) a smartwatch-based gesture recognizer to identify eating gestures, (b) a smartwatch-based image capturer that obtains a small set of relevant and useful images with a low energy overhead, and (c) a server-based image filtering engine that removes irrelevant uploaded images, and then catalogs them through a portal. Our primary challenge is to make the system robust to the huge diversity in natural eating habits and food choices. We show how we address this by an appropriate coupling between a smartwatch's camera sensor and inertial sensor-based tracking of eating gestures, thereby helping to capture multiple likely-to-be-useful images with low energy overhead. Through a series of real-world, in-the-wild studies, we demonstrate the end-to-end working of Annapurna, which captures useful images in over 95% of all natural eating episodes.