Variance-aware learning based material recognition using stereo cameras

• Main Author: Ren, Jiawei
• Other Authors: Chau Lap Pui
• Format: Final Year Project
• Language: English
• Published: 2019
• Subjects: DRNTU::Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision
• Online Access: http://hdl.handle.net/10356/78133
• Institution: Nanyang Technological University

Material recognition exploits rich information from surfaces, which general object recognition fails to stress on, and therefore has wide application in real-world scenarios like construction management, autonomous agent navigation and image editing of interior design. Employing deep neural networks as a robust feature extractor, recent methods successfully decode reflectance and texture cues from the angular variation of materials, with equipment like light-field cameras and robotic arms. Stereo cameras, as a two-lens system, also provide differential angular images that reveal material’s latent characteristic. Moreover, a depth map can be estimated from a stereo camera system. Inspired by the unprecedented mobilization of stereo cameras, we explore the potentiality of deploying algorithms that incorporate both differential angular information and geometry information on a common multi-camera mobile phone and benchmark their performance. The task demands a new dataset that has not been collected before. Therefore, we build a data collection tool, which is a mobile app that captures each camera’s output image from a multi-camera system and processes the images with calibration information. With this tool, we collected a mini-dataset for benchmarking. Lastly, to tackle the challenge posed by the complex natural condition in outdoor scenes and decouple confounding patterns from local invariant features, we propose a novel method that models intra-class variance with a variance encoder. The variance encoder also serves to mine hard negative samples from easy negative samples, which leads toward a more robust material recognition.