Lighting geometry aware environment matting and 3D reconstruction

• Main Author: Duan, Qi
• Other Authors: Cai Jianfei
• Format: Theses and Dissertations
• Language: English
• Published: 2013
• Subjects: DRNTU::Engineering::Computer science and engineering::Computing methodologies::Computer graphics; DRNTU::Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision
• Online Access: https://hdl.handle.net/10356/54835
• Institution: Nanyang Technological University

Lighting geometry, which refers to the relationship between lighting conditions, object geometry or appearance and various lighting interaction phenomenons, is a fundamental issue in many computer graphics and computer vision problems. The challenges of lighting geometry related problems lie in the complex interactions between environment lights and objects, the diversity of lighting sources, the high computational cost and so on. This thesis investigates two specific lighting geometry related topics: environment matting and 3D reconstruction, where the former is with controllable lighting conditions and the latter is with general unknown illumination conditions. Our goal is to design and develop some effective and efficient algorithms which exploit lighting geometry property to improve the performance of the existing algorithms. First of all, considering that the state-of-the-art real-time environment matting and compositing method is short of flexibility, in the sense that it has to repeat the entire complex matte acquisition process if the distance between the object and the background is different from that in the acquisition stage, and also lacks accuracy, in the sense that it can only remove noises but not errors, we introduce the concept of refractive vector and propose to use a refractive vector field as a new representation for environment matte. Such refractive vector field provides great flexibility for transparent object environment matting and compositing. Particularly, with only one process of the matte acquisition and the refractive vector field extraction, we are able to composite the transparent object into an arbitrary background at any distance. Furthermore, we introduce novel light vector field fitting algorithms to simultaneously remove both noises and errors contained in the extracted matte data. Experimental results show that our method is less sensitive to artifacts and can generate perceptually good composition results for more general scenarios. Second, considering that the existing high-quality environment matting methods usually require the capturing of a few thousand sample images and spends a few hours in data acquisition, we propose a novel environment matting algorithm to capture and extract the environment matte data effectively and efficiently. Particularly, the recently developed compressive sensing theory is incorporated to reformulate the environment matting problem and simplify the data acquisition process. In addition, taking into account the special properties of light refraction and reflection effects of transparent objects, two advanced priors, group clustering and Gaussian priors, as well as other basic constraints are introduced during the matte data recovery process to combat with the limited image samples, suppress the effects of the measurement noise resulted from data acquisition, and faithfully recover the sparse environment matte data. Compared with most of the existing environment matting methods, our algorithm significantly simplifies and accelerates the environment matting extraction process while still achieving high-accuracy composition results. Finally, we consider the problem of high-quality 3D reconstruction under unknown illumination using the joint multi-view stereo (MVS) and photometric stereo (PS) technique. We take into account the property of lighting geometry and propose to use total variation term to constrain the light function recovery. Our algorithm can refine the 3D object model and recover the lighting conditions simultaneously. Comparing with many previous methods, our method can provide a significant computational saving and is compatible with traditional MVS methods without the need for extra images.