Improving 3D reconstruction via RGB-D camera registration and shading-based surface refinement

• Main Author: Deng, Teng
• Other Authors: Cai Jianfei
• Format: Theses and Dissertations
• Language: English
• Published: 2019
• Subjects: DRNTU::Engineering::Computer science and engineering
• Online Access: https://hdl.handle.net/10356/88852; http://hdl.handle.net/10220/47633
• Institution: Nanyang Technological University

The process of generating a three-dimensional (3D) computer model of a real-world object or scene from two-dimensional images is called \emph{3D Reconstruction}. Nowadays, the emergence of consumer-grade RGB-D cameras brings opportunities and challenges in 3D reconstruction. On one hand, geometric information can be captured with ease using these consumer-grade RGB-D camera, resulting in a number of interesting and exciting 3D applications. On the other hand, the quality of the 3D data suffers from distortions and noises, limiting the quality of the 3D reconstruction. Shapes obtained from consumer-grade RGB-D cameras are in general of low quality and usually lack of enough surface details. Meanwhile, great progresses have been made on the surface recovery using shape-from-shading (SfS), in which surface details can be obtained by analyzing the relationship between lighting conditions, object materials and geometries. This thesis enhances the recent 3D reconstruction techniques by improving the calibration and registration of the depth capturing devices and refining surface details of rough and noisy initial shape through SfS. Firstly, we introduce a multiple RGB-D camera registration method using local rigid transforms. While impressive 3D reconstruction results have been obtained, combining data acquired by multiple RGB-D cameras constitutes a technical challenge. Several methods have been proposed to estimate the internal parameters of each RGB-D camera (such as depth mapping function and focal length). Despite that the textured geometry obtained by each RGB-D camera individually is visually attractive, even state-of-the-art methods have difficulties in correctly combining the textured geometries obtained by several RGB-D cameras via a rigid transformation. Based on this observation, our approach registers the RGB-D cameras by a smooth field of rigid transformations, instead of a single rigid transformation. Experimental results on challenging data demonstrate the validity of the proposed approach. Secondly, we propose a method for directly registering multiple depth cameras using everyday objects. It is a more practical method for conducting and maintaining registration of multi-depth sensors, via replacing checkerboards with everyday objects found in the scene, such as regular furniture. Particularly, high quality pre-scanned 3D shapes of standard furniture are used as calibration targets. We propose a unified framework that jointly computes the optimal extrinsic calibration and depth correction parameters. Experimental results show that our proposed method significantly outperforms state-of-the-art depth camera registration methods. Thirdly, we present subdivision-based representations for both lighting and geometry in shape-from-shading. A very recent shading-based method introduced a per-vertex overall illumination model for surface reconstruction, which has the advantages of conveniently handling complicated lighting condition and avoiding explicit estimation of visibility and varied albedo. However, due to its discrete nature, the per-vertex overall illumination requires a large amount of memory and lacks intrinsic coherence. To overcome these problems, we propose to use classic subdivision to define the basic smooth lighting function and surface and introduce additional independent variables into the subdivision to adaptively model sharp changes of illumination and geometry. Compared to previous works, the new model not only preserves the merits of the per-vertex illumination model, but also greatly reduces the number of variables required in surface recovery and intrinsically regularizes the illumination vectors and the surface. These features make the new model very suitable for surface detail recovery under general, unknown illumination condition. Particularly, a variational surface reconstruction method built upon the subdivision representations for lighting and geometry is developed. The experiments on both synthetic and real-world datasets have demonstrated that the proposed method can achieve memory efficiency and improve surface detail recovery. Lastly, we improve the subdivision-based lighting model and exploit parallel scheme of the algorithm for GPU implementation to achieve real-time performance. Particularly, we present a spatial-varying illumination model for shading-based depth refinement based on a smooth Spherical Harmonics (SH) lighting field. The proposed lighting model can recover shading under challenging unknown lighting conditions, thus improving the quality of recovered surface detail. To avoid over-parameterization, local lighting coefficients are treated as a vector-valued function which is represented by subdivided surfaces using Catmull-Clark subdivision. We solve our lighting model utilizing a highly parallelized scheme that recovers lighting in a few milliseconds. A real-time shading-based depth recovery system is implemented with the integration of our proposed lighting model. We conduct quantitative and qualitative evaluations on both synthetic and real-world datasets under challenging illumination. The experimental results show our method outperforms the state-of-the-art real-time shading-based depth refinement system.