Diffusion time-step curriculum for one image to 3D generation

• Main Authors: YI, Xuanyu, WU, Zike, XU, Qingshan, ZHOU, Pan, LIM, Joo Hwee, ZHANG, Hanwang
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2024
• Subjects: Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/9020; https://ink.library.smu.edu.sg/context/sis_research/article/10023/viewcontent/2024_CVPR_Image_3D.pdf
• Institution: Singapore Management University

Score distillation sampling (SDS) has been widely adopted to overcome the absence of unseen views in reconstructing 3D objects from a single image. It leverages pretrained 2D diffusion models as teacher to guide the reconstruction of student 3D models. Despite their remarkable success, SDS-based methods often encounter geometric artifacts and texture saturation. We find out the crux is the overlooked indiscriminate treatment of diffusion time-steps during optimization: it unreasonably treats the studentteacher knowledge distillation to be equal at all time-steps and thus entangles coarse-grained and fine-grained modeling. Therefore, we propose the Diffusion Time-step Curriculum one-image-to-3D pipeline (DTC123), which involves both the teacher and student models collaborating with the time-step curriculum in a coarse-to-fine manner. Extensive experiments on NeRF4, RealFusion15, GSO and Level50 benchmark demonstrate that DTC123 can produce multiview consistent, high-quality, and diverse 3D assets. Codes and more generation demos will be released in https: //github.com/yxymessi/DTC123