SceneCrafter - 3D scene generation and stylization

• Main Author: Yew Fu Yen
• Other Authors: Liu Ziwei
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; 3D stylization; Score distillation sampling; Neural rendering; 3D generative model; Unbounded scene generation
• Online Access: https://hdl.handle.net/10356/176074
• Institution: Nanyang Technological University

3D landscape assets are widely used in modern games and virtual reality worlds. Often, a 3D landscape is unbounded in nature. To design such a vast landscape manually requires much time and effort, thus it is favourable to develop a generative approach that can synthesize a realistic 3D world from noise inputs. To tackle this issue, SceneDreamer was proposed as a generative model trained from in-the-wild images for the task of unbounded landscape generation. Though SceneDreamer is robust in its ability to render photorealistic views of 3D scenes, it does not offer much controllability in its ability to stylize rendering outputs and lightings. As an improvement to the original SceneDreamer model, I propose SceneCrafter, a finetuning approach based on Score Distillation Sampling (SDS), which allows the model to conditionally stylize 3D generated scenes according to user text prompt inputs. SceneCrafter utilizes a pretrained Stable Diffusion model to guide the finetuning of SceneDreamer. Different parameter groups in SceneDreamer are finetuned at appropriately set learning rates to maximize stylization effects while retaining 3D consistency. The proposed loss formulation also allows for flexible adjustment in the degree of stylization.