3D snapshot: Invertible embedding of 3D neural representations in a single image

• Main Authors: LU, Yuqin, DENG, Bailin, ZHONG, Zhixuan, ZHANG, Tianle, QUAN, Yuhui, CAI, Hongmin, HE, Shengfeng
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2024
• Subjects: Three Dimensional Displays; Rendering Computer Graphics; Solid Modeling; Image Reconstruction; Image Color Analysis; Neural Networks; Metaverse; Invertible Image Processing; Neural Representations; Single Image; Neural Coding; 3 D Snapshots; Neural Network; Dynamic Network; Image Representation; Reconstruction Quality; 3 D Scene; Compact Network; Scene Representation; Dynamic Neural Network; Neural Image; Loss Function; Data Storage; Model Size; Wavelet Transform; Spatial Domain; Spatial Coordinates; Volume Density; Short Video; Image Embedding; View Synthesis; Steganography; Noisy Images; Dynamic Update; Least Significant Bit; View Direction; Spherical Harmonics; Intermediate Representation; Half Of The Channel; Artificial Intelligence and Robotics; Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/9765; https://ink.library.smu.edu.sg/context/sis_research/article/10765/viewcontent/6197abae_9c0c_4818_a024_0ea407ca20f9__1_.pdf
• Institution: Singapore Management University

3D neural rendering enables photo-realistic reconstruction of a specific scene by encoding discontinuous inputs into a neural representation. Despite the remarkable rendering results, the storage of network parameters is not transmission-friendly and not extendable to metaverse applications. In this paper, we propose an invertible neural rendering approach that enables generating an interactive 3D model from a single image (i.e., 3D Snapshot). Our idea is to distill a pre-trained neural rendering model (e.g., NeRF) into a visualizable image form that can then be easily inverted back to a neural network. To this end, we first present a neural image distillation method to optimize three neural planes for representing the original neural rendering model. However, this representation is noisy and visually meaningless. We thus propose a dynamic invertible neural network to embed this noisy representation into a plausible image representation of the scene. We demonstrate promising reconstruction quality quantitatively and qualitatively, by comparing to the original neural rendering model, as well as video-based invertible methods. On the other hand, our method can store dozens of NeRFs with a compact restoration network (5 MB), and embedding each 3D scene takes up only 160 KB of storage. More importantly, our approach is the first solution that allows embedding a neural rendering model into image representations, which enables applications like creating an interactive 3D model from a printed image in the metaverse.