3D snapshot: Invertible embedding of 3D neural representations in a single image
3D neural rendering enables photo-realistic reconstruction of a specific scene by encoding discontinuous inputs into a neural representation. Despite the remarkable rendering results, the storage of network parameters is not transmission-friendly and not extendable to metaverse applications. In this...
Saved in:
Main Authors: | , , , , , , |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2024
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/9765 https://ink.library.smu.edu.sg/context/sis_research/article/10765/viewcontent/6197abae_9c0c_4818_a024_0ea407ca20f9__1_.pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-10765 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-107652024-12-16T02:41:42Z 3D snapshot: Invertible embedding of 3D neural representations in a single image LU, Yuqin DENG, Bailin ZHONG, Zhixuan ZHANG, Tianle QUAN, Yuhui CAI, Hongmin HE, Shengfeng 3D neural rendering enables photo-realistic reconstruction of a specific scene by encoding discontinuous inputs into a neural representation. Despite the remarkable rendering results, the storage of network parameters is not transmission-friendly and not extendable to metaverse applications. In this paper, we propose an invertible neural rendering approach that enables generating an interactive 3D model from a single image (i.e., 3D Snapshot). Our idea is to distill a pre-trained neural rendering model (e.g., NeRF) into a visualizable image form that can then be easily inverted back to a neural network. To this end, we first present a neural image distillation method to optimize three neural planes for representing the original neural rendering model. However, this representation is noisy and visually meaningless. We thus propose a dynamic invertible neural network to embed this noisy representation into a plausible image representation of the scene. We demonstrate promising reconstruction quality quantitatively and qualitatively, by comparing to the original neural rendering model, as well as video-based invertible methods. On the other hand, our method can store dozens of NeRFs with a compact restoration network (5 MB), and embedding each 3D scene takes up only 160 KB of storage. More importantly, our approach is the first solution that allows embedding a neural rendering model into image representations, which enables applications like creating an interactive 3D model from a printed image in the metaverse. 2024-12-01T08:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/9765 info:doi/10.1109/TPAMI.2024.3411051 https://ink.library.smu.edu.sg/context/sis_research/article/10765/viewcontent/6197abae_9c0c_4818_a024_0ea407ca20f9__1_.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Three Dimensional Displays Rendering Computer Graphics Solid Modeling Image Reconstruction Image Color Analysis Neural Networks Metaverse Invertible Image Processing Neural Representations Single Image Neural Coding 3 D Snapshots Neural Network Dynamic Network Image Representation Reconstruction Quality 3 D Scene Compact Network Scene Representation Dynamic Neural Network Neural Image Loss Function Data Storage Model Size Wavelet Transform Spatial Domain Spatial Coordinates Volume Density Short Video Image Embedding View Synthesis Steganography Noisy Images Dynamic Update Least Significant Bit View Direction Spherical Harmonics Intermediate Representation Half Of The Channel Artificial Intelligence and Robotics Graphics and Human Computer Interfaces |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
Three Dimensional Displays Rendering Computer Graphics Solid Modeling Image Reconstruction Image Color Analysis Neural Networks Metaverse Invertible Image Processing Neural Representations Single Image Neural Coding 3 D Snapshots Neural Network Dynamic Network Image Representation Reconstruction Quality 3 D Scene Compact Network Scene Representation Dynamic Neural Network Neural Image Loss Function Data Storage Model Size Wavelet Transform Spatial Domain Spatial Coordinates Volume Density Short Video Image Embedding View Synthesis Steganography Noisy Images Dynamic Update Least Significant Bit View Direction Spherical Harmonics Intermediate Representation Half Of The Channel Artificial Intelligence and Robotics Graphics and Human Computer Interfaces |
spellingShingle |
Three Dimensional Displays Rendering Computer Graphics Solid Modeling Image Reconstruction Image Color Analysis Neural Networks Metaverse Invertible Image Processing Neural Representations Single Image Neural Coding 3 D Snapshots Neural Network Dynamic Network Image Representation Reconstruction Quality 3 D Scene Compact Network Scene Representation Dynamic Neural Network Neural Image Loss Function Data Storage Model Size Wavelet Transform Spatial Domain Spatial Coordinates Volume Density Short Video Image Embedding View Synthesis Steganography Noisy Images Dynamic Update Least Significant Bit View Direction Spherical Harmonics Intermediate Representation Half Of The Channel Artificial Intelligence and Robotics Graphics and Human Computer Interfaces LU, Yuqin DENG, Bailin ZHONG, Zhixuan ZHANG, Tianle QUAN, Yuhui CAI, Hongmin HE, Shengfeng 3D snapshot: Invertible embedding of 3D neural representations in a single image |
description |
3D neural rendering enables photo-realistic reconstruction of a specific scene by encoding discontinuous inputs into a neural representation. Despite the remarkable rendering results, the storage of network parameters is not transmission-friendly and not extendable to metaverse applications. In this paper, we propose an invertible neural rendering approach that enables generating an interactive 3D model from a single image (i.e., 3D Snapshot). Our idea is to distill a pre-trained neural rendering model (e.g., NeRF) into a visualizable image form that can then be easily inverted back to a neural network. To this end, we first present a neural image distillation method to optimize three neural planes for representing the original neural rendering model. However, this representation is noisy and visually meaningless. We thus propose a dynamic invertible neural network to embed this noisy representation into a plausible image representation of the scene. We demonstrate promising reconstruction quality quantitatively and qualitatively, by comparing to the original neural rendering model, as well as video-based invertible methods. On the other hand, our method can store dozens of NeRFs with a compact restoration network (5 MB), and embedding each 3D scene takes up only 160 KB of storage. More importantly, our approach is the first solution that allows embedding a neural rendering model into image representations, which enables applications like creating an interactive 3D model from a printed image in the metaverse. |
format |
text |
author |
LU, Yuqin DENG, Bailin ZHONG, Zhixuan ZHANG, Tianle QUAN, Yuhui CAI, Hongmin HE, Shengfeng |
author_facet |
LU, Yuqin DENG, Bailin ZHONG, Zhixuan ZHANG, Tianle QUAN, Yuhui CAI, Hongmin HE, Shengfeng |
author_sort |
LU, Yuqin |
title |
3D snapshot: Invertible embedding of 3D neural representations in a single image |
title_short |
3D snapshot: Invertible embedding of 3D neural representations in a single image |
title_full |
3D snapshot: Invertible embedding of 3D neural representations in a single image |
title_fullStr |
3D snapshot: Invertible embedding of 3D neural representations in a single image |
title_full_unstemmed |
3D snapshot: Invertible embedding of 3D neural representations in a single image |
title_sort |
3d snapshot: invertible embedding of 3d neural representations in a single image |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2024 |
url |
https://ink.library.smu.edu.sg/sis_research/9765 https://ink.library.smu.edu.sg/context/sis_research/article/10765/viewcontent/6197abae_9c0c_4818_a024_0ea407ca20f9__1_.pdf |
_version_ |
1819113132193218560 |