Sem2NeRF: converting single-view semantic masks to neural radiance fields
Image translation and manipulation have gain increasing attention along with the rapid development of deep generative models. Although existing approaches have brought impressive results, they mainly operated in 2D space. In light of recent advances in NeRF-based 3D-aware generative models, we intro...
Saved in:
Main Authors: | , , , , |
---|---|
Other Authors: | |
Format: | Conference or Workshop Item |
Language: | English |
Published: |
2023
|
Subjects: | |
Online Access: | https://hdl.handle.net/10356/172663 |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Nanyang Technological University |
Language: | English |
id |
sg-ntu-dr.10356-172663 |
---|---|
record_format |
dspace |
spelling |
sg-ntu-dr.10356-1726632023-12-19T05:55:42Z Sem2NeRF: converting single-view semantic masks to neural radiance fields Chen, Yuedong Wu, Qianyi Zheng, Chuanxia Cham, Tat-Jen Cai, Jianfei School of Computer Science and Engineering 17th European Conference on Computer Vision (ECCV 2022) Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision Neural Networks Computer Graphics Image translation and manipulation have gain increasing attention along with the rapid development of deep generative models. Although existing approaches have brought impressive results, they mainly operated in 2D space. In light of recent advances in NeRF-based 3D-aware generative models, we introduce a new task, Semantic-to-NeRF translation, that aims to reconstruct a 3D scene modelled by NeRF, conditioned on one single-view semantic mask as input. To kick-off this novel task, we propose the Sem2NeRF framework. In particular, Sem2NeRF addresses the highly challenging task by encoding the semantic mask into the latent code that controls the 3D scene representation of a pre-trained decoder. To further improve the accuracy of the mapping, we integrate a new region-aware learning strategy into the design of both the encoder and the decoder. We verify the efficacy of the proposed Sem2NeRF and demonstrate that it outperforms several strong baselines on two benchmark datasets. Code and video are available at https://donydchen.github.io/sem2nerf/. 2023-12-19T05:55:42Z 2023-12-19T05:55:42Z 2022 Conference Paper Chen, Y., Wu, Q., Zheng, C., Cham, T. & Cai, J. (2022). Sem2NeRF: converting single-view semantic masks to neural radiance fields. 17th European Conference on Computer Vision (ECCV 2022), 730-748. https://dx.doi.org/10.1007/978-3-031-19781-9_42 9783031197802 https://hdl.handle.net/10356/172663 10.1007/978-3-031-19781-9_42 2-s2.0-85142676989 730 748 en © 2022 Association for Computing Machinery. All rights reserved. |
institution |
Nanyang Technological University |
building |
NTU Library |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
NTU Library |
collection |
DR-NTU |
language |
English |
topic |
Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision Neural Networks Computer Graphics |
spellingShingle |
Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision Neural Networks Computer Graphics Chen, Yuedong Wu, Qianyi Zheng, Chuanxia Cham, Tat-Jen Cai, Jianfei Sem2NeRF: converting single-view semantic masks to neural radiance fields |
description |
Image translation and manipulation have gain increasing attention along with the rapid development of deep generative models. Although existing approaches have brought impressive results, they mainly operated in 2D space. In light of recent advances in NeRF-based 3D-aware generative models, we introduce a new task, Semantic-to-NeRF translation, that aims to reconstruct a 3D scene modelled by NeRF, conditioned on one single-view semantic mask as input. To kick-off this novel task, we propose the Sem2NeRF framework. In particular, Sem2NeRF addresses the highly challenging task by encoding the semantic mask into the latent code that controls the 3D scene representation of a pre-trained decoder. To further improve the accuracy of the mapping, we integrate a new region-aware learning strategy into the design of both the encoder and the decoder. We verify the efficacy of the proposed Sem2NeRF and demonstrate that it outperforms several strong baselines on two benchmark datasets. Code and video are available at https://donydchen.github.io/sem2nerf/. |
author2 |
School of Computer Science and Engineering |
author_facet |
School of Computer Science and Engineering Chen, Yuedong Wu, Qianyi Zheng, Chuanxia Cham, Tat-Jen Cai, Jianfei |
format |
Conference or Workshop Item |
author |
Chen, Yuedong Wu, Qianyi Zheng, Chuanxia Cham, Tat-Jen Cai, Jianfei |
author_sort |
Chen, Yuedong |
title |
Sem2NeRF: converting single-view semantic masks to neural radiance fields |
title_short |
Sem2NeRF: converting single-view semantic masks to neural radiance fields |
title_full |
Sem2NeRF: converting single-view semantic masks to neural radiance fields |
title_fullStr |
Sem2NeRF: converting single-view semantic masks to neural radiance fields |
title_full_unstemmed |
Sem2NeRF: converting single-view semantic masks to neural radiance fields |
title_sort |
sem2nerf: converting single-view semantic masks to neural radiance fields |
publishDate |
2023 |
url |
https://hdl.handle.net/10356/172663 |
_version_ |
1787136690848006144 |