Half-body portrait relighting with overcomplete lighting representation

We present a neural-based model for relighting a half-body portrait image by simply referring to another portrait image with the desired lighting condition. Rather than following classical inverse rendering methodology that involves estimating normals, albedo and environment maps, we implicitly e...

Full description

Saved in:
Bibliographic Details
Main Authors: Song, Guoxian, Cham, Tat-Jen, Cai, Jianfei, Zheng, Jianmin
Other Authors: School of Computer Science and Engineering
Format: Article
Language:English
Published: 2023
Subjects:
Online Access:https://hdl.handle.net/10356/172647
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-172647
record_format dspace
spelling sg-ntu-dr.10356-1726472023-12-19T01:47:25Z Half-body portrait relighting with overcomplete lighting representation Song, Guoxian Cham, Tat-Jen Cai, Jianfei Zheng, Jianmin School of Computer Science and Engineering Engineering::Computer science and engineering Image Processing Computational Photography We present a neural-based model for relighting a half-body portrait image by simply referring to another portrait image with the desired lighting condition. Rather than following classical inverse rendering methodology that involves estimating normals, albedo and environment maps, we implicitly encode the subject and lighting in a latent space, and use these latent codes to generate relighted images by neural rendering. A key technical innovation is the use of a novel overcomplete lighting representation, which facilitates lighting interpolation in the latent space, as well as helping regularize the self-organization of the lighting latent space during training. In addition, we propose a novel multiplicative neural render that more effectively combines the subject and lighting latent codes for rendering. We also created a large-scale photorealistic rendered relighting dataset for training, which allows our model to generalize well to real images. Extensive experiments demonstrate that our system not only outperforms existing methods for referral-based portrait relighting, but also has the capability generate sequences of relighted images via lighting rotations. Agency for Science, Technology and Research (A*STAR) This research was conducted at Singtel Cognitive and Artificial Intelligence Lab for Enterprises (SCALE@NTU), which is a collaboration between Singapore Telecommunications Limited (Singtel) and Nanyang Technological University (NTU) that is supported by A*STAR under its Industry Alignment Fund (LOA Award number:I1701E0013). 2023-12-19T01:47:25Z 2023-12-19T01:47:25Z 2021 Journal Article Song, G., Cham, T., Cai, J. & Zheng, J. (2021). Half-body portrait relighting with overcomplete lighting representation. Computer Graphics Forum, 40(6), 371-381. https://dx.doi.org/10.1111/cgf.14384 0167-7055 https://hdl.handle.net/10356/172647 10.1111/cgf.14384 2-s2.0-85109282950 6 40 371 381 en I1701E0013 Computer Graphics Forum © 2021 Eurographics - The European Association for Computer Graphics and John Wiley & Sons Ltd. All rights reserved.
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Engineering::Computer science and engineering
Image Processing
Computational Photography
spellingShingle Engineering::Computer science and engineering
Image Processing
Computational Photography
Song, Guoxian
Cham, Tat-Jen
Cai, Jianfei
Zheng, Jianmin
Half-body portrait relighting with overcomplete lighting representation
description We present a neural-based model for relighting a half-body portrait image by simply referring to another portrait image with the desired lighting condition. Rather than following classical inverse rendering methodology that involves estimating normals, albedo and environment maps, we implicitly encode the subject and lighting in a latent space, and use these latent codes to generate relighted images by neural rendering. A key technical innovation is the use of a novel overcomplete lighting representation, which facilitates lighting interpolation in the latent space, as well as helping regularize the self-organization of the lighting latent space during training. In addition, we propose a novel multiplicative neural render that more effectively combines the subject and lighting latent codes for rendering. We also created a large-scale photorealistic rendered relighting dataset for training, which allows our model to generalize well to real images. Extensive experiments demonstrate that our system not only outperforms existing methods for referral-based portrait relighting, but also has the capability generate sequences of relighted images via lighting rotations.
author2 School of Computer Science and Engineering
author_facet School of Computer Science and Engineering
Song, Guoxian
Cham, Tat-Jen
Cai, Jianfei
Zheng, Jianmin
format Article
author Song, Guoxian
Cham, Tat-Jen
Cai, Jianfei
Zheng, Jianmin
author_sort Song, Guoxian
title Half-body portrait relighting with overcomplete lighting representation
title_short Half-body portrait relighting with overcomplete lighting representation
title_full Half-body portrait relighting with overcomplete lighting representation
title_fullStr Half-body portrait relighting with overcomplete lighting representation
title_full_unstemmed Half-body portrait relighting with overcomplete lighting representation
title_sort half-body portrait relighting with overcomplete lighting representation
publishDate 2023
url https://hdl.handle.net/10356/172647
_version_ 1787136787591725056