Real-time shadow-aware portrait relighting in virtual backgrounds for realistic telepresence

While using virtual backgrounds has recently become a very popular feature in videoconferencing, there often exists a jarring mismatch between the lighting of the user and the illumination condition of the virtual background. Existing portrait relighting methods can alleviate the problem, but do not...

Full description

Saved in:
Bibliographic Details
Main Authors: Song, Guoxian, Cham, Tat-Jen, Cai, Jianfei, Zheng, Jianmin
Other Authors: School of Computer Science and Engineering
Format: Conference or Workshop Item
Language:English
Published: 2023
Subjects:
Online Access:https://hdl.handle.net/10356/172662
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-172662
record_format dspace
spelling sg-ntu-dr.10356-1726622023-12-19T05:40:00Z Real-time shadow-aware portrait relighting in virtual backgrounds for realistic telepresence Song, Guoxian Cham, Tat-Jen Cai, Jianfei Zheng, Jianmin School of Computer Science and Engineering 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Engineering::Computer science and engineering::Computing methodologies::Computer graphics Shadow Generation Relighting While using virtual backgrounds has recently become a very popular feature in videoconferencing, there often exists a jarring mismatch between the lighting of the user and the illumination condition of the virtual background. Existing portrait relighting methods can alleviate the problem, but do not have the capacity to deal with difficult shadow effects. In this paper, we present a new shadow-aware portrait relighting system that can relight an input portrait to be consistent with a given desired background image with shadow effects. Our system consists of four major components: portrait neutralization, illumination estimation, shadow generation and hierarchical neural rendering, which are all based on deep neural networks, and the whole system is end-to-end trainable. In addition, we created a large-scale photorealistic synthetic dataset with shadow, illumination and depth annotations for training, which allows our model to generalize well to real images. The extensive experiments demonstrate that our shadow-aware relight system outperforms the state-of-the-art portrait relighting solutions in terms of producing more lighting-consistent relighted images with shadow effects. This study is supported under the RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from Singapore Telecommunications Limited (Singtel), through Singtel Cognitive and Artificial Intelligence Lab for Enterprises (SCALE@NTU). 2023-12-19T05:39:59Z 2023-12-19T05:39:59Z 2022 Conference Paper Song, G., Cham, T., Cai, J. & Zheng, J. (2022). Real-time shadow-aware portrait relighting in virtual backgrounds for realistic telepresence. 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 729-738. https://dx.doi.org/10.1109/ISMAR55827.2022.00091 9781665453257 https://hdl.handle.net/10356/172662 10.1109/ISMAR55827.2022.00091 2-s2.0-85146439023 729 738 en IAF-ICP © 2022 IEEE. All rights reserved.
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Engineering::Computer science and engineering::Computing methodologies::Computer graphics
Shadow Generation
Relighting
spellingShingle Engineering::Computer science and engineering::Computing methodologies::Computer graphics
Shadow Generation
Relighting
Song, Guoxian
Cham, Tat-Jen
Cai, Jianfei
Zheng, Jianmin
Real-time shadow-aware portrait relighting in virtual backgrounds for realistic telepresence
description While using virtual backgrounds has recently become a very popular feature in videoconferencing, there often exists a jarring mismatch between the lighting of the user and the illumination condition of the virtual background. Existing portrait relighting methods can alleviate the problem, but do not have the capacity to deal with difficult shadow effects. In this paper, we present a new shadow-aware portrait relighting system that can relight an input portrait to be consistent with a given desired background image with shadow effects. Our system consists of four major components: portrait neutralization, illumination estimation, shadow generation and hierarchical neural rendering, which are all based on deep neural networks, and the whole system is end-to-end trainable. In addition, we created a large-scale photorealistic synthetic dataset with shadow, illumination and depth annotations for training, which allows our model to generalize well to real images. The extensive experiments demonstrate that our shadow-aware relight system outperforms the state-of-the-art portrait relighting solutions in terms of producing more lighting-consistent relighted images with shadow effects.
author2 School of Computer Science and Engineering
author_facet School of Computer Science and Engineering
Song, Guoxian
Cham, Tat-Jen
Cai, Jianfei
Zheng, Jianmin
format Conference or Workshop Item
author Song, Guoxian
Cham, Tat-Jen
Cai, Jianfei
Zheng, Jianmin
author_sort Song, Guoxian
title Real-time shadow-aware portrait relighting in virtual backgrounds for realistic telepresence
title_short Real-time shadow-aware portrait relighting in virtual backgrounds for realistic telepresence
title_full Real-time shadow-aware portrait relighting in virtual backgrounds for realistic telepresence
title_fullStr Real-time shadow-aware portrait relighting in virtual backgrounds for realistic telepresence
title_full_unstemmed Real-time shadow-aware portrait relighting in virtual backgrounds for realistic telepresence
title_sort real-time shadow-aware portrait relighting in virtual backgrounds for realistic telepresence
publishDate 2023
url https://hdl.handle.net/10356/172662
_version_ 1787136561360404480