Parsing-Conditioned Anime Translation: A New Dataset and Method

Parsing-Conditioned Anime Translation: A New Dataset and Method

Anime is an abstract art form that is substantially different from the human portrait, leading to a challenging misaligned image translation problem that is beyond the capability of existing methods. This can be boiled down to a highly ambiguous unconstrained translation between two domains. To this...

Full description

Saved in:
Bibliographic Details
Main Authors: LI, Zhansheng, XU, Yangyang, ZHAO, Nanxuan, ZHOU, Yang, LIU, Yongtuo, LIN, Dahua, HE, Shengfeng
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2023
Subjects:
Abstract arts
Aggregation methods
Anchorings
Feasible solution
Image editing
Image translation
Image-to-image translation
Labeled images
Prior-knowledge
Two domains
Databases and Information Systems
Online Access:https://ink.library.smu.edu.sg/sis_research/8434
https://ink.library.smu.edu.sg/context/sis_research/article/9437/viewcontent/Parsing_Conditioned_Anime_Translation__A_New_Dataset_and_Method.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
Description
Summary:Anime is an abstract art form that is substantially different from the human portrait, leading to a challenging misaligned image translation problem that is beyond the capability of existing methods. This can be boiled down to a highly ambiguous unconstrained translation between two domains. To this end, we design a new anime translation framework by deriving the prior knowledge of a pre-Trained StyleGAN model. We introduce disentangled encoders to separately embed structure and appearance information into the same latent code, governed by four tailored losses. Moreover, we develop a FaceBank aggregation method that leverages the generated data of the StyleGAN, anchoring the prediction to produce in-domain animes. To empower our model and promote the research of anime translation, we propose the first anime portrait parsing dataset, Danbooru-Parsing, containing 4,921 densely labeled images across 17 classes. This dataset connects the face semantics with appearances, enabling our new constrained translation setting. We further show the editability of our results, and extend our method to manga images, by generating the first manga parsing pseudo data. Extensive experiments demonstrate the values of our new dataset and method, resulting in the first feasible solution on anime translation.

Similar Items