DreamAnime: Learning style-identity textual disentanglement for anime and beyond
Text-to-image generation models have significantly broadened the horizons of creative expression through the power of natural language. However, navigating these models to generate unique concepts, alter their appearance, or reimagine them in unfamiliar roles presents an intricate challenge. For ins...
Saved in:
Main Authors: | , , , , |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2024
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/9799 |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-10799 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-107992024-12-12T09:00:03Z DreamAnime: Learning style-identity textual disentanglement for anime and beyond XU, Chenshu XU, Yangyang ZHANG, Huaidong XU, Xuemiao HE, Shengfeng Text-to-image generation models have significantly broadened the horizons of creative expression through the power of natural language. However, navigating these models to generate unique concepts, alter their appearance, or reimagine them in unfamiliar roles presents an intricate challenge. For instance, how can we exploit language-guided models to transpose an anime character into a different art style, or envision a beloved character in a radically different setting or role? This paper unveils a novel approach named DreamAnime, designed to provide this level of creative freedom. Using a minimal set of 2-3 images of a user-specified concept such as an anime character or an art style, we teach our model to encapsulate its essence through novel "words" in the embedding space of a pre-existing text-to-image model. Crucially, we disentangle the concepts of style and identity into two separate "words", thus providing the ability to manipulate them independently. These distinct "words" can then be pieced together into natural language sentences, promoting an intuitive and personalized creative process. Empirical results suggest that this disentanglement into separate word embeddings successfully captures a broad range of unique and complex concepts, with each word focusing on style or identity as appropriate. Comparisons with existing methods illustrate DreamAnime's superior capacity to accurately interpret and recreate the desired concepts across various applications and tasks. Code is available at https://github.com/chnshx/DreamAnime. 2024-05-07T07:00:00Z text https://ink.library.smu.edu.sg/sis_research/9799 info:doi/10.1109/TVCG.2024.3397712 Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Data Models Training Task Analysis Computational Modeling Natural Languages Visualization Shape Customization Diffusion Image Synthesis Style Disentanglement Artificial Intelligence and Robotics Graphics and Human Computer Interfaces |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
Data Models Training Task Analysis Computational Modeling Natural Languages Visualization Shape Customization Diffusion Image Synthesis Style Disentanglement Artificial Intelligence and Robotics Graphics and Human Computer Interfaces |
spellingShingle |
Data Models Training Task Analysis Computational Modeling Natural Languages Visualization Shape Customization Diffusion Image Synthesis Style Disentanglement Artificial Intelligence and Robotics Graphics and Human Computer Interfaces XU, Chenshu XU, Yangyang ZHANG, Huaidong XU, Xuemiao HE, Shengfeng DreamAnime: Learning style-identity textual disentanglement for anime and beyond |
description |
Text-to-image generation models have significantly broadened the horizons of creative expression through the power of natural language. However, navigating these models to generate unique concepts, alter their appearance, or reimagine them in unfamiliar roles presents an intricate challenge. For instance, how can we exploit language-guided models to transpose an anime character into a different art style, or envision a beloved character in a radically different setting or role? This paper unveils a novel approach named DreamAnime, designed to provide this level of creative freedom. Using a minimal set of 2-3 images of a user-specified concept such as an anime character or an art style, we teach our model to encapsulate its essence through novel "words" in the embedding space of a pre-existing text-to-image model. Crucially, we disentangle the concepts of style and identity into two separate "words", thus providing the ability to manipulate them independently. These distinct "words" can then be pieced together into natural language sentences, promoting an intuitive and personalized creative process. Empirical results suggest that this disentanglement into separate word embeddings successfully captures a broad range of unique and complex concepts, with each word focusing on style or identity as appropriate. Comparisons with existing methods illustrate DreamAnime's superior capacity to accurately interpret and recreate the desired concepts across various applications and tasks. Code is available at https://github.com/chnshx/DreamAnime. |
format |
text |
author |
XU, Chenshu XU, Yangyang ZHANG, Huaidong XU, Xuemiao HE, Shengfeng |
author_facet |
XU, Chenshu XU, Yangyang ZHANG, Huaidong XU, Xuemiao HE, Shengfeng |
author_sort |
XU, Chenshu |
title |
DreamAnime: Learning style-identity textual disentanglement for anime and beyond |
title_short |
DreamAnime: Learning style-identity textual disentanglement for anime and beyond |
title_full |
DreamAnime: Learning style-identity textual disentanglement for anime and beyond |
title_fullStr |
DreamAnime: Learning style-identity textual disentanglement for anime and beyond |
title_full_unstemmed |
DreamAnime: Learning style-identity textual disentanglement for anime and beyond |
title_sort |
dreamanime: learning style-identity textual disentanglement for anime and beyond |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2024 |
url |
https://ink.library.smu.edu.sg/sis_research/9799 |
_version_ |
1819113142125330432 |