Cocktail: mixing multi-modality controls for text-conditional image generation

Text-conditional diffusion models are able to generate high-fidelity images with diverse contents. However, linguistic representations frequently exhibit ambiguous descriptions of the envisioned objective imagery, requiring the incorporation of additional control signals to bolster the efficacy of t...

Full description

Saved in:
Bibliographic Details
Main Authors: Hu, Minghui, Zheng, Jianbin, Liu, Daqing, Zheng, Chuanxia, Wang, Chaoyue, Tao, Dacheng, Cham, Tat-Jen
Other Authors: School of Computer Science and Engineering
Format: Conference or Workshop Item
Language:English
Published: 2023
Subjects:
Online Access:https://hdl.handle.net/10356/172668
https://nips.cc/virtual/2023/calendar
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-172668
record_format dspace
spelling sg-ntu-dr.10356-1726682023-12-19T06:36:41Z Cocktail: mixing multi-modality controls for text-conditional image generation Hu, Minghui Zheng, Jianbin Liu, Daqing Zheng, Chuanxia Wang, Chaoyue Tao, Dacheng Cham, Tat-Jen School of Computer Science and Engineering 37th Annual Conference on Neural Information Processing Systems (NeurIPS 2023) Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision Text-Conditional Image Generation Computer Graphics Text-conditional diffusion models are able to generate high-fidelity images with diverse contents. However, linguistic representations frequently exhibit ambiguous descriptions of the envisioned objective imagery, requiring the incorporation of additional control signals to bolster the efficacy of text-guided diffusion models. In this work, we propose Cocktail, a pipeline to mix various modalities into one embedding, amalgamated with a generalized ControlNet (gControlNet), a controllable normalisation (ControlNorm), and a spatial guidance sampling method, to actualize multi-modal and spatially-refined control for text-conditional diffusion models. Specifically, we introduce a hyper-network gControlNet, dedicated to the alignment and infusion of the control signals from disparate modalities into the pre-trained diffusion model. gControlNet is capable of accepting flexible modality signals, encompassing the simultaneous reception of any combination of modality signals, or the supplementary fusion of multiple modality signals. The control signals are then fused and injected into the backbone model according to our proposed ControlNorm. Furthermore, our advanced spatial guidance sampling methodology proficiently incorporates the control signal into the designated region, thereby circumventing the manifestation of undesired objects within the generated image. We demonstrate the results of our method in controlling various modalities, proving high-quality synthesis and fidelity to multiple external signals. 2023-12-19T06:34:21Z 2023-12-19T06:34:21Z 2023 Conference Paper Hu, M., Zheng, J., Liu, D., Zheng, C., Wang, C., Tao, D. & Cham, T. (2023). Cocktail: mixing multi-modality controls for text-conditional image generation. 37th Annual Conference on Neural Information Processing Systems (NeurIPS 2023). https://hdl.handle.net/10356/172668 https://nips.cc/virtual/2023/calendar en © 2023 Neural Information Processing Systems. All rights reserved.
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision
Text-Conditional Image Generation
Computer Graphics
spellingShingle Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision
Text-Conditional Image Generation
Computer Graphics
Hu, Minghui
Zheng, Jianbin
Liu, Daqing
Zheng, Chuanxia
Wang, Chaoyue
Tao, Dacheng
Cham, Tat-Jen
Cocktail: mixing multi-modality controls for text-conditional image generation
description Text-conditional diffusion models are able to generate high-fidelity images with diverse contents. However, linguistic representations frequently exhibit ambiguous descriptions of the envisioned objective imagery, requiring the incorporation of additional control signals to bolster the efficacy of text-guided diffusion models. In this work, we propose Cocktail, a pipeline to mix various modalities into one embedding, amalgamated with a generalized ControlNet (gControlNet), a controllable normalisation (ControlNorm), and a spatial guidance sampling method, to actualize multi-modal and spatially-refined control for text-conditional diffusion models. Specifically, we introduce a hyper-network gControlNet, dedicated to the alignment and infusion of the control signals from disparate modalities into the pre-trained diffusion model. gControlNet is capable of accepting flexible modality signals, encompassing the simultaneous reception of any combination of modality signals, or the supplementary fusion of multiple modality signals. The control signals are then fused and injected into the backbone model according to our proposed ControlNorm. Furthermore, our advanced spatial guidance sampling methodology proficiently incorporates the control signal into the designated region, thereby circumventing the manifestation of undesired objects within the generated image. We demonstrate the results of our method in controlling various modalities, proving high-quality synthesis and fidelity to multiple external signals.
author2 School of Computer Science and Engineering
author_facet School of Computer Science and Engineering
Hu, Minghui
Zheng, Jianbin
Liu, Daqing
Zheng, Chuanxia
Wang, Chaoyue
Tao, Dacheng
Cham, Tat-Jen
format Conference or Workshop Item
author Hu, Minghui
Zheng, Jianbin
Liu, Daqing
Zheng, Chuanxia
Wang, Chaoyue
Tao, Dacheng
Cham, Tat-Jen
author_sort Hu, Minghui
title Cocktail: mixing multi-modality controls for text-conditional image generation
title_short Cocktail: mixing multi-modality controls for text-conditional image generation
title_full Cocktail: mixing multi-modality controls for text-conditional image generation
title_fullStr Cocktail: mixing multi-modality controls for text-conditional image generation
title_full_unstemmed Cocktail: mixing multi-modality controls for text-conditional image generation
title_sort cocktail: mixing multi-modality controls for text-conditional image generation
publishDate 2023
url https://hdl.handle.net/10356/172668
https://nips.cc/virtual/2023/calendar
_version_ 1787136810105700352