Beyond textual constraints : Learning novel diffusion conditions with fewer examples

• Main Authors: YU, Yuyang, LIU, Bangzhen, ZHENG, Chenxi, XU, Xuemiao, ZHANG, Huaidong, HE, Shengfeng
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2024
• Subjects: Prompt-free conditional learning; Conditional negative rectification; Training; Computer vision; Adaptation models; Codes; Text to image; Diffusion processes; Diffusion model; Image synthesis; Controllable image generation; Artificial Intelligence and Robotics; Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/9774; https://ink.library.smu.edu.sg/context/sis_research/article/10774/viewcontent/Yu_Beyond_CVPR_2024_paper.pdf
• Institution: Singapore Management University

In this paper, we delve into a novel aspect of learning novel diffusion conditions with datasets an order of magnitude smaller. The rationale behind our approach is the elimination of textual constraints during the few-shot learning process. To that end, we implement two optimization strategies. The first, prompt-free conditional learning, utilizes a prompt-free encoder derived from a pre-trained Stable Diffusion model. This strategy is designed to adapt new conditions to the diffusion process by minimizing the textual-visual cor-relation, thereby ensuring a more precise alignment between the generated content and the specified conditions. The second strategy entails condition-specific negative rectification, which addresses the inconsistencies typically brought about by Classifier-free guidance in few-shot training con-texts. Our extensive experiments across a variety of condition modalities demonstrate the effectiveness and efficiency of our framework, yielding results comparable to those obtained with datasets a thousand times larger.