Drag your noise: Interactive point-based editing via diffusion semantic propagation

Point-based interactive editing serves as an essential tool to complement the controllability of existing generative models. A concurrent work, DragDiffusion, updates the diffusion latent map in response to user inputs, causing global latent map alterations. This results in imprecise preservation of...

Full description

Saved in:
Bibliographic Details
Main Authors: LIU, Haofeng, XU, Chenshu, YANG, Yifei, ZENG, Lihua, HE, Shengfeng
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2024
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/9773
https://ink.library.smu.edu.sg/context/sis_research/article/10773/viewcontent/2404.01050v1.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.sis_research-10773
record_format dspace
spelling sg-smu-ink.sis_research-107732024-12-16T02:28:41Z Drag your noise: Interactive point-based editing via diffusion semantic propagation LIU, Haofeng XU, Chenshu YANG, Yifei ZENG, Lihua HE, Shengfeng Point-based interactive editing serves as an essential tool to complement the controllability of existing generative models. A concurrent work, DragDiffusion, updates the diffusion latent map in response to user inputs, causing global latent map alterations. This results in imprecise preservation of the original content and unsuccessful editing due to gradient vanishing. In contrast, we present DragNoise, offering robust and accelerated editing without retracing the latent map. The core rationale of DragNoise lies in utilizing the predicted noise output of each U-Net as a semantic editor. This approach is grounded in two critical observations: firstly, the bottleneck features of U-Net inherently possess semantically rich features ideal for interactive editing; secondly, highlevel semantics, established early in the denoising process, show minimal variation in subsequent stages. Leveraging these insights, DragNoise edits diffusion semantics in a single denoising step and efficiently propagates these changes, ensuring stability and efficiency in diffusion editing. Comparative experiments reveal that DragNoise achieves superior control and semantic retention, reducing the optimization time by over 50% compared to DragDiffusion. 2024-06-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/9773 https://ink.library.smu.edu.sg/context/sis_research/article/10773/viewcontent/2404.01050v1.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Point-based interactive editing Diffusion latent map alterations Semantic editor Artificial Intelligence and Robotics Graphics and Human Computer Interfaces
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic Point-based interactive editing
Diffusion latent map alterations
Semantic editor
Artificial Intelligence and Robotics
Graphics and Human Computer Interfaces
spellingShingle Point-based interactive editing
Diffusion latent map alterations
Semantic editor
Artificial Intelligence and Robotics
Graphics and Human Computer Interfaces
LIU, Haofeng
XU, Chenshu
YANG, Yifei
ZENG, Lihua
HE, Shengfeng
Drag your noise: Interactive point-based editing via diffusion semantic propagation
description Point-based interactive editing serves as an essential tool to complement the controllability of existing generative models. A concurrent work, DragDiffusion, updates the diffusion latent map in response to user inputs, causing global latent map alterations. This results in imprecise preservation of the original content and unsuccessful editing due to gradient vanishing. In contrast, we present DragNoise, offering robust and accelerated editing without retracing the latent map. The core rationale of DragNoise lies in utilizing the predicted noise output of each U-Net as a semantic editor. This approach is grounded in two critical observations: firstly, the bottleneck features of U-Net inherently possess semantically rich features ideal for interactive editing; secondly, highlevel semantics, established early in the denoising process, show minimal variation in subsequent stages. Leveraging these insights, DragNoise edits diffusion semantics in a single denoising step and efficiently propagates these changes, ensuring stability and efficiency in diffusion editing. Comparative experiments reveal that DragNoise achieves superior control and semantic retention, reducing the optimization time by over 50% compared to DragDiffusion.
format text
author LIU, Haofeng
XU, Chenshu
YANG, Yifei
ZENG, Lihua
HE, Shengfeng
author_facet LIU, Haofeng
XU, Chenshu
YANG, Yifei
ZENG, Lihua
HE, Shengfeng
author_sort LIU, Haofeng
title Drag your noise: Interactive point-based editing via diffusion semantic propagation
title_short Drag your noise: Interactive point-based editing via diffusion semantic propagation
title_full Drag your noise: Interactive point-based editing via diffusion semantic propagation
title_fullStr Drag your noise: Interactive point-based editing via diffusion semantic propagation
title_full_unstemmed Drag your noise: Interactive point-based editing via diffusion semantic propagation
title_sort drag your noise: interactive point-based editing via diffusion semantic propagation
publisher Institutional Knowledge at Singapore Management University
publishDate 2024
url https://ink.library.smu.edu.sg/sis_research/9773
https://ink.library.smu.edu.sg/context/sis_research/article/10773/viewcontent/2404.01050v1.pdf
_version_ 1819113134654226432