Sketch-based image synthesis with pre-trained text-to-image models

• Main Author: Ng, Samuel I-Shen
• Other Authors: Xingang Pan
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; Image generation
• Online Access: https://hdl.handle.net/10356/181525
• Institution: Nanyang Technological University

This paper presents the development of Inpainting with ControlNet - ComfyUI, a novel workflow designed to seamlessly integrate the capabilities of stable diffusion models with ControlNets in the ComfyUI platform. This approach enables users to generate edited images by providing a combination of an image, a mask, and a sketch, resulting in coherent and context-aware outputs that closely match the surrounding area. By leveraging the strengths of both stable diffusion models and ControlNets, our method provides a more efficient, effective, and user-friendly approach to image inpainting. The integration of ControlNets with inpainting models allows users to harness the power of text-to-image models while also providing additional guidance through image input, bridging the gap between user intent and the editing process. This solution has far-reaching implications, particularly in the context of image editing and manipulation. Our method has shown promising results, but there are still areas that require further investigation and improvement. Potential avenues for future research include exploring the feasibility of integrating models lacking ControlNets, investigating the benefits and limitations of using different models, and developing strategies for accurately specifying desired colours via a coloured sketch. The potential applications of this approach are vast, and further research and development could lead to even more innovative and powerful tools for image editing and manipulation.