DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation

Citations Over TimeTop 1% of 2022 papers

Abstract

Recently, GAN inversion methods combined with Contrastive Language-Image Pretraining (CLIP) enables zeroshot image manipulation guided by text prompts. However, their applications to diverse real images are still difficult due to the limited GAN inversion capability. Specifically, these approaches often have difficulties in reconstructing images with novel poses, views, and highly variable contents compared to the training data, altering object identity, or producing unwanted image artifacts. To mitigate these problems and enable faithful manipulation of real images, we propose a novel method, dubbed DiffusionCLIP, that performs textdriven image manipulation using diffusion models. Based on full inversion capability and high-quality image generation power of recent diffusion models, our method performs zeroshot image manipulation successfully even between unseen domains and takes another step towards general application by manipulating images from a widely varying ImageNet dataset. Furthermore, we propose a novel noise combination method that allows straightforward multi-attribute manipulation. Extensive experiments and human evaluation confirmed robust and superior manipulation performance of our methods compared to the existing baselines. Code is available at https://github.com/gwang-kim/DiffusionCLIP.git

Related Papers

• → Progressive Image Enhancement under Aesthetic Guidance(2019)11 cited
• → Towards demystifying dimensions of source code embeddings(2020)3 cited
• → Embedding of QR code image for arbitrary image by CNN(2019)2 cited
• From cages to wedges and clefts : Design of some novel hosts based on Image , Image , Image -triquinane framework(1989)
• → A Research on Performance Improvement of QR Code Embedding Method into Any Image Using Deep Learning(2022)