SDEdit: Image Synthesis and Editing with Stochastic Differential Equations

SDEdit: Image Synthesis and Editing with Stochastic Differential Equations


Recently, generative modeling with stochastic differential equations (SDEs) has demonstrated some advantages against generative adversarial networks (GANs). However, there is still a lack of real-world applications.

A recent paper proposes a unified approach to image editing and synthesis inspired by the previously-mentioned method.

Image editing. Picture credit: Max Pixel, CC0 Public Domain

Given an input image with user edits, such as a stroke painting, a suitable amount of noise is added to smooth out undesirable distortions. Then, reverse SDE is used to obtain a denoised result of high quality. The suggested framework enables applications as image compositing, stroke-based image synthesis, and stroke-based editing.

The method is particularly suitable for tasks where GAN inversion losses are hard to design or optimize. It is demonstrated that the novel method outperforms GAN baselines on stroke-based image synthesis and achieves competitive performance on other tasks.

We introduce a new image editing and synthesis framework, Stochastic Differential Editing (SDEdit), based on a recent generative model using stochastic differential equations (SDEs). Given an input image with user edits (e.g., hand-drawn color strokes), we first add noise to the input according to an SDE, and subsequently denoise it by simulating the reverse SDE to gradually increase its likelihood under the prior. Our method does not require task-specific loss function designs, which are critical components for recent image editing methods based on GAN inversion. Compared to conditional GANs, we do not need to collect new datasets of original and edited images for new applications. Therefore, our method can quickly adapt to various editing tasks at test time without re-training models. Our approach achieves strong performance on a wide range of applications, including image synthesis and editing guided by stroke paintings and image compositing.

Research paper: Meng, C., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S., “SDEdit: Image Synthesis and Editing with Stochastic Differential Equations”, 2021. Link:


Source link

Share This


Wordpress (0)
Disqus (0 )