Posts

TL;DR A new video representation by (i) a canonical image that aggregates the static contents and (ii) a temporal deformation field that reconstructs the video frames when applied to the static image. Problem statements Video processing comes at a high cost,and naively processing frames results in poor cross-frame consistency. Method High level objective. The proposed representations should have the following characteristics: Fitting capability for faithful video reconstruction. Semantic correctness of the canonical image to ensure the performance of image processing algorithms....

TL;DR This work enables interactive editing of a GAN’s generated image by translating (“dragging”) any point in the image to a target location. Problem statements GAN based image generation takes a noise vector to generate an image. There is a need of a localized controlled image manipulation as moving a region to a different location in the image. Method Given a GAN generated image, a user input of the source coordinates (q) and the coordinates of the destination (p)...

title: “[Summary] MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation” date: 2023-05-19 tags: - Diffusion Models - Image Editing - Controllability TL;DR To enable a more controllable image diffusion, MultiDiffusion introduce patches generation with a global constrain. Problem statements Diffusion models lack user controllability and methods that offer such control require a costly fine-tuning. Method The method can be reduced to the following algorithm: At each time step t: Extract patches from the global image I_{t-1} Execute the de-noising step to generate the patches J_{i,t} Combine the patches by average their pixel values to create the global image I_t For the panorama use case: simply generate N images with overlapping regions between them....

title: “[Summary] Break-A-Scene: Extracting Multiple Concepts from a Single Image” date: 2023-07-21 tags: - Diffusion Models - Concept Extraction - Image Generation TL;DR Fine-tuning of a diffusion model using a single image to generate images conditions on user-provided concepts. Problem statements Diffusion models are not able to generate a new image of user-provided concepts. Methods (DreemBooth) that enable this capabilities require several input images that contain the desired concept. Method The method consists of two phases....