The utilization of scene graphs for image manipulation represents a highly promising direction in the field of computer vision. However, generating high-quality images from scene graphs can be challenging due to the complexity of the scenes and the high diversity of the objects in datasets such as Visual Genome. To address these challenges, we present a novel Progressive Restoration framework for Scene Graph-based Image Manipulation (PRISM). As a part of PRISM, we've developed and extensively evaluated several novel approaches that individually improve the image manipulation capabilities of the system. Our end-to-end framework leverages image reconstruction through a progressive restoration process, providing additional context information that enables more precise image manipulation. We took advantage of the outer part of the masked to-be-manipulated area as they have a stronger correlation with the context of the scene, and in our end-to-end framework, we designed a progressive denoising process for image reconstruction that continuously decreases the size of the masked region in the image. Moreover, our multi-task architecture simultaneously reconstructs the entire image as well as selected image objects in detail, generating high-quality and detailed images. Our model outperforms the state-of-the-art methods in the semantic image manipulation task on the CLEVR and Visual Genome datasets. Our results demonstrate the potential of our approach for enhancing the quality and precision of scene graph-based image manipulation. Finally, we propose a new research avenue by showcasing the benefits of incorporating progressive generation into diffusion processes.
«
The utilization of scene graphs for image manipulation represents a highly promising direction in the field of computer vision. However, generating high-quality images from scene graphs can be challenging due to the complexity of the scenes and the high diversity of the objects in datasets such as Visual Genome. To address these challenges, we present a novel Progressive Restoration framework for Scene Graph-based Image Manipulation (PRISM). As a part of PRISM, we've developed and extensively ev...
»