Feature learning and patch matching for diverse image inpainting

We present an image inpainting approach to generate diverse high-quality inpainting results. Recent advances in deep adversarial networks have led to significant improvements in the challenging task of filling large holes in natural images. Although deep generative models can generate visually plausible structures and textures, most of them are not interpretable, making it difficult to control the inpainting output. In addition, deep generative models do not have capacity to produce diverse results for each input. To address such limitations, we design a novel free-form image inpainting framework with two sequential steps: the first step formulates the inpainting process as a regression problem and utilizes a U-Net-like convolutional neural network to map an input to a coarse inpainting output, and the second step utilizes the nearest neighbor based pixel-wise matching to map the coarse output to diverse high-quality outputs. The second step allows our approach to compose novel high-quality content by copy-pasting high-frequency missing information from different training exemplars. Experiments on multiple datasets, i.e., CelebA-HQ, AFHQ, and Paris StreetView, show that our approach is able to offer multiple natural outputs with higher diversity in a controllable manner.