Contextual-assisted scratched photo restoration

• Main Authors: CAI, Weiwei, ZHANG, Huaidong, XU, Xuemiao, HE, Shengfeng, ZHANG, Kun, QIN, Jing
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: Image restoration; Feature extraction; Deep learning; Electronic mail; Context modeling; Videos; Task analysis; Computational Engineering; Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/8377
• Institution: Singapore Management University

Printed photographs can be easily warped, wrinkled, and even deteriorated over time. Existing methods treat the restoration of scratches as a pure inpainting problem that neglects the underlying corrupted contextual knowledge. However, important underlying contents are hidden behind the scratches, which are essential hints for producing a semantically consistent result. Motivated by this insight, we explore how to harmonize the scratch-free features and noisy but essential scratch features to produce a visually consistent restoration. Specifically, in this paper, we propose an automatic retouching approach for scratched photographs with the aid of scratch/background context. We explicitly process scratch and background context in two stages. In the first stage, we mainly extract global scratch features, while the mask is introduced in the second stage to filter out and inpaint the scratches. Both contexts are carefully reciprocated for a faithful restoration. Particularly, we propose a Scratch Contextual Assisted Module (SCAM) to adaptively learn texture within the detected mask. This module utilizes the distance between the scratch mask-out feature and scratch encoder feature for modeling the pixel-wise correspondence, which determines the importance of the encoder feature within the scratch mask. Furthermore, to facilitate the evaluation of scratch restoration methods, we create two new scratched photo datasets which have 238 scratch/scratch-free photo pairs to promote the development in the scratch restoration field, namely Old Scratched Photo Dataset (OSPD) and Modern Scratched Photo Dataset (MSPD). Extensive experimental results on the proposed datasets demonstrate that our model outperforms existing methods. To extend the application, we also perform the proposed method on video samples and obtain visual-pleasing results. The code can be found at https://github.com/cwyyt/Contextual-assisted-Scratched-Photo-Restoration