Hierarchical damage correlations for old photo restoration

• Main Authors: CAI, Weiwei, XU, Xuemiao, XU, Jiajia, ZHANG, Huaidong, YANG, Haoxin, ZHANG, Kun, HE, Shengfeng
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2024
• Subjects: Image inpainting; Old photo restoration; Transformer; Graphics and Human Computer Interfaces; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/8730; https://ink.library.smu.edu.sg/context/sis_research/article/9733/viewcontent/HierarchicalDamageOldPhoto_av.pdf
• Institution: Singapore Management University

Restoring old photographs can preserve cherished memories. Previous methods handled diverse damages within the same network structure, which proved impractical. In addition, these methods cannot exploit correlations among artifacts, especially in scratches versus patch-misses issues. Hence, a tailored network is particularly crucial. In light of this, we propose a unified framework consisting of two key components: ScratchNet and PatchNet. In detail, ScratchNet employs the parallel Multi-scale Partial Convolution Module to effectively repair scratches, learning from multi-scale local receptive fields. In contrast, the patch-misses necessitate the network to emphasize global information. To this end, we incorporate a transformer-based encoder and decoder architecture. In the encoder phase, we introduce a Non-local Inpainting Attention Module, replacing the multi-head attention, to facilitate holistic context inpainting. In the decoder phase, the Mask-aware Instance Norm Module replaces the Layer Normalization, ensuring style consistency between foreground and background. Finally, the outcomes of ScratchNet are integrated into the PatchNet pipeline to supplement contextual information hierarchically. Mining damage correlations assists in training the network in an easy-to-hard manner. Extensive experiments demonstrate the superiority of our method over state-of-the-art approaches. The code is available at https://github.com/cwyyt/Hierarchical-Damage-Correlations-for-OldPhoto-Restoration.