End-to-end varifocal multiview images coding framework from data acquisition end to vision application end

• Main Authors: Wu, Kejun, Liu, Qiong, Wang, Yi, Yang, You
• Other Authors: School of Electrical and Electronic Engineering
• Format: Article
• Language: English
• Published: 2023
• Subjects: Engineering::Electrical and electronic engineering; Microscopy; Camera
• Online Access: https://hdl.handle.net/10356/171474
• Institution: Nanyang Technological University

The emerging data, varifocal multiview (VFMV) has an exciting prospect in immersive multimedia. However, the distinctive data redundancy of VFMV derived from dense arrangements and blurriness differences among views causes difficulty in data compression. In this paper, we propose an end-to-end coding scheme for VFMV images, which provides a new paradigm for VFMV compression from data acquisition (source) end to vision application end. VFMV acquisition is first conducted in three ways at the source end, including conventional imaging, plenoptic refocusing, and 3D creation. The acquired VFMV has irregular focusing distributions due to varying focal planes, which decreases the similarity among adjacent views. To improve the similarity and the consequent coding efficiency, we rearrange the irregular focusing distributions in descending order and accordingly reorder the horizontal views. Then, the reordered VFMV images are scanned and concatenated as video sequences. We propose 4-directional prediction (4DP) to compress the reordered VFMV video sequences. Four most similar adjacent views from the left, upper left, upper and upper right directions serve as reference frames to improve the prediction efficiency. Finally, the compressed VFMV is transmitted and decoded at the application end, benefiting potential vision applications. Extensive experiments demonstrate that the proposed coding scheme is superior to the comparison scheme in objective quality, subjective quality and computational complexity. Experiments on new view synthesis show that VFMV can achieve extended depth of field than conventional multiview at the application end. Validation experiments show the effectiveness of view reordering, the advantage over typical MV-HEVC, and the flexibility on other data types, respectively.