Compressed video action recognition with refined motion vectors

• Main Author: Yu, Shining
• Other Authors: Jiang Xudong
• Format: Final Year Project
• Language: English
• Published: 2019
• Subjects: DRNTU::Engineering::Electrical and electronic engineering
• Online Access: http://hdl.handle.net/10356/78259
• Institution: Nanyang Technological University

Although CNN has reached satisfactory performance in image-related tasks, using CNN to process videos is much more challenging due to the enormous size of raw video streams. In this Final Year Project, we propose to use motion vectors and residuals from modern video compression techniques to effectively learn representation of the raw frames and greatly remove the temporal redundancy, giving faster video processing model. Compressed Video Action Recognition (CoViAR) has explored to directly use compressed video to train the deep neural network, where the motion vectors was utilized to present temporal information. However, motion vector is designed for minimizing video size where precious motion information are not obligatory. Compared with optical flow, motion vectors contain noisy and unreliable motion information. Inspired by the mechanism of video compression codecs, we propose an approach to refine the motion vectors where unreliable movement will be removed while temporal information is largely reserved. We prove that replacing the original motion vector with refined one and using the same network as CoViAR has achieved state-of-art performance on the UCF-101 and HMDB-51 with negligible efficiency degrades comparing with original CoViAR.