Compressed video action recognition with refined motion vectors

Although CNN has reached satisfactory performance in image-related tasks, using CNN to process videos is much more challenging due to the enormous size of raw video streams. In this Final Year Project, we propose to use motion vectors and residuals from modern video compression techniques to effecti...

Full description

Saved in:
Bibliographic Details
Main Author: Yu, Shining
Other Authors: Jiang Xudong
Format: Final Year Project
Language:English
Published: 2019
Subjects:
Online Access:http://hdl.handle.net/10356/78259
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
Description
Summary:Although CNN has reached satisfactory performance in image-related tasks, using CNN to process videos is much more challenging due to the enormous size of raw video streams. In this Final Year Project, we propose to use motion vectors and residuals from modern video compression techniques to effectively learn representation of the raw frames and greatly remove the temporal redundancy, giving faster video processing model. Compressed Video Action Recognition (CoViAR) has explored to directly use compressed video to train the deep neural network, where the motion vectors was utilized to present temporal information. However, motion vector is designed for minimizing video size where precious motion information are not obligatory. Compared with optical flow, motion vectors contain noisy and unreliable motion information. Inspired by the mechanism of video compression codecs, we propose an approach to refine the motion vectors where unreliable movement will be removed while temporal information is largely reserved. We prove that replacing the original motion vector with refined one and using the same network as CoViAR has achieved state-of-art performance on the UCF-101 and HMDB-51 with negligible efficiency degrades comparing with original CoViAR.