Action-centric relation transformer network for video question answering

• Main Authors: ZHANG, Jipeng, SHAO, Jie, CAO, Rui, GAO, Lianli, XU, Xing, SHEN, Heng Tao
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2022
• Subjects: Knowledge discovery; multi-modal reasoning; Proposals; relation reasoning; Task analysis; temporal action detection; Video question answering; video representation; Visualization; Cognition; Encoding; Feature extraction; Broadcast and Video Studies; Databases and Information Systems; Numerical Analysis and Scientific Computing
• Online Access: https://ink.library.smu.edu.sg/sis_research/6020; https://ink.library.smu.edu.sg/context/sis_research/article/7023/viewcontent/Action_Centric_Relation_Video_Question_Answering_av.pdf
• Institution: Singapore Management University

Video question answering (VideoQA) has emerged as a popular research topic in recent years. Enormous efforts have been devoted to developing more effective fusion strategies and better intra-modal feature preparation. To explore these issues further, we identify two key problems. (1) Current works take almost no account of introducing action of interest in video representation. Additionally, there exists insufficient labeling data on where the action of interest is in many datasets. However, questions in VideoQA are usually action-centric. (2) Frame-to-frame relations, which can provide useful temporal attributes (e.g., state transition, action counting), lack relevant research. Based on these observations, we propose an action-centric relation transformer network (ACRTransformer) for VideoQA and make two significant improvements. (1) We explicitly consider the action recognition problem and present a visual feature encoding technique, action-based encoding (ABE), to emphasize the frames with high actionness probabilities (the probability that the frame has actions). (2) We better exploit the interplays between temporal frames using a relation transformer network (RTransformer). Experiments on popular benchmark datasets in VideoQA clearly establish our superiority over previous state-of-the-art models. Code could be found at https://github.com/op-multimodal/ACRTransformer.