Learning to match anchor-target video pairs with dual attentional holographic networks

Video hyperlinking is the task of linking two video fragments/clips based on their multi-modal contents. Specifically, given an anchor video as a query, machine techniques automatically generate links between the anchor and target videos by modeling and comparing their content aboutness. The term &q...

Full description

Saved in:
Bibliographic Details
Main Authors: HAO, Yan Bin, NGO, Chong-wah, ZHU, Bin
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2021
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/6821
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.sis_research-7824
record_format dspace
spelling sg-smu-ink.sis_research-78242022-01-27T03:48:03Z Learning to match anchor-target video pairs with dual attentional holographic networks HAO, Yan Bin NGO, Chong-wah ZHU, Bin Video hyperlinking is the task of linking two video fragments/clips based on their multi-modal contents. Specifically, given an anchor video as a query, machine techniques automatically generate links between the anchor and target videos by modeling and comparing their content aboutness. The term "aboutness" specifically refers to contextually relevant multimedia content, i.e., a fragment is on or of something. Since video contents are multi-modal (e.g., audio and vision), the content aboutness may be reflected across different modalities. Existing approaches regard hyperlinking as a retrieval task, by embedding multi-modal video contents into one or multiple common video representation space(s) for cross-modal comparison. As a result, the aboutness between videos is scored by computing the vector-distance based similarity in the learnt common feature space. However, these methods suffer from two main limitations: (1) the video modality descriptors/features are treated equally in representation learning, which hinders the effective modeling of their respective capabilities in linking; and (2) directly using the vector-distance based similarity to measure aboutness bears the risk of returning more duplicates. This paper focuses on addressing these two problems. Specifically, we firstly build attentional neural networks to learn a compact fragment-level representation, assigning different importance weights to different descriptor/feature contents by an attention mechanism. We believe that the potentially interesting content(s) should be highlighted in the representation. Furthermore, instead of directly computing the similarity of two representation embeddings, we secondly build a holographic composition network to model the aboutness for link establishment, with the core use of circular correlation. The two networks string together to form the final hyperlinking matching system. The entire model is trained in an end-to-end fashion. We examine its effectiveness by creating four train/validate/test partitioning schemes on the Blip10000 dataset and employing two video fragmentation methods. 2021-01-01T08:00:00Z text https://ink.library.smu.edu.sg/sis_research/6821 info:doi/10.1109/TIP.2021.3113165 Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Task analysis Neural networks Correlation Semantics Generative adversarial networks Feature extraction Benchmark testing Video hyperlinking hierarchical content attention holographic composition triplet loss Graphics and Human Computer Interfaces OS and Networks
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic Task analysis
Neural networks
Correlation
Semantics
Generative adversarial networks
Feature extraction
Benchmark testing
Video hyperlinking
hierarchical content attention
holographic composition
triplet loss
Graphics and Human Computer Interfaces
OS and Networks
spellingShingle Task analysis
Neural networks
Correlation
Semantics
Generative adversarial networks
Feature extraction
Benchmark testing
Video hyperlinking
hierarchical content attention
holographic composition
triplet loss
Graphics and Human Computer Interfaces
OS and Networks
HAO, Yan Bin
NGO, Chong-wah
ZHU, Bin
Learning to match anchor-target video pairs with dual attentional holographic networks
description Video hyperlinking is the task of linking two video fragments/clips based on their multi-modal contents. Specifically, given an anchor video as a query, machine techniques automatically generate links between the anchor and target videos by modeling and comparing their content aboutness. The term "aboutness" specifically refers to contextually relevant multimedia content, i.e., a fragment is on or of something. Since video contents are multi-modal (e.g., audio and vision), the content aboutness may be reflected across different modalities. Existing approaches regard hyperlinking as a retrieval task, by embedding multi-modal video contents into one or multiple common video representation space(s) for cross-modal comparison. As a result, the aboutness between videos is scored by computing the vector-distance based similarity in the learnt common feature space. However, these methods suffer from two main limitations: (1) the video modality descriptors/features are treated equally in representation learning, which hinders the effective modeling of their respective capabilities in linking; and (2) directly using the vector-distance based similarity to measure aboutness bears the risk of returning more duplicates. This paper focuses on addressing these two problems. Specifically, we firstly build attentional neural networks to learn a compact fragment-level representation, assigning different importance weights to different descriptor/feature contents by an attention mechanism. We believe that the potentially interesting content(s) should be highlighted in the representation. Furthermore, instead of directly computing the similarity of two representation embeddings, we secondly build a holographic composition network to model the aboutness for link establishment, with the core use of circular correlation. The two networks string together to form the final hyperlinking matching system. The entire model is trained in an end-to-end fashion. We examine its effectiveness by creating four train/validate/test partitioning schemes on the Blip10000 dataset and employing two video fragmentation methods.
format text
author HAO, Yan Bin
NGO, Chong-wah
ZHU, Bin
author_facet HAO, Yan Bin
NGO, Chong-wah
ZHU, Bin
author_sort HAO, Yan Bin
title Learning to match anchor-target video pairs with dual attentional holographic networks
title_short Learning to match anchor-target video pairs with dual attentional holographic networks
title_full Learning to match anchor-target video pairs with dual attentional holographic networks
title_fullStr Learning to match anchor-target video pairs with dual attentional holographic networks
title_full_unstemmed Learning to match anchor-target video pairs with dual attentional holographic networks
title_sort learning to match anchor-target video pairs with dual attentional holographic networks
publisher Institutional Knowledge at Singapore Management University
publishDate 2021
url https://ink.library.smu.edu.sg/sis_research/6821
_version_ 1770576075660722176