Interactive video corpus moment retrieval using reinforcement learning

Known-item video search is effective with human-in-the-loop to interactively investigate the search result and refine the initial query. Nevertheless, when the first few pages of results are swamped with visually similar items, or the search target is hidden deep in the ranked list, finding the know...

Full description

Saved in:
Bibliographic Details
Main Authors: MA, Zhixin, NGO, Chong-wah
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2022
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/7506
https://ink.library.smu.edu.sg/context/sis_research/article/8509/viewcontent/3503161.3548277.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.sis_research-8509
record_format dspace
spelling sg-smu-ink.sis_research-85092022-11-18T08:04:52Z Interactive video corpus moment retrieval using reinforcement learning MA, Zhixin NGO, Chong-wah Known-item video search is effective with human-in-the-loop to interactively investigate the search result and refine the initial query. Nevertheless, when the first few pages of results are swamped with visually similar items, or the search target is hidden deep in the ranked list, finding the know-item target usually requires a long duration of browsing and result inspection. This paper tackles the problem by reinforcement learning, aiming to reach a search target within a few rounds of interaction by long-term learning from user feedbacks. Specifically, the system interactively plans for navigation path based on feedback and recommends a potential target that maximizes the long-term reward for user comment. We conduct experiments for the challenging task of video corpus moment retrieval (VCMR) to localize moments from a large video corpus. The experimental results on TVR and DiDeMo datasets verify that our proposed work is effective in retrieving the moments that are hidden deep inside the ranked lists of CONQUER and HERO, which are the state-of-the-art auto-search engines for VCMR. 2022-10-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/7506 info:doi/10.1145/3503161.3548277 https://ink.library.smu.edu.sg/context/sis_research/article/8509/viewcontent/3503161.3548277.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Interactive search video corpus moment retrieval reinforcement learning user simulation Artificial Intelligence and Robotics Graphics and Human Computer Interfaces
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic Interactive search
video corpus moment retrieval
reinforcement learning
user simulation
Artificial Intelligence and Robotics
Graphics and Human Computer Interfaces
spellingShingle Interactive search
video corpus moment retrieval
reinforcement learning
user simulation
Artificial Intelligence and Robotics
Graphics and Human Computer Interfaces
MA, Zhixin
NGO, Chong-wah
Interactive video corpus moment retrieval using reinforcement learning
description Known-item video search is effective with human-in-the-loop to interactively investigate the search result and refine the initial query. Nevertheless, when the first few pages of results are swamped with visually similar items, or the search target is hidden deep in the ranked list, finding the know-item target usually requires a long duration of browsing and result inspection. This paper tackles the problem by reinforcement learning, aiming to reach a search target within a few rounds of interaction by long-term learning from user feedbacks. Specifically, the system interactively plans for navigation path based on feedback and recommends a potential target that maximizes the long-term reward for user comment. We conduct experiments for the challenging task of video corpus moment retrieval (VCMR) to localize moments from a large video corpus. The experimental results on TVR and DiDeMo datasets verify that our proposed work is effective in retrieving the moments that are hidden deep inside the ranked lists of CONQUER and HERO, which are the state-of-the-art auto-search engines for VCMR.
format text
author MA, Zhixin
NGO, Chong-wah
author_facet MA, Zhixin
NGO, Chong-wah
author_sort MA, Zhixin
title Interactive video corpus moment retrieval using reinforcement learning
title_short Interactive video corpus moment retrieval using reinforcement learning
title_full Interactive video corpus moment retrieval using reinforcement learning
title_fullStr Interactive video corpus moment retrieval using reinforcement learning
title_full_unstemmed Interactive video corpus moment retrieval using reinforcement learning
title_sort interactive video corpus moment retrieval using reinforcement learning
publisher Institutional Knowledge at Singapore Management University
publishDate 2022
url https://ink.library.smu.edu.sg/sis_research/7506
https://ink.library.smu.edu.sg/context/sis_research/article/8509/viewcontent/3503161.3548277.pdf
_version_ 1770576360097447936