Interactive video search with multi-modal LLM video captioning

• Main Authors: CHENG, Yu-Tong, WU, Jiaxin, MA, Zhixin, HE, Jiangshan, WEI, Xiao-Yong, NGO, Chong-wah
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2025
• Subjects: Interactive Video Retrieval; Multi-modal LLM; Video Captioning; Artificial Intelligence and Robotics; Databases and Information Systems
• Online Access: https://ink.library.smu.edu.sg/sis_research/10105; https://ink.library.smu.edu.sg/context/sis_research/article/11105/viewcontent/InteractiveVideo_LLM_av.pdf
• Institution: Singapore Management University

Cross-modal representation learning is essential for interactive text-to-video search tasks. However, the representation learning is limited by the size and quality of video-caption pairs. To improve the search accuracy, we propose to enlarge the size of available video-caption pairs by leveraging multi-model LLM on video captioning. Specifically, we use LLM to generate video captions for a large video collection (i.e., WebVid dataset) and use the generated video-caption pairs to pre-train a text-to-video search model. Additionally, we use LLM to generate fine-grained captions for test video collections to enable text-to-caption retrieval. Furthermore, we build a semantic overview of the retrieved rank list based on the detailed captions in our interactive video retrieval system which act as hints for user to refine their query. Experimental results show that the generated captions are effective in improving the search accuracy of both AVS and T-KIS tasks on the TRECVid datasets.