SQL-like interpretable interactive video search

• Main Authors: WU, Jiaxin, NGUYEN, Phuong Anh, MA, Zhixin, NGO, Chong-wah
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2021
• Subjects: Concept-based search; Concept-free search; Interactive video search; SQL-like interpretable search; Video browser showdown; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/6831; https://ink.library.smu.edu.sg/context/sis_research/article/7834/viewcontent/978_3_030_67835_7_34.pdf
• Institution: Singapore Management University

Concept-free search, which embeds text and video signals in a joint space for retrieval, appears to be a new state-of-the-art. However, this new search paradigm suffers from two limitations. First, the search result is unpredictable and not interpretable. Second, the embedded features are in high-dimensional space hindering real-time indexing and search. In this paper, we present a new implementation of the Vireo video search system (Vireo-VSS), which employs a dual-task model to index each video segment with an embedding feature in a low dimension and a concept list for retrieval. The concept list serves as a reference to interpret its associated embedded feature. With these changes, a SQL-like querying interface is designed such that a user can specify the search content (subject, predicate, object) and constraint (logical condition) in a semi-structured way. The system will decompose the SQL-like query into multiple sub-queries depending on the constraint being specified. Each sub-query is translated into an embedding feature and a concept list for video retrieval. The search result is compiled by union or pruning of the search lists from multiple sub-queries. The SQL-like interface is also extended for temporal querying, by providing multiple SQL templates for users to specify the temporal evolution of a query.