Query-document-dependent fusion: A case study of multimodal music retrieval

In recent years, multimodal fusion has emerged as a promising technology for effective multimedia retrieval. Developing the optimal fusion strategy for different modality (e.g. content, metadata) has been the subject of intensive research. Given a query, existing methods derive a unified fusion stra...

Full description

Saved in:
Bibliographic Details
Main Authors: LI, Zhonghua, ZHANG, Bingjun, YU, Yi, SHEN, Jialie, WANG, Ye
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2013
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/1822
https://ink.library.smu.edu.sg/context/sis_research/article/2821/viewcontent/Query_Document_Dependent_Fusion_2013_afv.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.sis_research-2821
record_format dspace
spelling sg-smu-ink.sis_research-28212020-03-25T08:48:23Z Query-document-dependent fusion: A case study of multimodal music retrieval LI, Zhonghua ZHANG, Bingjun YU, Yi SHEN, Jialie WANG, Ye In recent years, multimodal fusion has emerged as a promising technology for effective multimedia retrieval. Developing the optimal fusion strategy for different modality (e.g. content, metadata) has been the subject of intensive research. Given a query, existing methods derive a unified fusion strategy for all documents with the underlying assumption that the relative significance of a modality remains the same across all documents. However, this assumption is often invalid. We thus propose a general multimodal fusion framework, query-document-dependent fusion (QDDF), which derives the optimal fusion strategy for each query-document pair via intelligent content analysis of both queries and documents. By investigating multimodal fusion strategies adaptive to both queries and documents, we demonstrate that existing multimodal fusion approaches are special cases of QDDF and propose two QDDF approaches to derive fusion strategies. The dual-phase QDDF explicitly derives and fuses query- and document-dependent weights, and the regression-based QDDF determines the fusion weight for a query-document pair via a regression model derived from training data. To evaluate the proposed approaches, comprehensive experiments have been conducted using a multimedia data set with around 17K full songs and over 236K social queries. Results indicate that the regression-based QDDF is superior in handling single-dimension queries. In comparison, the dual-phase QDDF outperforms existing approaches for most query types. We found that document-dependent weights are instrumental in enhancing multimedia fusion performance. In addition, efficiency analysis demonstrates the scalability of QDDF over large data sets. 2013-12-01T08:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/1822 info:doi/10.1109/TMM.2013.2280437 https://ink.library.smu.edu.sg/context/sis_research/article/2821/viewcontent/Query_Document_Dependent_Fusion_2013_afv.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Information retrieval multimodal query-document-dependent fusion Databases and Information Systems Music
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic Information retrieval
multimodal
query-document-dependent fusion
Databases and Information Systems
Music
spellingShingle Information retrieval
multimodal
query-document-dependent fusion
Databases and Information Systems
Music
LI, Zhonghua
ZHANG, Bingjun
YU, Yi
SHEN, Jialie
WANG, Ye
Query-document-dependent fusion: A case study of multimodal music retrieval
description In recent years, multimodal fusion has emerged as a promising technology for effective multimedia retrieval. Developing the optimal fusion strategy for different modality (e.g. content, metadata) has been the subject of intensive research. Given a query, existing methods derive a unified fusion strategy for all documents with the underlying assumption that the relative significance of a modality remains the same across all documents. However, this assumption is often invalid. We thus propose a general multimodal fusion framework, query-document-dependent fusion (QDDF), which derives the optimal fusion strategy for each query-document pair via intelligent content analysis of both queries and documents. By investigating multimodal fusion strategies adaptive to both queries and documents, we demonstrate that existing multimodal fusion approaches are special cases of QDDF and propose two QDDF approaches to derive fusion strategies. The dual-phase QDDF explicitly derives and fuses query- and document-dependent weights, and the regression-based QDDF determines the fusion weight for a query-document pair via a regression model derived from training data. To evaluate the proposed approaches, comprehensive experiments have been conducted using a multimedia data set with around 17K full songs and over 236K social queries. Results indicate that the regression-based QDDF is superior in handling single-dimension queries. In comparison, the dual-phase QDDF outperforms existing approaches for most query types. We found that document-dependent weights are instrumental in enhancing multimedia fusion performance. In addition, efficiency analysis demonstrates the scalability of QDDF over large data sets.
format text
author LI, Zhonghua
ZHANG, Bingjun
YU, Yi
SHEN, Jialie
WANG, Ye
author_facet LI, Zhonghua
ZHANG, Bingjun
YU, Yi
SHEN, Jialie
WANG, Ye
author_sort LI, Zhonghua
title Query-document-dependent fusion: A case study of multimodal music retrieval
title_short Query-document-dependent fusion: A case study of multimodal music retrieval
title_full Query-document-dependent fusion: A case study of multimodal music retrieval
title_fullStr Query-document-dependent fusion: A case study of multimodal music retrieval
title_full_unstemmed Query-document-dependent fusion: A case study of multimodal music retrieval
title_sort query-document-dependent fusion: a case study of multimodal music retrieval
publisher Institutional Knowledge at Singapore Management University
publishDate 2013
url https://ink.library.smu.edu.sg/sis_research/1822
https://ink.library.smu.edu.sg/context/sis_research/article/2821/viewcontent/Query_Document_Dependent_Fusion_2013_afv.pdf
_version_ 1770571597542850560