Generating expensive relationship features from cheap objects

We investigate the problem of object relationship classification of visual scenes. For a relationship object1-predicate-object2 that captures the object interaction, its representation is composed by the combination of object1 and object2 features. As a result, relationship classification models usu...

Full description

Saved in:
Bibliographic Details
Main Authors: WANG, Xiaogang, SUN, Qianru, CHUA, Tat-Seng, ANG, Marcelo
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2019
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/4446
https://ink.library.smu.edu.sg/context/sis_research/article/5449/viewcontent/BMVC2019_0657_paper_published.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.sis_research-5449
record_format dspace
spelling sg-smu-ink.sis_research-54492021-02-19T04:40:22Z Generating expensive relationship features from cheap objects WANG, Xiaogang SUN, Qianru CHUA, Tat-Seng ANG, Marcelo We investigate the problem of object relationship classification of visual scenes. For a relationship object1-predicate-object2 that captures the object interaction, its representation is composed by the combination of object1 and object2 features. As a result, relationship classification models usually bias to the frequent objects, leading to poor generalization to rare or unseen objects. Inspired by the data augmentation methods, we propose a novel Semantic Transform Generative Adversarial Network (ST-GAN) that synthesizes relationship features for rare objects, conditioned on the features from random instances of the objects. Specifically, ST-GAN essentially offers a semantic transform function from cheap object features to expensive relationship features. Here, “cheap” means any easy-to-collect object which possesses an original but undesired relationship attribute, e.g., a sitting person; “expensive” means a target relationship on this object, e.g., person-riding-horse. By generating massive triplet combinations from any object pair with larger variance, ST-GAN can reduce the data bias. Extensive experiments on two benchmarks – Visual Relationship Detection (VRD) and Visual Genome (VG), show that using our synthesized features for data augmentation, the relationship classification model can be consistently improved in various settings such as zero-shot and low-shot. 2019-09-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/4446 https://ink.library.smu.edu.sg/context/sis_research/article/5449/viewcontent/BMVC2019_0657_paper_published.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Visual relationship object detection feature generation Artificial Intelligence and Robotics Numerical Analysis and Scientific Computing
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic Visual relationship
object detection
feature generation
Artificial Intelligence and Robotics
Numerical Analysis and Scientific Computing
spellingShingle Visual relationship
object detection
feature generation
Artificial Intelligence and Robotics
Numerical Analysis and Scientific Computing
WANG, Xiaogang
SUN, Qianru
CHUA, Tat-Seng
ANG, Marcelo
Generating expensive relationship features from cheap objects
description We investigate the problem of object relationship classification of visual scenes. For a relationship object1-predicate-object2 that captures the object interaction, its representation is composed by the combination of object1 and object2 features. As a result, relationship classification models usually bias to the frequent objects, leading to poor generalization to rare or unseen objects. Inspired by the data augmentation methods, we propose a novel Semantic Transform Generative Adversarial Network (ST-GAN) that synthesizes relationship features for rare objects, conditioned on the features from random instances of the objects. Specifically, ST-GAN essentially offers a semantic transform function from cheap object features to expensive relationship features. Here, “cheap” means any easy-to-collect object which possesses an original but undesired relationship attribute, e.g., a sitting person; “expensive” means a target relationship on this object, e.g., person-riding-horse. By generating massive triplet combinations from any object pair with larger variance, ST-GAN can reduce the data bias. Extensive experiments on two benchmarks – Visual Relationship Detection (VRD) and Visual Genome (VG), show that using our synthesized features for data augmentation, the relationship classification model can be consistently improved in various settings such as zero-shot and low-shot.
format text
author WANG, Xiaogang
SUN, Qianru
CHUA, Tat-Seng
ANG, Marcelo
author_facet WANG, Xiaogang
SUN, Qianru
CHUA, Tat-Seng
ANG, Marcelo
author_sort WANG, Xiaogang
title Generating expensive relationship features from cheap objects
title_short Generating expensive relationship features from cheap objects
title_full Generating expensive relationship features from cheap objects
title_fullStr Generating expensive relationship features from cheap objects
title_full_unstemmed Generating expensive relationship features from cheap objects
title_sort generating expensive relationship features from cheap objects
publisher Institutional Knowledge at Singapore Management University
publishDate 2019
url https://ink.library.smu.edu.sg/sis_research/4446
https://ink.library.smu.edu.sg/context/sis_research/article/5449/viewcontent/BMVC2019_0657_paper_published.pdf
_version_ 1770574840792612864