Concept-oriented transformers for visual sentiment analysis

In the richly multimedia Web, detecting sentiment signals expressed in images would support multiple applications, e.g., measuring customer satisfaction from online reviews, analyzing trends and opinions from social media. Given an image, visual sentiment analysis aims at recognizing positive or neg...

Full description

Saved in:
Bibliographic Details
Main Authors: TRUONG, Quoc Tuan, LAUW, Hady Wirawan
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2023
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/7799
https://ink.library.smu.edu.sg/context/sis_research/article/8802/viewcontent/wsdm23.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.sis_research-8802
record_format dspace
spelling sg-smu-ink.sis_research-88022023-04-04T03:03:19Z Concept-oriented transformers for visual sentiment analysis TRUONG, Quoc Tuan LAUW, Hady Wirawan In the richly multimedia Web, detecting sentiment signals expressed in images would support multiple applications, e.g., measuring customer satisfaction from online reviews, analyzing trends and opinions from social media. Given an image, visual sentiment analysis aims at recognizing positive or negative sentiment, and occasionally neutral sentiment as well. A nascent yet promising direction is Transformer-based models applied to image data, whereby Vision Transformer (ViT) establishes remarkable performance on largescale vision benchmarks. In addition to investigating the fitness of ViT for visual sentiment analysis, we further incorporate concept orientation into the self-attention mechanism, which is the core component of Transformer. The proposed model captures the relationships between image features and specific concepts. We conduct extensive experiments on Visual Sentiment Ontology (VSO) and Yelp.com online review datasets, showing that not only does the proposed model significantly improve upon the base model ViT in detecting visual sentiment but it also outperforms previous visual sentiment analysis models with narrowly-defined orientations. Additional analyses yield insightful results and better understanding of the concept-oriented self-attention mechanism. 2023-03-01T08:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/7799 info:doi/10.1145/3539597.3570437 https://ink.library.smu.edu.sg/context/sis_research/article/8802/viewcontent/wsdm23.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University visual sentiment analysis concept orientation transformers Databases and Information Systems Graphics and Human Computer Interfaces
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic visual sentiment analysis
concept orientation
transformers
Databases and Information Systems
Graphics and Human Computer Interfaces
spellingShingle visual sentiment analysis
concept orientation
transformers
Databases and Information Systems
Graphics and Human Computer Interfaces
TRUONG, Quoc Tuan
LAUW, Hady Wirawan
Concept-oriented transformers for visual sentiment analysis
description In the richly multimedia Web, detecting sentiment signals expressed in images would support multiple applications, e.g., measuring customer satisfaction from online reviews, analyzing trends and opinions from social media. Given an image, visual sentiment analysis aims at recognizing positive or negative sentiment, and occasionally neutral sentiment as well. A nascent yet promising direction is Transformer-based models applied to image data, whereby Vision Transformer (ViT) establishes remarkable performance on largescale vision benchmarks. In addition to investigating the fitness of ViT for visual sentiment analysis, we further incorporate concept orientation into the self-attention mechanism, which is the core component of Transformer. The proposed model captures the relationships between image features and specific concepts. We conduct extensive experiments on Visual Sentiment Ontology (VSO) and Yelp.com online review datasets, showing that not only does the proposed model significantly improve upon the base model ViT in detecting visual sentiment but it also outperforms previous visual sentiment analysis models with narrowly-defined orientations. Additional analyses yield insightful results and better understanding of the concept-oriented self-attention mechanism.
format text
author TRUONG, Quoc Tuan
LAUW, Hady Wirawan
author_facet TRUONG, Quoc Tuan
LAUW, Hady Wirawan
author_sort TRUONG, Quoc Tuan
title Concept-oriented transformers for visual sentiment analysis
title_short Concept-oriented transformers for visual sentiment analysis
title_full Concept-oriented transformers for visual sentiment analysis
title_fullStr Concept-oriented transformers for visual sentiment analysis
title_full_unstemmed Concept-oriented transformers for visual sentiment analysis
title_sort concept-oriented transformers for visual sentiment analysis
publisher Institutional Knowledge at Singapore Management University
publishDate 2023
url https://ink.library.smu.edu.sg/sis_research/7799
https://ink.library.smu.edu.sg/context/sis_research/article/8802/viewcontent/wsdm23.pdf
_version_ 1770576515906404352