Interactive change-aware transformer network for remote sensing image change captioning

Remote sensing image change captioning (RSICC) aims to automatically generate sentences describing the difference in content in remote sensing bitemporal images. Recent works extract the changes between bitemporal features and employ a hierarchical approach to fuse multiple changes of interest, yiel...

Full description

Saved in:
Bibliographic Details
Main Authors: Cai, Chen, Wang, Yi, Yap, Kim-Hui
Other Authors: School of Electrical and Electronic Engineering
Format: Article
Language:English
Published: 2024
Subjects:
Online Access:https://hdl.handle.net/10356/172986
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-172986
record_format dspace
spelling sg-ntu-dr.10356-1729862024-01-12T15:41:57Z Interactive change-aware transformer network for remote sensing image change captioning Cai, Chen Wang, Yi Yap, Kim-Hui School of Electrical and Electronic Engineering Engineering::Electrical and electronic engineering Image Change Captioning Remote Sensing Remote sensing image change captioning (RSICC) aims to automatically generate sentences describing the difference in content in remote sensing bitemporal images. Recent works extract the changes between bitemporal features and employ a hierarchical approach to fuse multiple changes of interest, yielding change captions. However, these methods directly aggregate all features, potentially incorporating non-change-focused information from each encoder layer into the change caption decoder, adversely affecting the performance of change captioning. To address this problem, we proposed an Interactive Change-Aware Transformer Network (ICT-Net). ICT-Net is able to extract and incorporate the most critical changes of interest in each encoder layer to improve change description generation. It initially extracts bitemporal visual features from the CNN backbone and employs an Interactive Change-Aware Encoder (ICE) to capture the crucial difference between these features. Specifically, the ICE captures the most change-aware discriminative information between the paired bitemporal features interactively through difference and content attention encoding. A Multi-Layer Adaptive Fusion (MAF) module is proposed to adaptively aggregate the relevant change-aware features in the ICE layers while minimizing the impact of irrelevant visual features. Moreover, we extend the ICE to extract multi-scale changes and introduce a novel Cross Gated-Attention (CGA) module into the change caption decoder to select essential discriminative multi-scale features to improve the change captioning performance. We evaluate our method on two RSICC datasets (e.g., LEVIR-CC and LEVIRCCD), and the experimental results demonstrate that our method achieves a state-of-the-art performance. Published version This research was funded by The Hong Kong Polytechnic University (PolyU) Start-up Fund for RAPs under the Strategic Hiring Scheme (P0047884). 2024-01-08T02:11:48Z 2024-01-08T02:11:48Z 2023 Journal Article Cai, C., Wang, Y. & Yap, K. (2023). Interactive change-aware transformer network for remote sensing image change captioning. Remote Sensing, 15(23), 5611-. https://dx.doi.org/10.3390/rs15235611 2072-4292 https://hdl.handle.net/10356/172986 10.3390/rs15235611 2-s2.0-85179131457 23 15 5611 en Remote Sensing © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/). application/pdf
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Engineering::Electrical and electronic engineering
Image Change Captioning
Remote Sensing
spellingShingle Engineering::Electrical and electronic engineering
Image Change Captioning
Remote Sensing
Cai, Chen
Wang, Yi
Yap, Kim-Hui
Interactive change-aware transformer network for remote sensing image change captioning
description Remote sensing image change captioning (RSICC) aims to automatically generate sentences describing the difference in content in remote sensing bitemporal images. Recent works extract the changes between bitemporal features and employ a hierarchical approach to fuse multiple changes of interest, yielding change captions. However, these methods directly aggregate all features, potentially incorporating non-change-focused information from each encoder layer into the change caption decoder, adversely affecting the performance of change captioning. To address this problem, we proposed an Interactive Change-Aware Transformer Network (ICT-Net). ICT-Net is able to extract and incorporate the most critical changes of interest in each encoder layer to improve change description generation. It initially extracts bitemporal visual features from the CNN backbone and employs an Interactive Change-Aware Encoder (ICE) to capture the crucial difference between these features. Specifically, the ICE captures the most change-aware discriminative information between the paired bitemporal features interactively through difference and content attention encoding. A Multi-Layer Adaptive Fusion (MAF) module is proposed to adaptively aggregate the relevant change-aware features in the ICE layers while minimizing the impact of irrelevant visual features. Moreover, we extend the ICE to extract multi-scale changes and introduce a novel Cross Gated-Attention (CGA) module into the change caption decoder to select essential discriminative multi-scale features to improve the change captioning performance. We evaluate our method on two RSICC datasets (e.g., LEVIR-CC and LEVIRCCD), and the experimental results demonstrate that our method achieves a state-of-the-art performance.
author2 School of Electrical and Electronic Engineering
author_facet School of Electrical and Electronic Engineering
Cai, Chen
Wang, Yi
Yap, Kim-Hui
format Article
author Cai, Chen
Wang, Yi
Yap, Kim-Hui
author_sort Cai, Chen
title Interactive change-aware transformer network for remote sensing image change captioning
title_short Interactive change-aware transformer network for remote sensing image change captioning
title_full Interactive change-aware transformer network for remote sensing image change captioning
title_fullStr Interactive change-aware transformer network for remote sensing image change captioning
title_full_unstemmed Interactive change-aware transformer network for remote sensing image change captioning
title_sort interactive change-aware transformer network for remote sensing image change captioning
publishDate 2024
url https://hdl.handle.net/10356/172986
_version_ 1789483182291156992