Autonomous soundscape augmentation with multimodal fusion of visual and participant-linked inputs

• Main Authors: Ooi, Kenneth, Watcharasupat, Karn, Lam, Bhan, Ong, Zhen-Ting, Gan, Woon-Seng
• Other Authors: School of Electrical and Electronic Engineering
• Format: Conference or Workshop Item
• Language: English
• Published: 2023
• Subjects: Engineering::Electrical and electronic engineering; Social sciences::Psychology::Affection and emotion; Auditory Masking; Neural Attention; Multimodal Fusion; Probabilistic Loss; Deep Learning
• Online Access: https://hdl.handle.net/10356/165017
• Institution: Nanyang Technological University

Autonomous soundscape augmentation systems typically use trained models to pick optimal maskers to effect a desired perceptual change. While acoustic information is paramount to such systems, contextual information, including participant demographics and the visual environment, also influences acoustic perception. Hence, we propose modular modifications to an existing attention-based deep neural network, to allow early, mid-level, and late feature fusion of participant-linked, visual, and acoustic features. Ablation studies on module configurations and corresponding fusion methods using the ARAUS dataset show that contextual features improve the model performance in a statistically significant manner on the normalized ISO Pleasantness, to a mean squared error of 0.1194±0.0012 for the best-performing all-modality model, against 0.1217±0.0009 for the audio-only model. Soundscape augmentation systems can thereby leverage multimodal inputs for improved performance. We also investigate the impact of individual participant-linked factors using trained models to illustrate improvements in model explainability.