SALSA: spatial cue-augmented log-spectrogram features for polyphonic sound event localization and detection

• Main Authors: Nguyen, Thi Ngoc Tho, Watcharasupat, Karn N., Nguyen, Ngoc Khanh, Jones, Douglas L., Gan, Woon-Seng
• Other Authors: School of Electrical and Electronic Engineering
• Format: Article
• Language: English
• Published: 2022
• Subjects: Engineering::Electrical and electronic engineering; Deep Learning; Microphone Array; Feature Extraction; Sound Event Localization and Detection; Spatial Cues
• Online Access: https://hdl.handle.net/10356/157118
• Institution: Nanyang Technological University

Sound event localization and detection (SELD) consists of two subtasks, which are sound event detection and direction-of-arrival estimation. While sound event detection mainly relies on time-frequency patterns to distinguish different sound classes, direction-of-arrival estimation uses amplitude and/or phase differences between microphones to estimate source directions. As a result, it is often difficult to jointly optimize these two subtasks. We propose a novel feature called Spatial cue-Augmented Log-SpectrogrAm (SALSA) with exact time-frequency mapping between the signal power and the source directional cues, which is crucial for resolving overlapping sound sources. The SALSA feature consists of multichannel log-spectrograms stacked along with the normalized principal eigenvector of the spatial covariance matrix at each corresponding time-frequency bin. Depending on the microphone array format, the principal eigenvector can be normalized differently to extract amplitude and/or phase differences between the microphones. As a result, SALSA features are applicable for different microphone array formats such as first-order ambisonics (FOA) and multichannel microphone array (MIC). Experimental results on the TAUNIGENS Spatial Sound Events 2021 dataset with directional interferences showed that SALSA features outperformed other state-of-the-art features. Specifically, the use of SALSA features in the FOA format increased the F1 score and localization recall by 6 % each, compared to the multichannel log-mel spectrograms with intensity vectors. For the MIC format, using SALSA features increased F1 score and localization recall by 16 % and 7 %, respectively, compared to using multichannel logmel spectrograms with generalized cross-correlation spectra.