SoftSkip: Empowering multi-modal dynamic pruning for single-stage referring comprehension

• Main Authors: WEERAKOON, Dulanga, SUBBARAJU, Vigneshwaran, TRAN, Tuan, MISRA, Archan
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2022
• Subjects: Human-Robot Interaction; Referring Expression Comprehension; Pruning; Computer Vision; Natural Language Processing; Computer Engineering; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/7707; https://ink.library.smu.edu.sg/context/sis_research/article/8710/viewcontent/multimedia_final.pdf
• Institution: Singapore Management University

Supporting real-time referring expression comprehension (REC) on pervasive devices is an important capability for human-AI collaborative tasks. Model pruning techniques, applied to DNN models, can enable real-time execution even on resource-constrained devices. However, existing pruning strategies are designed principally for uni-modal applications, and suffer a significant loss of accuracy when applied to REC tasks that require fusion of textual and visual inputs. We thus present a multi-modal pruning model, LGMDP, which uses language as a pivot to dynamically and judiciously select the relevant computational blocks that need to be executed. LGMDP also introduces a new SoftSkip mechanism, whereby 'skipped' visual scales are not completely eliminated but approximated with minimal additional computation. Experimental evaluation, using 3 benchmark REC datasets and an embedded device implementation, shows that LGMDP can achieve 33% latency savings, with an accuracy loss 0.5% - 2%.