Pruning meta-trained networks for on-device adaptation

• Main Authors: GAO, Dawei, HE, Xiaoxi, ZHOU, Zimu, TONG, Yongxin, THIELE, Lothar
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2021
• Subjects: deep neural networks; meta learning; network pruning; OS and Networks; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/6702; https://ink.library.smu.edu.sg/context/sis_research/article/7705/viewcontent/cikm21_gao.pdf
• Institution: Singapore Management University

Adapting neural networks to unseen tasks with few training samples on resource-constrained devices benefits various Internet-of-Things applications. Such neural networks should learn the new tasks in few shots and be compact in size. Meta-learning enables few-shot learning, yet the meta-trained networks can be overparameterised. However, naive combination of standard compression techniques like network pruning with meta-learning jeopardises the ability for fast adaptation. In this work, we propose adaptation-aware network pruning (ANP), a novel pruning scheme that works with existing meta-learning methods for a compact network capable of fast adaptation. ANP uses weight importance metric that is based on the sensitivity of the meta-objective rather than the conventional loss function, and adopts approximation of derivatives and layer-wise pruning techniques to reduce the overhead of computing the new importance metric. Evaluations on few-shot classification benchmarks show that ANP can prune meta-trained convolutional and residual networks by 85% without affecting their fast adaptation.