Neural architecture search as sparse supernet

• Main Authors: WU, Y., LIU, A., HUANG, Zhiwu, ZHANG, S., VAN, Gool L.
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2021
• Subjects: OS and Networks; Systems Architecture
• Online Access: https://ink.library.smu.edu.sg/sis_research/6411; https://ink.library.smu.edu.sg/context/sis_research/article/7414/viewcontent/Neural_architecture_search_as_sparse_supernet.pdf
• Institution: Singapore Management University

This paper aims at enlarging the problem of Neural Architecture Search (NAS) from Single-Path and Multi-Path Search to automated Mixed-Path Search. In particular, we model the NAS problem as a sparse supernet using a new continuous architecture representation with a mixture of sparsity constraints. The sparse supernet enables us to automatically achieve sparsely-mixed paths upon a compact set of nodes. To optimize the proposed sparse supernet, we exploit a hierarchical accelerated proximal gradient algorithm within a bi-level optimization framework. Extensive experiments on Convolutional Neural Network and Recurrent Neural Network search demonstrate that the proposed method is capable of searching for compact, general and powerful neural architectures.