Work-in-progress: what to expect of early training statistics? An investigation on hardware-aware neural architecture search

• Main Authors: Luo, Xiangzhong, Liu, Di, Kong, Hao, Huai, Shuo, Chen, Hui, Liu, Weichen
• Other Authors: School of Computer Science and Engineering
• Format: Conference or Workshop Item
• Language: English
• Published: 2023
• Subjects: Engineering::Computer science and engineering; Neural Architecture Search; Early Training Statistics
• Online Access: https://hdl.handle.net/10356/165389
• Institution: Nanyang Technological University

Neural architecture search (NAS) is an emerging paradigm to automate the design of top-performing deep neural networks (DNNs). Specifically, the increasing success of NAS is attributed to the reliable performance estimation of different architectures. Despite significant progress to date, previous relevant methods suffer from prohibitive computational overheads. To avoid this, we propose an effective yet computationally efficient proxy, namely Trained Batchwise Estimation (TBE), to reliably estimate the performance of different architectures using the early batchwise training statistics. We then integrate TBE into the hardware-aware NAS scenario to search for hardware-efficient architecture solutions. Experimental results clearly show the superiority of TBE over previous relevant state-of-the-art approaches.