Designing efficient DNNs via hardware-aware neural architecture search and beyond

Hardware systems integrated with deep neural networks (DNNs) are deemed to pave the way for future artificial intelligence (AI). However, manually designing efficient DNNs involves non-trivial computation resources since significant trial-and-errors are required to finalize the network configuration...

Full description

Saved in:
Bibliographic Details
Main Authors: Luo, Xiangzhong, Liu, Di, Huai, Shuo, Kong, Hao, Chen, Hui, Liu, Weichen
Other Authors: School of Computer Science and Engineering
Format: Article
Language:English
Published: 2022
Subjects:
Online Access:https://hdl.handle.net/10356/155785
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-155785
record_format dspace
spelling sg-ntu-dr.10356-1557852023-12-15T01:27:33Z Designing efficient DNNs via hardware-aware neural architecture search and beyond Luo, Xiangzhong Liu, Di Huai, Shuo Kong, Hao Chen, Hui Liu, Weichen School of Computer Science and Engineering HP-NTU Digital Manufacturing Corporate Lab Engineering::Computer science and engineering Deep Neural Networks Neural Architecture Search Hardware systems integrated with deep neural networks (DNNs) are deemed to pave the way for future artificial intelligence (AI). However, manually designing efficient DNNs involves non-trivial computation resources since significant trial-and-errors are required to finalize the network configuration. To this end, we, in this paper, introduce a novel hardware-aware neural architecture search (NAS) framework, namely GoldenNAS, to automate the design of efficient DNNs. To begin with, we present a novel technique, called dynamic channel scaling, to enable the channel-level search since the number of channels has non-negligible impacts on both accuracy and efficiency. Besides, we introduce an efficient progressive space shrinking method to raise the awareness of the search space towards target hardware and alleviate the search overheads as well. Moreover, we propose an effective hardware performance modeling method to approximate the runtime latency of DNNs upon target hardware, which is further integrated into GoldenNAS to avoid the tedious on-device measurements. Then, we employ the evolutionary algorithm (EA) to search for the optimal operator/channel configurations of DNNs, denoted as GoldenNets. Finally, to enable the depthwise adaptiveness of GoldenNets under dynamic environments, we propose the adaptive batch normalization (ABN) technique, followed by the self-knowledge distillation (SKD) approach to improve the accuracy of adaptive sub-networks. We conduct extensive experiments directly on ImageNet, which clearly demonstrate the advantages of GoldenNAS over existing state-of-the-art approaches. Ministry of Education (MOE) Nanyang Technological University Submitted/Accepted version This work is partially supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (MOE2019-T2-1-071) and Tier 1 (MOE2019-T1-001-072), and partially supported by Nanyang Technological University, Singapore, under its NAP (M4082282) and SUG (M4082087). 2022-03-22T00:48:53Z 2022-03-22T00:48:53Z 2021 Journal Article Luo, X., Liu, D., Huai, S., Kong, H., Chen, H. & Liu, W. (2021). Designing efficient DNNs via hardware-aware neural architecture search and beyond. IEEE Transactions On Computer-Aided Design of Integrated Circuits and Systems. https://dx.doi.org/10.1109/TCAD.2021.3100249 0278-0070 https://hdl.handle.net/10356/155785 10.1109/TCAD.2021.3100249 en MOE2019-T2-1-071 MOE2019-T1-001-072 M4082282 M4082087 IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 10.21979/N9/GZJ0PW © 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. The published version is available at: https://doi.org/10.1109/TCAD.2021.3100249. application/pdf
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Engineering::Computer science and engineering
Deep Neural Networks
Neural Architecture Search
spellingShingle Engineering::Computer science and engineering
Deep Neural Networks
Neural Architecture Search
Luo, Xiangzhong
Liu, Di
Huai, Shuo
Kong, Hao
Chen, Hui
Liu, Weichen
Designing efficient DNNs via hardware-aware neural architecture search and beyond
description Hardware systems integrated with deep neural networks (DNNs) are deemed to pave the way for future artificial intelligence (AI). However, manually designing efficient DNNs involves non-trivial computation resources since significant trial-and-errors are required to finalize the network configuration. To this end, we, in this paper, introduce a novel hardware-aware neural architecture search (NAS) framework, namely GoldenNAS, to automate the design of efficient DNNs. To begin with, we present a novel technique, called dynamic channel scaling, to enable the channel-level search since the number of channels has non-negligible impacts on both accuracy and efficiency. Besides, we introduce an efficient progressive space shrinking method to raise the awareness of the search space towards target hardware and alleviate the search overheads as well. Moreover, we propose an effective hardware performance modeling method to approximate the runtime latency of DNNs upon target hardware, which is further integrated into GoldenNAS to avoid the tedious on-device measurements. Then, we employ the evolutionary algorithm (EA) to search for the optimal operator/channel configurations of DNNs, denoted as GoldenNets. Finally, to enable the depthwise adaptiveness of GoldenNets under dynamic environments, we propose the adaptive batch normalization (ABN) technique, followed by the self-knowledge distillation (SKD) approach to improve the accuracy of adaptive sub-networks. We conduct extensive experiments directly on ImageNet, which clearly demonstrate the advantages of GoldenNAS over existing state-of-the-art approaches.
author2 School of Computer Science and Engineering
author_facet School of Computer Science and Engineering
Luo, Xiangzhong
Liu, Di
Huai, Shuo
Kong, Hao
Chen, Hui
Liu, Weichen
format Article
author Luo, Xiangzhong
Liu, Di
Huai, Shuo
Kong, Hao
Chen, Hui
Liu, Weichen
author_sort Luo, Xiangzhong
title Designing efficient DNNs via hardware-aware neural architecture search and beyond
title_short Designing efficient DNNs via hardware-aware neural architecture search and beyond
title_full Designing efficient DNNs via hardware-aware neural architecture search and beyond
title_fullStr Designing efficient DNNs via hardware-aware neural architecture search and beyond
title_full_unstemmed Designing efficient DNNs via hardware-aware neural architecture search and beyond
title_sort designing efficient dnns via hardware-aware neural architecture search and beyond
publishDate 2022
url https://hdl.handle.net/10356/155785
_version_ 1787136481132806144