DeepArc: Modularizing neural networks for the model maintenance

• Main Authors: REN, Xiaoning, LIN, Yun, XUE, Yinxing, LIU, Ruofan, SUN, Jun, FENG, Zhiyong, DONG, Jinsong
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: architecture; modularization; neural networks; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/9316; https://ink.library.smu.edu.sg/context/sis_research/article/10316/viewcontent/icse23_DeepArc_av.pdf
• Institution: Singapore Management University

Neural networks are an emerging data-driven programming paradigm widely used in many areas. Unlike traditional software systems consisting of decomposable modules, a neural network is usually delivered as a monolithic package, raising challenges for some maintenance tasks such as model restructure and re-adaption. In this work, we propose DeepArc, a novel modularization method for neural networks, to reduce the cost of model maintenance tasks. Specifically, DeepArc decomposes a neural network into several consecutive modules, each of which encapsulates consecutive layers with similar semantics. The network modularization facilitates practical tasks such as refactoring the model to preserve existing features (e.g., model compression) and enhancing the model with new features (e.g., fitting new samples). The modularization and encapsulation allow us to restructure or retrain the model by only pruning and tuning a few localized neurons and layers. Our experiments show that (1) DeepArc can boost the runtime efficiency of the state-of-the-art model compression techniques by 14.8%; (2) compared to the traditional model retraining, DeepArc only needs to train less than 20% of the neurons on average to fit adversarial samples and repair under-performing models, leading to 32.85% faster training performance while achieving similar model prediction performance.