OPTIMIZING INFERENCE PERFORMANCE OF BERT ON CPUS USING APACHE TVM

• Main Author: Legowo, Setyo
• Format: Theses
• Language: Indonesia
• Online Access: https://digilib.itb.ac.id/gdl/view/56144
• Institution: Institut Teknologi Bandung

BERT as a model that composed of Transformer layer is game changing for field of natural language processing (NLP). There has been a lot of study to speedup training the model however only relatively little efforts are made to improve their inference performance. Also, not all machine learning framework that commonly used by modeler in an instance able to run and optimized on particular target that not supported by those frameworks. Apache TVM offers good benefits to optimize any kind of machine learning model aside from its primary objective to convert those various models that already implemented on a ML framework to be able to run on any backend. Therefore, we analyze several factors that affect inference time of BERT after converted by TVM, that is SIMD extensions in the CPU, library that used by TVM runtime, and combination between core CPU and threads. With good settings, we can speedup 44% from default implementation.