Visualisation and management tool for supercomputer resource

• Main Author: Shao, Yiyang
• Other Authors: Lee Bu Sung, Francis
• Format: Final Year Project
• Language: English
• Published: 2018
• Subjects: DRNTU::Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence
• Online Access: http://hdl.handle.net/10356/73967
• Institution: Nanyang Technological University

The machine learning, especially deep learning has gained an astonishing amount of popularity over the last few years. Because of the success of machine learning in many other fields. The interest of applying machine learning to the job scheduling of high performance supercomputer has been raised. The reason is that current scheduling schemes are mostly heuristic based approach with various strategies such as backfilling and they are relatively fixed and not flexible enough to handle the changing characteristics of large amount of jobs belonging to different types. Therefore, a more dynamic job scheduler is needed. In this report, two of the recent machine learning approaches are examined on real world workload from National Supercomputer Center of Singapore(NSCC). One of them used a nonlinear regression machine learning strategy. The other one made use of the popular deep reinforcement learning technique. The nonlinear regression functions performed fairly well on the real world workload, showing its robustness and generalization ability. The deep reinforcement learning model could not easily adapt the real world workload due to its complicated architecture and difficulty and time consuming in training. Its performance on NSCC real world data is not ideal, either. Therefore, the nonlinear regression method is preferred to be further improved and adapted for production use.