Long-term multi-resource fairness for pay-as-you use computing systems

• Main Authors: Tang, Shanjiang, Niu, Zhaojie, He, Bingsheng, Lee, Bu-Sung, Yu, Ce
• Other Authors: School of Computer Science and Engineering
• Format: Article
• Language: English
• Published: 2020
• Subjects: Engineering::Computer science and engineering; Long-term Multi-resource Fairness; Cloud Computing
• Online Access: https://hdl.handle.net/10356/142090
• Institution: Nanyang Technological University

Many current computing systems such as clouds and supercomputers charge users for their resource usages. A user's demand is often changing over time, indicating that it is difficult to keep the high resource utilization all the time for cost efficiency. Resource sharing is a classical and effective approach for high resource utilization. In view of the heterogeneous resource demands of users' workloads, multi-resource allocation fairness is a must for resource sharing in such pay-as-you-use computing systems. However, we find that, existing multi-resource fair policies such as Dominant Resource Fairness (DRF), implemented in currently popular resource management systems such as Apache YARN [4] and Mesos [23] , are not suitable for the pay-as-you-use computing systems. We show that this is because of their memoryless characteristic that can cause the following problems in the pay-as-you-use computing systems: 1). users can get resource benefits by cheating; 2). users might not be able to get the total amount of resources that they are entitled to in terms of their resource contributions. In this paper, we propose a new policy called H-MRF, which generalizes DRF and Asset Fairness with the long-term notion. We show that it can address these problems and is suitable for pay-as-you-use computing systems. We have implemented it into YARN by developing a prototype called MRYARN. Finally, we evaluate H-MRF using both testbed and simulated experiments. The experimental results show that there are about 1.1 ∼ 1.5 sharing benefit degrees and 1.2 × ∼ 1.8 × performance improvement for users with H-MRF, better than existing fair schedulers.