Hadoop job scheduling with dynamic task splitting
Job scheduling affects the fairness and performance of shared Hadoop clusters. Fairness measures how fair the resources in the cluster are shared among different users in the Hadoop cluster. In Hadoop, schedulers will always attempt to maximize data locality. Data locality refers to the processing o...
Saved in:
Main Author: | |
---|---|
Other Authors: | |
Format: | Theses and Dissertations |
Language: | English |
Published: |
2015
|
Subjects: | |
Online Access: | https://hdl.handle.net/10356/65309 |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Nanyang Technological University |
Language: | English |
Summary: | Job scheduling affects the fairness and performance of shared Hadoop clusters. Fairness
measures how fair the resources in the cluster are shared among different users in the
Hadoop cluster. In Hadoop, schedulers will always attempt to maximize data locality.
Data locality refers to the processing of data by tasks on nodes where the data is stored.
Processing of data on data-local nodes improves performance, as there is no need to
transfer data from one node to another.
However, fairness and data locality are often in conflict. During scheduling, it is not
always possible that the available nodes contain the data that a user’s job requires. In such
cases, a scheduler may choose to schedule the tasks on these nodes regardless of data
locality thus sacrificing performance. Alternatively, a scheduler may choose to give up
the user’s slot and wait for a data-local node thus sacrificing fairness. Achieving pure
fairness may compromise the data locality of the tasks that will in turn negatively affects
performances, and vice-versa. Delay scheduling is a technique that attempts to improve
data locality by waiting for a data-local node to be available. It violates the fairness
criteria.
The Dynamic Task Splitting Scheduler (DTSS) is proposed to mitigate the tradeoffs
between fairness and data locality during job scheduling. DTSS does so by dynamically
splitting a task and executing the split task immediately, on a non-data-local node, to
improve the fairness. Analysis and experiments results show that it is possible to improve
both fairness and the performance by adjusting the proportion of the task split. DTSS is
shown to improve the makespan of different users in a cluster by 2% to 11% as compared
to delay scheduling under conditions that is difficult to obtain data-local nodes on a
cluster. Lastly, experiments show that DTSS is not a suitable scheduler under conditions
where jobs are able to obtain data-local nodes easily. |
---|