Dynamic Workload Balancing for Hadoop MapReduce
Citations Over TimeTop 10% of 2014 papers
Abstract
Hadoop has two components which are HDFS and MapReduce. HDFS is a distributed file system for storing data for users of Hadoop and MapReduce is the framework that executes jobs from users. Hadoop stores user data based on space utilization of data nodes on the cluster rather than the processing capability of the data nodes. Furthermore Hadoop runs in a heterogeneous environment as all data nodes may not be homogeneous. For these reasons, workload imbalances will occur when Hadoop runs resulting in poor performance. In this paper, we propose a dynamic algorithm to balance the workload between different racks on a Hadoop cluster based on information obtained from analyzing the log files of Hadoop. Moving tasks from the busiest rack to another rack improves the performance of Hadoop MapReduce by reducing the running time of jobs. Our simulations indicate that using our algorithm, we can decrease by more than 50% the remaining time of the tasks belonged to a job running on the busiest rack.
Related Papers
- → Dynamic behaviours of rack vehicle-track system considering rack flexibility under gear-rack mesh impact(2024)19 cited
- → A Convenient Storage Rack for Graduated Cylinders(2004)2 cited
- Research on the method of small-module rack measurement based on machine vision(2011)
- Influences of Logix Rack's parameters on tooth profile and their selection(2003)
- Study on Vibration Characteristics of Single Rack in Trash、rack(2004)