ARIA
Citations Over TimeTop 1% of 2011 papers
Abstract
MapReduce and Hadoop represent an economically compelling alternative for efficient large scale data processing and advanced analytics in the enterprise. A key challenge in shared MapReduce clusters is the ability to automatically tailor and control resource allocations to different applications for achieving their performance goals. Currently, there is no job scheduler for MapReduce environments that given a job completion deadline, could allocate the appropriate amount of resources to the job so that it meets the required Service Level Objective (SLO). In this work, we propose a framework, called ARIA, to address this problem. It comprises of three inter-related components. First, for a production job that is routinely executed on a new dataset, we build a job profile that compactly summarizes critical performance characteristics of the underlying application during the map and reduce stages. Second, we design a MapReduce performance model, that for a given job (with a known profile) and its SLO (soft deadline), estimates the amount of resources required for job completion within the deadline. Finally, we implement a novel SLO-based scheduler in Hadoop that determines job ordering and the amount of resources to allocate for meeting the job deadlines.
Related Papers
- → Job Scheduling without Prior Information in Big Data Processing Systems(2017)19 cited
- → Improving batch schedulers with node stealing for failed jobs(2024)2 cited
- → Job Scheduling Simulator for Assisting the Mapping Configuration Between Queue and Computing Nodes(2019)3 cited
- → Automating Job Monitoring System for an Ecosystem of High Performance Computing(2017)1 cited
- Multi-queue Hybrid Job Scheduling Mechanism in Grid Computing(2007)