Improving batch schedulers with node stealing for failed jobs
Citations Over TimeTop 15% of 2024 papers
Abstract
Summary After a machine failure, batch schedulers typically re‐schedule the job that failed with a high priority. This is fair for the failed job but still requires that job to re‐enter the submission queue and to wait for enough resources to become available. The waiting time can be very long when the job is large and the platform highly loaded, as is the case with typical HPC platforms. We propose another strategy: when a job fails, if no platform node is available, we steal one node from another job , and use it to continue the execution of despite the failure. In this work, we give a detailed assessment of this node stealing strategy using traces from the Mira supercomputer at Argonne National Laboratory. The main conclusion is that node stealing improves the utilization of the platform and dramatically reduces the flow of large jobs, at the price of slightly increasing the flow of small jobs.
Related Papers
- → Job Scheduling without Prior Information in Big Data Processing Systems(2017)19 cited
- → Approbation of Methods for Supercomputer Job Queue Wait Time Estimation(2023)5 cited
- → End-to-End Predictions-Based Resource Management Framework for\n Supercomputer Jobs(2020)1 cited
- Multi-queue Hybrid Job Scheduling Mechanism in Grid Computing(2007)
- → End-to-End Predictions-Based Resource Management Framework for Supercomputer Jobs(2020)