Hiding Checkpoint Overhead in HPC Applications with a Semi-Blocking Algorithm

Citations Over TimeTop 10% of 2012 papers

Abstract

The HPC community has seen a steady increase in the number of components in every generation of supercomputers. Assembling a large number of components into a single cluster makes a machine more powerful, but also much more prone to failures. Therefore, fault tolerance has become a major concern in HPC. To deal with node crashes in large systems, checkpoint/restart is by far the preferred method. A typical way to implement checkpoints is by using a blocking algorithm, which suspends the execution of the application while the checkpoint is safely stored. One limitation of the blocking algorithm is that it saturates the network bandwidth at the time of checkpoint. This problem will become even more critical because the projected network bandwidth increase will not match the increase in memory per node. To alleviate this problem, we have developed a semi-blocking checkpoint algorithm that overlaps execution of the application with transmission of checkpoints. Our implementation decomposes a checkpoint into small messages that are interleaved with application messages. The experimental results show a dramatic reduction in the checkpoint overhead for various applications. We present a model for our approach and use this model to compute the benefit of the semi-blocking algorithm for different failure rates predicted at Exascale. We estimate our method can reduce up to 22% the total execution time of an iterative scientific application.

Related Papers

• → The performance of consistent checkpointing(2003)355 cited
• → Blocking action in the northern hemisphere: A Climatological study(1981)123 cited
• → Synonymy blocking and the elsewhere condition: lexical morphology and the speaker(2001)43 cited
• Discussion on Miss Operation Blocking Device in Substation(2008)
• New method for detecting and dividing vehicles blocking(2012)