Combining in-situ and in-transit processing to enable extreme-scale scientific analysis
Citations Over TimeTop 1% of 2012 papers
Abstract
With the onset of extreme-scale computing, I/O constraints make it increasingly difficult for scientists to save a sufficient amount of raw simulation data to persistent storage. One potential solution is to change the data analysis pipeline from a post-process centric to a concurrent approach based on either in-situ or in-transit processing. In this context computations are considered in-situ if they utilize the primary compute resources, while in-transit processing refers to offloading computations to a set of secondary resources using asynchronous data transfers. In this paper we explore the design and implementation of three common analysis techniques typically performed on large-scale scientific simulations: topological analysis, descriptive statistics, and visualization. We summarize algorithmic developments, describe a resource scheduling system to coordinate the execution of various analysis workflows, and discuss our implementation using the DataSpaces and ADIOS frameworks that support efficient data movement between in-situ and in-transit computations. We demonstrate the efficiency of our lightweight, flexible framework by deploying it on the Jaguar XK6 to analyze data generated by S3D, a massively parallel turbulent combustion code. Our framework allows scientists dealing with the data deluge at extreme scale to perform analyses at increased temporal resolutions, mitigate I/O costs, and significantly improve the time to insight.
Related Papers
- → Big Data 4.0: The Era of Big Intelligence(2024)4 cited
- → Large-scale simulation of the three-dimensional Navier-Stokes equations(1992)2 cited
- → Electromagnetic scattering simulations using overset grids on massively parallel computing platforms(2002)1 cited
- → Studies on the Efficiency of Massively Parallel Computational Method for Large-Scale Computation based on Unstructured Grid(1996)
- → M3E_LINSOL: A Scalable Linear Solver Package for Reservoir Simulations on Massively Parallel Computers(2017)