High Performance Threaded Data Streaming for Large Scale Simulations
2005pp. 243–250
Citations Over TimeTop 10% of 2005 papers
Abstract
We have developed a threaded parallel data streaming approach using logistical networking (LN) to transfer multiterabyte simulation data from computers at NERSC to our local analysis/visualization cluster, as the simulation executes, with negligible overhead. Data transfer experiments show that this concurrent data transfer approach is more favorable compared with writing to local disk and later transferring this data to be post-processed. Our algorithms are network aware, and can stream data at up to 97 Mbs on a 100 Mbs link from CA to NJ during a live simulation, using less than 5% CPU overhead at NERSC. This method is the first step in setting up a pipeline for simulation workflow and data management.
Related Papers
- → Study of high-speed data transfer utilizing flexible and parallel transmission lines(2005)1 cited
- → Recordable digital versatile disc for high data transfer rate(2003)1 cited
- Research on Data Transfer in ISCS of Metro(2011)
- Design of multiple data transfer model of GridFTP(2008)