Integrating Online Compression to Accelerate Large-Scale Data Analytics Applications

Citations Over TimeTop 10% of 2013 papers

Abstract

Compute cycles in high performance systems are increasing at a much faster pace than both storage and wide-area bandwidths. To continue improving the performance of large-scale data analytics applications, compression has therefore become promising approach. In this context, this paper makes the following contributions. First, we develop a new compression methodology, which exploits the similarities between spatial and/or temporal neighbors in a popular climate simulation dataset and enables high compression ratios and low decompression costs. Second, we develop a framework that can be used to incorporate a variety of compression and decompression algorithms. This framework also supports a simple API to allow integration with an existing application or data processing middleware. Once a compression algorithm is implemented, this framework automatically mechanizes multi-threaded retrieval, multi-threaded data decompression, and the use of informed prefetching and caching. By integrating this framework with a data-intensive middleware, we have applied our compression methodology and framework to three applications over two datasets, including the Global Cloud-Resolving Model (GCRM) climate dataset. We obtained an average compression ratio of 51.68%, and up to 53.27% improvement in execution time of data analysis applications by amortizing I/O time by moving compressed data.

Related Papers

• → Expansion of Premixed Compression Ignition Combustion Region by Supercharging Operation and Lower Compression Ratio Piston(2007)23 cited
• → The Impact of Fuel Injection Strategies and Compression Ratio on Combustion and Performance of a Heavy-Duty Gasoline Compression Ignition Engine(2022)3 cited
• → Compression Ratio Estimation Based on Cylinder Pressure Data(2004)3 cited
• Dynamic parameter control method for process data compression ratio(2013)
• → Iterative Data Compression with Calculated Codes-IEEE_1col.docx(2023)