PageRank Pipeline Benchmark: Proposal for a Holistic System Benchmark for Big-Data Platforms

Citations Over TimeTop 10% of 2016 papers

Abstract

The rise of big data systems has created a need for benchmarks to measure and compare the capabilities of these systems. Big data benchmarks present unique scalability challenges. The supercomputing community has wrestled with these challenges for decades and developed methodologies for creating rigorous scalable benchmarks (e.g., HPC Challenge). The proposed PageRank pipeline benchmark employs supercomputing benchmarking methodologies to create a scalable benchmark that is reflective of many real-world big data processing systems. The PageRank pipeline benchmark builds on existing prior scalable benchmarks (Graph500, Sort, and PageRank) to create a holistic benchmark with multiple integrated kernels that can be run together or independently. Each kernel is well defined mathematically and can be implemented in any programming environment. The linear algebraic nature of PageRank makes it well suited to being implemented using the GraphBLAS standard. The computations are simple enough that performance predictions can be made based on simple computing hardware models. The surrounding kernels provide the context for each kernel that allows rigorous definition of both the input and the output for each kernel. Furthermore, since the proposed PageRank pipeline benchmark is scalable in both problem size and hardware, it can be used to measure and quantitatively compare a wide range of present day and future systems. Serial implementations in C++, Python, Python with Pandas, Matlab, Octave, and Julia have been implemented and their single threaded performance has been measured.

Related Papers

• Strategic Benchmarking: How to Rate Your Company's Performance against the World's Best(1993)
• → COMPARISON OF BEST PRACTICE BENCHMARKING MODELS(2011)11 cited
• → A guide for mental health clinicians to develop and undertake benchmarking activities(2010)2 cited
• Theoretical Aspects of Benchmarking Theory(2004)
• → Comparing ourselves: using benchmarking techniques to measure performance between academic libraries(2009)1 cited