SCOPE

Citations Over TimeTop 1% of 2008 papers

Abstract

Companies providing cloud-scale services have an increasing need to store and analyze massive data sets such as search logs and click streams. For cost and performance reasons, processing is typically done on large clusters of shared-nothing commodity machines. It is imperative to develop a programming model that hides the complexity of the underlying system but provides flexibility by allowing users to extend functionality to meet a variety of requirements. In this paper, we present a new declarative and extensible scripting language, SCOPE (Structured Computations Optimized for Parallel Execution), targeted for this type of massive data analysis. The language is designed for ease of use with no explicit parallelism, while being amenable to efficient parallel execution on large clusters. SCOPE borrows several features from SQL. Data is modeled as sets of rows composed of typed columns. The select statement is retained with inner joins, outer joins, and aggregation allowed. Users can easily define their own functions and implement their own versions of operators: extractors (parsing and constructing rows from a file), processors (row-wise processing), reducers (group-wise processing), and combiners (combining rows from two inputs). SCOPE supports nesting of expressions but also allows a computation to be specified as a series of steps, in a manner often preferred by programmers. We also describe how scripts are compiled into efficient, parallel execution plans and executed on large clusters.

Related Papers

• → Application of Filters to Multiway Joins in MapReduce(2014)6 cited
• → A comparative evaluation of techniques for N-way joins in wireless sensors networks(2020)3 cited
• → Faster joins, self-joins and multi-way joins using join indices(1998)15 cited
• The Network Joins Each Other and Joins Installation to Use Dwelling(2007)
• USING SEMI-JOINS TO REALIZE JOINS AND MIXED-SEMI(?)JOINS(1987)