Meteor Shower: A Reliable Stream Processing System for Commodity Data Centers

Citations Over TimeTop 12% of 2012 papers

Abstract

Large-scale failures are commonplace in commodity data centers, the major platforms for Distributed Stream Processing Systems (DSPSs). Yet, most DSPSs can only handle single-node failures. Here, we propose Meteor Shower, a new fault-tolerant DSPS that overcomes large-scale burst failures while improving overall performance. Meteor Shower is based on checkpoints. Unlike previous schemes, Meteor Shower orchestrates operators' check pointing activities through tokens. The tokens originate from source operators, trickle down the stream graph, triggering each operator that receives these tokens to checkpoint its own state. Meteor Shower is a suite of three new techniques: 1) source preservation, 2) parallel, asynchronous check pointing, and 3) application-aware check pointing. Source preservation allows Meteor Shower to avoid the overhead of redundant tuple saving in prior schemes, parallel, asynchronous check pointing enables Meter Shower operators to continue processing streams during a checkpoint, while application-aware check pointing lets Meteor Shower learn the changing pattern of operators' state size and initiate checkpoints only when the state size is minimal. All three techniques together enable Meteor Shower to improve throughput by 226% and lower latency by 57% vs prior state-of-the-art. Our results were measured on a prototype implementation running three real world applications in the Amazon EC2 Cloud.

Related Papers

• → Mesospheric temperature estimation from meteor decay times during Geminids meteor shower(2016)14 cited
• → The 2004 June Bootid meteor shower(2005)5 cited
• → The “Mobile Observation of Meteor” (MoMET) device(2022)1 cited
• The Third Peak of the 1998 Leonid Meteor Shower(2001)
• An Evaluation of the Accuracy of Meteor Shower Forecasts(2004)