Crossing analytics systems: A case for integrated provenance in data lakes
Citations Over TimeTop 10% of 2016 papers
Abstract
The volumes of data in Big Data, their variety and unstructured nature, have had researchers looking beyond the data warehouse. The data warehouse, among other features, requires mapping data to a schema upon ingest, an approach seen as inflexible for the massive variety of Big Data. The Data Lake is emerging as an alternate solution for storing data of widely divergent types and scales. Designed for high flexibility, the Data Lake follows a schema-on-read philosophy and data transformations are assumed to be performed within the Data Lake. During its lifecycle in a Data Lake, a data product may undergo numerous transformations performed by any number of Big Data processing engines leading to questions of traceability. In this paper we argue that provenance contributes to easier data management and traceability within a Data Lake infrastructure. We discuss the challenges in provenance integration in a Data Lake and propose a reference architecture to overcome the challenges. We evaluate our architecture through a prototype implementation built using our distributed provenance collection tools.
Related Papers
- → Data Integration Processes(2014)8 cited
- Merging Multidimensional Data Models: A Practical Approach for Schema and Data Instances(2013)
- Research On Spatial Data Integration Orientated To Spatial Data Warehouse(2003)
- Research into spatial data integration——Taking the building of the spatial data warehouse as an example(2004)
- Data Integration in A Data Warehouse(2002)