Probabilistic noise identification and data cleaning

Citations Over TimeTop 10% of 2004 papers

Abstract

Real world data is never as perfect as we would like it to be and can often suffer from corruptions that may impact interpretations of the data, models created from the data, and decisions made based on the data. One approach to this problem is to identify and remove records that contain corruptions. Unfortunately, if only certain fields in a record have been corrupted then usable, uncorrupted data will be lost. We present LENS, an approach for identifying corrupted fields and using the remaining noncorrupted fields for subsequent modeling and analysis. Our approach uses the data to learn a probabilistic model containing three components: a generative model of the clean records, a generative model of the noise values, and a probabilistic model of the corruption process. We provide an algorithm for the unsupervised discovery of such models and empirically evaluate both its performance at detecting corrupted fields and, as one example application, the resulting improvement this gives to a classifier.

Related Papers

• → Noise Measurements of Tactical UAVs(2010)36 cited
• → Noise-assisted data processing in measurement science: Part one part 40 in a series of tutorials on instrumentation and measurement(2012)11 cited
• → A new method of measurement of the different types of noise altering the output signal of oscillators(1993)6 cited
• → Accuracy of nonoscillating one-port noise measurements(1995)4 cited
• → Identification of measurement and maneuver noise in α-β target tracking systems(1992)