On-line data validation in distributed data fusion

Citations Over TimeTop 10% of 2013 papers

Abstract

Data acquisition and data fusion systems are becoming increasingly complex, being in fact systems of systems, where every component may be a system with varying levels of autonomy by themselves. Possible changes in system configuration by entities joining or being removed from the system make the system complex. As synchronous operation cannot be expected in such a system configuration, the temporal and spatial correctness of data must be achieved via other means. This paper presents the concept of mediated interactions as a method for ensuring correctness of computation in a distributed system. The mediator associated with each computing entity is responsible for online checking of the data both before it is sent out at the sender side and before it is received at the receiver side, ensuring that only data satisfying the validity constraints of the receiver-side data processing algorithm is used in computation. This assumes that each data item is augmented with metadata, which enables online data validation. The validity and quality dimensions in use depend on the system requirements defined by a specific problem and situational context; they may be temporal, spatial and involve various data quality dimensions, such as accuracy, confidence, relevance, credibility, and reliability. Among other capabilities, the mediator is able to cope with the unknowns in the temporal dimension that occur at runtime and are not predictable, such as channel delay, jitter of clocks and processing delays. This capability becomes an especially relevant factor in multi-tasking systems and in configurations in which a computing entity may have to process a variable number of parallel streams of data. Both the architecture and a simulation case study of a distributed data fusion scenario are presented in the paper.

Related Papers

• Automated Data Validation for Data Migration Security(2011)
• → Data Quality Assessment(2018)2 cited
• Automated Data Validation Framework for Data Quality in Big Data Migration Projects(2014)
• → Evaluation Method of Data Quality for Background Data in LCA(2009)1 cited
• → Beyond Data Validation – Advanced Strategies for Assessing Data Quality for Oil Spill Investigations(2021)