Efficient record linkage in large data sets

Citations Over TimeTop 1% of 2003 papers

Abstract

This paper describes an efficient approach to record linkage. Given two lists of records, the record-linkage problem consists of determining all pairs that are similar to each other where the overall similarity between two records is defined based on domain-specific similarities over individual attributes constituting the record. The record-linkage problem arises naturally in the context of data cleansing that usually precedes data analysis and mining. We explore a novel approach to this problem. For each attribute of records, we first map values to a multidimensional Euclidean space that preserves domain-specific similarity. Many mapping algorithms can be applied, and we use the FastMap approach as an example. Given the merging rule that defines when two records are Similar a set of attributes are chosen along which the merge will proceed A multidimensional similarity join over the chosen attributes is used to determine similar pairs of records. Our extensive experiments using real data sets show that our solution has very good efficiency and accuracy.

Related Papers

• → Population Data Centre Profiles: Centre for Data Linkage(2020)17 cited
• → Use of record linkage in small-area studies(1996)74 cited
• → Record Linkage Theory and Practice: an Experiment in the Application of Multiple Pass Linkage Algorithms(1996)6 cited
• → Determining First Admissions in a Hospital Discharge File via Record Linkage(1998)16 cited
• → Understanding the linking possibilities in Scottish Records and an algorithmic approach to full linkage(2018)