Record linkage

Under duplicate detection or object identifier ( also English Record Linkage ) refers to various automatic methods by which we can identify cases in data sets that represent the same object in the real world. This is for example for merging multiple data sources ( deduplication ) or during data cleanup necessary.

Duplicates can occur, for example by entering and transmission errors, due to various spellings and abbreviations, or due to different data schemas. Example can be taken from different sources addresses in an address database, with one and the same address with variations can be taken multiple times. By means of duplicate detection are now found these duplicates and the actual addresses are identified as objects.

There are two different types of duplicates: identical duplicates, in which all values ​​are identical and non-identical duplicates, which differ from one to several values. The detection and cleanup is trivial in the first case, the extra duplicates can be easily removed without loss of information. Difficult and complex, the second case, since the duplicate can not be identified is a simple direct comparison, such as the first case. For this reason, heuristics have to be applied. In the second case, the excess data can not simply be deleted, it must first be consolidated and the values ​​are summarized.

The process for identifying and consolidating duplicates

The process for identifying and consolidating duplicates may take the following four steps ( Apel, 2009, p 164):

To detect duplicates different similarity measures are applied, for example, the Levenshtein Distance, or the typewriter distance. Because usually for reasons of cost, each record with each other can not be compared, there are methods such as the sorted neighborhood (English Sorted Neighborhood), in which only potentially similar records are checked to see if they are duplicates.

There are phonetic algorithms that assign words to their voice sound a string, the phonetic code to implement a similarity search, such as Soundex and Cologne phonetics.

Examples

The following entries from a list of names may be possibly involve duplicates:

With a library duplicates can occur when multiple library catalogs are merged.