Comparing & Merging Duplicates

Viewing Cluster Content

To view the cluster content:

To view the contents of a cluster, click on the row with the cluster name in the "Duplicates" section.

As a result of the action, a drawer opens with detailed information about the cluster: by which matching rule or set of rules the cluster was formed, the date of formation, and the number of records in the cluster (Figure 1). A table with the list of records included in the cluster will be generated below, consisting of 5 columns:

  • Display name. Name of the main displayed attribute of the record.

  • Record ID. Record unique identification number.

  • Type. Type of space in which the record is located (entity/reference set).

  • Name. Name of the entity/reference set containing the record.

  • Matching rate. Level of correspondence of the found record to the configured rules.

Notes:

  • The transition to the record card is available by clicking on the record ID. The card will open in a new browser tab.

  • The parent node is also compared in the hierarchical reference sets.

  • When comparing referenses, only the main displayed attribute of the linked record is displayed by default. Clicking on the arrow icon next to the link name enables a detailed view of the linked record.

Drawer with contents of duplicate cluster

Figure 1. Drawer with contents of duplicate cluster

Comparing Duplicates

To compare cluster records:

  1. Open the contents of the required cluster.

  2. Select the checkboxes of the records you want to compare. The records must be within the same space - a certain entity or reference set, otherwise the "Actions" button will be inactive.

  3. Press the "Actions" button, which becomes active after selecting two or more records, and select "Compare and merge".

  4. As a result of the action, a screen for comparing duplicate records in tabular form will be displayed (Figure 2).

    • The cells that differ will be highlighted in blue.

    • To display only the differing items, click image5 "Differences only".

    • To change the tabular view to a list view, click image6 "Change view".

Merging Duplicates

To merge cluster records:

  1. If needed, exclude unnecessary records from the comparison. Note that all duplicate records from the comparison will later be merged into a single winning record.

  2. On the comparing screen, the winning record is marked with the image1 icon. If necessary, designate another record as the winner by clicking image2 "Set as winner" to the left of the record ID header (Figure 2).

  3. If necessary, select attribute values (incl. complex attributes) of the other source records. Click image6 "Select", which appears when you hover over the attribute row. As a result of the action, the selected attribute will be marked with the image3 icon.

  4. The final result of the merge is displayed in the leftmost column and is grayed out.

  5. Click the "Merge records" button in the upper right corner of the window and confirm the action.

  6. As a result of the action, the records will be merged and the cluster will be deleted.

  7. The result of the merging will be available in the "Data" section.

Tip

Known issue. When merge duplicates, the "Set as winner" flag doesn't work

Screen to compare and merge cluster records

Figure 2. Screen to compare and merge cluster records

Scheme for Automatic Detection of Winning Record

Algorithm for consolidating a cluster of duplicate records:

  • The winner's Etalon ID is selected randomly (as a rule, the first one from the list of Etalon IDs in the cluster is selected).

  • Records are consolidated according to the principle of referring to the best version of the truth: when choosing a winner, the attributes from the source system with the highest weight have priority; if the source system is the same or the weight is the same, then the attribute wins by date (priority for a newer attribute).

  • Relations are consolidated according to the principle of referring to the best value of the record: the relations are taken entirely from the winning entry.

  • Relations are grouped by destination (to).

    • If there are relations that match the destination, then the winner of the Etalon ID of the relation is randomly selected among them, and all the Origin ID of the relation is transferred to it.

    • The relations of losers in the cluster of records that do not have a match at the destination point are rewritten to the winner.

  • When combining hierarchical reference set records: the parent node of the combined record becomes the parent node of the winning record.

Notes:

  • When comparing relations, only the main displayed attribute of the related record is displayed by default. When you click on the arrow icon next to the relation name and select the related record, the relation attributes are also displayed.

  • When merging records in the hierarchical reference set, the parent node of the merged record becomes the parent node of the winning record

  • It is not possible to select individual relations as winners in the duplicate merging table. The relations in the combined record will be taken from the winning record.

  • If one of the compared attributes is empty, but still selected as a winner, the attribute will still be completed in the final record (for situations where the winning record already had an attribute value). An empty attribute cannot override an existing attribute when merging.