Data Quality Pipelines

Quality Rules Execution

  1. Start segment: [RECORD_UPSERT_QUALITY_START] - defines an entity/reference set for which the quality rules will be executed.

  2. Point segment: [RECORD_UPSERT_QUALITY_POINT] - perfoms the assigned quality rules for all validity periods of the record.

  3. Gate segment: [RECORD_UPSERT_QUALITY_GATE] - indexes the calculated quality rule errors and applies the enrichment results to the record.

  4. Finish segment: [RECORD_UPSERT_QUALITY_FINISH] - returns the result of quality rule execution.

When Saving Record Changes

  1. Start segment: [RECORD_UPSERT_START] - saves record changes.

  2. Connector segment: [RECORD_QUALITY_CONNECTOR] - checks quality rule sets assigned to the entity/reference set and starts the quality rules execution pipeline for the records (if the records have assigned rule sets).

  3. Finish segment: [RECORD_UPSERT_FINISH] - returns the result of saving record changes.

Forming Quality Rule Errors for Records

  1. Start segment: [RECORD_GET_START].

  2. Connector segment: [RECORD_GET_QUALITY_CONNECTOR] - allows you to retrieve the calculated quality rule errors for the requested record.

  3. Finish segment: [RECORD_GET_FINISH].

Example of completed pipeline

Figure 1. Example of completed pipeline