Changelog

Standard Edition

6.10 Version

Frontend SDK

The following methods were added to the Dialog class:

• static setMaxNestedItems(count: number) - allows you to change the maximum number of errors that are displayed in the popup message (3 by default). Also, the popup message now has the option to view all errors by clicking on the "Show all'' button.

• static getMaxNestedItems(): number - getter for the value set by the previous method.

Matching Duplicates by Relations

The "Value Set Matching" algorithm was added. Based on the physical column type, hstore matches sets of values and returns true if the sets have an intersection (?| operator).

• Support for HSTORE type in columns was added to support row sets. Added SetAlgorithm to support row sets.

• Support for using attributes and special relation fields that can be referenced from mapping was added. Added extension points to link execution threads to support link changes without changing records and separate data publishing. Insertion now works asynchronously after the transaction closes, as it requires the final state of all data, which is not available in record connectors.

• Automatic fields were added to data and relation records.

Search for Tasks by "Initiator" Criterion

Added "Initiator" field for tasks on backend. For old tasks the field will be missing in the index - it is necessary to reindex it.

Re-indexing of tasks is implemented in com.unidata.mdm.workflow.data module, which will be added to the Universe DG system in future releases.

Customizing the Display of Incoming Relations in Record Card

1. Modifications to the RegisterEntityRestService REST service:

RegisterEntityRO:

• Field relations, methods getRelations, setRelations are marked deprecated;

• Added fields outgoingRelations, incomingRelations.

RelationEntityRO:

• Added attributes incomingNameForDisplay, showIncoming, showIncomingAttributes.

Influences:

• /v2/data/model/register-entities/{id}

• /v2/data/model/register-entities/

2. Modifications to the DataRelationsRestService REST service:

fetchByToSide:

• Added query parameter fetchByToSide to get information about incoming relation.

• If fetchByToSide = true, incoming relation is queried; if fetchByToSide = false, outgoing relation is queried. The default value remains false.

Influences:

• GET /v2/data/relations/timelines/date/{etalonId}

• GET /v2/data/relations/timelines/range/{etalonId}

• GET /v2/data/relations/relation-bulk/{id}/{name}

For GET /v2/data/relations/relation-bulk/{id}/{name}:

• Added etalonDisplayNameFrom field (in EtalonRelationToRO) for the name of the incoming relation;

• If showIncomingAttributes = false for an incoming relation - the attributes of the relation are not returned;

• Added query parameter fetchByToSide to get information about the incoming relation.

SmartETL Message Sending Pipeline

1. Segments (not included in pipelines.json of the standard build) have been added to the pipeline for sending messages to SmartETL about record merge and detach events:

   • com.universe.mdm.notifications[SEND_MERGE_NOTIFICATION] - segment for record merge event.

   • com.universe.mdm.notifications[SEND_UNMERGE_NOTIFICATION] is the segment for the record detach event.

To register a listener endpoint for receiving messages, you must specify it in the com.universe.mdm.notifications.notifications.messaging.endpoint parameter in backend.properties. For example, com.universe.mdm.notifications.messaging.endpoint = smart-etl:80/consumer.

• The /consumer will be appended to the address specified in the parameter to send notifications. In the example: smart-etl:80/consumer/consumer. POST requests containing JSON with the event will be sent there.

2. Changed the name of the listener endpoint parameter in backend.properties:

   • com.unidata.smartetl.mdm.notifications.messaging.endpoint changed to com.universe.mdm.notifications.messaging.endpoint.

For new messages, the integrator may need to change the listener controller to be able to receive additional information of new messages.

• Record Merge Event

To enable event sending, the com.universe.mdm.notifications[SEND_MERGE_NOTIFICATION] segment must be added to the org.unidata.mdm.data[RECORD_MERGE_START] pipeline before the final org.unidata.mdm.data[RECORD_MERGE_FINISH] segment to ensure that the result is unchanged after the notification is sent.

The format of the request body received by the listener is:

{
  "id": "messageID",
  "entityName": "Entity/reference set name",
  "etalonId": "Etalon key of the winner record",
  "originId": "Original key of the winner record",
  "timestamp": "Event timestamp",
  "type": "MERGE",
 "winner": { "//Winner Record Key".
    "entityName": "Entity/reference set name",
    "etalonId": "Record etalon key",
    "externalKeys": [ //ExternalKeys of Record
      "External Keys of Record"
    ]
  },
  }, "merged": [] //Emerged Record Keys
}

Example of a request:

{
  "id": "5341c39f-749b-4136-b8e1-bac3d73aef36",
  "entityName": "regTest",
  "etalonId": "fcd26202-5947-11ee-b77a-f58a37793af4",
  "originId": "fcd26203-5947-11ee-b77a-f58a37793af4",
  "timestamp": "2023-09-22T16:00:39.082",
  "type": "MERGE",
  "winner": {
    "entityName": "regTest",
    "etalonId": "fcd26202-5947-11ee-b77a-f58a37793af4",
    "externalKeys": [
      "universe|test-doc-10"
    ]
  },
  "merged": [
    {
      "entityName": "regTest",
      "etalonId": "fcd23aef-5947-11ee-b77a-f58a37793af4",
      "externalKeys": [
        "universe|test-doc-9"
      ]
    },
    {
      "entityName": "regTest",
     "etalonId": "fcd06617-5947-11ee-b77a-f58a37793af4",
      "externalKeys": [
        "universe|test-doc-1"
      ]
   },
    {
      "entityName": "regTest",
      "etalonId": "fcd17796-5947-11ee-b77a-f58a37793af4",
      "externalKeys": [
        "universe|test-doc-6"
      ]
    },
    {
      "entityName": "regTest",
      "etalonId": "fcd1eccc-5947-11ee-b77a-f58a37793af4",
      "externalKeys": [
        "universe|test-doc-8"
      ]
    },
    {
      "entityName": "regTest",
      "etalonId": "fcd1025d-5947-11ee-b77a-f58a37793af4",
      "externalKeys": [
        "universe|test-doc-3"
      ]
    },
    {
      "entityName": "regTest",
      "etalonId": "fcd15083-5947-11ee-b77a-f58a37793af4",
      "externalKeys": [
        "universe|test-doc-5"
      ]
    },
    {
      "entityName": "regTest",
      "etalonId": "fcd12970-5947-11ee-b77a-f58a37793af4",
      "externalKeys": [
        "universe|test-doc-4"
      ]
    },
    {
      "entityName": "regTest",
      "etalonId": "fcd1c5b9-5947-11ee-b77a-f58a37793af4",
      "externalKeys": [
        "universe|test-doc-7"
      ]
    },
    {
      "entityName": "regTest",
      "etalonId": "fcd0db4a-5947-11ee-b77a-f58a37793af4",
      "externalKeys": [
        "universe|test-doc-2"
      ]
    }
  ]
}

• Record disconnect event

To enable event sending, the com.universe.mdm.notifications[SEND_UNMERGE_NOTIFICATION] segment must be added to the org.unidata.mdm.data[RECORD_UNMERGE_START] pipeline before the final org.unidata.mdm.data[RECORD_UNMERGE_FINISH] segment to ensure that the result is unchanged after the notification is sent.

The format of the request body received by the listener is:

{
  "id": "messageID",
  "entityName": "Entity/reference set name",
  "etalonId": "Etalon key of the original record from which the other record was detached",
  "originId": "Original key of the winning record",
  "timestamp": "Event timestamp",
  "type": "UNMERGE",
  "from": { //Keys of the original record from which the other record was unmerged.
    "entityName": "Entity/reference set name",
    "etalonId": "Record etalon key",
    "externalKeys": [ //ExternalKeys of Record
      "ExternalKeys of Record"
    ]
  },
  "remaining": {}, //Keys of the original record after disconnection
  "excluded": {}, //Keys of the disconnected record
}

Example of a request:

{
  "id": "373427ba-8566-4429-8630-ba5b81f17b6b",
  "entityName": "regTest",
  "etalonId": "fcd26202-5947-11ee-b77a-f58a37793af4",
  "originId": "fcd26203-5947-11ee-b77a-f58a37793af4",
  "timestamp": "2023-09-22T16:06:55.881",
  "type": "UNMERGE",
  "from": {
    "entityName": "regTest",
    "etalonId": "fcd26202-5947-11ee-b77a-f58a37793af4",
    "externalKeys": [
      "universe|test-doc-9",
      "universe|test-doc-3",
      "universe|test-doc-2",
      "universe|test-doc-1",
      "universe|test-doc-8",
      "universe|test-doc-7",
      "universe|test-doc-6",
      "universe|test-doc-10",
      "universe|test-doc-5"
    ]
  },
  "remaining": {
    "entityName": "regTest",
    "etalonId": "fcd26202-5947-11ee-b77a-f58a37793af4",
    "externalKeys": [
      "universe|test-doc-9",
      "universe|test-doc-3",
      "universe|test-doc-2",
      "universe|test-doc-1",
      "universe|test-doc-8",
      "universe|test-doc-7",
      "universe|test-doc-6",
      "universe|test-doc-10",
      "universe|test-doc-5"
    ]
  },
  "excluded": {
   "entityName": "regTest",
    "etalonId": "fcd12970-5947-11ee-b77a-f58a37793af4",
    "externalKeys": [
      "universe|test-doc-4"
    ]
  }
}

REST for Duplicates Check

There are two execution threads needed to call online matching in case of insertion/update and recovery, respectively:

• RECORD_UPSERT_START, ${online-matching-upsert-pipeline}

• RECORD_RESTORE_START, ${online-matching-restore-pipeline}

Only the API extension has been performed, backward compatibility is not broken.

Transliteration Support in Search

To update, you need to install the analysis-icu plugin for OpenSearch.

When using Docker.

When launching Universe MDM from the Docker repository via Docker Compose, the plugin will be installed automatically.

Manual installation.

The commands below are executed from the directory where OpenSearch is installed.

Command to check installed plugins:

• bin/opensearch-plugin list

The command to install the analysis-icu plugin:

• bin/opensearch-plugin install analysis-icu

Example of a command to install the plugin offline (you need to download a zip file with the plugin in advance, for example, by following the link https://artifacts.opensearch.org/releases/plugins/analysis-icu/2.7.0/analysis-icu-2.7.0.zip):

• bin/opensearch-plugin install file:C:/Downloads/analysis-icu-2.7.0.zip

Example of a command to install a plugin by specifying the URL to the zip file with the plugin:

• bin/opensearch-plugin install https://artifacts.opensearch.org/releases/plugins/analysis-icu/2.7.0/analysis-icu-2.7.0.zip

Tool for Detaching Records (unmerge)

Added execution threads (related to connector classification com.unidata.mdm.classifiers[CLASSIFICATIONS_UNMERGE_CONNECTOR] and the flow com.unidata.mdm.classifiers[CLASSIFICATION_UNMERGE_START] added only in EE):

 {
 "startId": "org.unidata.mdm.data[RECORD_UNMERGE_START]",
 "subjectId": "",
 "description": "org.unidata.mdm.data.record.unmerge.start.description",
 "segments": [
   {
     "segmentType": "START",
     "id": "org.unidata.mdm.data[RECORD_UNMERGE_START]"
   },
   {
     "segmentType": "POINT",
     "id": "org.unidata.mdm.data[RECORD_UNMERGE_ACCESS]"
   },
   {
     "segmentType": "POINT",
     "id": "org.unidata.mdm.data[RECORD_UNMERGE_TIMELINE]"
   },
   {
     "segmentType": "CONNECTOR",
     "id": "org.unidata.mdm.dq.data[RECORD_UNMERGE_QUALITY_CONNECTOR]"
   },
   {
     "segmentType": "POINT",
     "id": "org.unidata.mdm.data[RECORD_UNMERGE_INDEXING]"
   },
   {
     "segmentType": "POINT",
     "id": "org.unidata.mdm.data[RECORD_UNMERGE_TRANSITION]"
   },
   {
     "segmentType": "CONNECTOR",
     "id": "org.unidata.mdm.data[RELATIONS_UNMERGE_CONNECTOR]"
   },
   {
     "segmentType": "CONNECTOR",
     "id": "com.unidata.mdm.classifiers[CLASSIFICATIONS_UNMERGE_CONNECTOR]"
   },
   {
     "segmentType": "POINT",
     "id": "org.unidata.mdm.data[RECORD_UNMERGE_PERSISTENCE]"
   },
   {
     "segmentType": "CONNECTOR",
     "id": "org.unidata.mdm.matching.data[RECORD_UNMERGE_MATCHING_CONNECTOR]"
   },
   {
     "segmentType": "POINT",
     "id": "org.unidata.mdm.data[RECORD_UNMERGE_AUDIT]"
   },
   {
     "segmentType": "FALLBACK",
     "id": "org.unidata.mdm.data[RECORD_UNMERGE_AUDIT_FALLBACK]"
   },
   {
     "segmentType": "FINISH",
     "id": "org.unidata.mdm.data[RECORD_UNMERGE_FINISH]"
   }

 ]
},
{
 "startId": "org.unidata.mdm.data[RELATION_UNMERGE_START]",
 "subjectId": "",
 "description": "org.unidata.mdm.data.relation.unmerge.start.description",
 "segments": [
   {
     "segmentType": "START",
     "id": "org.unidata.mdm.data[RELATION_UNMERGE_START]"
   },
   {
     "segmentType": "POINT",
     "id": "org.unidata.mdm.data[RELATION_UNMERGE_TIMELINE]"
   },
   {
     "segmentType": "POINT",
     "id": "org.unidata.mdm.data[RELATION_UNMERGE_INDEXING]"
   },
   {
     "segmentType": "POINT",
     "id": "org.unidata.mdm.data[RELATION_UNMERGE_PERSISTENCE]"
   },
   {
     "segmentType": "FINISH",
     "id": "org.unidata.mdm.data[RELATION_UNMERGE_FINISH]"
   }
 ]
},
{
 "startId": "com.unidata.mdm.classifiers[CLASSIFICATION_UNMERGE_START]",
 "subjectId": "",
 "description": "com.unidata.mdm.classifiers.classification.unmerge.start.description",
 "segments": [
   {
     "segmentType": "START",
     "id": "com.unidata.mdm.classifiers[CLASSIFICATION_UNMERGE_START]"
   },
   {
     "segmentType": "POINT",
     "id": "com.unidata.mdm.classifiers[CLASSIFICATION_UNMERGE_TIMELINE]"
   },
   {
     "segmentType": "POINT",
     "id": "com.unidata.mdm.classifiers[CLASSIFICATION_UNMERGE_INDEXING]"
   },
   {
     "segmentType": "POINT",
     "id": "com.unidata.mdm.classifiers[CLASSIFICATION_UNMERGE_PERSISTENCE]"
   },
   {
     "segmentType": "FINISH",
     "id": "com.unidata.mdm.classifiers[CLASSIFICATION_UNMERGE_FINISH]"
   }
 ]

}

Security Resource for Custom Batch Operations

Resources have been added to the "Records Management":

Custom batch operations that include importing, exporting, modifying, and deleting records.

If there are no response rights, the user receives a response :

The user [user1] does not have sufficient rights to (import/export/modify/delete - depending on the selected operation) records. Access is denied. ( The user [user1] has no rights to (import/export/modify/delete records). Access denied.)

UI Roles → Access rights → Records Management - there are rights settings for added resources.

In the absence of rights, the corresponding actions ("Import", "Export", "Modification", "Deletion") are inactive.

SE/EE. Displaying Instances of Complex Attributes

Added the complexValues field to the DTO and RO objects of the search result.

SE/EE Ability to Attach Multiple Files

Added support for arrays of file type attributes.

• org.unidata.mdm.rest.system.upload.attachment.directory

• org.unidata.mdm.rest.system.upload.attachment.memory.threshold

• org.unidata.mdm.rest.system.upload.attachment.max.size

Have been changed to:

- ``org.unidata.mdm.core.upload.attachment.directory``
- ``org.unidata.mdm.core.upload.attachment.memory.threshold``
- ``org.unidata.mdm.core.upload.attachment.max.size``

A field has been added:

• org.unidata.mdm.core.file.max.count

It sets the maximum number of files for the attribute (10 by default).

Notification module org.universe.mdm.notifications

The segment for sending notifications about inserting a record com.universe.mdm.notifications[SEND_UPSERT_NOTIFICATION] :ref:`` notifications module com.universe.mdm.notifications <mdm_notifications>`` has changed:

• Added system parameters:

  • com.universe.universe.mdm.notifications.notifications.upsert.version - segment version used;

  • com.universe.universe.mdm.notifications.notifications.notifications.fetchLookupLinkAttributesAsObjects - fill/not fill lookupLinkAttributes field in notification;

  • com.universe.mdm.notifications.notifications.notifications.showAbsentLookupLinkAttributes - fetch/unfetch missing attributes as null in lookupLinkAttributes (not included in ChangeDiffResult);

  • com.universe.mdm.notifications.notifications.notifications.fetchRecordKeys - fill in/not fill in the attributes of the lookupLinkAttributes field of the recordKeys field.

• The following objects were added:

  • NotificationExternalKey - com.universe.mdm.notifications.types.NotificationExternalKey;

  • LookupLinkAttribute - com.universe.mdm.notifications.types.upsert.LookupLinkAttribute;

  • LookupLinkValueChanges - com.universe.mdm.notifications.types.upsert.LookupLinkValueChanges.

• Objects changed:

  • RecordChanges - added lookupLinkAttributes field, in version 1 always null - com.universe.mdm.notifications.notifications.types.upsert.RecordChanges;

  • RecordDiffResult - added lookupLinkAttributes field, in version 1 always null - com.universe.mdm.notifications.notifications.types.upsert.RecordDiffResult;

  • UpsertRecordEvent - added lookupLinkAttributes field, in version 1 always null - com.universe.mdm.notifications.notifications.types.upsert.UpsertRecordEvent;

  • NotificationRecordKeys - externalKeys field now lists NotificationExternalKey, not strings - com.universe.mdm.notifications.notifications.types.NotificationRecordKeys.

Note

After the update, the com.universe.mdm.notifications.notifications.notifications.upsert.version=1 parameter must be set to com.universe.mdm.notifications.notifications.upsert.version=1 in order to use the old version of the segment.

Enterprise Edition

Enterprise Edition

6.10 Version

Module com.unidata.mdm.rest.v1.classifiers

• REST endpoint: POST /api/v1/classifiers/model/reindex was added.

Displaying Classification Data in Search Results

The REST endpoint POST api/v1/classifiers/classification/search is supposed to be used to search for classifications of retrieved records. The point has been finalized for better usability.

• Classification attribute values are stored in a map within the document. It should be transferred to the search response, thus enriching it with mapped values and other necessary information. Classification attributes are now added to the search response in the same way as record fields.

• Classification attributes are rendered in the same order in which they are defined in the model.

• The fieldDisplayName field was added to get the displayed attribute name without loading the classifier model. For homogeneity, the field is added to the general search output (records, relations, ...).

• The fieldValueType field was added to get the attribute type without loading the classifier model. For homogeneity, the field was added to the general search output (records, relationships, ...).

• Extended the search API (v2/search, v1/classifiers/classification/search) by adding additional fields, backward compatibility is not broken.

• Dedicated component for enriching search output SearchResultHitModifierCommonComponent.