Description of Modular Infrastructure

Technical Infrastructure

Universe products use Java 9 Jigsaw + proprietary extensions for cross-module communication. This solution has the advantages of:

  • Jigsaw is part of the Java platform.

  • Jigsaw fully delivers what is required: modularity in the form of jar files without introducing additional complexity in the form of new APIs to integrate and use and additional infrastructure to build into.

  • Proprietary extensions do not introduce unnecessary dependencies, are lightweight and simple: as opposed to OSGi's complex and cumbersome inter-module management platform.

  • Running module code in-process avoids the "constant added complexity" problem inherent in microservice architectures. This helps for performance.

Module Management

To manage the module subsystem, the system uses ModuleService, which is a module logger, a source of information about modules and a means of module subsystem management.

ModuleService Schema

Figure 1. ModuleService Schema

It is the job of the ModuleService to load and run modules during system startup.

  1. During startup, only the Spring root context is automatically started and only com.unidata.mdm.core is scanned.

  2. Once the core is fully read and initialized, the ModuleService scans the archives looking for classes - descriptors implementing org.unidata.mdm.common.types.Module and the jar files specified in the descriptor in the META-INF/MANIFEST.MF jar file (OSGi style, see below).

  3. For all found modules, a dependency tree is built, packages are scanned and Spring child contexts are initialized in order of mutual dependencies.

ModuleService stores the installation status of each module in the database. The status table stores:

  • The version of the installed module (MANIFEST.MF or descriptor)

  • String module ID (MANIFEST.MF or descriptor)

  • Module tag (part of the ID indicating possible customization for the customer, MANIFEST.MF or descriptor)

  • Installation date

  • Installation result (success/failure)

  • Path to the file from which the installation was run

At system startup, modules that were transferred for installation and failed during the installation are no longer started. On the UI the status of such modules is marked accordingly.

The service also performs the tasks of installing modules in the cluster - downloading, verification and distribution of the installable archive between the cluster nodes.

Modules Interaction

Explicit dependencies are defined between modules. Deleting a module on which other modules depend also causes the deletion of dependent modules. The presence of unsatisfied dependencies causes the module to fail to be installed on the system.

Support for standard Spring annotations like @Autowire or JSR330 @Inject is expected, as they are a necessary and familiar element of everyday development. It's about modifying the Spring ContextFactory so that module services can be injected into class instances of other modules.

Alternatively, if the approach above fails, you can use dynamic retrieval of service instances from other modules, based on boolean<M extends Module> ModuleService.isActive(Class<M> module) and <T> T Module.getService(Class<T> serviceType), by handling @Module annotations, or similar methods by module id (string). It is assumed that if a module is active, all its services are also active.

Modules interact with each other and with client code only through public - or, for internal system needs, semi-public (*Ext) interfaces.

Module Description

A module is a jar file that can be installed into the platform, and whose details will be visible on the UI in the "About" section. The distinctive feature of a module is the presence in the module archive of the org.unidata.mdm.common.types.Module interface implementation specified in the descriptor in META-INF/MANIFEST.MF of the module jar file.

A module can be executable code, a set of resources, a data model patch, a set of REST or SOAP endpoints, or any other functionality. It can also be completely custom modules written for a specific customer.

In the source code, the module is formalized as a separate gradle (IDEA / Eclipse) subproject, which is built in a separate jar file. Which particular module will be included in the build for a particular customer depends on the build configuration.

The module performs installation, migration and environment update tasks in the install() method, which is called by ModuleService during the installation / update process.

The module performs the tasks of removing information about itself on the system via the uninstall() method.

The start() method is executed during module startup and may be needed for non-standard initializations.

The stop() method is called when the module is stopped for some reason and is used for non-standard resource release.

Rules for Modules

Several rules apply to all modules:

  • A module has a string id that matches the root package name of the module (e.g. org.unidata.mdm.data). By this id it is accessible from the ModuleService.

  • The module must be able to return information about itself - its version consisting of major.minor.revision, a localized name, a localized description, a tag potentially indicating build specificity, the integration points it supports (Pipelines).

  • If the module uses a database, it maintains its data in a fully isolated schema (e.g. org_unidata_mdm_data or org_unidata_mdm_core) whose name is the same as the module id, where the dots are replaced by an underscore.

  • If the module uses a database, it keeps its own connection pool, which is however available for configuration from backend.properties in the style org.unidata.mdm.data.datasource.minPoolSize = 10.

  • The module populates MANIFEST.MF in a similar style to the OSGi specification. The following metadata attributes are supported in META-INF/MANIFEST.MF:

Manifest-Version: 1.0
Unidata-Module-Class: org.unidata.mdm.data.module.DataModule

  • The module maintains its data schema migrations independently. Schema installation / update is performed when the module is installed / updated by the module itself (it is suggested to go to database-migrator).

  • The module maintains its i18n resources independently.

  • Module defines its own exception classes (Unidata Exception API refactoring will be required).

  • The module publishes supported features (ModuleFeature) whose activity or inactivity is governed by a valid license. To do this, the module interacts with LicenseService at startup, which allows you to read the necessary information from the license.

  • The module publishes the supported integration points (Segment(s) in Pipelines).

  • The module stores its private settings in module.properties, which are not available in other contexts for other modules and are not editable at runtime.

  • All PostgreSQL extensions are installed in the public schema, which is automatically added to the search_path of all datasource configurations of all modules.

Module Format

All modules must adhere to the same standards in code structuring.

  • Module project names begin with the module root package name org.unidata.mdm.* or com.unidata.mdm.* (org.unidata.mdm.core, org.unidata.mdm.data, org.unidata.mdm.dq, org.unidata.mdm.data.rest, org.unidata.mdm.data.soap, etc.).

  • All projects place code in packages with a common prefix - com.unidata.mdm for commercial modules and org.unidata.mdm for open source modules.

  • The next element of the path must be a subpackage of details (core, model, dq, search, data, matching, job, classifier etc.) indicating the purpose of the module. This package name segment is the root package of the module and forms the module id.

  • Private classes of service implementations, DAOs and similar code should be in the next level impl subpackages - com.unidata.mdm.dq.service.impl, com.unidata.mdm.dq.dao.impl, etc.

  • The next part of the path should be subpackages of the class categories. See the table below:

Table 1. Description of subpackages of class categories

type

Representation of data in the middle layer, intermediate and service types not intended to be transported up or down. Sub-packages of granularity (org.unidata.mdm.core.type.calculables, org.unidata.mdm.core.type.module, org.unidata.mdm.core.type.data, etc.) may follow below. That is the place of the domain model

dao

If the module interacts with the database

po

If the module interacts with the database. The following may be followed by detail subpackages (org.unidata.mdm.core.po.security, org.unidata.mdm.core.po.module, etc.)

service

Public service interfaces. Implementations are located in the impl subpackage and have the corresponding suffix

dto result

Transfer - objects (return types). Subpackages of detail (org.unidata.mdm.core.dto.security, org.unidata.mdm.core.dto.module, etc.) may follow. The classes must end with Result and have a common module prefix

util

Uncategorized static utilities

сonvert

Data view conversion utilities

module

Types related to the metadata of the module itself

exception

Module exception IDs + local types

context

Contexts (parameters of service calls). Classes must end with Context and have a common module prefix

configuration

Anything related to module configuration, including classes annotated @Configuration

migration

Data schema migrations

serialization

For objects used in serialization

The purpose of this arrangement is to concentrate all the implementation details of a module within the module itself, making it airtight, and to define a common pattern, allowing rapid entry of each team member into any area of the system, including previously unknown areas.

Modules have several characteristics:

  • Modules come in mandatory and optional forms. Mandatory modules are always present in the system in one or another realization. Optional ones may not be present. Their operability is regulated by the product license.

  • Modules can be standard or non-standard (edition). This is also regulated by the license metadata.

  • Customization of a module may be indicated by a tag when it has a value other than "standard".

  • Modules can store private metadata in the license data space.