Target and Source Map

Modified on Mon, 09 Oct 2023 at 11:06 PM

A fundamental element in any data migration scenario is the way it is specified how to migrate the source data to the target data.

The success of any data migration is directly linked to the quality of this specification and how it is translated into the executable that performs the actual data migration. This is only underlined by the fact that this specification often grows to enormous size and complexity. It is difficult to maintain validity and coherence in the specification itself. Most importantly: In many cases, it proves impossible to maintain complete fidelity in the consistency between the specification and the executable.

A key component in hopp is Studio. This component is a dedicated multiuser productivity application providing a complete and consistent interface to produce the mapping. Using the Studio, a team of users can collaborate to produce the mapping for the framework executables.

The Studio contains rich cross-reference and cross-validation functionality to ensure a very high degree of consistency and coherence in the mapping. Most importantly, the Studio enforces mapping of an extremely structured nature. In fact, the specifications are so structured that they serve as input to a code generator that generates the migration executables.

It is a key quality of hopp that the consistency between the mapping and the actual executable is inherently guaranteed.

Collaboration

Studio is a Windows application running locally on a PC or laptop. The mapping produced by the Studio is a collection of (xml) files residing locally on the user’s machine.

While this enables an individual user to work on a given mapping locally on his/her Windows machine, the Studio can be backed by a central repository (a SQL Server database). The repository provides the functionality necessary for a team to collaborate on the same mapping (checkout, check-in and get-latest).

The investment in the mapping can be safeguarded by implementing a suitable backup scheme using the given facilities in SQL Server.

Mapping types

In the case of repeated data migrations from varying source systems to the same target system, it is crucial that a clear separation exists between the mapping for source data and the mapping for target data.

Using Studio, the mapping for any data migration is separated into two different mapping types ensuring the highest degree of reuse of these specifications from migration project to migration project.

Target Map

The Target Map is founded on the description of the Target data.

The Target Map eliminates internal references and data that can be derived from other data and exposes the data that cannot be derived and thus must be received. In addition, the Target Map can implement a wide host of runtime validations to ensure the highest quality possible of the target data being produced by the data migration.

The Target Map is strongly linked to the target system and this mapping can be reused in all migrations to the same target system. The value of improving/extending the Target Map is retained over time, from project to project.

From Studio, it is possible to export the Target Map in two ways:

As an interface specification that can be imported into Studio when working on the Source Map (see below)
As a complete, structured specification that serves as input for the engine generator generating the target migration engine

Source Map

Source Map is based on both the source data descriptions as well as the data requirements exposed by the Target Map.

While the target mapping exposes the data that must be received, it does so in terms of the target system. In addition, all validation is founded on value sets known by the target system.

On the other hand, the Source Map describes how - based on the source data - to produce the data required by the target map.

Finally, the export can be exported from the Studio as a complete, structured specification as input for the engine generator generating the export engine.

Quality tools

Mapping commonly grows to significant size and complexity. In many cases users need to communicate around the specifications – for instance, users with knowledge of the target system may need to communicate with users with knowledge of the source system.

Studio works as a frame around the different mapping types providing rich facilities useable across the different mapping types.

Cross-reference

Studio provides where-used, cross-referencing capabilities. These capabilities are useful as the mapping grows in size, providing support for where-used analysis and general control over complex mapping often lacking in other specification tools.

Validation

Using Studio any user can perform a complete validation of the consistency of the mapping. Validation errors indicate that code generation may fail or the generated code may fail to build.

For this reason, the validation is an integrated part of the workflow. In addition, the validation in combination with the checkout/check-in collaboration facilities provides support for what-if scenarios.

A user can check out an item (or any number of items), perform some modification – for instance import a new version of the target data structure - and then perform a validation. The validation report will give a good indication of the impact of the changes, and in case of unforeseen consequences for the consistency of the mapping, the user can simply undo the previous checkout and revert to the previous state of the mapping.

Reporting

Studio provides a palette of reports common for all mapping types as well as reports specific for each mapping type. The reports provide support for communication with other users not in contact with Studio.

Using Studio, items can be annotated with descriptions and comments adding value to extracted reports.