Agile Workflow

How does it all fit together in a workflow when a data migration project is underway? 

How does it even start up the first time hopp is put to the task in a new installation context?

How can changes to the mapping be done concurrently with ongoing migration iterations and tests of the migration results? 

From scratch, little by little

The complete target-source mapping complements each other. The Source Map sits on top of the Target Map. This may give the impression that the entire target specification must be completed before progressing to the Source Map. However, this impression is far from the truth.

In fact, the iterative nature of the entire framework makes it entirely feasible to proceed in a less daunting and more efficient manner. 

The best practice is to start out with the Target Map or just with one Business Object hierarchy and even within this one just a minor part of the entire hierarchy. Moreover, quickly proceed to the Source Map of the same, limited Business Object hierarchy.

In this way, it is very fast to start the iterative process of the entire framework. The framework calls for this vertical approach where elements in the specification in an incremental fashion are added little by little in Target and Source Maps. Due to the ease of iterating the entire process of migration specification modifications, code generation, deployment and execution, new areas can constantly and seamlessly be added, and existing areas deepened and enhanced.

Up and running

Once a first limited Business Object has paved the way and a migration track is operational and the results are shown in the Portal, the migration project is in process and the normal workflow is in place. This is an easy iterative flow involving the tasks shown below:

• Mapping is modified. New Business Objects may be added, existing Business Objects may be modified. Mapping is published and imported upwards in the mapping type chain as described. Modifications are constantly checked into the common mapping repository
• New code is generated. A user gets the latest version of the mapping from the common repository and publishes the generator input and runs the code generators
• New manual rules may be developed, and/or existing rules modified
• The engines are built and deployed in the Runtime environment
• The relevant track is reset so the new engines are loaded, and execution iterated as needed. Either everything or cherry-picking business objects in one way or another
• The migration results and events are shown in the Portal
• Based on the Portal feedback flows back and results in modifications to the mapping and the cycle repeats

This iterative workflow is typical when working with hopp. Moreover, it can be very fast. Not counting the time needed for modifying mapping and manual rules, the inherent processes performed by hopp to generate, build, deploy and execute can be measured in minutes.

It is common for a migration team to iterate the migration process in this manner many times daily.

Feedback and problem tracing

Feedback may be introduced into the workflow from several sources. Unexpected occurrences of certain events may be shown in the Portal; a test in a target system test instance may uncover issues, etc.

hopp provides strong support for tracing problems, right from the event or problem in the target system, through the data flowing through the migration steps and back to the area in the mapping in need of a review.

This support is greatly enhanced by the concept of Business Objects:

• Events in the Portal and/or problems in the target system are directly related to a given business object
• In the Portal, all information for any specific business object is easily located. This information includes:
  1. All events that occurred during the migration
  2. All data for each of the migration steps (source data, export result, transformation result and target result) 
• For each of the migration steps the data is clearly organized in the same hierarchy of child Business Objects as defined in the mapping

In this manner, hopp and the partitioning of the entire mapping and execution flow in Business Objects provide efficient means of analyzing problems and locating the exact area in the mapping in need of a review.

Snapshots

While the above workflow enables the migration team to iterate to a very high degree, in many instances it is relevant to freeze a snapshot of the data migration iteration at a certain point in time.

Typically, whenever the target data result is unloaded from hopp and delivered to a test instance of the target system, it is beneficial to freeze a snapshot exactly corresponding to the data that were offloaded and delivered.  When the target system test instance is then undergoing tests to verify the quality of the migrated data, this snapshot can be used as described above to analyze and trace any problems uncovered by these tests. 

At the same time, iterations in the original can resume allowing the overall forward process of the entire agile workflow – now including corrective actions fed back from the target system test instance.

A special case of this snapshot is when the migration project is finished and the migrated data are finally delivered to the production instance of the target system. In this case, the snapshot may be kept for an extended period. Analysis of any future issues in the target system may be significantly supported by the knowledge of exactly what data was delivered by the migration project as well as all migration events.  

Freezing any migration snapshot in this manner is a simple matter of copying the databases and other artefacts that comprise the Runtime track used by the migration iterations for the project.