migFx Support

This post is meant to float the idea of Automated Execution in the Director. This is hopp's initial suggestion on how to implement such a feature. We hope to receive valuable feedback in order to mature the idea into a proposal to include in the migFx road map.

Use case

One of the major characteristics of migFx is the ease of iteration. The Studio, Engine and Director components in unison significantly simplify the task of iterating the migration of selected Business Object instances.

The complete list of steps in a migFx iteration is

1. Modify mapping (Source- and or Target) as needed
2. Get latest version from Repository to retrieve modifications by other team members  
3. Validate and correct validation errors (if any)
4. Publish mapping to generator and generate Engines
5. Implement any new Rules or Bags or modify existing as needed
6. Deploy engines to the Director Runtime environment
7. Restart Track to load new Engine
8. Setup new Engines to synchronize Engine code and Staging Database
9. Ensure Runtime Parameters are set to correct values
10. Load Dynamic and Translation Valuesets
11. Load Source Tables
12. Load Source Views
13. Run Export
14. Run Import
15. Publish to Tracker 
16. Unload migration results

In addition, migFx iterations are typically performed in one of these distinct contexts

1. Individual iterations of migration subsets by developers modifying mapping and/or rule implementations to verify and debug modifications
2. Internal iterations of the entire migration to asses the current quality and state of the migration project
3. Official iterations of the entire migration involving stakeholders outside the migration team, such as Business Users and Testers. Typically performed upon receipt of a new set of source data 
4. Go-live iteration
5. Correctional iterations of subsets in order to correct issues in situation 2 to 5

At present, migFx offers no facilities for Automated Execution. This makes it necessary to manually prepare and manually monitor the execution of the subsequent jobs in the Director in order to launch jobs as predecessors finish. Especially regarding the internal iterations - of which there ideally is quite a few - this places strain on the team members assigned to the actual execution of complete, internal migration iterations.

Scope

Given the use case above, it is clear that some of the steps are job running in the Director runtime and as such obvious candidates for automation. Other steps are more or less manual and to a lesser degree suitable for automation.

Based on these considerations, our ambition and scope for automation is initially limited to handle

• Automatic execution of dependent Director jobs - but not the manual steps leading up to these jobs. Referring to the list above, this would only include the steps concerned with actually running the migration in Director - from 'Load Dynamic and Translation Valuesets' and out
  
  
• The iterations concerned with the entire migration. These usually takes quite some time and in the present situation requires somebody to monitor job execution in order to submit subsequent jobs when ready. This would exclude the 'Individual migrations' as well as the 'Correctional iterations' above from the initial scope of automated execution

Given this scope, the workflow for an initial implementation would be like

1. One team member is designated as in charge of the upcoming iteration and the team agrees on a deadline for modifications to be included.
   
   
2. Before the agreed deadline, all other members of the migration team finalize their of test-iterations, check in their modifications in Studio, and Visual Studio and finally signal ready to the designated team member
   
   
3. The designated team member 
   1. Does a get latest version in Studio, validates and publishes
   2. Does a get latest in Visual Studio, generates and deploys new engines
   3. Restarts the Track and verifies Runtime parameters (can also be delegated to another team member)
   4. Launches the automated execution of
      1. Load Dynamic and Translation Valuesets
      2. Load Source Tables
      3. Load Source Views 
      4. Run Export
      5. Run Import
      6. Publish to Tracker 
      7. Unload migration results

Launching an automated iteration as above will typically be done 

• For internal iterations: As the last action before the end of the working day for the iteration to run overnight and be ready for evaluation the next morning
• For official and Go-live iterations, when external dependencies are resolved (such as the reception of new source data)

Errors and mistakes

It is true that the present situation today with no automated execution does place an unnecessary strain on the team member that is tasked with overseeing especially the internal test iterations. These are typically run overnight, and somebody must remain awake (or repeatedly wake up) in order to launch jobs when ready. In practice, this places a limit to how often internal test iterations can reasonably be executed. This is a shame, because frequent iterations undoubtedly lead to improved quality.

However, having a set of human eyes on the progressing execution does have the advantage of a constant assessment of the quality of the running migration. It happens frequently that the iteration is manually paused due to some issue rendering the results meaningless, some corrective action performed, and the iteration resumed to a significantly improved conclusion.

Automated Execution may in similar situations result in 'wasted' iterations, where the migration team upon completion the next morning discovers that the iteration is useless. Hopefully, Automated Execution will mitigate this by making it possible to perform more iterations without wearing out the migration team.    

Design

As outline above, the initial scope of Automated Execution is limited to initiate and monitor the steps of a complete migration iteration. The steps to be executed depends on whether the migration project is an 'Export Only' project or an 'Export and Import' project. The steps and their dependencies are illustrated below.

Each of the boxes in the figure above illustrates a step of an Automated Execution. Some of the step consists of only 1 job, other steps of multiple jobs, all of which must complete before releasing any succeeding steps.

There is a duality here that is important to understand and master in the design of Automated Execution The migFx Director already contains the functionality to parameterise, execute and monitor jobs. But an automated execution is not just a series of jobs, but rather a series of steps, each of which consists of one or multiple jobs, that must complete before subsequent steps can be released/submitted.

New: Restart from scratch 

The existing job scheduling infrastructure can handle the execution of the jobs of each step as outlined above. However, in case one or more jobs of an Automated Execution fails, the existing ability to restart a failed job from the point of failure may not be enough. In many cases, it will be desirable to be able to perform some corrective action and re-run the failed job from start - not just from point of failure.

It is not possible in the present job scheduling infrastructure to restart a failed/canceled job from scratch. In the present infrastructure, a new job must be submitted in this case - raising the question how this new job would relate back to the governing Automated Execution of the original job.

To overcome this limitation, we suggest to extend the existing functionality to either restart a job with an option to restart from point of failure/cancellation (as today) or restart from scratch (new). Restarting from scratch would simply mean erasing the committed restart restart information for the job before submitting a new job instance as usual.

This new restart from scratch option will be available in the Job List for all jobs, not limited to jobs submitted by an Automated Execution.

Automated Execution

An Automated Execution is a new element in the Director, Client and Runtime. In the client, the Automated Execution will probably end up having is own, dedicated panel to create new and monitor existing Automated Executions.

Creating a new Automated Execution includes

• Setting up the Automated Execution
  • Execute now
  • Execute on Trigger
    • Time
    • External (some external dependency, for instance from OPC or similar)
• Setting up each of the steps:
  • To skip or include the step
  • If the step is included, parameterise the step. For instance, Which (or all) Valuesets to load, which (or all) Source Tables to Load etc 

An Automated Execution can assume the following states below until completion

Director UI

A new panel for Automated Execution will be added to the Director UI. The panel shows a list of Automated Executions with their current state and enables the creation and parameterisation of new Automated Executions.

Parameterisation

When setting up an automated execution, each job must be parameterised:

• Should the job be included or skipped
• If included, which (or all) objects should be processed by the job. 
• The parameterisation of each job should ideally be similar to the way the job is parameterised when submitted on its own today

In addition, the automated execution itself could be parameterised. For instance to define the time on which the execution should start, but perhaps other parameters could be relevant (callout for suggestions, start on OPC trigger?)

Monitoring

The individual jobs constituting the automated execution will appear as all other jobs in the Director job list. Once a given step is ready to run, the jobs of the step will be submitted (not at once in a held state). This allows for the edit of parameters of steps that has not started yes, as described above. 

The Automated Executions will appear as a list in a new panel in the Director UI. In this list, it will be possible to drill down:

• Automated Execution
  • Steps in Automated Execution
    • Jobs in Step

The actions and information available on each level in this drill-down are: