This post is meant to float the idea of Automated Execution in the Director. This is hopp's initial suggestion on how to implement such a feature. We hope to receive valuable feedback in order to mature the idea into a proposal to include in the migFx road map.
One of the major characteristics of migFx is the ease of iteration. The Studio, Engine and Director components in unison significantly simplify the task of iterating the migration of selected Business Object instances.
The complete list of steps in a migFx iteration is
In addition, migFx iterations are typically performed in one of these distinct contexts
At present, migFx offers no facilities for Automated Execution. This makes it necessary to manually prepare and manually monitor the execution of the subsequent jobs in the Director in order to launch jobs as predecessors finish. Especially regarding the internal iterations - of which there ideally is quite a few - this places strain on the team members assigned to the actual execution of complete, internal migration iterations.
Given the use case above, it is clear that some of the steps are job running in the Director runtime and as such obvious candidates for automation. Other steps are more or less manual and to a lesser degree suitable for automation.
Based on these considerations, our ambition and scope for automation is initially limited to handle
Given this scope, the workflow for an initial implementation would be like
Launching an automated iteration as above will typically be done
Errors and mistakes
It is true that the present situation today with no automated execution does place an unnecessary strain on the team member that is tasked with overseeing especially the internal test iterations. These are typically run overnight, and somebody must remain awake (or repeatedly wake up) in order to launch jobs when ready. In practice, this places a limit to how often internal test iterations can reasonably be executed. This is a shame, because frequent iterations undoubtedly lead to improved quality.
However, having a set of human eyes on the progressing execution does have the advantage of a constant assessment of the quality of the running migration. It happens frequently that the iteration is manually paused due to some issue rendering the results meaningless, some corrective action performed, and the iteration resumed to a significantly improved conclusion.
Automated Execution may in similar situations result in 'wasted' iterations, where the migration team upon completion the next morning discovers that the iteration is useless. Hopefully, Automated Execution will mitigate this by making it possible to perform more iterations without wearing out the migration team.
As outline above, the initial scope of Automated Execution is limited to initiate and monitor the steps of a complete migration iteration. The steps to be executed depends on whether the migration project is an 'Export Only' project or an 'Export and Import' project. The steps and their dependencies are illustrated below.
Each of the boxes in the figure above illustrates a step of an Automated Execution. Some of the step consists of only 1 job, other steps of multiple jobs, all of which must complete before releasing any succeeding steps.
|Load Valuesets||One job loading all selected valusets|
|Load Source Tables||Multiple jobs, one per Source Table|
|Load Source Views||One job loading all Source Views, resolving dependencies|
|Export||Multiple jobs, one per Business Object|
|Import||One job importing all selected Business Objects, resolving dependencies|
|Unload||One job unloading all selected Business Objects|
|Publish||One job publishing all selected Business Objects|
There is a duality here that is important to understand and master in the design of Automated Execution The migFx Director already contains the functionality to parameterise, execute and monitor jobs. But an automated execution is not just a series of jobs, but rather a series of steps, each of which consists of one or multiple jobs, that must complete before subsequent steps can be released/submitted.
New: Restart from scratch
The existing job scheduling infrastructure can handle the execution of the jobs of each step as outlined above. However, in case one or more jobs of an Automated Execution fails, the existing ability to restart a failed job from the point of failure may not be enough. In many cases, it will be desirable to be able to perform some corrective action and re-run the failed job from start - not just from point of failure.
It is not possible in the present job scheduling infrastructure to restart a failed/canceled job from scratch. In the present infrastructure, a new job must be submitted in this case - raising the question how this new job would relate back to the governing Automated Execution of the original job.
To overcome this limitation, we suggest to extend the existing functionality to either restart a job with an option to restart from point of failure/cancellation (as today) or restart from scratch (new). Restarting from scratch would simply mean erasing the committed restart restart information for the job before submitting a new job instance as usual.
This new restart from scratch option will be available in the Job List for all jobs, not limited to jobs submitted by an Automated Execution.
An Automated Execution is a new element in the Director, Client and Runtime. In the client, the Automated Execution will probably end up having is own, dedicated panel to create new and monitor existing Automated Executions.
Creating a new Automated Execution includes
An Automated Execution can assume the following states below until completion
|Created||The Automated Execution has been parameterised and created and is ready to run. In this state it is possible to edit the parameters of all steps|
|Running||The Automated execution is running, submitting, monitoring jobs and releasing new steps when ready. In this state it is not possible to edit the parameters of any step|
|Paused||User can pause and resume the Automated Execution. When Paused, steps will not be released even when ready. In this state it is possible to edit the parameters of steps not released yet|
|Halted||The Automated Execution cannot continue due to faulted or cancelled jobs. Jobs must be restarted and run to completion for execution to continue. In this state it is possible to edit the parameters of steps not released yet (job restart should be available fro the Automated Execution panel, see below)|
|Completed||All Steps and jobs have run to completion.|
It can be considered, whether it should possible to rerun a completed job from scratch, thereby resuming the remaining Automated Execution from that point. However, the same result can easily be obtained by parameterising and submitting a new Automated Execution
(callout: comments welcome)
A new panel for Automated Execution will be added to the Director UI. The panel shows a list of Automated Executions with their current state and enables the creation and parameterisation of new Automated Executions.
When setting up an automated execution, each job must be parameterised:
In addition, the automated execution itself could be parameterised. For instance to define the time on which the execution should start, but perhaps other parameters could be relevant (callout for suggestions, start on OPC trigger?)
The individual jobs constituting the automated execution will appear as all other jobs in the Director job list. Once a given step is ready to run, the jobs of the step will be submitted (not at once in a held state). This allows for the edit of parameters of steps that has not started yes, as described above.
The Automated Executions will appear as a list in a new panel in the Director UI. In this list, it will be possible to drill down:
The actions and information available on each level in this drill-down are:
(also available in Job List)