This post is meant to float the idea of Automated Execution in the Director. This is hopp's initial suggestion on how to implement such a feature. We hope to receive valuable feedback in order to mature the idea into a proposal to include in the migFx road map.
Use case
One of the major characteristics of migFx is the ease of iteration. The Studio, Engine and Director components in unison significantly simplify the task of iterating the migration of selected Business Object instances.
The complete list of steps in a migFx iteration is
Modify mapping (Source- and or Target) as needed
Get latest version from Repository to retrieve modifications by other team members
Validate and correct validation errors (if any)
Publish mapping to generator and generate Engines
Implement any new Rules or Bags or modify existing as needed
Deploy engines to the Director Runtime environment
Restart Track to load new Engine
Setup new Engines to synchronize Engine code and Staging Database
Ensure Runtime Parameters are set to correct values
Load Dynamic and Translation Valuesets
Load Source Tables
Load Source Views
Run Export
Run Import
Publish to Tracker
Unload migration results
In addition, migFx iterations are typically performed in one of these distinct contexts
Individual iterations of migration subsets by developers modifying mapping and/or rule implementations to verify and debug modifications
Internal iterations of the entire migration to asses the current quality and state of the migration project
Official iterations of the entire migration involving stakeholders outside the migration team, such as Business Users and Testers. Typically performed upon receipt of a new set of source data
Go-live iteration
Correctional iterations of subsets in order to correct issues in situation 2 to 5
At present, migFx offers no facilities for Automated Execution. This makes it necessary to manually prepare and manually monitor the execution of the subsequent jobs in the Director in order to launch jobs as predecessors finish. Especially regarding the internal iterations - of which there ideally is quite a few - this places strain on the team members assigned to the actual execution of complete, internal migration iterations.
Scope
Given the use case above, it is clear that some of the steps are job running in the Director runtime and as such obvious candidates for automation. Other steps are more or less manual and to a lesser degree suitable for automation.
Based on these considerations, our ambition and scope for automation is initially limited to handle
Automatic execution of dependent Director jobs - but not the manual steps leading up to these jobs. Referring to the list above, this would only include the steps concerned with actually running the migration in Director - from 'Load Dynamic and Translation Valuesets' and out
The iterations concerned with the entire migration. These usually takes quite some time and in the present situation requires somebody to monitor job execution in order to submit subsequent jobs when ready. This would exclude the 'Individual migrations' as well as the 'Correctional iterations' above from the initial scope of automated execution
Given this scope, the workflow for an initial implementation would be like
One team member is designated as in charge of the upcoming iteration and the team agrees on a deadline for modifications to be included.
Before the agreed deadline, all other members of the migration team finalize their of test-iterations, check in their modifications in Studio, and Visual Studio and finally signal ready to the designated team member
The designated team member
Does a get latest version in Studio, validates and publishes
Does a get latest in Visual Studio, generates and deploys new engines
Restarts the Track and verifies Runtime parameters (can also be delegated to another team member)
Launches the automated execution of
Load Dynamic and Translation Valuesets
Load Source Tables
Load Source Views
Run Export
Run Import
Publish to Tracker
Unload migration results
Launching an automated iteration as above will typically be done
For internal iterations: As the last action before the end of the working day for the iteration to run overnight and be ready for evaluation the next morning
For official and Go-live iterations, when external dependencies are resolved (such as the reception of new source data)
Errors and mistakes
It is true that the present situation today with no automated execution does place an unnecessary strain on the team member that is tasked with overseeing especially the internal test iterations. These are typically run overnight, and somebody must remain awake (or repeatedly wake up) in order to launch jobs when ready. In practice, this places a limit to how often internal test iterations can reasonably be executed. This is a shame, because frequent iterations undoubtedly lead to improved quality.
However, having a set of human eyes on the progressing execution does have the advantage of a constant assessment of the quality of the running migration. It happens frequently that the iteration is manually paused due to some issue rendering the results meaningless, some corrective action performed, and the iteration resumed to a significantly improved conclusion.
Automated Execution may in similar situations result in 'wasted' iterations, where the migration team upon completion the next morning discovers that the iteration is useless. Hopefully, Automated Execution will mitigate this by making it possible to perform more iterations without wearing out the migration team.
Design
As outline above, the initial scope of Automated Execution is limited to initiate and monitor the steps of a complete migration iteration. The steps to be executed depends on whether the migration project is an 'Export Only' project or an 'Export and Import' project. The steps and their dependencies are illustrated below.
Each of the boxes in the figure above illustrates a step of an Automated Execution. Some of the step consists of only 1 job, other steps of multiple jobs, all of which must complete before releasing any succeeding steps.
Load Valuesets
One job loading all selected valusets
Load Source Tables
Multiple jobs, one per Source Table
Load Source Views
One job loading all Source Views, resolving dependencies
Export
Multiple jobs, one per Business Object
Import
One job importing all selected Business Objects, resolving dependencies
Unload
One job unloading all selected Business Objects
Publish
One job publishing all selected Business Objects
There is a duality here that is important to understand and master in the design of Automated Execution The migFx Director already contains the functionality to parameterise, execute and monitor jobs. But an automated execution is not just a series of jobs, but rather a series of steps, each of which consists of one or multiple jobs, that must complete before subsequent steps can be released/submitted.
New: Restart from scratch
The existing job scheduling infrastructure can handle the execution of the jobs of each step as outlined above. However, in case one or more jobs of an Automated Execution fails, the existing ability to restart a failed job from the point of failure may not be enough. In many cases, it will be desirable to be able to perform some corrective action and re-run the failed job from start - not just from point of failure.
It is not possible in the present job scheduling infrastructure to restart a failed/canceled job from scratch. In the present infrastructure, a new job must be submitted in this case - raising the question how this new job would relate back to the governing Automated Execution of the original job.
To overcome this limitation, we suggest to extend the existing functionality to either restart a job with an option to restart from point of failure/cancellation (as today) or restart from scratch (new). Restarting from scratch would simply mean erasing the committed restart restart information for the job before submitting a new job instance as usual.
This new restart from scratch option will be available in the Job List for all jobs, not limited to jobs submitted by an Automated Execution.
Automated Execution
An Automated Execution is a new element in the Director, Client and Runtime. In the client, the Automated Execution will probably end up having is own, dedicated panel to create new and monitor existing Automated Executions.
Creating a new Automated Execution includes
Setting up the Automated Execution
Execute now
Execute on Trigger
Time
External (some external dependency, for instance from OPC or similar)
Setting up each of the steps:
To skip or include the step
If the step is included, parameterise the step. For instance, Which (or all) Valuesets to load, which (or all) Source Tables to Load etc
An Automated Execution can assume the following states below until completion
Created
The Automated Execution has been parameterised and created and is ready to run. In this state it is possible to edit the parameters of all steps
Running
The Automated execution is running, submitting, monitoring jobs and releasing new steps when ready. In this state it is not possible to edit the parameters of any step
Paused
User can pause and resume the Automated Execution. When Paused, steps will not be released even when ready. In this state it is possible to edit the parameters of steps not released yet
Halted
The Automated Execution cannot continue due to faulted or cancelled jobs. Jobs must be restarted and run to completion for execution to continue. In this state it is possible to edit the parameters of steps not released yet (job restart should be available fro the Automated Execution panel, see below)
Completed
All Steps and jobs have run to completion.
It can be considered, whether it should possible to rerun a completed job from scratch, thereby resuming the remaining Automated Execution from that point. However, the same result can easily be obtained by parameterising and submitting a new Automated Execution
(callout: comments welcome)
Director UI
A new panel for Automated Execution will be added to the Director UI. The panel shows a list of Automated Executions with their current state and enables the creation and parameterisation of new Automated Executions.
Parameterisation
When setting up an automated execution, each job must be parameterised:
Should the job be included or skipped
If included, which (or all) objects should be processed by the job.
The parameterisation of each job should ideally be similar to the way the job is parameterised when submitted on its own today
In addition, the automated execution itself could be parameterised. For instance to define the time on which the execution should start, but perhaps other parameters could be relevant (callout for suggestions, start on OPC trigger?)
Monitoring
The individual jobs constituting the automated execution will appear as all other jobs in the Director job list. Once a given step is ready to run, the jobs of the step will be submitted (not at once in a held state). This allows for the edit of parameters of steps that has not started yes, as described above.
The Automated Executions will appear as a list in a new panel in the Director UI. In this list, it will be possible to drill down:
Automated Execution
Steps in Automated Execution
Jobs in Step
The actions and information available on each level in this drill-down are:
Automated Execution
Created
Start: Start now, disregarding any wait on trigger or time Pause: Will not start on trigger or time Resume: Will start on trigger or time (or if trigger or time already occurred, start now) Delete
Running
Pause
Paused
Resume Delete
Halted
Delete Restart faulted/canceled jobs to resume
Completed
Delete
Step
Created
Pause: Will not start when ready Edit parameters: Will Pause step and Resume when done editing
Running
(No action possible, but individual jobs may be cancelled to enter Halted state)
Lars Kjaersgaard
This post is meant to float the idea of Automated Execution in the Director. This is hopp's initial suggestion on how to implement such a feature. We hope to receive valuable feedback in order to mature the idea into a proposal to include in the migFx road map.
Use case
One of the major characteristics of migFx is the ease of iteration. The Studio, Engine and Director components in unison significantly simplify the task of iterating the migration of selected Business Object instances.
The complete list of steps in a migFx iteration is
In addition, migFx iterations are typically performed in one of these distinct contexts
At present, migFx offers no facilities for Automated Execution. This makes it necessary to manually prepare and manually monitor the execution of the subsequent jobs in the Director in order to launch jobs as predecessors finish. Especially regarding the internal iterations - of which there ideally is quite a few - this places strain on the team members assigned to the actual execution of complete, internal migration iterations.
Scope
Given the use case above, it is clear that some of the steps are job running in the Director runtime and as such obvious candidates for automation. Other steps are more or less manual and to a lesser degree suitable for automation.
Based on these considerations, our ambition and scope for automation is initially limited to handle
Given this scope, the workflow for an initial implementation would be like
Launching an automated iteration as above will typically be done
Errors and mistakes
It is true that the present situation today with no automated execution does place an unnecessary strain on the team member that is tasked with overseeing especially the internal test iterations. These are typically run overnight, and somebody must remain awake (or repeatedly wake up) in order to launch jobs when ready. In practice, this places a limit to how often internal test iterations can reasonably be executed. This is a shame, because frequent iterations undoubtedly lead to improved quality.
However, having a set of human eyes on the progressing execution does have the advantage of a constant assessment of the quality of the running migration. It happens frequently that the iteration is manually paused due to some issue rendering the results meaningless, some corrective action performed, and the iteration resumed to a significantly improved conclusion.
Automated Execution may in similar situations result in 'wasted' iterations, where the migration team upon completion the next morning discovers that the iteration is useless. Hopefully, Automated Execution will mitigate this by making it possible to perform more iterations without wearing out the migration team.
Design
As outline above, the initial scope of Automated Execution is limited to initiate and monitor the steps of a complete migration iteration. The steps to be executed depends on whether the migration project is an 'Export Only' project or an 'Export and Import' project. The steps and their dependencies are illustrated below.
Each of the boxes in the figure above illustrates a step of an Automated Execution. Some of the step consists of only 1 job, other steps of multiple jobs, all of which must complete before releasing any succeeding steps.
There is a duality here that is important to understand and master in the design of Automated Execution The migFx Director already contains the functionality to parameterise, execute and monitor jobs. But an automated execution is not just a series of jobs, but rather a series of steps, each of which consists of one or multiple jobs, that must complete before subsequent steps can be released/submitted.
New: Restart from scratch
The existing job scheduling infrastructure can handle the execution of the jobs of each step as outlined above. However, in case one or more jobs of an Automated Execution fails, the existing ability to restart a failed job from the point of failure may not be enough. In many cases, it will be desirable to be able to perform some corrective action and re-run the failed job from start - not just from point of failure.
It is not possible in the present job scheduling infrastructure to restart a failed/canceled job from scratch. In the present infrastructure, a new job must be submitted in this case - raising the question how this new job would relate back to the governing Automated Execution of the original job.
To overcome this limitation, we suggest to extend the existing functionality to either restart a job with an option to restart from point of failure/cancellation (as today) or restart from scratch (new). Restarting from scratch would simply mean erasing the committed restart restart information for the job before submitting a new job instance as usual.
This new restart from scratch option will be available in the Job List for all jobs, not limited to jobs submitted by an Automated Execution.
Automated Execution
An Automated Execution is a new element in the Director, Client and Runtime. In the client, the Automated Execution will probably end up having is own, dedicated panel to create new and monitor existing Automated Executions.
Creating a new Automated Execution includes
An Automated Execution can assume the following states below until completion
It can be considered, whether it should possible to rerun a completed job from scratch, thereby resuming the remaining Automated Execution from that point. However, the same result can easily be obtained by parameterising and submitting a new Automated Execution
(callout: comments welcome)
Director UI
A new panel for Automated Execution will be added to the Director UI. The panel shows a list of Automated Executions with their current state and enables the creation and parameterisation of new Automated Executions.
Parameterisation
When setting up an automated execution, each job must be parameterised:
In addition, the automated execution itself could be parameterised. For instance to define the time on which the execution should start, but perhaps other parameters could be relevant (callout for suggestions, start on OPC trigger?)
Monitoring
The individual jobs constituting the automated execution will appear as all other jobs in the Director job list. Once a given step is ready to run, the jobs of the step will be submitted (not at once in a held state). This allows for the edit of parameters of steps that has not started yes, as described above.
The Automated Executions will appear as a list in a new panel in the Director UI. In this list, it will be possible to drill down:
The actions and information available on each level in this drill-down are:
Pause: Will not start on trigger or time
Resume: Will start on trigger or time (or if trigger or time already occurred, start now)
Delete
Delete
Restart faulted/canceled jobs to resume
Edit parameters: Will Pause step and Resume when done editing
Edit parameters
(also available in Job List)