Orchestrating a complete migration

Modified on Sat, 9 Dec, 2023 at 12:43 AM

This is a detailed topic in our support portal in the Using Hopp series and assumes that you have some prior knowledge or experience using Hopp. 


3rd party Orchestration applications like Octopus Deploy, Control-M, Automic Automation, Tivoli Workload Scheduler and others are built to handle the submission an monitoring of complicated flows of individual tasks (or steps/jobs/operations/etc depending on the terminology of the application in question).


The Hopp.Automation PowerShell module provides at set of PowerShell cmdlets that makes it possible to orchestrate the entire migration job flow in the Hopp Runtime using a 3rd party application.


A full end-to-end migration in the Hopp Runtime includes the steps in the figure below. This is of course just the bare bones. In many cases, it is beneficial to add manual stops to the flow for verification to take place before proceeding. 


And a great benefit of orchestration is of course that the execution of the migration in the Hopp Runtime can become part of a much larger flow with both predecessors and successors.



This guide will provide the PowerShell script for each of the single steps in the migration flow above and should serve as a good starting point for potential further customization.


Establish connection

Before any command can be issued, it is necessary to connect to the Hopp Portal. For the purpose of 3rd party applications, it is necessary to configure the Hopp Portal to accept connections from the application (client).


This is done by configuring a Client Id and a Client Secret in to configuration of the Hopp Portal. The application can then connect simply by providing this client id and secret.


Important: You must configure your Orchestration application to safely store and use the client secret, so the secret is not unduly exposed.


Every single step in the flow above must establish a connection, execute the Hopp Automation commands for the step and finally tear down the connection. This means that all the steps will use this template:


$portalUrl = "(the url of the Hopp Portal)" 
$trackID = "(The Guid of the track the step should act on)" 
$clientId = "(the Client Id configured in the Hopp Portal)" 
$clientSecret = "(the Client Secret configured in the Hopp Portal)"

# Connect to the Portal and the Track 
Connect-Hp -portalUrl $portalUrl -trackID $trackID -clientId $clientId -clientSecret $clientSecret

try {
    # (Hopp Automation commands go here)
}
finally {
    # Tear down the connection
    Disconnect-Hp
}

In all the following samples, it is assumed that this template is followed, and each sample will only include the Hopp Automation commands that goes inside the try { } block in the sample above.


Wait-HpJob

The Wait-HpJob cmdlet is used in all samples below but the first one. 


Wait-HpJob will return 0 if all the awaited jobs are successfully completed, otherwise 1. The samples will exit with this exit code to be handled by the orchestration application


If a job is faulted, Wat-HpJob will always output the error from the log of the failed job. If Wait-HpJob is called with the option -showLog $true, it will output the full log of all awaited jobs.


Restart Track

It is a good idea to restart the track at the beginning of the flow to ensure that the latest deployed Source and Target Engines are loaded into the Track.


Stop-HpTrack
Start-HpTrack

Remarks

  • Stop-HpTrack issues a stop request and waits for the track to stop. Does nothing if the track is already stopped
  • Start-HpTrack issues a start request and waits for the track to start



Setup Source Engine

Run the setup of the Source Engine to ensure the deployed engine and the migration databases for the track are in sync


exit New-HpSetupSourceJob | Submit-HpJob | Wait-HpJob
Remarks
  • Creates and submits a job to setup the Source Engine
  • Waits for the job to stop running. If the job completes successfully, the script will exit with code = 0, otherwise 1



Setup Target Engine

Run the setup of the Target Engine to ensure the deployed engine and the migration databases for the track are in sync


exit New-HpSetupTargetJob | Submit-HpJob | Wait-HpJob
Remarks
  • Creates and submits a job to setup the Target Engine
  • Waits for the job to stop running. If the job completes successfully, the script will exit with code = 0, otherwise 1



Load Source Valuesets

Load all dynamic Valuesets for the Source Engine. 


exit Get-HpValuesetList -engine "Source" | New-HpLoadValuesetJob | Submit-HpJob | Wait-HpJob
Remarks
  • Creates and submits a job to load all valuesets in the Source Engine
  • Waits for the job to stop running. If the job completes successfully, the script will exit with code = 0, otherwise 1



Load Source Tables

Load all source tables for the Source Engine. 


exit Get-HpSourceTableList | New-HpLoadSourceTableJob | Submit-HpJob | Wait-HpJob
Remarks
  • Creates and submits a series of jobs to load all source tables  in the Source Engine. Will submit 1 job for each table
  • Waits for the jobs to stop running. If all jobs complete successfully, the script will exit with code = 0, otherwise 1



Load Target Valuesets

Load all dynamic Valuesets for the Target Engine. 


exit Get-HpValuesetList -engine "Target" | New-HpLoadValuesetJob | Submit-HpJob | Wait-HpJob
Remarks
  • Creates and submits a job to load all valuesets in the Target Engine
  • Waits for the job to stop running. If the job completes successfully, the script will exit with code = 0, otherwise 1



Load Source Views

Load all Valuesets for the Source Engine. 


exit Get-HpSourceViewList | New-HpLoadViewJob | Submit-HpJob | Wait-HpJob
Remarks
  • Creates and submits a job to load all views in the Source Engine
  • Waits for the job to stop running. If the job completes successfully, the script will exit with code = 0, otherwise 1



Export

Export all entities in the track 


exit Get-HpEntityList | New-HpExportEntityJob | Submit_HpJob | Wait-HpJob
Remarks
  • Creates and submits a series of jobs to export all entities in the track. Will submit 1 job for each entity
  • Waits for the jobs to stop running. If all jobs complete successfully, the script will exit with code = 0, otherwise 1



Import

Import all entities in the track. 


exit New-HpImportFullMontyJob | Submit-HpJob | Wait-HpJob
Remarks
  • Creates and submits a job to run a Full Monty import
  • Waits for the job to stop running. If the job completes successfully, the script will exit with code = 0, otherwise 1



Publish

Publish migration result to the Portal. 


exit New-HpPublishToPortalJob | Submit-HpJob | Wait-HpJob
Remarks
  • Creates and submits a job to publish the migration result to the Portal
  • Waits for the job to stop running. If the job completes successfully, the script will exit with code = 0, otherwise 1



Unload

Unloads all migrates entities. The Guid of the Unloader extension to use must be specified in the -unloaderID option


Get-HpEntityList | New-HpUnloadTargetJob -unloaderID "(Guid)" | Submit-HpJob | Wait-HpJob
Remarks
  • Creates and submits a job to load all valuesets in the Target Engine
  • Waits for the job to stop running. If the job completes successfully, the script will exit with code = 0, otherwise 1


Summary

In many cases, the samples above are exactly what is needed to orchestrate the migration.


If not, the modular nature of the Automation module makes it easy an straght forward to create deeply customized automation to accommodate your requirements.


Check out other articles in this section for useful guidance.

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select at least one of the reasons
CAPTCHA verification is required.

Feedback sent

We appreciate your effort and will try to fix the article