Pipelines in Automation

Modified on Sun, 10 Dec, 2023 at 2:12 AM

This is a detailed topic in our support portal in the Using Hopp series and assumes that you have some prior knowledge or experience using Hopp.

Automation module leverages the powerful concept of PowerShell pipelines. This feature allows the output of one command to be passed directly as input to another. Using the module means stringing together the module cmdlets to pipelines.

The module provides these cmdlets for this purpose:

Get-Hp*List Cmdlets

These cmdlets are used to retrieve a list of items. For example, Get-HpEntityList returns a list of entities that can be piped to the next New-Hp*Job cmdlet in the pipeline.

Similarly, there are cmdlets to get the list of Valuesets, Source Tables, View etc.

New-Hp*Job Cmdlets

After retrieving a list, the next step is to create parameterized jobs for these items. The New-Hp*Job cmdlets are designed to create one job per item in the list or a single job for all items, depending on the type of item being processed.

For instance, the New-HpExportEntityJob creates one job for each entity in the list whereas the New-HpImportEntityJob creates a single job importing all the entities in the list.

Submit-HpJob Cmdlet

Once the jobs are created, they need to be submitted for execution. The Submit-HpJob cmdlet takes the jobs created in the previous step and submits them in the Hopp Runtime.

Wait-HpJob Cmdlet

Finally, the Wait-HpJob cmdlet may be used to wait for the completion of the submitted jobs. This ensures that PowerShell will not continue with subsequent commands until all jobs are completed, allowing for synchronous control of jobs.

Here's an example: Exporting All Entities

To illustrate the use of these cmdlets in a pipeline, consider the following example that exports all entities in the connected track:

Get-HpEntityList | New-HpExportEntityJob | Submit-HpJob | Wait-HpJob

Here’s an overview of what happens in this pipeline:

Get-HpEntityList retrieves a list of all entities in the track
New-HpExportEntityJob creates a parameterized export job for each entity in the list
Submit-HpJob submits these jobs for execution
Wait-HpJob blocks until all export jobs are complete (or faulted/cancelled)

The flow in the pipeline is from left to right - but an important aspect is that the return value of the entire pipeline is the return value of the last cmdlet in the pipeline. So in this case, it is the return value of the Wait-HpJob cmdlet that is the result of the pipeline

Important point: Wait-HpJob - and thus the entire pipeline - returns 0 if all jobs succeed, otherwise 1

If you are creating a step in an orchestrator like Octopus Deploy or similar to submit and monitor all the jobs to export all entities, you can capture and exit with this return code

exit Get-HpEntityList | New-HpExportEntityJob | Submit-HpJob | Wait-HpJob

Embellishment

This example is straight forward and very useful. Indeed, it will in most case be used just as-is.

To illustrate how you can customize this by leveraging PowerShell, here's an example to show how to limit the export to only entities that are copies of the Business Object Account.

Firstly, executing the Get-HpEntityList cmdlet on its own will output a list of the entities migrating in the track. For instance:

ItemID FullName Name Entity

------ ----------------- ---- ------

(Guid) Account [Current] Current Account

(Guid) Account [Loan] Loan Account

(Guid) Card Card

(Guid) Customer Customer

So the copies of the Business Object Account are Account [Loan] and Account [Current]. In order to only export these, the output from Get-HpEntityList can be filtered on the Entity property in the list by inserting a native PowerShell cmdlet into the pipeline like so :

Get-HpEntityList | ForEach-Object { 
    if ($_.Entity -eq "Account") { 
        $_ 
    } 
}

The output from Get-HpEntityList is piped to the ForEach-Object cmdlet that will iterate through the list and only send an item down the pipeline if the Entity property of the item equals 'Account'.

To summarize, to submit and await the export jobs for all copies of the Account Business Object, the pipeline would look like this:

Get-HpEntityList | ForEach-Object { 
    if ($_.Entity -eq "Account") { 
        $_ 
    } 
} | New-HpExportEntityJob | Submit-HpJob | Wait-HpJob

The Automation module contains Get-Hp*List cmdlets to get these relevant lists:

Get-HpEntityList	Gets a list of the entities migration in the track ItemID FullName Name Entity ------ ----------------- ---- ------ (Guid) Account [Current] Current Account (Guid) Account [Loan] Loan Account (Guid) Card Card (Guid) Customer Customer
Get-HpSourceTableList	Gets a list of the source tables to load in the track ItemID Alias Name FullName InScope ------ ----- ---- -------- ------- (Guid) Src Account Src.Account True (Guid) Src AccountBalance Src.AccountBalance True (Guid) Src AlternateAddress Src.AlternateAddress True
Get-HpSourceViewList	Gets a list of views to load in the track ItemID Name FullName InScope ------ ---- -------- ------- (Guid) Account Vw.Account True (Guid) AccountRelatedPart Vw.AccountRelatedPart True (Guid) Address Vw.Address True
Get-HpValuesetList	Gets a list of the dynamic valuesets to load in the track ItemID ProjectID Engine Type Name ------ --------- ------ ---- ---- (Guid) (Guid) Target Dynamic AccountProducts (Guid) (Guid) Target Dynamic BankUnits (Guid) (Guid) Source Translation TranslateBankUnits (Guid) (Guid) Source Translation TranslateCardTypes This cmdlet can be called with the option -engine set to either "Target" or "Source" or omitted for both

Summary

With this pipeline sample, we hope you get the idea of how to use the Automation cmdlets to string together your own pipelines.

In following articles, we will explore how to combine these pipelines in to larger flows.