Pipelines in Automation

Modified on Sun, 10 Dec 2023 at 02:12 AM

This is a detailed topic in our support portal in the Using Hopp series and assumes that you have some prior knowledge or experience using Hopp. 

Automation module leverages the powerful concept of PowerShell pipelines. This feature allows the output of one command to be passed directly as input to another. Using the module means stringing together the module cmdlets to pipelines. 

The module provides these cmdlets for this purpose:

Get-Hp*List Cmdlets

These cmdlets are used to retrieve a list of items. For example, Get-HpEntityList returns a list of entities that can be piped to the next New-Hp*Job cmdlet in the pipeline. 

Similarly, there are cmdlets to get the list of Valuesets, Source Tables, View etc.

New-Hp*Job Cmdlets

After retrieving a list, the next step is to create parameterized jobs for these items. The New-Hp*Job cmdlets are designed to create one job per item in the list or a single job for all items, depending on the type of item being processed. 

For instance, the New-HpExportEntityJob creates one job for each entity in the list whereas the New-HpImportEntityJob creates a single job importing all the entities in the list.

Submit-HpJob Cmdlet

Once the jobs are created, they need to be submitted for execution. The Submit-HpJob cmdlet takes the jobs created in the previous step and submits them in the Hopp Runtime.

Wait-HpJob Cmdlet

Finally, the Wait-HpJob cmdlet may be used to wait for the completion of the submitted jobs. This ensures that PowerShell will not continue with subsequent commands until all jobs are completed, allowing for synchronous control of jobs.

Here's an example: Exporting All Entities

To illustrate the use of these cmdlets in a pipeline, consider the following example that exports all entities in the connected track:

Get-HpEntityList | New-HpExportEntityJob | Submit-HpJob | Wait-HpJob

Here’s an overview of what happens in this pipeline:

  1. Get-HpEntityList retrieves a list of all entities in the track

  2. New-HpExportEntityJob creates a parameterized export job for each entity in the list

  3. Submit-HpJob submits these jobs for execution

  4. Wait-HpJob blocks until all export jobs are complete (or faulted/cancelled)

The flow in the pipeline is from left to right - but an important aspect is that the return value of the entire pipeline is the return value of the last cmdlet in the pipeline. So in this case, it is the return value of the Wait-HpJob cmdlet that is the result of the pipeline

Important point: Wait-HpJob - and thus the entire pipeline - returns 0 if all jobs succeed, otherwise 1

If you are creating a step in an orchestrator like Octopus Deploy or similar to submit and monitor all the jobs to export all entities, you can capture and exit with this return code

exit Get-HpEntityList | New-HpExportEntityJob | Submit-HpJob | Wait-HpJob


This example is straight forward and very useful. Indeed, it will in most case be used just as-is. 

To illustrate how you can customize this by leveraging PowerShell, here's an example to show how to limit the export to only entities that are copies of the Business Object Account.

Firstly, executing the Get-HpEntityList cmdlet on its own will output a list of the entities migrating in the track. For instance:

ItemID FullName          Name    Entity

------ ----------------- ----    ------

(Guid) Account [Current] Current Account

(Guid) Account [Loan]    Loan    Account

(Guid) Card                      Card

(Guid) Customer                  Customer

So the copies of the Business Object Account are Account [Loan] and Account [Current]. In order to only export these, the output from Get-HpEntityList can be filtered on the Entity property in the list by inserting a native PowerShell cmdlet into the pipeline like so  :

Get-HpEntityList | ForEach-Object { 
    if ($_.Entity -eq "Account") { 

The output from Get-HpEntityList is piped to the ForEach-Object cmdlet that will iterate through the list and only send an item down the pipeline if the Entity property of the item equals 'Account'.

To summarize, to submit and await the export jobs for all copies of the Account Business Object, the pipeline would look like this:

Get-HpEntityList | ForEach-Object { 
    if ($_.Entity -eq "Account") { 
} | New-HpExportEntityJob | Submit-HpJob | Wait-HpJob

The Automation module contains Get-Hp*List cmdlets to get these relevant lists:

Get-HpEntityListGets a list of the entities migration in the track

ItemID FullName          Name    Entity

------ ----------------- ----    ------

(Guid) Account [Current] Current Account

(Guid) Account [Loan]    Loan    Account

(Guid) Card                      Card

(Guid) Customer                  Customer

Get-HpSourceTableListGets a list of the source tables to load in the track

ItemID Alias Name             FullName             InScope

------ ----- ----             --------             -------

(Guid) Src   Account          Src.Account             True

(Guid) Src   AccountBalance   Src.AccountBalance      True

(Guid) Src   AlternateAddress Src.AlternateAddress    True

Get-HpSourceViewListGets a list of views to load in the track

ItemID Name               FullName              InScope

------ ----               --------              -------

(Guid) Account            Vw.Account               True

(Guid) AccountRelatedPart Vw.AccountRelatedPart    True

(Guid) Address            Vw.Address               True

Get-HpValuesetListGets a list of the dynamic valuesets to load in the track

ItemID ProjectID Engine Type        Name

------ --------- ------ ----        ----

(Guid) (Guid)    Target Dynamic     AccountProducts

(Guid) (Guid)    Target Dynamic     BankUnits

(Guid) (Guid)    Source Translation TranslateBankUnits

(Guid) (Guid)    Source Translation TranslateCardTypes

This cmdlet can be called with the option -engine set to either "Target" or "Source" or omitted for both


With this pipeline sample, we hope you get the idea of how to use the Automation cmdlets to string together your own pipelines.

In following articles, we will explore how to combine these pipelines in to larger flows.

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select atleast one of the reasons
CAPTCHA verification is required.

Feedback sent

We appreciate your effort and will try to fix the article