New and improved features

This section includes a summary of the new functionality provided in this release, it is broken down into the 5 main components of the software:

• Portal
• Studio
• Engines
• Runtime
• Automation

Where available links are included to other articles in the Support Portal that provide more detailed information about the new features and how they can be utilised. If the change relates to a Ticket in the Support Portal the number of the Ticket is included in the header after the has symbol (#).

Studio

Move stuff around in the Project Explorer tree view

We all make mistakes or change our minds. Sometimes, a Validation Rule is created but at a later point you realize that it should really have been a Mapping Rule. And sometimes you may want to change the Business Object hierarchy. For instance, mMove a child to another parent - or out as a separate root Business Object. Sometimes you want to move a Target Object to a different Business Object.

Previously, this was a major, manual undertaking where you would have to create a copy in the new location and manually replicate all the properties from the original to the copy.

Not so anymore! With version 2.5, you can now simply do this by drag and drop or copy and paste in the Studio Project Explorer.

You can read more about this new feature here: Moving things around and changing rule types.

Note: This new feature has introduced new Copy and Paste items in the Project Explorer context menu. To make things clearer, we have renamed the previous Copy menu item to Duplicate.

We have just changed the label of the menu item. The function does exactly the same as before.

Add and delete Target Objects and Child Business Objects directly from the Project Explorer

While it is still possible to add child Business Objects and Target Objects to a Business Object by opening it and go to the Children and Target tabs, this can now also be done directly from the Project Explorer

Likewise, a child can now be deleted directly from the Project Explorer as well.

Note that all these actions will cause the Business Object to be automatically checked out. It follows, that they are disabled, if the Business Object is currently checked out by someone else.

Improved usability on Views in the Source Map

Views in the Source Map is a powerful feature enabling Hopp to overcome all kinds of challenges when bridging the - sometimes large - gap between on one side the Business Object hierarchies required by the Target Interface and on the other side the structure of the legacy source data available to the Source Map.

While the views are certainly very useful, they have up until now in some cases been quite work intense.

The main task when creating a view is to write and test the SQL query for the view. This is commonly done in Sql Server Management Studio and - when satisfactory - the sql is pasted into the Studio view

Previously, when defining a view in the Source Map, there remained these 2 tasks to be done:

1. All the objects (Metadata, Views, Valuesets) referenced by the Sql Statement has to be added to the Studio view
2. All the columns returned by the Sql Statement has to be added as Exposed Fields to the Studio view

These 2 points - and especially point 2 - would potentially require a lot of work when creating and maintaining views with a larger number of Referenced Objects and Exposed Fields.

This release removes the requirement to manually maintain the list of Referenced Objects and introduces the capability to automatically synchronize the list of Exposed Fields with the SQL query of the view.

Read more about the new synchronize features on the Views here: Creating Views in the Studio Source Map

Improved cross-reference on columns used in Views

The cross-reference in Studio will now discover references to columns done in the SQL query of a view - even when these columns are only present in the query and not expressly exposed by the view.

Given a view named Sample containing this SQL query:

select
  A.[BankId]
,  A.[AccountNumber]
,  B.[OfficialId] as [CustomerNumber]
from
  Src.[Account] as A
  inner join Src.[Client] as B
  on
    A.[BankId] = B.[BankId]
  and  A.[ClientId] = B.[ClientId]

Stricter validation of Exposed Fields on Views

The new capability to automatically synchronize the exposed fields of views makes it easy to ensure that the fields have correct data types. This has enabled a stricter validation of the data types of the exposed fields significantly reducing the risk of incompatible data returned by the view at runtime.

Examples of problems previously discovered at runtime that is now caught by the Studio validation:

• Exposed Field is not nullable, but the SQL query returns null
• Length of Exposed Field is too short, resulting in a truncation error 
• Etc

NB This stricter validation will probably require you to fix some of your views where the data types of the exposed fields are not 100% compatible with the values returned by the view. The refactor tools provided with this release has a switch to optionally do this automatically, see below.

Portal

Comparison of object versions now include Events and Dependencies

When comparing 2 historic versions of an object, the comparison now includes the Event and Dependency differences between the 2 versions

Runtime

Constant and Secret substitution added to parameters to Extension Usages.

It is now possible to use Constants (including RunParameters) with the moniker {constant:Name} and and Secrets with the moniker {secret:Name} inside the values of the parameters for extension usages - for instance Target Unloader extensions.

Read more: Secrets and Constant replacement

Direct load of Staging Tables

Up until this version, loading of data from a file to a staging table consisted of 2 steps:

1. The file was read by a reformatter extension specific for the format of the input file. Hopp had reformatters to parse CSV, XML, Excel and DB2 unload files. The reformatter would then pass the data read from the input file back to the Runtime that would write it out to a load file in the in a common format
   
   
2. The load file would be bilk inserted into the staging table by the Runtime
   
   

This is obviously not the most efficient way to go about this and has long been an target for optimization. We finally got around to fixing it in version 2.5.

Now, the Reformatter extensions are a thing of the past and replaced by the new BulkDataProvider extension. Instead of reformatting the input file and writing an intermediate load file, the new extension is responsible for handing the Runtime a reader implementing the .NET IDataReader interface.

This reader is then connected by the Runtime directly to the bulk insert. The end result is that the separate reformat step to write the intermediate load file is gone and the necessary reformatting takes place 'on-the-fly' as the file is bulk loaded.

Expect to see significant improvements in elapse time when loading large files to the staging database.

No more Reformatted Files

With the replacement of the Reformatters with the new BulkDataProviders, there are no reformatted files for bulk insert into Sql Server any more and the corresponding Path setting has been removed from the configuration of individual tracks.

In cases were the Hopp installation has separate execution and the data base servers, there is an an added benefit. With the previous solution, it was necessary for the formatted files to be in a folder on the database server in order for the Sql Server user to have access to the file in order to load it.

To accommodate this, it was necessary in these dual server installations for the execution server to write the formatted files in a folder on the database server. This cross-machine file write is no longer required, and and shared folders created to enable this can be decommissioned.   

NB This change requires a rewrite of any existing, custom Reformatter implementations. Hopp will do this rewrite free of charge for all our customers. Just provide the existing reformatter code.

Major rework of unload 

Internally in the Runtime, the unload of the migration results has changed significantly. The runtime no longer depends on the Microsoft Bulk Copy Program (BCP) command line tool to unload the data. 

Some improvement in unload performance is expected and this generally a more resilient and robust solution.

Issues fixed

The following bugs have been fixed in this release, where there is a Ticket on the Support Portal its number is included after a hash symbol (#):

• Studio Target map: Key field validation
  Target validation gives warning 0087 (Interface field is never used in any items) on Keyfields even they are mapped in manual rule
  
  
• ExcelReformatter hardcoded to write load file in ISO-8859-1
  Track context is UTF-8 and this the code page passed to the Bulk Load for the formatted file. Breaks when the formatter file is in fact written in ISO-8859-1. So the bulk load reads as UTF-8 but the file was in fact ISO-8859-1 resulting in character problems in the loaded data in the sql table.
  
  
• Sort column error when changing track
  On the Job List and change the Track. This resulted in a number of errors on the client
  
  
• Not Done does not exclude Resolved events
  The Not Done filter wrongly includes resolved events
  
  
• Check Auth on Dependencies
  When viewing the dependencies for an Object, the authorization for the partition of the dependencies should be taken into account. If the user is not authorized for the partition of a dependency, the UI should not show a link to jump
  
  
• Studio validation code 0087 raised on interface fields used in manual exit rule
  Interface field gets validation warning 0087 even it is used in manual Exit rule
  
  
• Studio crash if 'Skip if Null' is checked before rule is selected
  The Skip if Null check box must be disabled until a validation rule is selected.
  
  
• The format of timestamps are overwritten by machine localization
  If the host machine has a localization that uses a different format than what is specified as the format, the localization format is used instead.
  
  
• Studio does not update Entity Label when importing new Metadata
  Label in the first list of entities in the metadata panel is not updated on import
  
  
• In Studio references on field level does not work
  Metadata fields only used as internal join fields in a view query are not found bt View Referebces
  
  
• Error in Portal edit of numeric value for Extension Usage
  Editing numeric values for extension usage parameters failed in some instances
  
  
• NRE when removing Source Map Exit Rule
  In some instances, removing a rule from a list of rule calls resulted in a Null references exception
  
  
• Target Unload fails on quotes in data
  A mapping rule in the target map assigned a string value to a target field. The assigned string contained double-quotes. When they ran the target unload, this failed because the double quotes in the string value were output into the value attribute without being escaped to " So the temporary unloaded file in the _Unload folder was not valid xml
  
  
• Export job failure when an Exposed field and a source object name in relationship in Extraction map are same#4564, #4483
  In some instances, naming a Source Object in the Extraction Map with the same name as parent Source Objectresulted in a uncaught error in the export
  
  
• Metadata.xml deleted by Target Generator
  A race condition sometimes caused the Target Engine Generator to clean up the Metadata.xml file it just created as obsolete
  
  
• System Event Impact resets to Critical
  The value for Disposition on System Events where never saved. So it resets to default value of Critical when item is closed and reopened
  
  
• Bulk events parameter values shifting left
  When bulk downloading Events with Values, the parameter columns for the events shift left, if any of the key fields for the line are null
  
  
• Studio Source export rule missing and return value changed
  If a rule is used to assign a value to an Interface Field. Then the return type of the rule is changed to a type that is not compatible with the type of the Interface Field. Now, when the rule dialog is opened for the field, the Rule is not show because it did not satisfy the filter of the rules allowed to appear in the rule drop down
  
  
• Track fails after clear files
  When running the clear files actions from the Manage panel, the Track Process fails when the schema folder is deleted
  
  

Installation and Refactoring

Artefacts can be downloaded from the download page . 

Please find attached to these release notes (below):

• Database refactor scripts
• An upgrade checklist Hopp.2.5.Upgrade.xlsx

Local customizations

We are always doing our outmost to ensure the quality of the Hopp Migration Platform. To the extent that our customers have custom implementations using our external APIs or are writing queries against the internal databases of the Hopp platform, we do expect our customers to perform rigorous testing of their own implementations.

Internal Database Tables

Our platform's internal database tables are not part of the public API and are subject to change at any time without notice. These tables are intended for internal use only.

If your custom code or integrations access these tables directly, you must test them thoroughly with each release to ensure continued compatibility. We strongly recommend relying on documented APIs wherever possible to reduce the risk of breaking changes.

Extension APIs

Our API for extensions is provided to enable advanced integrations and custom functionality. However, this API may evolve over time, and changes may occur between releases.

We advise all customers using the Extension API to validate their custom extensions and integrations against each new version of the platform. Proper testing will help identify and resolve any issues arising from API modifications.

Database

• The Sql scripts to run are all attached below
  
  
• Please back up all databases before running refactor scripts
  
  
• Run attached MigrationDb.16.sql script in all Migration databases
  
  
• Run attached MasterDb.10.sql script in Master database on the Master server

Please note that our internal staging and migration databases are designed to support our data processing workflows and are not intended to function as stable, external APIs. While we aim to maintain consistency across releases, we do not guarantee backward compatibility or structural stability of these databases. Clients who choose to access these databases directly are responsible for validating that their integrations continue to function as expected after any updates or releases. 

Master Server

• Redeploy the Portal web application
  • Preserve and re-establish the appsettings.json file
  • If you are hosting the Portal in an IIS sub-folder, you should also preserve and re-establish the wwwroot/index.html
• Upload new runtime.zip to the Portal

Execution Servers

• Reinstall the Hopp Agent Service 
  • Stop the Hopp Agent Service before uninstall
  • Preserve and re-establish the MigFx.Agent.dll.config file
  • Re-enter the application credentials as logon for the Agent service
  • Start the agent service
    
    
• Reinstall any DataServices extensions (if installed)
  • Hopp.DataServices.SqlServer
  • Hopp.DataServices.DB2
  • Hopp.DataServices.Oracle

Workstations

Studio

• Reinstall Studio from the download page
  
  

DataServices

• Reinstall any DataServices extensions (if installed)
  • Hopp.DataServices.SqlServer
  • Hopp.DataServices.DB2
  • Hopp.DataServices.Oracle

Studio Maps

Version 2.5 of Studio includes the following refactorings. When running a refactor, it is important to assure that nobody in your team has anything checked out in any maps in Studio.

Refactor removal of Referenced Objects on Views

Since it is no longer required to maintain a list of Referenced Objects on views in the Source Map, all Source Maps must be refactored to remove these references and avoid false flagging of stale items in Studio.

To automatically refactor your Source Maps, run this refactor command from a command line:

"(Studio Install Path)\MigFx.Studio.Refactor" 2.5 --Fix --DataSource (sql server instance) --InitialCatalog (repository database) --WorkingFolder (temporary working folder) --Commit --CommitMessage "Refactor: Upgrade to 2.5" 

NB The command line switch --Fix is optional.

Because this release imposes stricter validation on the Exposed Fields in Views in the Source Map, it is very probable that a number of views will trigger validation errors.

Adding the --Fix switch will cause the refactor to automatically synchronize the Exposed Fields on all views.

And in turn, this synchronize may alter the data type on Exposed Fields. The --Fix switch will also cause the refactor to fix all Interface Fields, adding or removing casts where required as a consequence of the changed data type on an Exposed Field.

Omit the --Fix switch if you prefer to manually fix this. In many cases, it may be a better option to modify the query of a view to return the data types as exposed instead.

If you omit the --Commit switch, the refactor will run and save the refactored maps in the --WorkingFolder - but it will not commit the changes back into the repository. this can be useful if you want to a test run before fully committing.

Visual Studio

• Reinstall the Hopp Visual Studio extension (VsExtension.2022.vsix)
  
  

Visual Studio Engine Projects

• In all *.SourceEngineGenerated.csproj files
  • Update Microsoft.EntityFrameworkCore.SqlServer package reference to 8.0.17
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="8.0.17" />
    
    
  • Update Microsoft.Data.SqlClient package reference to 5.2.3
    <PackageReference Include="Microsoft.Data.SqlClient" Version="5.2.3" />
    
    
• In all *.TargetEngine.csproj files
  • Update Microsoft.EntityFrameworkCore.SqlServer package reference to 8.0.17
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="8.0.17" />
    
    
• In all custom library projects implementing Hopp extensions
  • Update any Hopp.* NuGet package references to 2.5.*
    
    
  • Update any Microsoft.Data.SqlClient package references to 5.2.3
    
    
  • Update Microsoft.EntityFrameworkCore.SqlServer package references to 8.0.17 
    
    
• If using Nuget
  • Update Hopp.SourceEngine package reference to 2.5.* in all *.SourceEngineGenerated.csproj files
    <PackageReference Include="Hopp.SourceEngine" Version="2.5.*"/>
    
    
  • Update Hopp.TargetEngine package reference to 2.5.* in all *.TargetEngine.csproj files
    <PackageReference Include="Hopp.TargetEngine" Version="2.5.*"/>
    
    
  • Update the Hopp.Generator tool
    dotnet tool update hopp.tools.enginegenerator --global
    
    
• If using local libraries
  • Reinstall MigFx.Libraries from download page
  • Reinstall MigFx.EngineGenerator from download page

Powershell Automation

• Re-install the PowerShell Automation module on relevant workstations and servers.
  
  

Extensions

BulkDataProviders

As mentioned above, the existing Reformatter extensions are being replaced by new BulkDataProvider extensions. This requires any existing, custom Reformatter extensions to be rewritten as BulkDataProviders. 

Hopp will provide this rewrite for all customers free of charge and assist with the deployment of these.

NB: The Db2Reformatter previously provided by Hopp is decommissioned, instead Hopp now provides an RFC4180 compliant BulkDataProvider that will do the same job, provided that the delimiter is set to semi-colon and the quote character is set to hash (#).

Audit Collection

For customers that have implemented custom audit collection, we have tightened down the expected return value from the custom Auditor. The return value was previously just a string containing the collected audit data that the runtime would store and hand back to the audit result processing.

This has been changed, and the custom auditor must now return an Xml Element containing the collected audit data. We have introduced this change now, before too many customer are implementing custom auditors. 

Tying down the collected audit data to an Xml Element opens up the possibility for us to implement history on the collected data in a future release.

If your custom auditor already internally builds an xml element, it should be a simple refactor to just return the element instead of the xml string. Likewise, your audit result implementation should be refactored to accept that the properties on the IAuditResultItem is now an xml element and not a string.

If the string value returned by your current auditor is not an xml string, then you will need to wrap the string value in a CDATA element inside an xml element to return from the auditor.

Wrapping string audit data

For your convenience, a sample implementation of an efficient way to wrap the audit string

using System.Xml.Linq;

// Instead of
return auditData;

// Do this
return string.IsNullOrEmpty(auditData) 
	? null 
	: new XElement("AuditData", new XCData(auditData));

public interface IAuditResultItem
{
  long ItemId { get; }
  Guid EntityID { get; }
  string PartitionValue { get; }
  string SourceKey { get; }
  string MigrationKey { get; }
  XElement SourceAuditData { get; } // Changed from string
  XElement TargetAuditData { get; } // Changed from string
}

var sourceAuditData = item.SourceAuditData.Value;
var targetAuditData = item.TargetAuditData.Value;

All extensions

In addition to this, a minor change in the call contract for extensions loaded by the Runtime requires a minor edit to any custom extension implementation.

In more detail, the change in the contract is that base class for all extensions now requires the context instance for the extension to be passed to its constructor.

So while extensions previously did not have to have a constructor defined, now all extensions must have a constructor accepting the context as an argument and pass that argument on to the base constructor.

Here's an example from the TargetCsvUnloader included with Hopp:

[Extension("migFx Target Csv")]
public class TargetCsvUnloader : MigFx.Director.Server.Import.Unload.TargetUnloader
{
  // The extension now must have a constructor defined to pass the 
  // context up to the base constructor
  public TargetCsvUnloader(IUnloadContext ctx)
  : base(ctx)
  {
    // As a benefit, the extension can now use the context in the constructor. This 
    // was not possible in earlier versions 
  }
}

Fear not, once you reference the new NuGet packages for version 2.5 - or install the 2.5 MigFx.Libraries if not using NuGet, the compiler will catch all this for you and ensure that all your extensions will be corrected. 

The effort and risk in this change is minimal.