|This is a detailed topic in our support portal in the Using migFx series and assumes that you have some prior knowledge or experience using migFx. If you want to know more about hopp tech and migFx, our comprehensive solution for complex data migration, please visit our web site hopp.tech. |
Manage Servers, Projects and Tracks
Finally, we got around to implementing the long-missing UI to manage Servers, Projects and Tracks in the Director. About time! No longer will you have to manually enter data into the master tables in Sql Server and be responsible for the data integrity yourself, the new UI is there to help. Even better, the new management of Projects also handles all the necessary trickery around authorizations, so you no longer have to open the .NET Sql Authorization Manager to do so manually.
Manage Engines too
As a brand new thing, you can now also manage the different Engines in the Director, and for a Project, you can identify the Engines this Project uses. Once you have done this, the Director Runtime - more precisely the Track Watcher service - will use this information when you start a Track and seamlessly edit the config file of the Track to identify and load the correct engines. This behaviour is new, if you wish to keep on manually editing the Track config files yourself, you can retain the old behaviour with a new setting in the config file of the Track Watcher Service.
Here's what it all looks like, the new management interface is presented in the Runtime section of the Tree Menu in the MigFx Administration form:
Let's go through the sections one by one.
Here you maintain all the Servers in your installations setup:
You can add new servers or modify existing ones. You also delete a server, but if it is in use by Tracks, the deletion will not succeed. Add or modify will show the Server dialog:
|Computer Name||This must be the name given locally on the server. If the server is an active server running one or more tracks, the migFx Track Watcher Windows Service will use this name to identify any track start requests destined for the server.|
In detail the Watcher Service will get the name of the machine from the operating system, and use this name to look for the Server in the Director - so if the machine Name is not correct, the Watcher will not find anything, and you will be unable to start any Tracks on the Server
|Description||The Description of the Server. The Director will show this description or the Machine Name of the Server in the UI as governed by the setting Show Servers in Director by described below (see Config)|
|Default track path||This is the default root folder for all the Tracks on the Server. When creating a new Track, the Director will automatically suggest best practice naming of all the folder paths for the new Track under this root|
|Network Name||The (optional) network name to use when connecting to the Sql Server instance on this Server. If blank, Machine Name will be used|
|Sql Instance||An (optional) name for the Sql Instance on the Server. Leave blank for the Default Instance|
|Sql Port||An (optional) port for the Sql Instance on the Server. Leave blank for the default port|
|DataSource||The label DataSource will update to show the exact DataSource that is constructed from the values of Machine Name, Network Name, Sql Instance and Port|
Here you create and maintain the Engines that your Projects can use. You can add or modify Engines. If an Engine is not in use by any Project, you can also delete it. Add or Modify will show the Engine dialog:
|Studio Project ID||The id assigned to this map in Studio. You can find this ID on the Settings of the root node in the Studio project Explorer tree view: |
|Engine name||The Engine name is displayed when you select Engines for your Projects|
|Engine type||The Engine can be either a Source- or a Target Engine|
|Installation path||This is the path from which the Director Runtime will copy the Engine when a Track starts. This can be a network location (starting with \\) and shouldreachable as a network location from the migration server(s)|
|Library||This is the name of the library that the Director will load in order to instantiate objects from the Engine. This must be the same name that is specified in Visual Studio as the assembly name for the Engine project. For the Source Engine, it must be the assembly name of the EngineCustom project.|
Maintaining projects engines happens here:
You can add and modify Projects. You can delete a Project if it is not running in any Tracks. Note that deleting a Project will only remove it from the Director management. It will not delete the Project database.
Add or Modify will show the Project dialog:
|ID||The Project ID is assigned when a new projects is added|
|Name||This is the name that is displayed for the Project everywhere: In the Director and in the Tracker|
|State||The State can be Active or Inactive. Inactive projects are not available in Director but are still available in Tracker. An Inactive project cannot have any Engines connected. Use the Inactive state for finished projects that only live as archives but do not have any execution tracks|
|Source- and Target Engine||Here you select the Source- and Target Engines that the Project will use|
|Mode||Mode can be Export or Migration:|
The project is only exporting data. In this mode there is no Import available in the Director for any Tracks running the Project and you cannot select a Target Engine for the Project
This is the full migration exporting and importing data.
|Default language||This is the language to display for the project in the Tracker, if a user has not selected any language in the Tracker user settings|
|Project Database: Server||The Server where the Project database is located|
|Project Database: Database||The name of the Project database. Note that you must create this database yourself.|
|Project Database: Connection string format||An optional connection string format to use when connecting to the database. If not specified, the global connection string maintained in the Config panel (see below) is used|
This is where you create and maintain Tracks on your Servers. You can add and modify Tracks, decide which Project uses a given Track and finally link 2 different Tracks for Archiving (see below). You can delete a Track if it is not in use. Note that deleting a Track only removes it from the Director management. Database or files for the track are not deleted.
Note the Filter dropdown to filter the list of tracks by:
- Runtime Tracks only
- Archive Tracks only
|ID||The Track ID is assigned when a new Track is added|
|Runtime version||The version of the Director Runtime the Track runs. This enables side-by-side installations of new versions of the Director Runtime. Current version at the time of writing is 1.0.|
|Execution Server||The Server running the Track Watcher Service that will monitor for start requests for the Track. All jobs in the Track will run on this server|
|Track Number||The number of the Track on the Execution Server. This must unique for the Server. If you try to create a Track with a duplicate number, you will get an error|
|Paths...||Click the Paths... button to specify the folders to use in the Track (see below). If you do not do this for a new Track, a set of default folders will automatically be created|
|Archive||Check this to specify that this Track is an Archive Track and will not be executing any migration jobs. You can only change this setting if the Track is not participating in any Archiving link (see below).|
Archive Tracks only show up in the Tracker and are hidden from the Director
|Migration Database: Server||The Server where the Migration database is located. It is default the same as the Execution Server, but you can choose another server|
|Migration Database: Database name||The name of the Migration database. Note that you must create this database yourself|
|Migration Database: Connection string format||An optional connection string format to use when connecting to the Migration database. If not specified, the global connection string maintained in the Config panel (see below) is used|
|Staging Database: Server||The Server where the Staging database is located. It is default the same as the Execution Server, but you can choose another server|
|Staging Database: Database name||The name of the Staging database. Note that you must create this database yourself|
|Staging Database: Connection string format||An optional connection string format to use when connecting to the Staging database. If not specified, the global connection string maintained in the Config panel (see below) is used|
Clicking the Paths... button brings up the Paths dialog. Here you can set the paths to the folders used by the Track. Note that these are the paths as seen from the Execution Server of the Track. It is possible to specify network share paths (starting with \\)
|Runtime||The Director will copy the Director Runtime as well as the Engines to be used by the Track to this folder and launch the Track governing process from here|
|Valuesets||The Default DataServices extension delivered with migFx will load file based Valuesets from here|
|Source files||Source to be loaded into the Staging database must be placed in sub folders under this folder - one subfolder pr Metadata alias in the Source Map|
|Result||The Director will place the unloaded results from the migration in subfolders under this folder, one subfolder pr unload job|
|Temp||Working folder for temporary files utilized by the Track|
|Reformatted files to bulk load||During the load process, source files will be reformatted to a common format understood by the Director. The reformatted files are placed in this folder|
|Bulk load format files||The Source Engine will generated format files to use for the Sql Server bulk load of the reformatted files into the Staging database. The Director will save the files to this folder|
|Bulk load error files||If the bulk load of source data into the Staging database produces any error files, these wil be placed in this folder|
|Source duplicates||The Director contains a facility to detect duplicate lines in Source files. The result of this detection is placed in this folder|
|Archiving||This folder is not active yet, but will be playing a role in the coming release of automatic archiving of the Runtime track|
|Extensions||This folder will be provided for any extensions running as jobs under the Director Runtime|
|Root folder||Clicking the Set Defaults button will automatically create a suggested best-practice hierarchy of paths under the specified Root folder|
|Set Defaults||Click the button automatically create a suggested best-practice hierarchy of paths under the specified Root folder|
Click the Usage button to assign a Track to be used by a given Project - or to release the Track for use by other Projects.
|Vacant||The Track is vacant and not in use by any Project|
|In Use||The Track is in use by a Project|
|Project||Choose the Project using the Track|
|Usage||Give this usage of the Track a label. This label will show up in the Director and in the Tracker|
|Publish||The Track will not show up anywhere in the Director nor the Tracker if Publish is unchecked. Use this setting to keep an new track hidden while you are setting everything up|
|Tracker Expiry||If set, the Usage will not be visible in the Tracker past this date|
Linking Tracks for Archiving will enable the automatic archiving:
You only can link a Runtime Track to an Archive Track being used by the same Project. An automatic Archive will:
- Backup the Archive databases and files to the Archiving folder of the Archive Track.
- Backup the Runtime databases and files and move them to the Archiving folder of the Archive Track
- Restore and replace the Archive databases with the Runtime backup
- Delete the backup of the Runtime databases
In the Config section you can edit a few global configuration settings. Obs: Remember to click the Save button if you make any changes.
|Default connection string format||This is the format the Director Runtime will use for the connection strings. In a given context, it will replace the placeholder string @server with the actual Data source of the given Sql Server instance and the placeholder string @database with the database name.|
Normally this global format is all you need to specify, but as you have seen above, it is possible to override this format in specific instances.
Notice that the connectionstring setting MultipleActiveResultsets is set to true. This setting is vital for the Director Runtime.
|Web Service URL for Translation Valuesets||This is the URL all Tracks will use when they need to load the Translation Valuesets from the Tracker web application to the Director Runtime. The server part of the URL will normally be the same as the URL type in the browser to get hold of the Tracker web application|
|Director client pulse interval||This interval determines how often all connected Director clients will poll the master server for configuration changes. It is preset to 2500 milliseconds which is normally fine. You only need to touch this setting if there are extraordinarily any clients connected and the polling is imposing a heavy load on the master server. Normally this will never be the case.|
|Show Servers in Director by||Determine whether Servers will be shown in the Director client UI by their Machine Name or by their Description. This will change the way Tracks are shown in the Director menu as well as in the Title bar of any Track window|
|Additional configuration settings||Here you can add additional settings that will be automatically added to the Track config file when a track is started. You can add either connection strings (will be added to the connectionStrings section) or application settings (will be added to the appSettings section).|
This is useful if you develop extensions to the Director Runtime that needs their own settings in the config file, for instance a connection string to your own Master database.