Start a new topic

Manage Servers, Projects, Tracks and Engines in Director

Manage Servers, Projects and Tracks

Finally we got around to implementing the long missing UI to manage Servers, Projects and Tracks in the Director. About time! No longer will you have to manually enter data into the master tables in Sql Server and be responsible for the data integrity yourself, the new UI is there to help. Even better, the new management of Projects also handles all the necessary trickery around authorizations, so you no longer have to open the .NET Sql Authorization Manager to do so manually.


Manage Engines too

As a brand new thing, you can now also manage the different Engines in the Director, and for a Project you can identify the Engines this Projects uses. Once you have done this, the Director Runtime - more precisely the Track Watcher service - will use this information when you start a Track and seamlessly edit the config file of the Track to identify and load the correct engines. This behavior is new, if you wish to keep on manually editing the Track config files yourself, you can retain the old behavior with a new setting in the config file of the Track Watcher Service.


Here's what it all looks like, the new management interface is presented in the Runtime section of the Tree Menu in the MigFx Administration form:



Let's go through the sections one by one.


Servers

Here you maintain all the Servers in your migFx setup:


You can add new servers or modify existing ones. You also delete a server, but if it is in use by Tracks, the deletion will not succeed. Add or modify will show the Server dialog:


Computer Name
This must be the name given locally on the server. If the server is an active server running one or more tracks, the migFx Track Watcher Windows Service will use this name to identify any track start requests destined for the server.

In detail the Watcher Service will get the name of the machine from the operating system, and use this name to look for the Server in the Director - so if the machine Name is not correct, the Watcher will not find anything, and you will be unable to start any Tracks on the Server
Description
The Description of the Server. The Director will show this description or the Machine Name of the Server in the UI as governed by the setting Show Servers in Director by described below (see Config)
Default track path
This is the default root folder for all the Tracks on the Server. When creating a new Track, the Director will automatically suggest best practice naming of all the folder paths for the new Track under this root
Network Name
The (optional) network name to use when connecting to the Sql Server instance on this Server. If blank, Machine Name will be used
Sql Instance
An (optional) name for the Sql Instance on the Server. Leave blank for the Default Instance
Sql Port
An (optional) port for the Sql Instance on the Server. Leave blank for the default port
DataSource
The label DataSource will update to show the exact DataSource that is constructed from the values of Machine Name, Network Name, Sql Instance and Port


Engines

Here you create and maintain the Engines that your Projects can use:



You can add or modify Engines. If an Engine is not in use by any Project, you can also delete it. Add or Modify will show the Engine dialog:

Engine name
The Engine name is displayed when you select Engines for your Projects
Engine type
The Engine can be either a Source- or a Target Engine
Installation path
This is the path from which the Director Runtime will copy the Engine when a Track starts. This can be a network location (starting with \\) and should identify the same location as the one specified in Visual Studio in the migFx.config file for the engine project
Library
This is the name of the library that the Director will load in order to instantiate objects from the Engine. This must be the same name that is specified in Visual Studio as the assembly name for the Engine project. For the Source Engine, it must be the assembly name of the EngineCustom project. Note that you do not have to specify the .dll extension. If you do it will be removed before saving


Projects

Maintaining projects happens here:



You can add and modify Projects. You can delete a Project if it is not running in any Tracks. Note that deleting a Project will only remove it from the Director management. It will not delete the Project database.


Add or Modify will show the Project dialog:



ID
The Project ID is assigned when a new projects is added
Name
This is the name that is displayed for the Project everywhere: In the Director and in the Tracker
State
The State can be Active or Inactive. Inactive projects are not available in Director but are still available in Tracker.  An Inactive project cannot have any Engines connected. Use the Inactive state for finished projects that only live as archives but do not have any execution tracks
Source- and Target Engine
Here you select the Source- and Target Engines that the Project will use
Mode
Mode can be Export or Migration:

Export 
The project is only exporting data. In this mode there is no Import available in the Director for any Tracks running the Project and you cannot select a Target Engine for the Project

Migration
This is the full migration exporting and importing data.
Default language
This is the language to display for the project in the Tracker, if a user has not selected any language in the Tracker user settings
Project Database: Server
The Server where the Project database is located
Project Database: Database
The name of the Project database. Note that you must create this database yourself.
Project Database: Connection string format
An optional connection string format to use when connecting to the database. If not specified, the global connection string maintained in the Config panel (see below) is used


Tracks

This is where you create and maintain Tracks on your Servers:



You can add and modify Tracks, decide which Project uses a given Track and finally link 2 different Tracks for Archiving (see below). You can delete a Track if it is not in use. Note that deleting a Track only removes it from the Director management. Database or files for the track are not deleted.


Note the Filter dropdown to filter the list of tracks by: 

  • All
  • Runtime Tracks only
  • Archive Tracks only


Add/Modify Track



ID
The Track ID is assigned when a new Track is added
Runtime version
The version of the Director Runtime the Track runs. This enables side-by-side installations of new versions of the Director Runtime. Current version at the time of writing is 1.0.
Execution Server
The Server running the Track Watcher Service that will monitor for start requests for the Track. All jobs in the Track will run on this server
Track Number
The number of the Track on the Execution Server. This must unique for the Server. If you try to create a Track with a duplicate number, you will get an error
Paths...
Click the Paths... button to specify the folders to use in the Track (see below). If you do not do this for a new Track, a set of default folders will automatically be created
Archive
Check this to specify that this Track is an Archive Track and will not be executing any migration jobs. You can only change this setting if the Track is not participating in any Archiving link (see below).

Archive Tracks only show up in the Tracker and are hidden from the Director
Migration Database: Server
The Server where the Migration database is located. It is default the same as the Execution Server, but you can choose another server
Migration Database: Database name
The name of the Migration database. Note that you must create this database yourself
Migration Database: Connection string format
An optional connection string format to use when connecting to the Migration database. If not specified, the global connection string maintained in the Config panel (see below) is used
Staging Database: Server
The Server where the Staging database is located. It is default the same as the Execution Server, but you can choose another server
Staging Database: Database name
The name of the Staging database. Note that you must create this database yourself
Staging Database: Connection string format
An optional connection string format to use when connecting to the Staging database. If not specified, the global connection string maintained in the Config panel (see below) is used


Paths

Clicking the Paths... button brings up the Paths dialog. Here you can set the paths to the folders used by the Track. Note that these are the paths as seen from the Execution Server of the Track. It is possible to specify network share paths (starting with \\)



Paths
Runtime
The Director will copy the Director Runtime as well as the Engines to be used by the Track to this folder and launch the Track governing process from here
Valuesets
The Default DataServices extension delivered with migFx will load file based Valuesets from here
Source files
Source to be loaded into the Staging database must be placed in sub folders under this folder - one subfolder pr Metadata alias in the Source Map
Result
The Director will place the unloaded results from the migration in subfolders under this folder, one subfolder pr unload job
Temp
Working folder for temporary files utilized by the Track
Reformatted files to bulk load
During the load process, source files will be reformatted to a common format understood by the Director. The reformatted files are placed in this folder
Bulk load format files
The Source Engine will generated format files to use for the Sql Server bulk load of the reformatted files into the Staging database. The Director will save the files to this folder
Bulk load error files
If the bulk load of source data into the Staging database produces any error files, these wil be placed in this folder
Source duplicates
The Director contains a facility to detect duplicate lines in Source files. The result of this detection is placed in this folder
Archiving
This folder is not active yet, but will be playing a role in the coming release of automatic archiving of the Runtime track
Extensions
This folder will be provided for any extensions running as jobs under the Director Runtime

Root folder
Clicking the Set Defaults button will automatically create a suggested best-practice hierarchy of paths under the specified Root folder
Set Defaults
Click the button automatically create a suggested best-practice hierarchy of paths under the specified Root folder


Usage

Click the Usage button to assign a Track to be used by a given Project - or to release the Track for use by other Projects.



Vacant
The Track is vacant and not in use by any Project
In Use
The Track is in use by a Project
Project
Choose the Project using the Track
Usage
Give this usage of the Track a label. This label will show up in the Director and in the Tracker
Publish
The Track will not show up anywhere in the Director nor the Tracker if Publish is unchecked.  Use this setting to keep an new track hidden while you are setting everything up
Tracker Expiry
If set, the Usage will not be visible in the Tracker past this date


Archiving

Linking Tracks for Archiving will enable the automatic archiving in the Director:


You only can link a Runtime Track to an Archive Track being used by the same Project. An automatic Archive in the Director will 

  • Backup the Archive databases and files to the Archiving folder of the Archive Track. 
  • Backup the Runtime databases and files and move them to the Archiving folder of the Archive Track
  • Restore and replace the Archive databases with the Runtime backup
  • Delete the backup of the Runtime databases


Config

In the Config section you can edit a few global configuration settings. Obs: Remember to click the Save button if you make any changes.



Default connection string format
This is the format the Director Runtime will use for the connection strings. In a given context, it will replace the placeholder string @server with the actual Data source of the given Sql Server instance and the placeholder string @database with the database name.

Normally this global format is all you need to specify, but as you have seen above, it is possible to override this format in specific instances.

Notice that the connectionstring setting MultipleActiveResultsets is set to true. This setting is vital for the Director Runtime.
Web Service URL for Translation Valuesets
This is the URL all Tracks will use when they need to load the Translation Valuesets from the Tracker web application to the Director Runtime. The server part of the URL will normally be the same as the URL type in the browser to get hold of the Tracker web application
Director client pulse interval
This interval determines how often all connected Director clients will poll the master server for configuration changes. It is preset to 2500 milliseconds which is normally fine. You only need to touch this setting if there are extraordinarily any clients connected and the polling is imposing a heavy load on the master server. Normally this will never be the case.
Show Servers in Director by
Determine whether Servers will be shown in the Director client UI by their Machine Name or by their Description. This will change the way Tracks are shown in the Director menu as well as in the Title bar of any Track window
Additional configuration settings
Here you can add additional settings that will be automatically added to the Track config file when a track is started. You can add either connection strings (will be added to the connectionStrings section) or application settings (will be added to the appSettings section).

This is useful if you develop extensions to the Director Runtime that needs their own settings in the config file, for instance a connection string to your own Master database.


Login or Signup to post a comment