Version 1.13 is finally getting 100% rid of the Director. While version 1.12 ported most of the Director client functionality to the Portal, the Visual Studio extension (VsExtension) still used the Director client as a command line in order to Deploy and Debug engines.
In preparation for the testing of 1.13, a new VM has been established in Azure to host the databases, Runtime and Studio of version 1.13. The portal is hosted in an Azure app service (staging) and is - via its configuration connection strings - accessing the databases on the new server.
However, due to the way the Runtime and Engines are currently deployed (also in 1.13), it is necessary to deploy a duplicate instance of the Portal locally on the new server. It is a great shame to have to deploy the Portal twice. The purpose of this epic is to implement a new and more decoupled Runtime and Engine deployment model.
This changes the way the VsExtension Deploys and Debugs - and also changes the way the Runtime Agent service retrieves the Runtime and Engine libraries when a track is started and ultimately how new versions of the Runtime is installed.
The Runtime contains the libraries used by the process running a track (the TrackProcess). Currently the Runtime is deployed with a Windows msi installer (from our download page), that is run on the master server and installs the libraries on a local folder on master server. The msi saves the install location in the Windows registry (HKLM\SOFTWARE\HoppTech\MigFx\Installation).
When deploying an engine in the VsExtension, the engine project is built in Visual Studio and afterwards, endpoints in the Portal are called to upload a zip file with the engine libraries and to unzip to the correct location for the engine (given by the [director].[Engine] table in the master database). This location is normally a local folder on the master server.
The Runtime Agent is a Windows Service that is installed on all execution servers. The agent runs under the migFx application user and polls the master database for any request from the Portal to start a Track that exists on the execution server (identified by machine name).
See Exercise 4.1 – Deploy and Setup New Engines : Support (hopp.tech) for a diagram
Upon request, the Agent copies (File.Copy) first the Engine libraries for the track and afterwards the Runtime libraries from the remote share folders to the local runtime folder for the Track and then launches the TrackProcess. The app.config file of the Runtime agent contains the folder location the Runtime libraries and the connection string to the master database (manually edited after installation). The remote share folder location of the Engine libraries is retrieved from the master database.
In a multi-server setup, it is necessary to create a network file share for the Runtime and Engine location and give the migFx application user full control.
The VsExtension retrieves the Runtime via a Portal endpoint. This endpoint queries the Windows registry for the install location of the Runtime and transfers it as a zip file to the VsExtension.
The VsExtension keeps a locally cached version of the Runtime and initially calls a Portal endpoint to query the remote version of the Runtime. The Runtime is retrieved only if remote and local versions differ.
In the current implementation, the Portal Api is accessing both local folders and the Windows Registry on the master server. This works when the Portal is deployed locally on the master server itself. But it is obviously a problem if the Portal is deployed elsewhere - for instance in an Azure app service with no visibility of neither folders nor registry on the master server.
Instead of deploying the Runtime libraries and the Engine libraries to a local folder on the master server and afterwards retrieve them by accessing this folder when starting the track or debugging in Visual Studio, this should be handled in the Portal in a way that is decoupled from the master and execution servers.
- Instead of releasing new versions of the Runtime as an msi that must be installed on the master server, the Runtime is released as a zip file on our download page. The Runtime is deployed by uploading this zip file in the Portal
- The Portal endpoint that stores the Runtime zip can inspect and save the version number (used by the VsExtension caching of the Runtime libraries)
- Likewise, the Portal Deploy endpoints called by the VsExtension cli stores the deployed engine as a zip file locally in the Portal
- The Portal can either store the zip files in an app_data folder under the Portal or - alternatively - in a VarBinary column in a new table in the master database (probably simpler to set up)
- The Portal URL is stored the app.config file of the Runtime Agent service instead of the share path to the Runtime installation folder
- It can be considered to trawl the Runtime Agent for all direct Sql connections to the master database and replace with calls to Portal endpoints. Then the master connection string can be removed from the app.config, leaving only the Portal URL
- The Agent retrieves the Runtime and Engine zip files by calling endpoints in the Portal and unzips in the local runtime folder for the Track (this replaces the file copy that is done today)
The VsExtension client should remain more or less unaltered. Only the Portal endpoints called by the client should be modified to conform to the new model.
- Install and configure the Agent Windows service on a new Execution server from the Portal similar to hosting an Azure DevOps build agent: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/windows-agent?view=azure-devops
- Use the 'Upload zip file' method to deploy Portal extensions
- On first install the appsettings.json must still be updated manually. Unless we can figure out a robust way to do this as part of the Upload and Deploy