Update TFS build environment variable from asynchronously executing PowerShell process

Question

I'm working on a TFS build process featuring adownload of a large ZIP archive from an Artifactory repository. The download typically takes upwards of 20 minutes, effectively doubling my build execution duration.

In order to speed up my build, I wanted to kick off an asynchronous process from a PowerShell build step to request the artifact download and move onto the next build steps whilst the download executes n the background.

I've managed this successfully (Start-Process powershell -WindowStyle Hidden etc...); the artifact does download successfully and, crucially, the download continues and completes after the PowerShell step has finished and the build has continued to subsequent steps.

Now, the downloaded archive will be required fore processing later in the build. So, in order to ensure that the processing does not begin until the download is completed, I want to update a build environment variable 'IsDownloadComplete' from 'False' to 'True'.

The usual PowerShell for environment variable update does not work (I suspect due to executing from orphaned async process?):

"##vso[task.setvariable variable='IsDownloadComplete';]'True'"

So, what to do? How can the environment variable be updated from the async process?

UPDATES: The async process call from my TFS build step looks like this:

$cmd = "$(SupportFunctionsLocation)\[script].ps1 -artifactUrl '$(artifactUrl)' -artifactoryApiKey '$(ArtifactoryApiKey)' -baseDir '$(Build.BinariesDirectory)\[base_dir]' -envVarName 'IsDownloadComplete' -envVarValue 'true'"

try {
    Start-Process PowerShell.exe -ArgumentList "-NoProfile -WindowStyle Hidden -Command $cmd"
    Start-Sleep -s 10
} catch {
    Write-Host $_.ErrorMessage
}

The call to update the build en vironment variable is located in the '$(SupportFunctionsLocation)[script].ps1' script, after the artifact pull is complete:

    param (
    [parameter(mandatory=$true)][string]$artifactUrl, 
    [parameter(mandatory=$true)][string]$apiKey,
    [parameter(mandatory=$true)][string]$baseDir,
    [parameter(mandatory=$true)][string]$envVarName,
    [parameter(mandatory=$true)][string]$envVarValue
)


function Pull-FromArtifactory ($url, $apiKey, $dir) {
    if (-not (Test-Path -Path $dir)) {
        New-Item -Path $dir -ItemType "Directory"
    }
    
    $archive = $null
    Write-Host "Pulling archive from Artifactory repository (this may take a minute or two)..."
    try {        
        $archive = Split-Path $url -leaf
        $headers = @{"X-JFrog-Art-Api" = $apiKey}
        Invoke-RestMethod -Uri $url -Headers $headers -OutFile "$dir\$archive"
    }
    catch {
        Write-Host "Failed to retrieve archive from Artifactory repository..."
        Write-Host $_.Exception.Message
        Exit 1
    }
}

Pull-FromArtifactory $artifactUrl $apiKey $baseDir

# Update 'IsDownloadComplete' build environment variable
Write-Host "##vso[task.setvariable variable=$envVarName;]$envVarValue"

And my subsequently executing debug build step looks like this:

if (Test-Path ("$(Build.BinariesDirectory)\[base_dir]")) {
    do {
    Get-ChildItem "$(Build.BinariesDirectory)\[base_dir]" -Include "test.zip" -Force -Recurse
    Write-Host "Download complete: " $env:IsDownloadComplete
    Start-Sleep -s 30
    } while ($env:IsDownloadComplete -ne 'true')
}

When my build executes, the async process kicks off successfully from the Start-Process call in the PowerShell step. The step doesn't wait; it terminates and the async process keeps executing in the background.

As the script in the the debug step loops, each execution of the Get-ChildItem call shows the file size increase as the async process continues until the download is complete (we know by file size).

But, even though the download is complete, the build environment variable fails to update; the debug script step continues to loop indefinitely.

Log output from debug step:

2020-09-11T13:46:23.5557526Z Directory: C:\BuildAgent_work\15\b[base_dir]

2020-09-11T13:46:23.5557526Z

2020-09-11T13:46:23.5687526Z Mode LastWriteTime Length Name

2020-09-11T13:46:23.5707484Z ---- ------------- ------ ----

2020-09-11T13:46:23.5737531Z -a---- 9/11/2020 9:46 AM 3572999 test.zip

2020-09-11T13:46:23.6027538Z Download complete: false

2020-09-11T13:46:53.6214501Z -a---- 9/11/2020 9:46 AM
30362999 test.zip

2020-09-11T13:46:53.6224502Z Download complete: false

2020-09-11T13:47:23.6271640Z -a---- 9/11/2020 9:46 AM
56742999 test.zip

2020-09-11T13:47:23.6291560Z Download complete: false

2020-09-11T13:47:53.6328511Z -a---- 9/11/2020 9:46 AM
83072999 test.zip

2020-09-11T13:47:53.6338508Z Download complete: false

2020-09-11T13:48:23.6377761Z -a---- 9/11/2020 9:48 AM
95577750 test.zip

2020-09-11T13:48:23.6387762Z Download complete: false

2020-09-11T13:48:53.6424119Z -a---- 9/11/2020 9:48 AM
95577750 test.zip

2020-09-11T13:48:53.6434120Z Download complete: false

2020-09-11T13:49:23.6473880Z -a---- 9/11/2020 9:48 AM 95577750 test.zip

2020-09-11T13:49:23.6483867Z Download complete: false

2020-09-11T13:49:53.6519497Z -a---- 9/11/2020 9:48 AM 95577750 test.zip

2020-09-11T13:49:53.6529503Z Download complete: false

2020-09-11T13:50:23.6560150Z -a---- 9/11/2020 9:48 AM 95577750 test.zip

Note increasing 'test.zip' file size with each iteration.

Answer 1

As described in OP and comments above, I could not get the environment variable update call from the asynchronous job to work. In the end, time constraints required a quick & dirty solution:

1. In the build step starting the async job, before the async process is created, I created a text file, containing the word "False":

   New-Item -Path $tempFilePath -Name "ArtifactDownloadComplete.txt" -ItemType "file" -Value "False"
   

2. Start the async download process; step ends and build progresses (async process executes in the background). After the Invoke-RestMethod call in the async function is complete, content of 'ArtifactDownloadComplete.txt' file is updated to "True":

   ((Get-Content -Path "$tempFilePath\ArtifactDownloadComplete.txt" -Raw) -replace "False","True") | Set-Content -Path "$tempFilePath\ArtifactDownloadComplete.txt"
   

3. Before the downloaded archive is required for processing in the build, I added a step to unpackage the archive. In the PowerShell implementation of this step is included a loop delay; the breaking condition for this loop in order to begin the step processing is the content of the 'ArtifactDownloadComplete.txt' file having being updated to read "True":

   do {
       if ((Get-Content $statusFile) -eq "False") {
           Start-Sleep -s 20
       }
       else {
           Write-ConsoleLog "Artifact download complete."
       }
   }
   while ((Get-Content $statusFile) -eq "False")
   

Quick and dirty indeed.

Possible area of interest for future development of his process: TFS REST API.

Answer 2

You're not far off. I'd do the following

1. Download the huge zip file with a script. Last step is to set an variable "Write-Host ##vso[task.setvariable variable=IsDownloadComplete]True"
2. Do the steps that run parallel with the large download, until the job that unites everything is about to start.
3. Add a step with a powershell script that checks for$env:IsDownloadComplete -eq 'True'. Let it loop until IsDownloadComplete is true, or you hit a timeout (you don't want to run your build forever when something goes wrong) . Don't forget to add an refreshenv in this step, because the value is updated by a different process.
4. The rest of the steps