CODEWITHSUNDEEP
powershellforeach-object
What am I
What am I

Reputation: 73

Powershell Foreach-Object -Parallel how to change the value of a variable outside the loop (track progress)

This code prints a simple progression, doing one thing at a time:

$files = 1..100
$i = 0
$files | Foreach-Object {
    $progress = ("#" * $i)
    Write-Host  "`r$progress" -NoNewLine
    $i ++
    Start-Sleep -s 0.1
}

But if I want to do two things in parallel at the same time, I can't output the progress because I can't change the variable outside the parallel loops. This doesn't do what's needed:

$files = 1..100
$i = 0
$files | Foreach-Object -ThrottleLimit 2 -Parallel {
    $progress = ("#" * $i)
    Write-Host  "`r$progress" -NoNewLine
    $i ++
    Start-Sleep -s 0.1
}

I can't find a good solution for accessing an external variable not only to read it with $Using, but also to change it. Is this even possible in Powershell 7?

Upvotes: 7

Views: 11159

Answers (4)

Mehrdad Mirreza
Mehrdad Mirreza

Reputation: 1082

I found this solution with thead-safe dictionaries on https://devblogs.microsoft.com/powershell/powershell-foreach-object-parallel-feature/. I think this is at the moment the safest solution:

$threadSafeDictionary = [System.Collections.Concurrent.ConcurrentDictionary[string,object]]::new()
Get-Process | ForEach-Object -Parallel {
    $dict = $using:threadSafeDictionary
    $dict.TryAdd($_.ProcessName, $_)
}

$threadSafeDictionary["pwsh"]

Upvotes: 2

FuegoJohnson
FuegoJohnson

Reputation: 392

If you are working with larger values of -ThrottleLimit (say 4+), using a synchronized queue (for thread safety), Write-Progress, and jobs can be a nice solution for tracking progress. As others have mentioned, the $Using keywords allows you access to variables within a scriptblock scope:

$files = 1..100
$queue = [System.Collections.Queue]::new()
1..$files.Count | ForEach-Object { $queue.Enqueue($_) }
$syncQueue = [System.Collections.Queue]::synchronized($queue)

$job = $files | ForEach-Object -AsJob -ThrottleLimit 6 -Parallel {
    $sqCopy = $Using:syncQueue
    #Simulating work...do stuff with files here
    Start-Sleep (Get-Random -Maximum 10 -Minimum 1)

    #Dequeue element to update progress
    $sqCopy.Dequeue()
}

#While $job is running, update progress bar
while ($job.State -eq 'Running') {
    if ($syncQueue.Count -gt 0) {
        $status = ((1 / $syncQueue.Count) * 100)
        Write-Progress -Activity "Operating on Files" -Status "Status" -PercentComplete $status
        Start-Sleep -Milliseconds 100
    }
}

In my experience, using a synchronized hashtable was too messy for several threads; I wanted a single, clean progress bar. It depends on your use case though. Thought I'd add my piece to the other excellent answers.

Upvotes: 6

mclayton
mclayton

Reputation: 9975

Per this article - PowerShell ForEach-Object Parallel Feature - you can reference variables from the "outer" script using the $using keyword:

e.g.

$files = 1..100
$i = 100;
$files | Foreach-Object -ThrottleLimit 2 -Parallel {
    write-host ($using:i)
    Start-Sleep -s .1
}
# 100
# 100
# etc

But if you try to update the value you'll get this:

$files | Foreach-Object -ThrottleLimit 2 -Parallel {
    $using:i += $using:i
    Start-Sleep -s .1
}

ParserError:
Line |
   2 |      $using:i += $using:i
     |      ~~~~~~~~
     | The assignment expression is not valid. The input to an assignment operator must be an object that is able to accept
     | assignments, such as a variable or a property.

Basically, you can't assign back to the $using:i variable.

What you could do is mutate the properties of a complex object instead - e.g. this:

$counter = @{ "i" = 0 }
$files | Foreach-Object -ThrottleLimit 2 -Parallel {
     ($using:counter).i = ($using:counter).i + 1
     Start-Sleep -s .1
}
$counter

# Name                           Value
# ----                           -----
# i                              100
#

which lets you update the value, but may not (probably won't be) be thread-safe.

Upvotes: 9

js2010
js2010

Reputation: 27423

I think this is right, based on Writing Progress across multiple threads with Foreach Parallel. It may be missing a lock, but just for writing progress it's probably not a big deal. In this case you can just use the filename for the progress too.

$hash = @{i = 1}
$sync = [System.Collections.Hashtable]::Synchronized($hash)

$files = 1..100
$files | Foreach-Object -throttlelimit 2 -parallel {
    $syncCopy = $using:sync
    $progress = '#' * $syncCopy.i
    #$progress = '#' * $_
    Write-Host  "`r$progress" -NoNewLine
    $syncCopy.i++
    Start-Sleep .1
}

Output:

####################################################################################################

Upvotes: 3

Related Questions