CODEWITHSUNDEEP
arraysmultithreadingpowershelljobs
Rhys Brennan
Rhys Brennan

Reputation: 13

Powershell - foreach to array as job (local / multi-threaded)

I'm trying to parse a site to collect price and product details. The script works in a loop however it's very slow. So I'm trying to run a multi-threaded powershell script as a job.

I've tried a lot of suggestions but I'm struggling to get the results out even though I can see its working (the web-request screen flashing up)

I'm only selecting the last 10 but I'll put in a throttle later. Just can't get it to output. Essentially I'd like all results to flow back into $arr. 

#Import Danmurphy Sitelist
[xml] $XmlDocument = (New-Object System.Net.WebClient).DownloadString("http://www.example.com/sites.xml")

#get websites listed
$ImportedProducts = $XmlDocument.DocumentElement.url | select -Last 10

"Killing existing jobs . . ."
Get-Job | Remove-Job -Force
"Done."

#loop through the products

#Create Array
$arr = @()

#$argumentlist 

#ScriptBlock
$ScriptBlock = {
Param($product,$arr)

if ($product.loc -like "http://www.example.com/product/*"){

$uri = $product.loc
$WebResponse = Invoke-WebRequest -Uri $uri -SessionVariable WS 


#mainpricetest
$mainprice = $WebResponse.AllElements | ? { $_.Class -eq 'price-main' } | select innerText

$MainPriceArray = $mainprice.innerText.Split(' ')

$MainUnitArry = $MainPriceArray[1..10]

$MainDollar = $MainPriceArray[0]

$MainUnit = $MainUnitArry -join ' '


$item = New-Object PSObject
$item | Add-Member -type NoteProperty -Name 'Product Site' -Value $($product.loc)
$item | Add-Member -type NoteProperty -Name 'Main Price' -Value $($MainDollar)
$item | Add-Member -type NoteProperty -Name 'Main Unit' -Value $($MainUnit)



$arr += $item

}
}

foreach ($product in $ImportedProducts){
Start-Job -InputObject $ImportedProducts -ScriptBlock $ScriptBlock -ArgumentList $product,$arr
}

$data = Get-Job * | Receive-Job 

#Show Array
$arr

Upvotes: 1

Views: 4038

Answers (2)

Bill Hurt
Bill Hurt

Reputation: 749

If you want to get the results into $arr, you can't do it from within the script block as you are attempting to do. Multiple script blocks running in parallel cannot be allowed to access a single copy of a variable without taking additional steps not worth getting into.

The answer to your problem is going to be to write the output of each script block as regular output. That output is buffered until you use Receive-Job to get the results out of the job at which time you capture it into the $arr variable in a single threaded manner. Below is cod which should get you most of the way there. 

#Import Danmurphy Sitelist
[xml] $XmlDocument = (New-Object System.Net.WebClient).DownloadString("http://www.example.com/sites.xml")

#get websites listed
$ImportedProducts = $XmlDocument.DocumentElement.url | select -Last 10

"Killing existing jobs . . ."
Get-Job | Remove-Job -Force
"Done."

#loop through the products

#Create Array
$arr = @()

#$argumentlist 

#ScriptBlock
$ScriptBlock = {
    Param($product)

    if ($product.loc -like "http://www.example.com/product/*"){

    $uri = $product.loc
    $WebResponse = Invoke-WebRequest -Uri $uri -SessionVariable WS 


    #mainpricetest
    $mainprice = $WebResponse.AllElements | ? { $_.Class -eq 'price-main' } | select innerText

    $MainPriceArray = $mainprice.innerText.Split(' ')

    $MainUnitArry = $MainPriceArray[1..10]

    $MainDollar = $MainPriceArray[0]

    $MainUnit = $MainUnitArry -join ' '


    $item = New-Object PSObject
    $item | Add-Member -type NoteProperty -Name 'Product Site' -Value $($product.loc)
    $item | Add-Member -type NoteProperty -Name 'Main Price' -Value $($MainDollar)
    $item | Add-Member -type NoteProperty -Name 'Main Unit' -Value $($MainUnit)



    Write-Output $item

    }
}

foreach ($product in $ImportedProducts){
    Start-Job -InputObject $ImportedProducts -ScriptBlock $ScriptBlock -ArgumentList $product
}

do {
    $arr += Get-Job -State Completed | Receive-Job -AutoRemoveJob
} while (Get-Job -State Running)

#Show Array
$arr

Upvotes: 1

4c74356b41
4c74356b41

Reputation: 72191

So you would want to use runspaces for that. Runspaces is a pretty complicated thing, luckily we have Posh-RSJob which handles everything for you. https://github.com/proxb/PoshRSJob

You can pass in the script block, so you would need very little adjustments. Probably something like this:

foreach ($product in $ImportedProducts){
    Start-RSJob -ScriptBlock $ScriptBlock
}
Get-RSjob | Receive-RSJob

Upvotes: 2

Related Questions