CODEWITHSUNDEEP
multithreadingpowershell
starcodex
starcodex

Reputation: 2298

Calling multiple URLs at a time using multithreading in powershell

I have one URL for which its query changes. The queries are stored in an array so changing the URLs isn't a problem within a loop (I'm not interested in any particular query).

I'm having a hard time creating jobs for each URL and starting a group of jobs at the same time and monitoring them.

I figure to start to iterate through the array of queries 5 at a time, I'd be calling 5 new URLs so every iteration needs to have an array of jobs whose elements are the URLs for that iteration.

Is my approach right? Any pointers will be appreciated!

This is sample code to demonstrate my approach:

$queries = 1..10
$jobs = @()

foreach ($i in $queries) {
  if ($jobs.Count -lt 5) {
     $ScriptBlock = { 
        $query = $queries[$i]
        $path = "http://mywebsite.com/$query"
        Invoke-WebRequest -Uri $path
     }
     $jobs += Start-Job -ScriptBlock $ScriptBlock
  } else {
      $jobs | Wait-Job -Any
  }
}

Upvotes: 3

Views: 2287

Answers (1)

Keith Hill
Keith Hill

Reputation: 201652

You will run into a couple of issues with the code above. The scriptblock gets transferred to a different PowerShell.exe process to execute so it won't have acess to $queries. You will to pass that it like so:

...
$scriptblock = {param($queries)
    ...
}
...
$jobs += Start-Job $scriptblock -Arg $queries

The other issue is that you never remove a completed job from $job so once this $jobs.Count -lt 5 expression evals to false because the count has reached 5, you'll never add anymore jobs. Try something like this:

$jobs | Wait-Job -Any
$jobs = $jobs | Where ($_.State -eq 'Running'}

Then you'll wind up with only the running jobs in $jobs which will allow you to start more jobs as previous jobs complete (or fail).

Upvotes: 2

Related Questions