CODEWITHSUNDEEP
powershell
user2749942
user2749942

Reputation: 43

Download multiple large files asynchronously with Powershell

I have four large OS installation media I need to download. This will take a long time if I wait for each download to finish before moving to the next one. Before downloading, I want to check if the media is already present.

The solution is likely a combination of a hash table, test-path and invoke-webrequest but I just can't crack it.

So in pseudo something on the lines of:

Check if file1 exists
if true then download and check file2
if false check file 2
check if file 2 exists...

So check if files exists and if not, start downloading all the ones that are missing.

I'm not very experienced with PS so all help is much appreciated, thank you very much! Researching the answer was fun but I feel I'm missing a keyword here...

Upvotes: 4

Views: 3634

Answers (2)

Garric
Garric

Reputation: 724

Based on code in https://blog.ironmansoftware.com/powershell-async-method/

[void][Reflection.Assembly]::LoadWithPartialName("System.Threading")
function Wait-Task {
    param([Parameter(Mandatory, ValueFromPipeline)][System.Threading.Tasks.Task[]]$Task)
    Begin {$Tasks = @()}
    Process {$Tasks += $Task}
    End {While(-not [System.Threading.Tasks.Task]::WaitAll($Tasks, 200)){};$Tasks.ForEach({$_.GetAwaiter().GetResult()})}
}
Set-Alias -Name await -Value Wait-Task -Force
@(
    (New-Object System.Net.WebClient).DownloadFileTaskAsync("https://github.com/Microsoft/TypeScript/archive/master.zip","$env:TEMP\TS.master.zip")
    (New-Object System.Net.WebClient).DownloadFileTaskAsync("https://github.com/Microsoft/calculator/archive/master.zip","$env:TEMP\calc.master.zip")
    (New-Object System.Net.WebClient).DownloadFileTaskAsync("https://github.com/Microsoft/vscode/archive/master.zip","$env:TEMP\Vscode.master.zip")
) | await

Upvotes: 1

Mike Twc
Mike Twc

Reputation: 2355

There is a fairly simple way for async downloads using WebClient class, although it's probably not available on older version of PS. See the example below

$files = @(
 @{url = "https://github.com/Microsoft/TypeScript/archive/master.zip"; path = "C:\temp\TS.master.zip"}
 @{url = "https://github.com/Microsoft/calculator/archive/master.zip"; path = "C:\temp\calc.master.zip"}
 @{url="https://github.com/Microsoft/vscode/archive/master.zip"; path = "C:\temp\Vscode.master.zip"}
)

$workers = foreach ($f in $files) { 

$wc = New-Object System.Net.WebClient

Write-Output $wc.DownloadFileTaskAsync($f.url, $f.path)

}

# wait until all files are downloaded
# $workers.Result

# or just check the status and then do something else
$workers | select IsCompleted, Status

Upvotes: 8

Related Questions