CODEWITHSUNDEEP
powershelldownload
Robert Franzke
Robert Franzke

Reputation: 11

Powershell Re-trying file download if it download encouters 404 error

So I have a powershell script which downloads a a set of files one after another using a foreach-object loop (and System.Net.WebClient). From time to time I get a 404 error from the source URL for a particular file. The error is non-termninating so the script moves on when encountering this type of error. The issue is that the fileset needs to be downloaded as a set and is no good unless all files are available. So what I need to do is have the script pause and retry the download if it encounters a 404 error. I am having trouble figuring out how best to have this happen. Here is the code I am using for the download. Originally it just did this:

Get-Content $shaFile | Foreach-Object { ($_ -split " ", 2)[0] } |
        Where-Object { ($adfs.$v.iniFile,$shaFile -NotContains $_) -and (Is-Match $_) } | Foreach-Object {
            #$null=Download-File $w $_ $dir

So this downloads the files properly....unless I get some type of error like a 404. What I want to do is have it pick the error and then try and download the missed file, and keep trying until its downloaded. Typically this seems to be caused by a timing issue whereby I am trying to download a file the source URL is still publishing (copying and is not yet available). So if I can just detect the error and keep trying to download the file until downloaded as the source URL will eventually publish it. What I came up with is below:

Get-Content $shaFile | Foreach-Object { ($_ -split " ", 2)[0] } |
        Where-Object { ($adfs.$v.iniFile,$shaFile -NotContains $_) -and (Is-Match $_) } | Foreach-Object {

        try {
            $null=Download-File $w $_ $dir
        }

        catch {
            sleep 60
            $null=Download-File $w $_ $dir
            "Re-downloading missed file"
    }
    }

So I think this will have the script try again in a minute if it encounters an error. Thats fine I guess but I wanted to actually try and check for when the file is actually downloaded before moving on. So like some kind of while loop that says if $_ does not exist in destination, keep trying to download it until it does. In the above code I am calling a function called "Download-file" and a function "Is-Match". Download file uses System.Net.WebClient to download files from a website.

I thought maybe the following might work but am not sure its the best way to handle this:

Get-Content $shaFile | Foreach-Object { ($_ -split " ", 2)[0] } |
            Where-Object { ($adfs.$v.iniFile,$shaFile -NotContains $_) -and (Is-Match $_) } | Foreach-Object {
do{
    $Failed = $false
    Try{
        $null=Download-File $w $_ $dir
    } 
    catch { $Failed = $true }
} while ($Failed)

Or perhaps I could do the following

Get-Content $shaFile | Foreach-Object { ($_ -split " ", 2)[0] } |
                Where-Object { ($adfs.$v.iniFile,$shaFile -NotContains $_) -and (Is-Match $_) } | Foreach-Object {
Try{
        $null=Download-File $w $_ $dir
}
   catch {   
    while ($error){
    Start-Sleep -s 60
    $null=Download-File $w $_ $dir
}
}

I can test the code but I cannot test for the error very easily as most times the files are all there. Anyway I would appreciate any help here. Thanks in advance.

Upvotes: 1

Views: 905

Answers (1)

Lee Exothermix
Lee Exothermix

Reputation: 356

I would try something like this. It assumes that the object returned for Download-File is null if the download fails - you'll have to check how it actually behaves. To simulate a failed download, you could re-write the returned object from Download-File to $null.

$maxRetries = 5
$retryWaitSeconds = 60
Get-Content $shaFile | Foreach-Object { ($_ -split " ", 2)[0] } |
        Where-Object { ($adfs.$v.iniFile,$shaFile -NotContains $_) -and (Is-Match $_) } | Foreach-Object {
            do {
                $file = Download-File $w $_ $dir
                #force the file download to fail 3 times for testing
                foreach ($i in 1..3) {
                    $file = $null
                }

                #sleep if the file download failed.
                if ($file -eq $null) {
                    start-sleep $retryWaitSeconds
                }
            }
            while (($file -eq $null) -and ($retryCount -le $maxRetries)) #limit the maximum number of retries in case download will never succeed.
        }

Upvotes: 1

Related Questions