CODEWITHSUNDEEP
powershell
Mark
Mark

Reputation: 2926

How can I download a file only if it has changed using powershell

I am using this simple function to download a file:

function DownloadFile([string]$url, [string]$file)
{
    $clnt = new-object System.Net.WebClient
    Write-Host "Downloading from $url to $file " 
    $clnt.DownloadFile($url, $file)
}

It works fine but the script I am using that calls it can be called many times and at present that can mean downloading the file(s) many times.

How can i modify the function to only download if the file doesn't exist locally or the server version is newer (e.g. the LastModifiedDate on the server is greater than the LastModifiedDate locally)?

EDIT: This is what I've got so far, seems to work but would like not to have 2 calls to the server.

function DownloadFile([string]$url, [string]$file)
{
    $downloadRequired = $true
    if ((test-path $file)) 
    {
        $localModified = (Get-Item $file).LastWriteTime 
        $webRequest = [System.Net.HttpWebRequest]::Create($url);
        $webRequest.Method = "HEAD";
        $webResponse = $webRequest.GetResponse()
        $remoteLastModified = ($webResponse.LastModified) -as [DateTime] 
        $webResponse.Close()

        if ($remoteLastModified -gt $localModified)
        {
            Write-Host "$file is out of date"
        }
        else
        {
            $downloadRequired = $false
        }

    }

    if ($downloadRequired)
    {
        $clnt = new-object System.Net.WebClient
        Write-Host "Downloading from $url to $file"
        $clnt.DownloadFile($url, $file)
    }
    else
    {
        Write-Host "$file is up to date."
    }
}

Upvotes: 8

Views: 8874

Answers (3)

Christopher G. Lewis
Christopher G. Lewis

Reputation: 4836

I've been beating this up this week, and came up with this

# ----------------------------------------------------------------------------------------------
# download a file
# ----------------------------------------------------------------------------------------------
Function Download-File {
    Param (
        [Parameter(Mandatory=$True)] [System.Uri]$uri,
        [Parameter(Mandatory=$True )] [string]$FilePath
    )

    #Make sure the destination directory exists
    #System.IO.FileInfo works even if the file/dir doesn't exist, which is better then get-item which requires the file to exist
    If (! ( Test-Path ([System.IO.FileInfo]$FilePath).DirectoryName ) ) { [void](New-Item ([System.IO.FileInfo]$FilePath).DirectoryName -force -type directory)}

    #see if this file exists
    if ( -not (Test-Path $FilePath) ) {
        #use simple download
        [void] (New-Object System.Net.WebClient).DownloadFile($uri.ToString(), $FilePath)
    } else {
        try {
            #use HttpWebRequest to download file
            $webRequest = [System.Net.HttpWebRequest]::Create($uri);
            $webRequest.IfModifiedSince = ([System.IO.FileInfo]$FilePath).LastWriteTime
            $webRequest.Method = "GET";
            [System.Net.HttpWebResponse]$webResponse = $webRequest.GetResponse()

            #Read HTTP result from the $webResponse
            $stream = New-Object System.IO.StreamReader($webResponse.GetResponseStream())
            #Save to file
            $stream.ReadToEnd() | Set-Content -Path $FilePath -Force 

        } catch [System.Net.WebException] {
            #Check for a 304
            if ($_.Exception.Response.StatusCode -eq [System.Net.HttpStatusCode]::NotModified) {
                Write-Host "  $FilePath not modified, not downloading..."
            } else {
                #Unexpected error
                $Status = $_.Exception.Response.StatusCode
                $msg = $_.Exception
                Write-Host "  Error dowloading $FilePath, Status code: $Status - $msg"
            }
        }
    }
}

Upvotes: 11

FHC
FHC

Reputation: 101

# If the local directory exists and it gets a response from the url,
# it checks the last modified date of the remote file. If the file
# already exists it compares the date of the file to the file from
# the url. If either the file doesn't exists or has a newer date, it
# downloads the file and modifies the file's date to match.

function download( $url, $dir, $file ) {
 if( Test-Path $dir -Ea 0 ) {
  $web = try { [System.Net.WebRequest]::Create("$url/$file").GetResponse() } catch [Net.WebException] {}
  if( $web.LastModified ) {
   $download = 0
   if(-Not(Test-Path "$dir\$file" -Ea 0)) { $download = 1 }
   elseif((gi "$dir\$file").LastWriteTime -ne $web.LastModified) { $download = 1 }
   if( $download ) {
    Invoke-WebRequest "$url/$file" -OutFile "$dir\$file" | Wait-Process
    (gi "$dir\$file").LastWriteTime = $web.LastModified
   }
   $web.Close()
  }
 }
}
download "https://website.com" "$env:systemdrive" "file.txt"

Upvotes: 0

Bacon Bits
Bacon Bits

Reputation: 32145

Last modified is in the HTTP response headers.

Try this:

$clnt.OpenRead($Url).Close();
$UrlLastModified = $clnt.ResponseHeaders["Last-Modified"];

If that's newer than the date on your file, your file is old.

The remote server doesn't have to respond with an accurate date or with the file's actual last modified date, but many will.

GetWebResponse() might be a better way to do this (or more correct way). Using OpenRead() and then Close() immediately afterwards bothers my sensibilities, but I may be crazy. I do mostly work on databases.

Upvotes: 1

Related Questions