CODEWITHSUNDEEP
linuxdownloadwget
cgtechuk
cgtechuk

Reputation: 11

wget to append already downloaded files and skip completed

I am trying to get wget to download all the content from a webserver and it seems to be going well however there are problems with the server I am currently downloading to running of of disk space and therefore stopping.

I am looking for a way to get wget to download all the website content and if it finds a file it has already processed to check and see if it is smaller than the one on the web server if it is continue the broken download or at worse re-download it. If the file does exist and is same size then move onto the next one

I am currently using

wget -r --no-parent -N http://www.website.com

Upvotes: 1

Views: 1520

Answers (1)

Rich Alloway - RogueWave
Rich Alloway - RogueWave

Reputation: 194

Check man wget for the --continue (or -c) option. The man page even shows this option used with recursive downloads.

From the man page:

   -c
   --continue
       Continue getting a partially-downloaded file.  This is useful
       when you want to finish up a download started by a previous
       instance of Wget, or by another program.  For instance:

               wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z

       If there is a file named ls-lR.Z in the current directory,
       Wget will assume that it is the first portion of the remote 
       file, and will ask the server to continue the retrieval from
       an offset equal to the length of the local file.

and, showing -r and -c used together:

   You may put several options that do not require arguments together, like:

           wget -drc <URL>

   This is completely equivalent to:

           wget -d -r -c <URL>

-Rich Alloway (RogueWave)

Upvotes: 2

Related Questions