CODEWITHSUNDEEP
curlwget
Hintswen
Hintswen

Reputation: 4007

trying to use curl to download a series of files

I'm trying to use curl to download a seried of files in the following format:

http://example.com/001.jpg

..

http://example.com/999.jpg

So I used this command:

time curl "http://example.com/[0-9][0-9][0-9].jpg" -o "#1#2#3.gif"

But some of the files don't exist, and that command will create the files on my end but really it will just contain the error page.

I need a way for curl to check if it exists on the remote server before downloading it.

I know wget can do it, but wget seems to take about 5 times longer to run. The command I used to try it in wget is this:

time wget http://example.com/{0..9}{0..9}{0..9}.jpg

Upvotes: 1

Views: 2751

Answers (1)

alex vasi
alex vasi

Reputation: 5344

Try -f/--fail option:

(HTTP) Fail silently (no output at all) on server errors. This is mostly done to better enable scripts etc to better deal with failed attempts. In normal cases when a HTTP server fails to deliver a document, it returns an HTML document stating so (which often also describes why and more). This flag will prevent curl from outputting that and return error 22.

This method is not fail-safe and there are occasions where non-successful response codes will slip through, especially when authentication is involved (response codes 401 and 407).

Upvotes: 1

Related Questions