CODEWITHSUNDEEP
shellcurl
jvilhena
jvilhena

Reputation: 1131

curl from shell: discard output file if transfer timed-out

Using curl from the shell, what is the best way to discard (or detect) files that are not completely downloaded because a timeout occurred? What I'm trying to do is: 

curl -m 2 --compress -o "dest/#1" "http://url/{$list}"

When a timeout occurs, the log shows it, and the part of the file that was downloaded is saved to disk:

[4/13]: http://URL/123.jpg --> dest/123.jpg
 99 97984   99 97189    0     0  45469      0  0:00:02  0:00:02 --:--:-- 62500
curl: (28) Operation timed out after 2000 milliseconds with 97189 bytes received

I'm trying to either get rid of the files that were not 100% downloaded, or have them listed to attempt a resume (-C flag), later.

Upvotes: 1

Views: 351

Answers (1)

jvilhena
jvilhena

Reputation: 1131

The best solution I have found so far is to capture the stderr of the curl call, and parse it with a combination of perl and grep to get the output file names:

curl -m 2 -o "dest/#1" "http://url/{$list}" 2>curl.out
perl -pe 's/[\t\n ]+/ /g ; s/--> /\n/g' curl.out | grep -i "Curl: (28)" | perl -pe 's/ .*//g'

Upvotes: 1

Related Questions