CODEWITHSUNDEEP
curlflurry
rfsk2010
rfsk2010

Reputation: 8611

Curl: sleep/delay between requests

I am trying to download flurry exception logs using the following command.

curl --cookie ./flurry.jar -k -L "https://dev.flurry.com/exceptionLogsCsv.do?projectID=49999&versionCut=versionsAll&intervalCut=allTime&direction=1&offset=[0-100:10]" --output "exception#1.csv"

It works fine and it downloads the csv files based on the offset(10,20,30 etc). I would like to insert a delay between each request. Is it possible to do that in CURL?

Upvotes: 17

Views: 35831

Answers (4)

Geremia
Geremia

Reputation: 5656

in bash, this will pause a random number of seconds in the range 0-60:

for d in {0..100..10}
do
    i=`printf "%03d" $d`
    curl --cookie ./flurry.jar -k -L 'https://dev.flurry.com/exceptionLogsCsv.do?projectID=49999&versionCut=versionsAll&intervalCut=allTime&direction=1&offset='$d --output 'exception'$i'.csv'
    sleep $(($RANDOM*60/32767))
done

Upvotes: 4

Muhammad Abrar
Muhammad Abrar

Reputation: 2322

Using bash shell (Linux) :

while :
do
    curl --cookie ./flurry.jar -k -L "https://dev.flurry.com/exceptionLogsCsv.do?projectID=49999&versionCut=versionsAll&intervalCut=allTime&direction=1&offset=[0-100:10]" --output "exception#1.csv"
    sleep 5m
done

It is an infinite loop, and the delay is given by the sleep command.

Edit. On Windows machine, you can do this trick instead :

for /L %i in (0,0,0) do (
    curl --cookie ./flurry.jar -k -L "https://dev.flurry.com/exceptionLogsCsv.do?projectID=49999&versionCut=versionsAll&intervalCut=allTime&direction=1&offset=[0-100:10]" --output "exception#1.csv"
    ping -n XX 127.0.0.1>NUL
)

The sleep command is not available on Windows. But you can use ping to "emulate" it. Just replace the XX above with the number of seconds you want to delay.

Upvotes: 9

legoscia
legoscia

Reputation: 41648

With curl 7.84.0 and later, you can use request rate limiting with the --rate option:

The request rate is provided as N/U where N is an integer number and U is a time unit. Supported units are s (second), m (minute), h (hour) and d (day, as in a 24 hour unit). The default time unit, if no /U is provided, is number of transfers per hour.

So to wait 10 seconds between each request, use curl --rate 6/m.

With curl 8.10.0 and later, you can use a number for the denominator as well:

Starting in curl 8.10.0 this option accepts an optional number of units. The request rate is then provided as N / Z U (with no spaces) where N is a number, Z is a number of time units and U is a time unit.

You could ask curl to send one request every 10 seconds with curl --rate 1/10s.

Upvotes: 4

Alberto
Alberto

Reputation: 2982

wget has delay options

wget --wait=seconds

and also random delay option

wget --random-wait

Upvotes: 5

Related Questions