CODEWITHSUNDEEP
c++linuxcurldownloadmultiprocessing
inbaly
inbaly

Reputation: 612

curl stops downloading using multiple connections on Linux

I have a problem using curl,from version 1.28.1 to 1.37.1 - same problem occurs: In Linux environment, using the command line, I am running the following commands simultaneously:

curl -o /dev/null "http://example.com/short_video.mp4" -o /dev/null "http://example.com/short_video.mp4" &
curl -o /dev/null "http://example.com/short_video.mp4" &
curl -o /dev/null "http://example.com/short_video.mp4" &
curl -o /dev/null "http://example.com/short_video.mp4" &

The problem is - it starts downloading successfully, but often, on a high speed internet connection, one of the downloads get stuck on a zero speed, and the whole process gets stuck.. the only thing that helps is using the speed limit:

curl --speed-limit 5 -o /dev/null "http://example.com/short_video.mp4" &

and then it recognizes that the speed is low and kills the download, but I want to know how to avoid this situation in the first place.

Upvotes: 1

Views: 356

Answers (1)

inbaly
inbaly

Reputation: 612

After a small research, it seemed it just sometimes happnes to any single connection, (Even without the use of multiple connections) It seems like a bug in TCP Flow Control refereed to as: Silly window syndrome: https://en.wikipedia.org/wiki/Silly_window_syndrome

Quote:

A serious problem can arise in the sliding window operation when the sending application program creates data slowly, the receiving application program consumes data slowly, or both. If a server with this problem is unable to process all incoming data, it requests that its clients reduce the amount of data they send at a time (the window setting on a TCP packet). If the server continues to be unable to process all incoming data, the window becomes smaller and smaller, sometimes to the point that the data transmitted is smaller than the packet header, making data transmission extremely inefficient.

I noticed that after a long time (2 minutes or more) it starts downloading again.

Upvotes: 1

Related Questions