CODEWITHSUNDEEP
httpnetwork-programmingoptimizationdownloadweb-crawler
xiaohan2012
xiaohan2012

Reputation: 10332

How to maximize downloading throughput by multi-threading

I am using multi-threading to speed up the process of downloading a bunch of files from the Web.

How might I determine how many threads I should use to maximize or nearly maximize the total download throughput?

PS:

I am using my own laptop and the bandwidth is 1Mb.

The data I want is the webpage source code of coursera.com

Upvotes: 0

Views: 167

Answers (1)

codewarrior
codewarrior

Reputation: 1269

There are much more factors than only number of threads if you want to speed up downloading files from network. Actually I don't believe that you will achieve this expect that there are some limitations you haven't described (like max bandwidth per connection on server side, you have multilink client and you can use different links do download different data, you want to download different parts from different servers, or similar).

In usual conditions having multiple threads to download something will slow the process. You will need to maintain couple of connections and somehow synchronise data (expect if you will download e.g. different files at the same time).

I would say that in "ordinary" conditions much bigger limitations are your bandwidth limit so using more threads will not make downloading faster. You will in this case share your whole bandwidth to many connections.

Upvotes: 1

Related Questions