CODEWITHSUNDEEP
.netperformanceoptimization
Asik
Asik

Reputation: 22133

Optimizing download of many html files

I have about a million urls pointing to HTML pages on a public web server that I want to save to my disk. Each of these is about the same size, ~30 kilobytes. My url lists are split about evenly in 20 folders on disk, so for simplicity I create one Task per folder, and in each task I download one URL after the other, sequentially. So that gives me about 20 parallel requests at any time. I'm on a relatively crappy DSL, 5mbps connection.

This represents several gigabytes of data so I'm expecting the process to take several hours, but I'm wondering if I could make the approach any more efficient. Is it likely I'm making the most out of my connection? How can I measure that? Is 20 parallel downloads a good number or should I dial up or down?

The language is F#, I'm using WebClient.DownloadFile for every url, one WebClient per task.

==================================

EDIT: One thing that made a huge difference was adding a certain header to the request:

let webClient = new WebClient()
webClient.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")

This cut the size of downloads from about 32k to 9k, resulting in enormous speed gains and disk space savings. Thanks to TerryE for mentioning it!

Upvotes: 1

Views: 264

Answers (2)

TerryE
TerryE

Reputation: 10878

If you are using a downloader API, then make sure that it is issuing a

Accept-Encoding: gzip, deflate

request header so that the site that you are scraping knows to return compressed HTML. (Most webservers wil be configured to compress the HTML data streams if the client uses this request header to let the server know that it will accept compressed data streams.)

This will reduce the data transferred by roughly a factor of 4. (E.g. this page was 40K raw HTML, but only 10K was transferred to my browser (the HTML is zipped).

Upvotes: 2

Joseph Sheedy
Joseph Sheedy

Reputation: 6726

I would only parallelize until reaching the limit of the connection speed. If each single request is saturating your DSL connection, running them in parallel won't gain you anything, and might get you blocked.

First measure your capacity with a tool like http://wowrack.speedtest.net. Then parallelize until your throughput reaches this value. There are a variety of ways to monitor your current network usage, the easiest is to go to the Windows Task Manager and hit the Network tab.

Also make sure you're keeping a connection open to the server, and not reopening it for every request. This will cause unnecessary overhead.

Upvotes: 1

Related Questions