CODEWITHSUNDEEP
c#.netwebdownload
alexmac
alexmac

Reputation: 19567

How to increase download speed of the file from web-sites?

I have a C# console app, which downloads the files from web-sites. Here are my methods:

public bool DownloadFile(string fileUri, string filePath)
{
    using (var response = _GetResponce(fileUri)) {
        if (response.StatusCode != HttpStatusCode.OK) {
            return false;
        } 
        using (var responseStrm = response.GetResponseStream()) {
            using (var fileStrm = new FileStream(filePath, FileMode.OpenOrCreate)) {
            var buffer = new byte[CommonConstants.StreamBufferSize];
            int bytesRead = responseStrm.Read(buffer, 0, CommonConstants.StreamBufferSize);      
            while(bytesRead > 0) {
                fileStrm.Write(buffer, 0, bytesRead);
                bytesRead = responseStrm.Read(buffer, 0, CommonConstants.StreamBufferSize);
            }
        }
    }
}

private HttpWebResponse _GetResponce(string requestedUri, string method = "GET")
{
    var request = (HttpWebRequest)WebRequest.Create(requestedUri);
    request.Method = method;
    request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13";
    request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
    request.Headers[HttpRequestHeader.AcceptLanguage] = "ru-ru,ru;q=0.8,en-us;q=0.5,en;q=0.3";
    request.Headers[HttpRequestHeader.AcceptEncoding] = "gzip,deflate";
    request.Headers[HttpRequestHeader.AcceptCharset] = "windows-1251,utf-8;q=0.7,*;q=0.7";
    return (HttpWebResponse)request.GetResponse();
}

I tried to increase the buffer size, to no avail. If I add threads, can it increase download speed?

Upvotes: 2

Views: 3505

Answers (2)

Cameron
Cameron

Reputation: 98736

Have you tried buffering more data before writing it to the file? In general you want to be polling the socket as quickly as possible -- it's been my experience that even if many bytes are available for reading, only a relatively small chunk of them may be returned, meaning the socket throughput ends up tied to the overhead of the rest of the loop, which could be significant.

Try this code and see if you get a measurable improvement:

using (var fileStrm = new FileStream(filePath, FileMode.OpenOrCreate))
using (var responseStrm = response.GetResponseStream()) {
    const int BufferSize = 1 * 1024 * 1024 + CommonConstants.StreamBufferSize;
    var buffer = new byte[BufferSize];
    int offset, bytesRead;
    do {    // Until we've read everything
        offset = 0;
        do {        // Until the buffer is very nearly full or there's nothing left to read
            bytesRead = responseStrm.Read(buffer, offset, BufferSize - offset);
            offset += bytesRead;
        } while (bytesRead != 0 && offset + CommonConstants.StreamBufferSize < BufferSize);

        // Empty the buffer
        if (offset != 0) {
            fileStrm.Write(buffer, 0, offset);
        }
    } while (bytesRead != 0);
}

Also, if you're accepting gzip and zlib streams, you'll need to turn on decompression(!):

request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

Upvotes: 1

BrunoLM
BrunoLM

Reputation: 100312

Depending on the server, you might gain some speed downloading the file with several segments.

For that you can use request.AddRange and have different threads to download the same file. Note that when you write the data to a local file you must start in the correct offset.

This is something that most download managers do.

Note that some servers will not allow that.

Upvotes: 1

Related Questions