CODEWITHSUNDEEP
c#multithreadingdownload
Saman Gholami
Saman Gholami

Reputation: 3512

Download a File through Multiple Threads

I want to download a file with 6 threads to speed up the process, so this is the part code which calculates the amount of file size for each thread:

string url = "http://somefile.mp3";
List<FileDownloader> filewonloadersList = new List<FileDownloader>();
System.Net.WebRequest req = System.Net.HttpWebRequest.Create(url);
var response = req.GetResponse();
req.Method = "HEAD";
System.Net.WebResponse resp = req.GetResponse();
int responseLength = int.Parse(resp.Headers.Get("Content-Length"));
int parts = 6;
var eachSize = responseLength / parts;
var lastPartSize = eachSize + (responseLength % parts);
for (int i = 0; i < parts - 1; i++)
{
    filewonloadersList.Add(new FileDownloader(url, i * eachSize, eachSize));
}
filewonloadersList.Add(new FileDownloader(url, (parts - 1) * eachSize, lastPartSize));
var threads = new List<Thread>();
foreach (var item in filewonloadersList)
{
    var newThread = new Thread(DoDownload);
    threads.Add(newThread);
    newThread.Start(item);
}

And this is the body of DoDownload function:

public static void DoDownload(object data)
{
    retry:
        try
        {
            var downloader = data as FileDownloader;
            HttpWebRequest req = (HttpWebRequest)WebRequest.Create(downloader.Url);
            if (downloader.Start > 0)
            {
                req.AddRange(downloader.Start, downloader.Start + downloader.Count - 1);
            }
            else
            {
                req.AddRange(downloader.Start, downloader.Start + downloader.Count - 1);
            }
            var response = req.GetResponse();
            using (var reponseStream = response.GetResponseStream())
            {
                using (var fs = new FileStream($"temp_{downloader.Start}.sth", FileMode.OpenOrCreate))
                {
                    var buffer = new byte[1024];
                    int bytesRead = 0;
                    do
                    {
                        bytesRead = reponseStream.Read(buffer, 0, 1024);
                        fs.Write(buffer, 0, bytesRead);
                        fs.Flush();
                    } while (bytesRead > 0);
                    fs.Close();
                }
            }
        }
        catch (WebException e)
        {
            if (e.Status == WebExceptionStatus.Timeout || e.Status == WebExceptionStatus.KeepAliveFailure)
                goto retry;
        }
    }

In addition this is the FileDownloader definition:

public class FileDownloader
{
    public int Start;
    public int Count;
    public string PathTemp;
    public string Url;
    public FileDownloader(string url, int start, int count)
    {
        Url = url;
        Start = start;
        Count = count;
    }
}

Everything is just working as I expected, the length of the file is exactly as much as it should be. Also after I merging the part of downloaded file, it's working properly. The problem is the threading part. I expected 6 files are being downloaded at the same time but they are being downloaded one by one, e.g. when the first part is completed, the second one will be downloaded. How should I fix that?

UPDATE

Based on suggestions, I have changed the function to async:

public async Task DoDownload(object data)
        {
            retry:
            try
            {
                var downloader = data as FileDownloader;
                HttpWebRequest req = (HttpWebRequest)WebRequest.Create(downloader.Url);
                req.AddRange(downloader.Start, downloader.Start + downloader.Count - 1);
                var response = await req.GetResponseAsync();
                using (var reponseStream = response.GetResponseStream())
                {
                    using (var fs = new FileStream($"temp_{downloader.Start}.sth", FileMode.OpenOrCreate))
                    {
                        var buffer = new byte[1024];
                        int bytesRead = 0;
                        do
                        {
                            //reponseStream.Seek(downloader.Start, SeekOrigin.Current);
                            bytesRead = await reponseStream.ReadAsync(buffer, 0, 1024);
                            await fs.WriteAsync(buffer, 0, bytesRead);
                            await fs.FlushAsync();
                        } while (bytesRead > 0);
                        fs.Close();
                    }
                }
            }
            catch (WebException e)
            {
                if (e.Status == WebExceptionStatus.Timeout || e.Status == WebExceptionStatus.KeepAliveFailure)
                    goto retry;
            }
        }

And the foreach loop, in which the parts are populating:

foreach (var item in filewonloadersList)
            {
                Task.WhenAll(DoDownload(item));
            }

But the result is the same! Just one part of a file is downloading at the same time.

Upvotes: 5

Views: 9133

Answers (1)

Saman Gholami
Saman Gholami

Reputation: 3512

Based on this link Trying to run multiple HTTP requests in parallel, but being limited by Windows (registry). In short, It is matter of ServicePoint. Which provides connection management for HTTP connections. The default maximum number of concurrent connections allowed by a ServicePoint object is 2.

I just needed to add a single nice line to the code:

System.Net.ServicePointManager.DefaultConnectionLimit = 1000;

It's working like a charm! There are multiple files at the same time, and also there is no difference between the way that async and Thread are working. At least both of them are producing my desired result.

Upvotes: 6

Related Questions