CODEWITHSUNDEEP
c#.netmultithreadingftpftpwebrequest
Vishnu
Vishnu

Reputation: 103

Download single file from FTP server in chunks in multiple threads using C#

I just wanted to download a single file from FTP server in multiple segments in multiple threads using C#.

Is it possible to give ranges for the file download like in HttpWebRequest?

Upvotes: 3

Views: 2069

Answers (3)

Martin Prikryl
Martin Prikryl

Reputation: 202292

You can use FtpWebRequest.ContentOffset to specify the starting offset.

But FtpWebRequest.ContentLength is not implemented. To workaround that you have to abort the download, once you receive the desired amount of bytes.

const string name = "bigfile.dat";
const int chunks = 3;
const string url = "ftp://example.com/remote/path/" + name;
NetworkCredential credentials = new NetworkCredential("username", "password");

Console.WriteLine("Starting...");

FtpWebRequest sizeRequest = (FtpWebRequest)WebRequest.Create(url);
sizeRequest.Credentials = credentials;
sizeRequest.Method = WebRequestMethods.Ftp.GetFileSize;
long size = sizeRequest.GetResponse().ContentLength;
Console.WriteLine($"File has {size} bytes");
long chunkLength = size / chunks;

List<Task> tasks = new List<Task>();

for (int chunk = 0; chunk < chunks; chunk++)
{
    int i = chunk;
    tasks.Add(Task.Run(() =>
        {
            FtpWebRequest request = (FtpWebRequest)WebRequest.Create(url);
            request.Credentials = credentials;
            request.Method = WebRequestMethods.Ftp.DownloadFile;
            request.ContentOffset = chunkLength * i;
            long toread =
                (i < chunks - 1) ? chunkLength : size - request.ContentOffset;
            Console.WriteLine(
                $"Downloading chunk {i + 1}/{chunks} with {toread} bytes ...");

            using (Stream ftpStream = request.GetResponse().GetResponseStream())
            using (Stream fileStream = File.Create(name + "." + i))
            {
                byte[] buffer = new byte[10240];
                int read;
                while (((read = (int)Math.Min(buffer.Length, toread)) > 0) &&
                       ((read = ftpStream.Read(buffer, 0, read)) > 0))
                {
                    fileStream.Write(buffer, 0, read);
                    toread -= read;
                }
            }
            Console.WriteLine($"Downloaded chunk {i + 1}/{chunks}");
        }));
}

Console.WriteLine(
    "Started all chunks downloads, waiting for them to complete...");
Task.WaitAll(tasks.ToArray());

Console.WriteLine("Done");
Console.ReadKey();

Upvotes: 0

Wilson Wong
Wilson Wong

Reputation: 1

The default connection limit is 2. Use ServicePoint to change the limit.

request.Method = WebRequestMethods.Ftp.DownloadFile;
ServicePoint sp = request.ServicePoint;
sp.ConnectionLimit = 4;

Upvotes: 0

Christopher
Christopher

Reputation: 9804

First the disclaimer: Multitasking is not a magical "go faster" bullet. If you apply it to the wrong problem, you end up with code that is more complex/prone to errors, more memory demanding and actually slower then the plain old singletasked/sequential approach. One alternate Task for a long running operation is genereally mandatory. But Massive parallelisation is only in very specific circumtances.

Generaly file operations are Disk or Network bound. Multitasking will not add any speedup to Disk or Network bound operations. And indeed might cause a slowdown, as NCQ and similar features have to straightern out your random access requests. That being said with Netowrking it sometimes can help. Some servers do apply a "per connection" limit, and thus splitting the download into multiple segments with their own connection can be a net speedup by. But be certain this is actually the case here. Consider all but point 1 of the Speed Rant.

Asuming FTPWebRequest is still the class you are using, it looks like ContentLenght and ContentOffset might be the droids you are looking for. You basically use it similar to substring - each connection/sub-request takes X bytes from Y offset.

Upvotes: 1

Related Questions