CODEWITHSUNDEEP
c#memoryftpstreamstreaming
Leon Cullens
Leon Cullens

Reputation: 12476

How to stream data to FTP

I'm running into an OutOfMemoryException. My flow was as follows:

  1. I read the data from the database (big collection, millions of rows).
  2. I loop through the retrieved data and append it to a StringBuilder.
  3. I call ToString on the StringBuilder and upload the string to a new file using SFTP (Renci.SSHnet).

This flow didn't work because there were too many files to keep in memory. I decided to make some changes:

This is where things started to get tricky. I want to append new data to the StreamWriter in my data processing code while the SFTP upload code reads from the same stream, GZIPs it and uploads the data. Note that it should only create one file on the remote server. In other words: I want to process data while uploading the processed data so I can keep a very low memory footprint. I can't get this working. This is what I came up with (simplified example), but it never reads the data:

static void Main()
{
    using (var stream = new MemoryStream())
    {
        WriteToStream(stream, GetData());
        ReadFromStream(stream);
    }
}

static IEnumerable<string> GetData()
{
    for (int i = 0; i < 10; i++)
    {
        var date = DateTime.Now;
        Console.WriteLine("{0:O} - Fetched some data: {1}", date, i);
        yield return string.Format("{0}", i);
    }
}

static void WriteToStream(Stream stream, IEnumerable<string> data)
{
    using (var writer = new StreamWriter(stream))
    {
        foreach (string str in data)
        {
            Console.WriteLine("{0:O} - Wrote some data: {1}", DateTime.Now, str);
            writer.WriteLine("{0}", str);
        }
    }
}

static void ReadFromStream(Stream stream)
{
    using (var reader = new StreamReader(stream))
    {
        while (reader.Peek() >= 0)
        {
            var res = reader.ReadLine();
            Console.WriteLine("{0} - Read some data: {1}", DateTime.Now, res);
        }
    }
}

Can anyone point me in the right direction?

Upvotes: 0

Views: 1580

Answers (1)

Robert Sirre
Robert Sirre

Reputation: 702

Instead of creating a new stream yourself and passing that stream to the ftp component, use the ftp components' method for initiating a new stream (SftpClient.Open(...)) and write to that. Similar to your WriteToStream method, just probably have to replace the writeline with write.

The rest of your issue is still interesting though :)

Upvotes: 1

Related Questions