CODEWITHSUNDEEP
c#wcfrestcsvmemorystream
user717236
user717236

Reputation: 5039

Building and sending very large CSV data-set over C# WCF REST Web Service as MemoryStream

This topic is related to the question here:

How to send a CSV file using C# WCF RESTful (i.e. Web) Service?

I was able to send a very small data set by utilizing the method discussed at http://blogs.msdn.com/b/carlosfigueira/archive/2008/04/17/wcf-raw-programming-model-web.aspx, which explains how to utilize a "RAW" programming model to send the response -- I'm using the MemoryStream class.

public Stream ReturnCSVSample()
{
    string csv = "Hello, World\nGoodBye, For, Now\n";
    byte[] csvBytes = Encoding.UTF8.GetBytes(csv);

    WebOperationContext.Current.OutgoingResponse.ContentType = "text/csv";

    return new MemoryStream(csvBytes);
}

This works fine. However, in reality, I'm retrieving about 144,000 records from a SQL database table, two columns wide, and using string concatenation to format it into CSV. Then, I take this string,encode it as a bytes array and return it as a MemoryStream object. I let the service run for over 20 minutes and it still does not finish! I've determined the bottleneck is the string concatenation. Why? Because I performed the operation of retrieving these 144,000 records and storing it to a class I created and it took a few minutes, at most, to complete.

Either a string that large is inefficient or there is a better method to perform string concatenation for the amount of records I specified.

How can I send a CSV stream over a web service with such as a large data set?

Thank you.

Upvotes: 1

Views: 1749

Answers (1)

Keith Payne
Keith Payne

Reputation: 3082

A StreamWriter() would do a better job than string concatenation:

public Stream LoadStreamSample(Stream ms)
{
    using (StreamWriter sw = new StreamWriter(ms, Encoding.UTF8)){
        // Call SQL and iterate over rows from SqlDataReader here...
        while (dr.Read()){
            // replace following with csv of one record
            string csv = "Hello, World\nGoodBye, For, Now\n";
            sw.WriteLine(csv)
        }
    }
    return ms;
}

Upvotes: 1

Related Questions