CODEWITHSUNDEEP
.net
Jonathan Rupp
Jonathan Rupp

Reputation: 15762

Looking for a .Net disk-buffered stream

I have an .NET 2.0 web application that is only used as the back-end for a client application that will send it a request with 1-500KB of data, and the server will do some processing on that data, then query a database to build a set of data to return the client -- anywhere from 200KB to 200MB of data. The data may take up to 15 minutes to fully load from the database, and up to an hour to fully transfer to the client (though typical times are quite a bit lower -- 20s query, 10s transmit). However, for some combinations of database query and client, sometimes the DB is faster, sometimes the link to the client is faster.

Clients connect over varying quality of links -- some over a 10MB LAN, some over a shared 64kbps ISDN link. I have four things I want to ensure:

  1. Database connection must be held open as short of a period as possible.
  2. Minimal server memory must be used.
  3. Data must be streamed to the client as quickly as possible.
  4. Total duration of the process should be as close to the maximum of the two processes (DB wait time, network transmission time) as possible, rather than the sum of the time of the two processes.

So far, we have three modes we can run the code in:

  1. As data is fetched from the DB, write it to the response stream.
  2. As data is fetched from the DB, save it in memory. When the DB read is complete, close the DB connection and write the data from memory to the response stream.
  3. As data is fetched from the DB, write it to a temporary file. When the read from the DB is complete, close the DB connection and read from the temp file and write to the response stream.

As you could imagine, none of these fully meet our requirements for all the scenarios. We've talked about implementing some code to guess based on some variables which of the three codepaths we should use, but I think there's a more general solution here, and I'm looking for some code/a library that can take care of this for us.

What I'm looking for is something I can wrap around a .NET stream (the output stream) that I can write to from our HttpHandler, and it will handle writing it the data to the underlying stream. If the underlying stream has too much pending data, it is stored in memory and sent later, when it can. Once that in-memory buffer gets too big, it falls back to a temporary file as a buffer. When I try to close the stream, it blocks until all the data has been written to the stream.

Do you know of any code like this I could use as a starting point, or a library that has something like this, or any guidance on things I should watch out for? Or, am I trying to completely over-engineer this solution?

Upvotes: 3

Views: 308

Answers (1)

Jeremy Stein
Jeremy Stein

Reputation: 19661

Operating systems are really good at swapping memory to disk at the right times. Rather than second-guess the OS, why don't you just use a regular buffered stream and let the swap space be your file backing?

Upvotes: 2

Related Questions