CODEWITHSUNDEEP
.netmultithreadingasynchronous.net-2.0
albert
albert

Reputation: 1571

Segmented FileStream Write

I use a segmented http download using ranges (HttpWebRequest.AddRanges()) splitting a file in parts and then joining all parts. If a file is 100 bytes I split it in 4 parts (0-24, 25-49, 50-74, 75-99) and I call four times DownloadPart function. Each part is downloaded using asynchronous programming model (BeginRead/EndRead). When all parts are completed I join them, part1+ part2+ ...

 public void DowloadPart(HttpWebResponse httpResp)
    {
        Stream httpStm = httpResp.GetResponseStream(); 
        Stream fileStm = new FileStream("myFile.part1", FileMode.Append, FileAccess.Write);
        byte[] buff = new byte[1024 * 16];

        AsyncCallback callback = null;
        callback = ar =>
        {
            int bytesRead = httpStm.EndRead(ar);
            fileStm.Write(buff, 0, bytesRead);
            if(bytesRead == 0)
                return;

            httpStm.BeginRead(buff, 0, buff.Length, callback, null);
        };
        httpStm.BeginRead(buff, 0, buff.Length, callback, null);
    }

I want to replace parts joining with directly writing to the file. Since all "DownloadPart" are running in different threads what is the best way to write concurrently to the FileStream in different positions from different threads?

Upvotes: 2

Views: 405

Answers (1)

Remus Rusanu
Remus Rusanu

Reputation: 294267

Ideally you would grow the file first to the desired size, which is not trivial. The API to use exists only as native API, namely SetFileValidData and requires special SE_MANAGE_VOLUME_PRIVILEGE for the process. The (huge) advantage is that the file grows to the desired size without being first filled with 0s. For those familiar with databases this is known as Instant File Initialization.

If you don't want to go to this trouble then you can grow the file the traditional way, but you'll pay the price of writing the content twice, once as 0s once as valid content.

Once you have the file at the right size the best API to write the content is WriteFileGather. Scatter/Gather IO allows you to write arbitrary file segments at arbitrary offsets. But, again, is native only and has no managed counterpart...

W/o scatter/gather IO another decent option is memory mapped access. Luckily this has a managed API,MemoryMappedFile... for .Net 4.0.

W/o memory mapped IO you must use one single stream for all the writing. This means you must use a producer-consumer pattern in which the web streams produce buffers to write (annotated with file offset to write at!) and one consumer takes these buffers and writes them at the appropriate position in the out stream.

Upvotes: 4

Related Questions