Using BinaryWriter to seek and insert data

Question

Background:

I'm fairly new to C# and currently working on a project intended primarily for learning. I am creating a library to work with .PAK files, as used by Quake and Quake II. I can read and extract them fine, as well as writing a blank archive, and I'm on to writing data to the archive.

The basic specification can be found here for reference: http://debian.fmi.uni-sofia.bg/~sergei/cgsr/docs/pak.txt

Issue:

I would like to seek to the end of the binary data portion and insert data from a given file. I use BinaryWriter in the method when creating an empty file, so this is what I'm looking to use for insertion. From what I found, seeking and writing with BinaryWriter will write at a given position and overwrite existing data.

My goal is to simply insert data at a given point without losing any existing data. If this is not possible, I imagine that I will need to extract the entire index containing the individual file information (name, offset, length), add binary data, then append the index and append information for the file I am writing.

Any thoughts or suggestions?

Answer 1

It is hard to do insertions efficiently. Especially if you want to do multiple insertions into the same stream. You could write your own Stream class that tracks any insertions you do and pretends to have inserted the changes when it really hasn't. For example, if you would have an A at position 10 and you insert an X at that position. Your class could, instead of writing the insertion to the stream and moving all subsequent bytes, return X whenever you ask for position 10 and A from the underlying stream when you ask for position 11 (while in the underlying stream it is still at position 10). Then when you call Flush or Close on the stream, the entire changed stream is flushed to the underlying stream all at once. This requires some very careful engineering to get right, as you may have insertions in insertions, insertions in any order, etc...

Here is some code I wrote for you that inserts arbitrary bytes into a stream and moves all subsequent bytes without any clever caching. Unfortunately it does not use a BinaryWriter but works directly on the Stream (which must be seekable) as that was easier to code. It inserts at the current position of the stream, and afterwards the position is right after the inserted bytes. To insert strings, convert them to bytes using the Encoding classes.

public static class StreamExtensions
{
    /// <summary>
    /// Inserts the specified binary data at the current position in the stream.
    /// </summary>
    /// <param name="stream">The stream.</param>
    /// <param name="array">The array.</param>
    /// <remarks>
    /// <para>Note that this is a very expensive operation, as all the data following the insertion point has to
    /// be moved.</para>
    /// <para>When this method returns, the writer will be positioned at the end of the inserted data.</para>
    /// </remarks>
    public static void Insert(this Stream stream, byte[] array)
    {
        #region Contract
        if (stream == null)
            throw new ArgumentNullException("writer");
        if (array == null)
            throw new ArgumentNullException("array");
        if (!stream.CanRead || !stream.CanWrite || !stream.CanSeek)
            throw new ArgumentException("writer");
        #endregion

        long originalPosition = stream.Position;
        long readingPosition = originalPosition;
        long writingPosition = originalPosition;

        int length = array.Length;
        int bufferSize = 4096;
        while (bufferSize < length)
            bufferSize *= 2;

        int currentBuffer = 1;
        int[] bufferContents = new int[2];
        byte[][] buffers = new [] { new byte[bufferSize], new byte[bufferSize] };
        Array.Copy(array, buffers[1 - currentBuffer], array.Length);
        bufferContents[1 - currentBuffer] = array.Length;

        bool done;

        do
        {
            bufferContents[currentBuffer] = ReadBlock(stream, ref readingPosition, buffers[currentBuffer], out done);

            // Switch buffers.
            currentBuffer = 1 - currentBuffer;

            WriteBlock(stream, ref writingPosition, buffers[currentBuffer], bufferContents[currentBuffer]);

        } while (!done);

        // Switch buffers.
        currentBuffer = 1 - currentBuffer;

        // Write the remaining data.
        WriteBlock(stream, ref writingPosition, buffers[currentBuffer], bufferContents[currentBuffer]);

        stream.Position = originalPosition + length;
    }

    private static void WriteBlock(Stream stream, ref long writingPosition, byte[] buffer, int length)
    {
        stream.Position = writingPosition;
        stream.Write(buffer, 0, length);
        writingPosition += length;
    }

    private static int ReadBlock(Stream stream, ref long readingPosition, byte[] buffer, out bool done)
    {
        stream.Position = readingPosition;
        int bufferContent = 0;
        int read;
        do
        {
            read = stream.Read(buffer, bufferContent, buffer.Length - bufferContent);
            bufferContent += read;
            readingPosition += read;
        } while (read > 0 && read < buffer.Length);
        done = read == 0;
        return bufferContent;
    }
}

To use this method, paste the code in the same namespace as your code, and then:

Stream someStream = ...;
byte[] dataToInsert = new byte[]{ 0xDE, 0xAD, 0xBE, 0xEF };
someStream.Position = 5;
someStream.Insert(dataToInsert);

Since you say that you have a BinaryWriter because you used that to create an empty file, to be able to work with this class you'll need the actual Stream. You can use the static File.Create() and File.Open() methods instead, and they return FileStream objects. If you later decide you'll still need a BinaryWriter, you can use provide the stream to the BinaryWriter constructor. To get a stream from a writer, use BinaryWriter.BaseStream. Note that closing a BinaryWriter will also close the underlying stream, unless you prevent that by using the NonClosingStreamWrapper from the MiscUtil library.

Answer 2

Yes, as you said, you cannot "insert" data in a file. You should use some reader-writer combo until the insertion point(s) and write your new data.

I don't know the .pak format but I think this examples can illustrate you.

    var sourceStream = System.IO.File.OpenRead("myFile.pak");
    var destStream = System.IO.File.Create("modified.pak");

    using (var reader = new System.IO.BinaryReader(sourceStream))
    {
        using (var writer = new System.IO.BinaryWriter(destStream))
        {
            while (reader.BaseStream.CanRead)
            {
                char data = reader.ReadChar();
                if (data == '?')
                {
                    writer.Write("<Found!>");
                }
                else
                {
                    writer.Write(data);
                }
            }
        }
    }

Answer 3

You can't insert data to a file directly using the C# IO classes. The easiest way would be to either read it into a buffer and manipulate the buffer, or write the first part to a temp file, then insert your new data, and then write the remainder of the file and then rename it. Alternatively you can append some data at the end of the file and move everything up to create the right amount of usable space where you want to insert your data.

Take a look at http://www.codeproject.com/Articles/17716/Insert-Text-into-Existing-Files-in-C-Without-Temp

Answer 4

Using Stream-based objects (like FileStream) directly is better option for file level manipulations.

There is no way to insert data in the middle into any file, you have to either read whole file and save it back with newly inserted portions or copy to new file up to insertion point, add new data and than copy the rest (you can delete original/rename new at this point).