In C# how do I get the file position of each text row in an output file (as it is being written)?

Question

As I write text lines to a temp file, I'd like to store the file position of each line for future random access.

This does NOT work

 StreamWriter outFile;

 { ...
          fPos[i++] = outFile.BaseStream.Position;
 }

fPos values in the above loop are 0,0,0,...,1024,1024,...2028,2048 etc - i.e. physical positions.

The following DOES work but does away with buffering:

 { ...
          outFile.Flush();
          fPos[i++] = outFile.BaseStream.Position;
 }

Is there a better way to get the (logical) file position as I go without giving up buffering ?

Answer 1

I would create a subclass that counts the output characters. According to this Inheriting from StreamWriter with smallest possible effort , you only need to override void Write(Char)

 class StreamWriterWithPosition : StreamWriter {
   override void Write(Char c) { pos += Encoding.GetByteCount(c.ToString()); base.Write(c);}
   int pos = 0;
 }

(not tested!)

Answer 2

I can't remember where this came from exactly, but it has worked pretty well for me in the past.

Replaced reader with writer:

/// <summary>
/// A stream writer that allows its current position to be recorded and 
/// changed. This is not generally possible for stream writers, because 
/// of the use buffered reads. 
/// This writer can be used only for text content. Theoretically it can 
/// process any encoding, but was checked only for ANSI and UTF8 If you 
/// want to use this class with any other encoding, please check it.
/// NOT SAFE FOR ASYNC WRITES.
/// </summary>   
public class PositionableStreamWriter : StreamWriter
{
    /// <summary>
    /// Out best guess counted position.
    /// </summary>
    private long _position;

    /// <summary>
    /// Guards against e.g. calling "Write(char[] buffer, int index, int count)" 
    /// as part of the implementation of "Write(string value)", which would cause
    /// double counting.
    /// </summary>
    private bool _guardRecursion; 

    public PositionableStreamWriter(string fileName, bool append, Encoding enc)
        : base(fileName, append, enc)
    {
        CommonConstruct();
    }
    public PositionableStreamWriter(Stream stream, Encoding enc)
        : base(stream, enc)
    {
        CommonConstruct();
    }

    /// <summary>
    /// Encoding can really haven't preamble
    /// </summary>        
    private bool CheckPreamble(long lengthBeforeFlush)
    {
        byte[] preamble = this.Encoding.GetPreamble();
        if (this.BaseStream.Length >= preamble.Length)
        {
            if (lengthBeforeFlush == 0)
                return true;
            else // we would love to read, but base stream is write-only.
                return true; // just assume a preamble is there.
        }
        return false;
    }

    /// <summary>
    /// Use this property to get and set real position in file.
    /// Position in BaseStream can be not right.
    /// </summary>
    public long Position
    {
        get { return _position; }
        set
        {
            this.Flush();
            ((Stream)base.BaseStream).Seek(value, SeekOrigin.Begin);
            _position = value;
        }
    }

    public override void Write(char[] buffer)
    {
        if (buffer != null && !_guardRecursion)
            _position += Encoding.GetByteCount(buffer);
        CallBase(() => base.Write(buffer));
    }
    public override void Write(char[] buffer, int index, int count)
    {
        if (buffer != null && !_guardRecursion)
            _position += Encoding.GetByteCount(buffer, index, count);
        CallBase(() => base.Write(buffer, index, count));
    }
    public override void Write(string value)
    {
        if (value != null && !_guardRecursion)
            _position += Encoding.GetByteCount(value);
        CallBase(() => base.Write(value));
    }

    public override void WriteLine()
    {
        if (!_guardRecursion)
            _position += Encoding.GetByteCount(NewLine);
        CallBase(() => base.WriteLine());
    }
    public override void WriteLine(char[] buffer)
    {
        if (buffer != null && !_guardRecursion)
            _position += Encoding.GetByteCount(buffer) + Encoding.GetByteCount(NewLine);
        CallBase(() => base.WriteLine(buffer));
    }
    public override void WriteLine(char[] buffer, int index, int count)
    {
        if (buffer != null && !_guardRecursion)
            _position += Encoding.GetByteCount(buffer, index, count) + Encoding.GetByteCount(NewLine);
        CallBase(() => base.WriteLine(buffer, index, count));
    }
    public override void WriteLine(string value)
    {
        if (value != null && !_guardRecursion)
            _position += Encoding.GetByteCount(value) + Encoding.GetByteCount(NewLine);
        CallBase(() => base.WriteLine(value));
    }

    private void CallBase(Action callBase)
    {
        if (_guardRecursion == true)
            callBase();
        else
        {
            try
            {
                _guardRecursion = true;
                callBase();
            }
            finally
            {
                _guardRecursion = false;
            }
        }
    }

    private void CommonConstruct()
    {
        var lenghtAtConstruction = BaseStream.Length;
        if (lenghtAtConstruction == 0)
            Flush(); // this should force writing the preamble if a preamble is being written.
        //we need to add length of symbol which is in begin of file and describes encoding of file                                    
        if (CheckPreamble(lenghtAtConstruction))
        {
            _position = this.Encoding.GetPreamble().Length;
        }
    }
}

Usage:

class Program
{
    static void Main(string[] args)
    {
        var pos = new List<long>();
        using (var writer = new PositionableStreamWriter("tst.txt", false, Encoding.Unicode))
        {
            pos.Add(writer.Position);
            writer.Write("abcde");
            pos.Add(writer.Position);
            writer.WriteLine("Nope");
            pos.Add(writer.Position);
            writer.WriteLine("Something");
            pos.Add(writer.Position);
            writer.WriteLine("Another thing");
            pos.Add(writer.Position);
        }

        using (var stream = File.Open("tst.txt", FileMode.Open))
        using (var reader = new BinaryReader(stream))
        {
            for (int i = 0; i < pos.Count - 1; i++)
            {
                stream.Position = pos[i];
                var len = (int)(pos[i + 1] - pos[i]);
                var buf = reader.ReadBytes(len);
                Console.Write(Encoding.Unicode.GetString(buf));
            }
        }
    }
}

Answer 3

If the data is small enough to fit in memory, you could have the StreamWriter write to a MemoryStream. Flush after each line, and get the memory stream's position before writing each line. And when you're done writing the data to the memory stream, copy it to a file.

If the data is larger than will fit in memory, you could use a MemoryStream as a buffer, and flush that memory stream to the disk file whenever it fills up. That would take a little bookkeeping, but it wouldn't be too terrible.

Slightly more involved would be to create your own class derived from BufferedStream that counts the bytes written to it. Open your stream writer with a buffer size of 0, and have it write to the buffered stream. You could then query the buffered stream for the number of bytes written. Of course, that stream would write to the file stream. So it'd be something like:

using (var fs = new FileStream(...))
{
    using (var buff = new CustomBufferedStream(fs))
    {
        using (var writer = new StreamWriter(buff, encoding))
        {
            ... do your output here
        }
    }
}

You might not be able to have a StreamWriter buffer size of 0. If that's the case, then you'll have to flush after every line, and you might have to override the custom stream's Flush method so that it doesn't automatically flush to the disk file. I don't recall if StreamWriter.Flush automatically calls BaseStream.Flush.