CODEWITHSUNDEEP
c#.netbinaryreader
Miloš Đošović
Miloš Đošović

Reputation: 188

Read with BinaryReader line by line

I am using the following code to serialize data, from dataTable.

var rows = new List<Dictionary<string, object[]>>();

I am filling the rows from DataTable and placing them in Dictionary. Don't ask why :)

using(var fileStream = new FileStream(@"D:\temp.bin", FileMode.Create, FileAccess.Write, FileShare.None))
using(var bw = new BinaryWriter(fileStream))
{
    foreach(Dictionary<string, object[]> row in rows)
    {
        byte[] bytes = ObjectToByteArray(row);
        bw.Write(bytes);
    }
}

With following method:

private static byte[] ObjectToByteArray(Dictionary<string, object[]> rows)
{
    var bf = new BinaryFormatter();
    using(var ms = new MemoryStream())
    {
        bf.Serialize(ms, rows);
        return ms.ToArray();
    }
}

What I am trying to do is to deserialize line by line, if that is possible with BinaryReader. The problem is I am stuck with reading only first row.

What I would like to achieve is:

using(BinaryReader reader = new BinaryReader(File.Open(@"D:\temp.bin", FileMode.Open)))
{
    int pos = 0;
    int length = (int)reader.BaseStream.Length;
    while(pos < length)
    {
        byte[] v = reader.ReadBytes(pos);
        Dictionary<string, object[]> row = FromByteArray(v);
        // Advance our position variable.
        pos += row.Count;
    }
}

The biggest problem is reader.ReadBytes(XXX) -> what should be the value to read? I don't know in advance that. I need to read whole line and convert to Dictionary. The method I am using for conversion back is:

public static Dictionary<string, object[]> FromByteArray(byte[] data)
{
    BinaryFormatter bf = new BinaryFormatter();
    using(MemoryStream ms = new MemoryStream(data))
    {
        object obj = bf.Deserialize(ms);
        return (Dictionary<string, object[]>)obj;
    }
}

As I said FromByteArray works fine for first line, I am not finding any way to read next line.

When I use BinarryFormatter to serialize complete file, it passes if the file is not that large. If it is OOM occurs. Same stands for deserialization. That is why I want it to serialize/deserialize partially.

Tried everything and searched everywhere. Thanks on helping with this one.

Upvotes: 5

Views: 7237

Answers (1)

Ofir Winegarten
Ofir Winegarten

Reputation: 9365

For each iteration save in the file also the length of the following serialized objects.

When reading, every iteration first read 4-bytes (reader.ReadInt32) to get this value and read this much bytes to de-serialize.

I think it should look like this:

using(var fileStream = new FileStream(@"D:\temp.bin", FileMode.Create, FileAccess.Write, FileShare.None))
{
    using(var bw = new BinaryWriter(fileStream))
    {
        foreach(Dictionary<string, object[]> row in rows)
        {
            byte[] bytes = ObjectToByteArray(row);
            bw.Write(bytes.Length);
            bw.Write(bytes);
        }
    }
}


using(BinaryReader reader = new BinaryReader(File.Open(@"D:\temp.bin", FileMode.Open)))
{
    int length = (int)reader.BaseStream.Length;
    while(reader.BaseStream.Position != length)
    {
        int bytesToRead = reader.ReadInt32();
        byte[] v = reader.ReadBytes(bytesToRead);
        Dictionary<string, object[]> row = FromByteArray(v);
    }
}

Upvotes: 6

Related Questions