Reading and writing two-dimensional array to file

Question

I get a 2D array from a device at specific time intervals. I keep these series in a list. I need to write the sequences in the list to the file on the computer. After reading these series I will put it in the list again. I found a method for this. I can write the series. I have some questions at the point of reading.

When I write a fixed array; Example short[,] array2D = new short[,] { { 1, 2 }, { 3, 4 }, { 5, 6 }, { 7, 8 } };

I read it properly when I want to read the same series knowing its dimensions.

But when I write more than one series in a row, I want to read it piece by piece. I failed on this. How can I read the sequences from the file?

Write Code

private void testWriteArray()
    {
        for (int i = 0; i<Globals.logArrayList.Count; i++)
        {
            array2D = Globals.logArrayList[i];

            FileStream writeStream = new FileStream(filename, FileMode.Append);
            BinaryWriter writeBinary = new BinaryWriter(writeStream);
            GCHandle arrHandle = GCHandle.Alloc(array2D, GCHandleType.Pinned);
            IntPtr arrPtr = arrHandle.AddrOfPinnedObject();

            unsafe
            {
                int arrLen = array2D.Length;
                if (arrLen > 0)
                {
                    IEnumerator enumerator = array2D.GetEnumerator();
                    enumerator.MoveNext();
                    int arrSize = System.Runtime.InteropServices.Marshal.SizeOf(enumerator.Current) * arrLen;

                    using (UnmanagedMemoryStream arrStream =
                    new UnmanagedMemoryStream((byte*)arrPtr.ToPointer(), arrSize))
                    {
                        arrStream.CopyTo(writeBinary.BaseStream, (int)arrStream.Length);
                    }
                }
            }

            arrHandle.Free();

            writeStream.Flush();
            writeStream.Close();

        }
    }

Read Code

   static Array testReadArray()
    {

        int[,] array2D = new int[1000, 2000];

        FileStream readStream = new FileStream(filename, FileMode.Open);
        BinaryWriter readBinary = new BinaryWriter(readStream);
        GCHandle arrHandle = GCHandle.Alloc(array2D, GCHandleType.Pinned);
        IntPtr arrPtr = arrHandle.AddrOfPinnedObject();

        unsafe
        {
            int arrLen = array2D.Length;
            if (arrLen > 0)
            {
                IEnumerator enumerator = array2D.GetEnumerator();
                enumerator.MoveNext();
                int arrSize =
                System.Runtime.InteropServices.Marshal.SizeOf(enumerator.Current) * arrLen;

                using (UnmanagedMemoryStream arrStream = new UnmanagedMemoryStream
                ((byte*)arrPtr.ToPointer(), arrSize, arrSize, FileAccess.Write))
                {
                    readBinary.BaseStream.CopyTo(arrStream, (int)arrStream.Length);
                }
            }
        }

        arrHandle.Free();
        readStream.Close();
        return array2D;
    }

In the reading code, I don't know how many dimensions the array is in the file. That's why I gave the array size 1000,2000. Normally, although my data is between 0-255, I am reading 6-7 digit numbers while reading.

Is it possible for me to know the number of strings in the file? If so how?

Answer 1

This is not really an answer, but to show you another approach since you are using unsafe and know the size and dimensions of the array. So I give you a minimal allocation, generic pointer span based constant multi dimensional multi array reader and writer

Given

public static unsafe void Write<T>(T[,] source, Stream stream) where T : unmanaged
{
   fixed (void* asd = source)
      stream.Write(new Span<byte>(asd, source.Length * sizeof(T)));
}
public static unsafe bool Read<T>(T[,] source, Stream stream) where T : unmanaged
{
   fixed (void* asd = source)
      return stream.Read(new Span<byte>(asd, source.Length * sizeof(T))) != 0;
}

The worlds most contrived example

var array = new short[,] {{1, 2,}, {3, 4}, {5, 6}};

using (var rs = new FileStream(@"D:\test.dat", FileMode.Create))
   for (int i = 0; i < 10; i++)
   {
      array[0, 0] = (short) i; // to prove its working
      Write(array, rs);
      Console.WriteLine(string.Join(", ", array.Cast<short>()));
   }

Console.WriteLine();

using (var ws = new FileStream(@"D:\test.dat", FileMode.Open))
   while (Read(array, ws))
      Console.WriteLine(string.Join(", ", array.Cast<short>())); // to prove its working

Output

0, 2, 3, 4, 5, 6
1, 2, 3, 4, 5, 6
2, 2, 3, 4, 5, 6
3, 2, 3, 4, 5, 6
4, 2, 3, 4, 5, 6
5, 2, 3, 4, 5, 6
6, 2, 3, 4, 5, 6
7, 2, 3, 4, 5, 6
8, 2, 3, 4, 5, 6
9, 2, 3, 4, 5, 6

0, 2, 3, 4, 5, 6
1, 2, 3, 4, 5, 6
2, 2, 3, 4, 5, 6
3, 2, 3, 4, 5, 6
4, 2, 3, 4, 5, 6
5, 2, 3, 4, 5, 6
6, 2, 3, 4, 5, 6
7, 2, 3, 4, 5, 6
8, 2, 3, 4, 5, 6
9, 2, 3, 4, 5, 6

_{Note : you can modify this to create and return an array. however this was just a low allocation version that can reuse an array}

_{Note : if you just write 2 ints before the array, you can read any size array and any amount packets}

Answer 2

The problem was so simple that I don't know how I missed it. The array type I defined during reading is integer, but my data is short. When the data type of the array was short the whole problem was gone.

I also solved the total number of sequences in the file as follows.

The lengths of the 2-dimensional array at the time of writing are 48 * 64. I divided the total number of bytes in the file by 48 * 64. Also, since my data is short (2 bytes) I divided it by 2 to get the total number of strings.

var arrayCount = readBinary.BaseStream.Length / (48*64) / 2;