CODEWITHSUNDEEP
c#arrays
Jonathan Wood
Jonathan Wood

Reputation: 67283

Best Way to Represent Large Byte Array

I'm looking for the most efficient way to store and manage a large byte array in memory. I will have the need to both insert and delete bytes from any position within the array.

At first, I was thinking that a regular array was best.

byte[] buffer = new byte[ArraySize];

This would allow me to access any byte within the array. I can also resize the array. However, there doesn't appear to be any built-in support for shifting or moving items within the array.

One option is to have a loop to move items one by one but that sounds horribly inefficient in C#. Another option is to create a new array and copy bytes over to the correct position, but that requires copying all data in the array.

Is there no better option?

Upvotes: 3

Views: 4606

Answers (2)

Earlz
Earlz

Reputation: 63895

I think the best option in this case is a hybrid between a regular array and a list. This would only be necessary with megabyte sized arrays though.

So you could do something like this:

List<byte[]> buffer;

And have each element of the list just a chunk of the data(say 64K or something small and manageable)

It'd require quite a bit of custom code, but would definitely be the fastest option when having to shift data around in a large array.

Also, if you're doing a lot more shifting of bytes than anything else, LinkedList<T> may work better (but it's famously bad for everything but a specific set of cases)

To clarify why this is more correct than an array, consider inserting 1 byte to the beginning of an array. You must allocate another array (double memory consumption) and then copy every byte to the new array after inserting the new byte, and then free the old array (possible heap corruption depending on size)

Consider now this method with lists.

If you have to insert a lot of bytes, you'll probably want to insert at the beginning of the buffer list. This is an O(n) operation, so your ending efficiency for this operation is O(n/CHUNK_SIZE)

Or, if you just need to insert a single byte, you can just get the first element of the list and copy the array as normal. Then, the speed is O(CHUNK_SIZE), which isn't horrible, especially if n in comparison is very large (megabytes of data)

Upvotes: 1

Jonathan Wood
Jonathan Wood

Reputation: 67283

Actually, I just found the Buffer Class, which appears ideal for what I need.

It looks like the BlockCopy method will block copy a bunch of items and supports copying within the same array, and even correctly handles overlapping items.

Upvotes: 2

Related Questions