CODEWITHSUNDEEP
c#.netmemory-mapped-files
Kieren Johnstone
Kieren Johnstone

Reputation: 42003

How can I quickly read bytes from a memory mapped file in .NET?

In some situations the MemoryMappedViewAccessor class just doesn't cut it for reading bytes efficiently; the best we get is the generic ReadArray<byte> which it the route for all structs and involves several unnecessary steps when you just need bytes.

It's possible to use a MemoryMappedViewStream, but because it's based on a Stream you need to seek to the correct position first, and then the read operation itself has many more unnecessary steps.

Is there a quick, high-performance way to read an array of bytes from a memory-mapped file in .NET, given that it should just be a particular area of the address space to read from?

Upvotes: 22

Views: 16445

Answers (5)

Yurii Palkovskii
Yurii Palkovskii

Reputation: 35

Just wanted to share a version with long l_offset (so it is possible to read\write files larger than int32 max size):

public static unsafe byte[] ReadBytes(long l_offset, int i_read_buf_size, MemoryMappedViewAccessor mmva)
    {
        byte[] arr = new byte[i_read_buf_size];
        byte* ptr = (byte*)0;
        mmva.SafeMemoryMappedViewHandle.AcquirePointer(ref ptr);
        IntPtr p = new(ptr);
        Marshal.Copy(new IntPtr(p.ToInt64() + l_offset), arr, 0, i_read_buf_size);
        mmva.SafeMemoryMappedViewHandle.ReleasePointer();
        return arr;
    }

Upvotes: 0

WBuck
WBuck

Reputation: 5503

I know this is an older question which has been answered but I wanted to add my two cents.

I ran a test with both the accepted answer (using the unsafe code) and with the MemoryMappedViewStream approach for reading a 200MB byte array.

MemoryMappedViewStream

        const int MMF_MAX_SIZE = 209_715_200;
        var buffer = new byte[ MMF_VIEW_SIZE ];

        using( var mmf = MemoryMappedFile.OpenExisting( "mmf1" ) )
        using( var view = mmf.CreateViewStream( 0, buffer.Length, MemoryMappedFileAccess.ReadWrite ) )  
        {
            if( view.CanRead )
            {
                Console.WriteLine( "Begin read" );
                sw.Start( );
                view.Read( buffer, 0, MMF_MAX_SIZE );
                sw.Stop( );
                Console.WriteLine( $"Read done - {sw.ElapsedMilliseconds}ms" );
            }
        }

I ran the test 3 times with each approach and received the following times.

MemoryMappedViewStream:

  1. 483ms
  2. 501ms
  3. 490ms

Unsafe method

  1. 531ms
  2. 517ms
  3. 523ms

From the small amount of testing it looks like the MemoryMappedViewStream has a very slight advantage. With that in mind for anyone reading this post down the road I would go with the MemoryMappedViewStream.

Upvotes: 3

AJ Venturella
AJ Venturella

Reputation: 4912

A safe version of this solution is:

var file = MemoryMappedFile.CreateFromFile(...);
var accessor = file.CreateViewAccessor();
var bytes = new byte[yourLength];

// assuming the string is at the start of the file
// aka position: 0
// https://msdn.microsoft.com/en-us/library/dd267761(v=vs.110).aspx
accessor.ReadArray<byte>(
    position: 0,      // The number of bytes in the accessor at which to begin reading
    array: bytes,     // The array to contain the structures read from the accessor
    offset: 0,        // The index in `array` in which to place the first copied structure
    count: yourLength // The number of structures of type T to read from the accessor.
);

var myString = Encoding.UTF8.GetString(bytes);

I have tested this, it does work. I cannot comment on it's performance or if it's the BEST overall solution just that it works.

Upvotes: 1

nim
nim

Reputation: 328

See this bug report: No way to determine internal offset used by MemoryMappedViewAccessor - Makes SafeMemoryMappedViewHandle property unusable.

From the report:

MemoryMappedViewAccessor has a SafeMemoryMappedViewHandle property, which returns the ViewHandle being used internally by the MemoryMappedView, but does not have any property to return the offset being used by the MemoryMappedView.

As the MemoryMappedView is page aligning the offset requested in MemoryMappedFile.CreateViewAccessor(offset,size) it is impossible to use the SafeMemoryMappedViewHandle for anything useful without knowing the offset.

Note that what we actually want to do is use the AcquirePointer(ref byte* pointer) method to allow some fast pointer based (possibly unmanaged) code to run. We're OK with the pointer being page aligned, but it must be possible to find out what the offset from the originally requested address is.

Upvotes: 2

Kieren Johnstone
Kieren Johnstone

Reputation: 42003

This solution requires unsafe code (compile with /unsafe switch), but grabs a pointer to the memory directly; then Marshal.Copy can be used. This is much, much faster than the methods provided by the .NET framework.

    // assumes part of a class where _view is a MemoryMappedViewAccessor object

    public unsafe byte[] ReadBytes(int offset, int num)
    {
        byte[] arr = new byte[num];
        byte *ptr = (byte*)0;
        this._view.SafeMemoryMappedViewHandle.AcquirePointer(ref ptr);
        Marshal.Copy(IntPtr.Add(new IntPtr(ptr), offset), arr, 0, num);
        this._view.SafeMemoryMappedViewHandle.ReleasePointer();
        return arr;
    }

    public unsafe void WriteBytes(int offset, byte[] data)
    {
        byte* ptr = (byte*)0;
        this._view.SafeMemoryMappedViewHandle.AcquirePointer(ref ptr);
        Marshal.Copy(data, 0, IntPtr.Add(new IntPtr(ptr), offset), data.Length);
        this._view.SafeMemoryMappedViewHandle.ReleasePointer();
    }

Upvotes: 34

Related Questions