CODEWITHSUNDEEP
c#winapireadfilekernel32
trickdev
trickdev

Reputation: 599

ReadFile Kernel32 Maximum Buffer Size

I am reading directly from a disk using C# and pinvoking the kernel32 ReadFile method. I notice that with larger reads (currently only reading in single chunks) the buffer size is out of range.

Does anyone know the maximum size of the read buffer here?

If so what is the purpose of limiting the buffer size when I have surplus memory I want to read into? I understand the concepts of buffering and keeping a small memory footprint but why is a small size forced upon us? Perhaps just an artefact of an old Win32 API?

EDIT: The error received from Marshal.GetLastWin32Error() is "Value does not fall within the expected range."

The upper limit before I receive this error is 8192 bytes (8KB - hence my confusion).

using System;
using System.IO;
using System.Runtime.InteropServices;
using Microsoft.Win32.SafeHandles;

namespace DiskRead
{
    class Program
    {
        public const uint GenericRead = 0x80000000;
        public const uint FileShareRead = 1;
        public const uint FileShareWrite = 2;
        public const uint OpenExisting = 3;

        [DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
        private static extern IntPtr CreateFile(string lpFileName, uint dwDesiredAccess,
          uint dwShareMode, IntPtr lpSecurityAttributes, uint dwCreationDisposition,
          uint dwFlagsAndAttributes, IntPtr hTemplateFile);

        [DllImport("kernel32.dll", SetLastError = true)]
        static extern bool ReadFile(IntPtr hFile, [Out] byte[] lpBuffer,
           uint nNumberOfBytesToRead, out uint lpNumberOfBytesRead, IntPtr lpOverlapped);

        static void Main(string[] args)
        {
            string path = @"\\.\PhysicalDrive0";

            IntPtr ptr = CreateFile(path, GenericRead, FileShareRead | FileShareWrite, IntPtr.Zero, OpenExisting, 0, IntPtr.Zero);

            SafeFileHandle handleValue = new SafeFileHandle(ptr, true);
            FileStream fileStream = new FileStream(handleValue, FileAccess.Read);

            const uint numberOfBytesToRead = 8193;

            uint bytesRead;
            byte[] buffer = new byte[numberOfBytesToRead];


            if (!ReadFile(handleValue.DangerousGetHandle(), buffer, numberOfBytesToRead, out bytesRead, IntPtr.Zero))
            {
                Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
            }

        }
    }
}

Thanks in advance.

Upvotes: 1

Views: 7948

Answers (4)

osirisgothra
osirisgothra

Reputation: 11

I'm not sure if this helps you, but it may help other readers of this post:

  1. When it comes to ReadFile(), it's behavior and success depends heavily on how the file was opened with CreateFile(). Just to give you a point in the right direction think about this; when you call CreateFile() for reading to a buffer, you usually are going to not want the FILE_FLAG_NO_BUFFERING unless you are prepared to deal with data alignment and a nice big detour into some of the hardware-specific information, though it probably would just be easier to not.

  2. Your code doesn't seem to do this, but it is interop, so there could be a garbled value getting though somewhere, and .NET is notorious for molesting(?) memory blocks, so you might want to try allocating the memory via interop to guarentee that the memory block wont move anywhere. Also, i would try minimizing interop as much as possible, it seems to be somewhat unreliable when it is needed most. Instead, you might consider writing a DLL that will play nicer with your code and interop it instead.. I have done this in many cases and it works pretty well and adds the bonus of not having to write the same code over, just reuse it in as many programs as you like... As for the root cause of your problem, I havent been able to duplicate it myself, the largest file Ive tried is about 800MB and it opened,read,closed ok... can you make available the source so it can be tested by others (or a very verbose built executable) to see if others have the same problems with it? Anyways hope this helped someone.

Resources

Upvotes: 1

David Heffernan
David Heffernan

Reputation: 613302

There is no such limit. You are mistaken.

Clearly you are limited by address space and the requirement that the buffer is a contiguous block of virtual memory. On a 32 bit system each process can only address 2GB of virtual memory. What's more you will not be able to allocate a 2GB contiguous block of memory.

But these are general limitations. The ReadFile API will happily read into as big a buffer as your can allocate.

You claim to have hit a limit of 8KB but I have just successfully written and read a 1GB file using WriteFile and ReadFile. Clearly you have some problem but it's just not what you think it is. If you could show the rest of the code, especially that which calls your p/invoke then I'm sure it would become obvious.

And now that you have posted your full code we can see what the issue is. You are not reading a file but instead performing read of the physical disk. I see now that's what you meant by "reading directly from a disk" but I think you could have been a bit more specific!

Anyway, I don't know the details of what's happening here, but the issue is clearly not ReadFile per se, but that fact that your handle is to the physical disk rather than a file.

The documentation for CreateFile states:

A volume contains one or more mounted file systems. Volume handles can be opened as noncached at the discretion of the particular file system, even when the noncached option is not specified in CreateFile. You should assume that all Microsoft file systems open volume handles as noncached. The restrictions on noncached I/O for files also apply to volumes.

A file system may or may not require buffer alignment even though the data is noncached. However, if the noncached option is specified when opening a volume, buffer alignment is enforced regardless of the file system on the volume. It is recommended on all file systems that you open volume handles as noncached, and follow the noncached I/O restrictions.

I think you should consider asking a new question about how to read from physical disks.

Upvotes: 1

Damon
Damon

Reputation: 70186

While ReadFile is running, the kernel must lock the buffer. It must do so, or evil things could happen (a page could not be present when it tries to copy data, or worse... DMA into it).

Every application has limitations on its working set, which you can also configure programmatically. If you (or the kernel on your behalf) try to lock more memory than allowed by your working set limits, this fails. You can't lock more than your minimum working set size, which defaults to a rather small value (IIRC something like 16MB).

Note that the "maximum working set size" is not how much memory your application is allowed to use (that would be tragic). It's rather a figure of when a page belonging to your process is considered as "might be paged out in case someone else wants memory".

The intent behind all this is to ensure that the system keeps working with many programs running concurrently, using unknown amounts of memory.

Upvotes: 0

Tony The Lion
Tony The Lion

Reputation: 63250

Not sure about the max size of the buffer to read into, however the reason for a size of the buffer at all, is so that internally in the function a check can be made so no data is written past the end of the buffer. If the function had no notion of the size of the input buffer, it could write past the end of the buffer, causing a buffer overflow. That would be a serious security flaw.

Upvotes: 0

Related Questions