CODEWITHSUNDEEP
delphifile-handling
Atak_Snajpera
Atak_Snajpera

Reputation: 629

"Out of memory while expanding memory stream"

I need to read files directly from drive without any system buffering. I tried this code here How to unload a file from cache? unfortunately I get error message saying "Out of memory while expanding memory stream". It happens with any file I tried. Later I've noticed that SrcStream.Size returns always value -1. So obviously problem is here and the question is why is this happening?

uses
  MMSystem;

function GetTimeForRead(ABuffered: boolean): single;
const
  FileToRead = // name of file with maybe 500 MByte size
var
  FlagsAndAttributes: DWORD;
  FileHandle: THandle;
  SrcStream, DestStream: TStream;
  Ticks: DWord;
begin
  if ABuffered then FlagsAndAttributes := FILE_ATTRIBUTE_NORMAL
  else FlagsAndAttributes := FILE_FLAG_NO_BUFFERING;
  FileHandle := CreateFile(FileToRead, GENERIC_READ, FILE_SHARE_READ, nil,OPEN_EXISTING, FlagsAndAttributes, 0);
 if FileHandle = INVALID_HANDLE_VALUE then 
 begin
   Result := 0.0;
   exit;
 end;

 SrcStream := THandleStream.Create(FileHandle);
 try
   DestStream := TMemoryStream.Create;
   try
   DestStream.Size := SrcStream.Size;

   Ticks := timeGetTime;
   DestStream.CopyFrom(SrcStream, SrcStream.Size);
   Result := 0.001 * (timeGetTime - Ticks);

 finally
  DestStream.Free;
 end;
  finally
    SrcStream.Free;
  end;
end;

Upvotes: 0

Views: 6519

Answers (1)

David Heffernan
David Heffernan

Reputation: 612884

FILE_FLAG_NO_BUFFERING places special requirements on use of the file handle, that are not compatible with all functionality of Delphi's THandleStream class. The principal such requirement is that all access is aligned. By which it is meant that the file pointer is always placced on sector boundaries, and all reads and writes are of multiples of the sector size. The specific failure point here is the Size property.

You are reading a file whose size is not an exact multiple of the sector size. The answer you refer to fails with the error you report when presented a file whose size is not an exact multiple of the sector size. Presumably the author of that code was not aware of the issue and by pure chance used a file whose size was an exact multiple of the sector size.

You can confirm all of this by executing the code with an input file whose size is an exact multiple of 4096.

You likely can use THandleStream with such a file handle but you will need to be careful. Obviously you must avoid Size. You have to respect the alignment requirements. You need to be careful when reading at the end of the file because you will need to read an entire sector even if you know that the logical file ends before the end of the sector. That means using Read rather than ReadBuffer.

Frankly, in my opinion, if you must use unbuffered file access then I don't believe that the stream abstraction is a good fit. I would work directly with the Windows API.

Upvotes: 3

Related Questions