CODEWITHSUNDEEP
c#memorystream
Prashant Kumar
Prashant Kumar

Reputation: 249

Memory stream out of memory exception when filesize greter than 4 gb

I am having a problem with memory stream. I am getting a exception of out of memory exception. How to handle this.

var file = VC.ReadStream(filename, true);
var memoryStream = new MemoryStream();
file.CopyTo(memoryStream);
var fileContentBytes = memoryStream.ToArray();
memoryStream = null;
LogUtil.Log(LogUtil.LogType.INFO, String.Format("File size: {0} bytes", fileContentBytes.Length));
var enc = new UTF8Encoding();
var filecontent = enc.GetString(fileContentBytes);

Upvotes: 1

Views: 7532

Answers (3)

Dmitrii Bychenko
Dmitrii Bychenko

Reputation: 186668

First, the size of String is restricted to 2Gb, and that's why

  var filecontent = enc.GetString(fileContentBytes);

will be OutOfMemory exception. Next, you've got a giant overhead at

  var fileContentBytes = memoryStream.ToArray();

Since both memoryStream and fileContentBytes array are about 4 Gb they are 8 Gb total. Yet another issue: when working with IDisposable you are supposed to dispose the instances:

  using (var memoryStream = new MemoryStream()) {
    file.CopyTo(memoryStream);
    var fileContentBytes = memoryStream.ToArray();
    ...
  }

If your task is to put down the file's size into a log you don't need to read the file at all:

  long length = new System.IO.FileInfo(filename).Length;
  LogUtil.Log(LogUtil.LogType.INFO, String.Format("File size: {0} bytes", length));

When working with file content use FileStream class istead of MemoryStream since it works with memory chunks (Kbs in size, usually 4 or 8 Kb)

Upvotes: 3

user3383479
user3383479

Reputation:

I can also advice you to you the StreamReader.ReadLineAsync() Method.

Upvotes: 0

Felice Pollano
Felice Pollano

Reputation: 33252

Reading a so big file at once in memory is not a so god idea. Since this is a text file you should read it and process line by line, using FileStream.ReadLine(). If it is a bynary file, find some chunking strategy. If you really can't do so, try with some Memory Mapping technique.

Upvotes: 0

Related Questions