CODEWITHSUNDEEP
matlabmemorybigdata
Eff Emm
Eff Emm

Reputation: 33

large data file in matlab doesn't load/import

I have been trying to load data file (csv) into matlab 64 bit running on win7(64 bit) but get memory related errors. The file size is about 3 GB, containing date ( dd/mm/yyyy hh:mm:ss) in first column and bid and ask prices in another two columns. The memory command returns the following : 

  Maximum possible array:              19629 MB (2.058e+010 bytes) *
  Memory available for all arrays:     19629 MB (2.058e+010 bytes) *
  Memory used by MATLAB:                 522 MB (5.475e+008 bytes)
  Physical Memory (RAM):               16367 MB (1.716e+010 bytes)

*  Limited by System Memory (physical + swap file) available.

Can somebody here please explain if the max possible array size is 19.6 GB then why would matlab throw a memory error while importing a data array that is just about 3GB. Apologies if this is a simple question to the experienced as I have little experience in process/app memory management.

I would greatly appreciate if someone would also suggest solution to being able to load this dataset into matlab workspace.

Thank you.

Upvotes: 2

Views: 8047

Answers (2)

Max
Max

Reputation: 2131

I've posted before showing how to use memmapfile() to read huge text files in matlab. This technique may help you as well.

Upvotes: 0

FakeDIY
FakeDIY

Reputation: 1445

I am no expert in memory management but from experience I can tell you that you will run into all kinds of problems if you're importing/exporting 3GB text files.

I would either use an external tool to split your data before you read it or look into storing that data in another format that is more suited to large datasets. Personally, I have used hdf5 in the past---this is designed for large sets of data and is also supported by matlab.

In the meantime, these links may help:

Working with a big CSV file in MATLAB

Handling Large Data Sets Efficiently in MATLAB

Upvotes: 1

Related Questions