CODEWITHSUNDEEP
pythonmemory-managementnumpy
K. C
K. C

Reputation: 745

using file/db as the buffer for very big numpy array to yield data prevent overflow?

In using the numpy.darray, I met a memory overflow problem due to the size of data，for example:

Suppose I have a 100000000 * 100000000 * 100000000 float64 array data source, when I want to read data and process it in memory with np. It will raise a Memoray Error because it works out all memory for storing such a big array in memory.

Then maybe using a disk file / database as a buffer to store the array is a solution, when I want to use data, it will get the necessary data from the file / database, otherwise, it is just a python object take few memory.

Is it possible write such a adapter?

Thanks.

Rgs, KC

Upvotes: 1

Views: 363

Answers (2)

Peter Prettenhofer
Peter Prettenhofer

Reputation: 1971

Take a look at pytables or numpy.memmap, maybe they fit your needs.

best, Peter

Upvotes: 1

baklarz2048
baklarz2048

Reputation: 10938

If You have matrices with lots of zeros use scipy.sparse.csc_matrix. It's possible to write everything, for example You can override numarray array class.

Upvotes: 0

Related Questions