CODEWITHSUNDEEP
pythonnumpymatrix
Harwee
Harwee

Reputation: 1650

Numpy cannot store matrix larger than 1GB in memory

I need to store large matrix of order 20000x20000. When I try to initialize a matrix of that order Python raises memory error.

Traceback (most recent call last):
    File "C:\Users\harwee\Desktop\Bubble\test_folder\test.py", line 3, in <module>
    a = numpy.ones((x,x),dtype=int)
  File "C:\Python27\lib\site-packages\numpy\core\numeric.py", line 183, in ones
    a = empty(shape, dtype, order)
MemoryError

My total ram is 8GB and when I check in Taskmanager it shows a memory usage around 1GB. Why is python raising a memory error when Memory is available. Mine is python 32bit version

here is the code I am using

import numpy
x= 16200
a = numpy.ones((x,x),dtype=int)

Upvotes: 2

Views: 770

Answers (2)

Dan
Dan

Reputation: 1227

This isn't really a limit enforced by Python but rather one of 32 bit processing. There is only so much space in a 32 bit address system. You can remove this restriction by moving to a 64 bit Python installation.

Upvotes: 1

Bernardo Meurer
Bernardo Meurer

Reputation: 2335

If you're using 32-bit Python you won't be able to allocate more than ~2GB, if you're doing other things in your code you may be exceeding that limit. Also, I think numpy requires memory allocation to be contiguous, which can be kind of hard for larger blocks.

Upvotes: 2

Related Questions