CODEWITHSUNDEEP
pythonarraysmemorynumpy
aensm
aensm

Reputation: 3607

How much memory in numpy array? Is RAM a limiting factor?

I'm using numpy to create a cube array with sides of length 100, thus containing 1 million entries total. For each of the million entries, I am inserting a 100x100 matrix whose entries are comprised of randomly generated numbers. I am using the following code to do so:

import random
from numpy import *

cube = arange(1000000).reshape(100,100,100)

for element in cube.flat:
    matrix = arange(10000).reshape(100,100)
    for entry in matrix.flat:
        entry = random.random()*100
    element = matrix

I was expecting this to take a while, but with 10 billion random numbers being generated, I'm not sure my computer can even handle it. How much memory would such an array take up? Would RAM be a limiting factor, i.e. if my computer doesn't have enough RAM, could it fail to actually generate the array?

Also, if there is a more efficient to implement this code, I would appreciate tips :)

Upvotes: 9

Views: 17122

Answers (2)

Phil Cooper
Phil Cooper

Reputation: 5877

for the "inner" part of your function, look at the numpy.random module

import numpy as np
matrix = np.random.random((100,100))*100

Upvotes: 2

David Wolever
David Wolever

Reputation: 154484

A couple points:

  • The size in memory of numpy arrays is easy to calculate. It's simply the number of elements times the data size, plus a small constant overhead. For example, if your cube.dtype is int64, and it has 1,000,000 elements, it will require 1000000 * 64 / 8 = 8,000,000 bytes (8Mb).
  • However, as @Gabe notes, 100 * 100 * 1,000,000 doubles will require about 80 Gb.
  • This will not cause anything to "break", per-se, but operations will be ridiculously slow because of all the swapping your computer will need to do.
  • Your loops will not do what you expect. Instead of replacing the element in cube, element = matrix will simply overwrite the element variable, leaving the cube unchanged. The same goes for the entry = random.rand() * 100.
  • Instead, see: http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html#modifying-array-values

Upvotes: 23

Related Questions