CODEWITHSUNDEEP
pythonarrays
user3435121
user3435121

Reputation: 655

How to create a very large array in Python

For image processing, I need an array (array type, not numpy array) for 2 million 32 bit words. If I use something like:

tb = array.array( 'i', ,(0,)*2000000)

it requires 126 msec. It's large and I don't even need to initialize the array. I don't know Python internals but I assume that the statement generate tons on malloc() (memory allocator) and free() (memory deallocator).

Is there another way to create a very large Python array?

Upvotes: 2

Views: 1107

Answers (2)

pts
pts

Reputation: 87201

This is much faster, because it doesn't create a long, temporary tuple:

tb = array.array('i', (0,)) * 2000000

Upvotes: 2

Tim Peters
Tim Peters

Reputation: 70582

This does the same thing, but "should" run at least 10 times faster, by avoiding the needless expense of creating, and crawling over, a multi-million element tuple of unbounded Python ints:

>>> tb = array.array('i')
>>> tb.frombytes(b'\0' * 8_000_000)
>>> len(tb)
2000000
>>> all(i == 0 for i in tb)
True

Note: I'm assuming you're running on a platform where the array typecode i denotes a 4-byte integer type (that's why I changed your 2 million to 8 million). That's very likely, but if you're not sure of that, then slightly fancier code is needed:

>>> tb = array('i')
>>> tb.frombytes(b'\0' * (2_000_000 * tb.itemsize))

Of course tb.itemsize there returns 4 on my box.

Upvotes: 1

Related Questions