CODEWITHSUNDEEP
pythonarrayslistpython-3.xmemory
jan-seins
jan-seins

Reputation: 1283

Preallocate very large array in Python leads to MemoryError

I am trying to preallocate a list in python

c=[1]*mM #preallocate array

My Problem is that I run in to a MemoryError since

mM=4999999950000000

What is the best way to deal with this. I am thinking about creating a new object where is split my list at about a value of 500000000. Is this what I should do or is there a best practice to create an array with a lot of inputs?

Upvotes: 1

Views: 377

Answers (1)

foslock
foslock

Reputation: 3939

Using a Generator

You are attempting to create an object that you very likely will not be able to fit into your computer's memory. If you truly need to represent a list of that length, you can use a generator that dynamically produces values as they are needed.

def ones_generator(length):
    for _ in range(length):
        yield 1

gen = ones_generator(4999999950000000)
for i in gen:
    print(i)  # prints 1, a lot

Note: The question is tagged for Python 3, but if you are using Python 2.7, you will want to use xrange instead of range.

Using a Dictionary

By the sound of your question, you do not actually need to preallocate a list of that length, but you want to store values very sparsely at indexes that are very large. This pattern matches the dict type in Python more so than the list. You can simply store values in a dictionary, without pre-allocating they keys/space, Python handles that under the hood for you.

dct = {}
dct[100000] = "A string"
dct[592091] = 123
dct[4999999950000000] = "I promise, I need to be at this index"

print(dct[4999999950000000])
# I promise, I need to be at this index

In that example, I just stored str and int values, but they can be any object in Python. The best part about this is that this dictionary will not consume memory based on the maximum index (like a list would) but instead based on how many values are stored within it.

Upvotes: 4

Related Questions