CODEWITHSUNDEEP
pythonmemory
Alxander
Alxander

Reputation: 323

Python list takes up too much memory

If I execute these lines of code in python:

states = itertools.product("012",repeat = 16)
states = list(states)

Then I use up more memory than I have on my laptop. Is there a way around this? I need this list of states so that when I generate a new state, I can update its value in the list.

Edit: I'm storing these states for a 4x4 grid,where 0, 1, and 2 are the possible states of each square on the grid. The value being stored is actually a 16 length list that says what the reward is for making a move to any of the squares on the grid from the current state. With impossible moves being marked with -np.inf. As the game is played the reward for moves that lead to winning from certain states increases, so that the bot is more likely to make that move in the future.

Ex: A simplified example for tic-tac-toe.

x| |o
 | | 
o| |

This state would be translated to a 9 length list, '102000200', and when it'd be looked up in the list of all possible states to see what the next best move is. Which in this case would be the middle spot for x.

Upvotes: 0

Views: 1247

Answers (2)

japreiss
japreiss

Reputation: 11251

itertools.product returns an iterator. converting to the list is the step that uses a lot of memory. can you write your algorithm to iterate over the product without storing it? like

for tuple16 in itertools.product("012", repeat = 16):
    do_something(tuple16)

Upvotes: 2

Tim Pietzcker
Tim Pietzcker

Reputation: 336168

I've just tested this on Python 3.4 (64 bit).

The resulting list is big, but not enormous (or so it seems):

>>> import itertools, sys
>>> states = itertools.product("012",repeat = 16)
>>> s = list(states)
>>> sys.getsizeof(s)
357571088

And my initial speculation that a list of strings would be smaller is incorrect - that didn't make much of a difference.

However, I can see that Python's memory usage increases from 4 MB (after launch) to about 8 GB after the call to list, and it only returns to the baseline state after del(s), not after gc.collect() so it appears that there is some enormous overhead associated with such a large, multi-element list. It may have something to do with what Alex Martelli described here, in which case any Python solution will become quite complicated.

Perhaps you need to think about a different approach to the problem. You don't really need to store all those states - it's easy to calculate what item number 123456 of that list will be, so perhaps you only need to store the ones that are modified during the program's run?

Upvotes: 2

Related Questions