CODEWITHSUNDEEP
pythonpython-2.7python-itertools
user99106
user99106

Reputation: 21

Cartesian Product memory error converting itertools.product to list

I'm trying to create the Cartesian product of a list of lists. When I try to convert the result to a list it will give me a memory error. If I run it without converting it to a list it runs fine.

lists = [['a','b','c' ],['a','b','c' ],['a','b','c' ],['a','b','c' ],['a','b','c' ],['a','b','c' ],['a','b','c' ],['a','b','c' ],['a','b','c' ]]
my_product = list(itertools.product(*lists))

I even tried to filter through some of the result using itertools.dropwhile to make it smaller before converting it into a list and I get the same result.

filtered = itertools.dropwhile(lambda x: x[1]!=x[2] and x[3]!=x[4] and x[3]!=x[5] and x[4]!=x[5], my_product)

Upvotes: 1

Views: 1956

Answers (1)

Martijn Pieters
Martijn Pieters

Reputation: 1122222

You are creating 19683 new tuples of 9 elements. Your computer does not have enough memory available for all those tuples at once.

If you do not use list(), then only a generator object is created, one that will produce the 19683 one-by-one as you iterate over it.

You should filter the output of itertools.product() without converting it to a list:

my_product = itertools.product(*lists)
filtered = itertools.dropwhile(lambda x: x[1]!=x[2] and x[3]!=x[4] and x[3]!=x[5] and x[4]!=x[5], my_product)

You could turn filtered into a list, but if all you do is loop over that list and process items one by one, you should not do that. Only turn it into a list if you need random access to the elements.

Upvotes: 1

Related Questions