Concatenate Big List Elements Efficiently

Question

I want to make a list of elements where each element starts with 4 numbers and ends with 4 letters with every possible combination. This is my code

import itertools

def char_range(c1, c2):
    """Generates the characters from `c1` to `c2`"""
    for c in range(ord(c1), ord(c2)+1):
        yield chr(c)


chars =list()
nums =list()
for combination in itertools.product(char_range('a','b'),repeat=4):
        chars.append(''.join(map(str, combination)))

for combination in itertools.product(range(10),repeat=4):
        nums.append(''.join(map(str, combination)))

c = [str(x)+y for x,y in itertools.product(nums,chars)]
for dd in c:
        print(dd)

This runs fine but when I use a bigger range of characters, such as (a-z) the program hogs the CPU and memory, and the PC becomes unresponsive. So how can I do this in a more efficient way?

Answer 1

The documentation of itertools says that "it is roughly equivalent to nested for-loops in a generator expression". So itertools.product is never an enemy of memory, but if you store its results in a list, that list is. Therefore:

for element in itertools.product(...):
    print element

is okay, but

myList = [element for itertools.product(...)]

or the equivalent loop of

for element in itertools.product(...):
    myList.append(element)

is not! So you want itertools to generate results for you, but you don't want to store them, rather use them as they are generated. Think about this line of your code:

c = [str(x)+y for x,y in itertools.product(nums,chars)]

Given that nums and chars can be huge lists, building another gigantic list of all combinations on top of them is definitely going to choke your system.

Now, as mentioned in the comments, if you replace all the lists that are too fat to fit into the memory with generators (functions that just yield), memory is not going to be a concern anymore.

Here is my full code. I basically changed your lists of chars and nums to generators, and got rid of the final list of c.

import itertools

def char_range(c1, c2):
    """Generates the characters from `c1` to `c2`"""
    for c in range(ord(c1), ord(c2)+1):
        yield chr(c)

def char(a):
    for combination in itertools.product(char_range(str(a[0]),str(a[1])),repeat=4):
        yield ''.join(map(str, combination))

def num(n):
    for combination in itertools.product(range(n),repeat=4):
        yield ''.join(map(str, combination))

def final(one,two):
    for foo in char(one):
        for bar in num(two):
            print str(bar)+str(foo)

Now let's ask what every combination of ['a','b'] and range(2) is:

final(['a','b'],2)

Produces this:

0000aaaa
0001aaaa
0010aaaa
0011aaaa
0100aaaa
0101aaaa
0110aaaa
0111aaaa
1000aaaa
1001aaaa
1010aaaa
1011aaaa
1100aaaa
1101aaaa
1110aaaa
1111aaaa
0000aaab
0001aaab
0010aaab
0011aaab
0100aaab
0101aaab
0110aaab
0111aaab
1000aaab
1001aaab
1010aaab
1011aaab
1100aaab
1101aaab
1110aaab
1111aaab
0000aaba
0001aaba
0010aaba
0011aaba
0100aaba
0101aaba
0110aaba
0111aaba
1000aaba
1001aaba
1010aaba
1011aaba
1100aaba
1101aaba
1110aaba
1111aaba
0000aabb
0001aabb
0010aabb
0011aabb
0100aabb
0101aabb
0110aabb
0111aabb
1000aabb
1001aabb
1010aabb
1011aabb
1100aabb
1101aabb
1110aabb
1111aabb
0000abaa
0001abaa
0010abaa
0011abaa
0100abaa
0101abaa
0110abaa
0111abaa
1000abaa
1001abaa
1010abaa
1011abaa
1100abaa
1101abaa
1110abaa
1111abaa
0000abab
0001abab
0010abab
0011abab
0100abab
0101abab
0110abab
0111abab
1000abab
1001abab
1010abab
1011abab
1100abab
1101abab
1110abab
1111abab
0000abba
0001abba
0010abba
0011abba
0100abba
0101abba
0110abba
0111abba
1000abba
1001abba
1010abba
1011abba
1100abba
1101abba
1110abba
1111abba
0000abbb
0001abbb
0010abbb
0011abbb
0100abbb
0101abbb
0110abbb
0111abbb
1000abbb
1001abbb
1010abbb
1011abbb
1100abbb
1101abbb
1110abbb
1111abbb
0000baaa
0001baaa
0010baaa
0011baaa
0100baaa
0101baaa
0110baaa
0111baaa
1000baaa
1001baaa
1010baaa
1011baaa
1100baaa
1101baaa
1110baaa
1111baaa
0000baab
0001baab
0010baab
0011baab
0100baab
0101baab
0110baab
0111baab
1000baab
1001baab
1010baab
1011baab
1100baab
1101baab
1110baab
1111baab
0000baba
0001baba
0010baba
0011baba
0100baba
0101baba
0110baba
0111baba
1000baba
1001baba
1010baba
1011baba
1100baba
1101baba
1110baba
1111baba
0000babb
0001babb
0010babb
0011babb
0100babb
0101babb
0110babb
0111babb
1000babb
1001babb
1010babb
1011babb
1100babb
1101babb
1110babb
1111babb
0000bbaa
0001bbaa
0010bbaa
0011bbaa
0100bbaa
0101bbaa
0110bbaa
0111bbaa
1000bbaa
1001bbaa
1010bbaa
1011bbaa
1100bbaa
1101bbaa
1110bbaa
1111bbaa
0000bbab
0001bbab
0010bbab
0011bbab
0100bbab
0101bbab
0110bbab
0111bbab
1000bbab
1001bbab
1010bbab
1011bbab
1100bbab
1101bbab
1110bbab
1111bbab
0000bbba
0001bbba
0010bbba
0011bbba
0100bbba
0101bbba
0110bbba
0111bbba
1000bbba
1001bbba
1010bbba
1011bbba
1100bbba
1101bbba
1110bbba
1111bbba
0000bbbb
0001bbbb
0010bbbb
0011bbbb
0100bbbb
0101bbbb
0110bbbb
0111bbbb
1000bbbb
1001bbbb
1010bbbb
1011bbbb
1100bbbb
1101bbbb
1110bbbb
1111bbbb

Which is the exact result you are looking for. Each element of this result is generated on the fly, hence never creates a memory problem. You can now try and see that much bigger operations such as final(['a','z'],10) are CPU-friendly.