Newbie
Newbie

Reputation: 121

High memory pressure in generator with random.sample()

DISCLAIMER (added later): I have modified the code to take into account @jasonharper and @user2357112supportsMonica comments here below. Still having the memory issue.

I'm running the following code:

import itertools
from tqdm import tnrange
import random

def perm_generator(comb1, comb2):
    seen = set()
    length1 = len(comb1)
    length2 = len(comb2)
    while True:
        perm1 = tuple(random.sample(comb1, length1))
        perm2 = tuple(random.sample(comb2, length2))
        perm_pair = perm1 + perm2
        if ( (perm_pair not in seen) ):
            seen.add(perm_pair)
            yield [perm1,perm2]            


seq_all = ('A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V') 
combinations_first_half = list(itertools.combinations(seq_all, int(len(seq_all)/2)))

n = 1000
random.seed(0)
all_rand_permutations = []
for i in tnrange(len(combinations_first_half), desc = 'rand_permutations'):
    comb1 = combinations_first_half[i]
    comb2 = tuple(set(seq_all) - set(comb1))
    gen = perm_generator(comb1,comb2)
    rand_permutations = [next(gen) for _ in range(n)]
    all_rand_permutations += rand_permutations

and for almost all the iterations of the for loop everything goes smoothly with about 33 iterations per second.

However, in some rare cases, the loop gets stuck and begins to build memory pressure for quite a few seconds. Eventually, on a given later iteration the kernel dies.

It seems to be related with random.sample() because if I start the loop from the index relative to the iteration in which the kernel dies or one of the high memory pressure iterations (hence, somehow consequently shifting the random.seed()), there is no issue and the loop goes through it like for the other fast iterations.

I attach here a few screenshots:

Iteration building high memory pressure

Iteration with no problem

Iteration in which the Kernel dies

Upvotes: 0

Views: 81

Answers (0)

Related Questions