CODEWITHSUNDEEP
pythondictionarymemorymemory-managementmemory-leaks
Can Geliş
Can Geliş

Reputation: 1474

dict pop does not free the memory

Looks like garbage collector does not collect the values pop'd from the dict() in python 2.7 (did not try on python 3). Here is the example:

a = dict()

# fill the memory (dict)
for i in xrange(0, 9999999):
    a[i] = i

# Memory usage is about 600 MB
# try to free the memory
for i in xrange(0, 9999999):
    a.pop(i)

# print the dict and see it is empty
print "%r" % a
# prints: {}
# Memory usage is about 600 MB

import copy
a = copy.copy(a)
# Memory usage decreased to about 200 MB

import gc
gc.collect()
# Memory usage decreased to about 10 MB

Anybody knows why this happens and how to solve this memory leak issue?

Upvotes: 3

Views: 2924

Answers (1)

Daniel
Daniel

Reputation: 42778

There is no memory leak issue, since the memory is freed, when the dictionary isn't used any more. Dictionaries use internal tables to store the entries. These tables are not freed when using pop, because every key is mapped to a hash modulo size of the internal table. So the last existing key could lie at the end of this table.

To illustrate this, I'll use sys.getsizeof:

>>> a= {}
>>> sys.getsizeof(a)
288
>>> for i in range(9999999): a[i]=i
... 
>>> sys.getsizeof(a)
402653280
>>> for i in range(9999999): del a[i]
... 
>>> sys.getsizeof(a)
402653280
>>> a = copy.copy(a)
>>> sys.getsizeof(a)
288
>>>

Instead of using excessive pops, you should create new dictionaries if necessary.

Upvotes: 7

Related Questions