CODEWITHSUNDEEP
pythonmemorygarbage-collection
lumbric
lumbric

Reputation: 9053

Why does Python 2 consume a lot of memory even if there are no large objects referenced?

Consider the following minimal example:

# used memory: Python2=7421 MB, Python3=7440 MB
a = list(range(10**8))
# used memory: Python2=10553 MB, Python3=11317 MB
a = 1
# used memory: Python2=9785 MB, Python3=7454 MB
# ---> why does Python2 need >2GB of RAM here?

# after python process terminates: Python2=7433 MB, Python3=7458 MB

A large object is created which should be garbage collected after the second line. The memory usage has been monitored using free -m (this is not an exact measurement of course).

Python 3 needs more memory (3.7GB instead of 3.05GB) to store the large object, but it does what I expected: memory usage drops after the object is not needed any longer. Python2 seems to delete only 768 MB and keep 2.3GB of memory allocated. Why?

This is repeatable: if the list is created a second time, it will use again 3.05 GB, not more and it will drop again to 2.3GB RAM usage. gc.collect() returns 0 and does not change the amount of used memory.

Please don't tell me to use Python 3 - I know... :)

Some links to documentation which did not answer my question:

Upvotes: 3

Views: 847

Answers (1)

user2357112
user2357112

Reputation: 280564

In the specific case of reclaimed ints on Python 2, the memory is stuck on an unbounded free list and not returned to the OS. The memory reserved for ints is thus proportional to the largest number of ints that have existed simultaneously in the program, not the number of ints that currently exist.

For other cases of memory not returning to the OS, that's probably due to details of the underlying malloc allocator. Most other free lists I can think of in Python are bounded.

Upvotes: 4

Related Questions