CODEWITHSUNDEEP
pythonpython-3.xmemorymemory-managementmemory-leaks
Tom Burrows
Tom Burrows

Reputation: 2365

Is it possible for a Python function to still use memory after being called?

If I run a function in Python 3 (func()) is it possible that objects that are created inside func() but cannot be accessed after it has finished would cause it to increase its memory usage?

For instance, will running

def func():
    # Objects being created, that are not able to be used after function call has ended.

while True:
    func()

ever cause the program run out of memory, no matter what is in func()?

If the program is continually using memory, what are some possible things that could be going on in func() to cause it to continue using memory after it has been called?

Edit: I'm only asking about creating objects that can no longer be accessed after the function has ended, so they should be deleted.

Upvotes: 1

Views: 5236

Answers (1)

ohmu
ohmu

Reputation: 19752

Yes, it is possible for a Python function to still use memory after being called.

  • Python uses garbage collection (GC) for memory management. Most GCs (I suppose there could be some exceptions) make no guarantee if or when they will free the memory of unreferenced objects. Say you have a function consume_lots_of_memory() and call it as:

    while True:
    consume_lots_of_memory()

    There is no guarantee that all of the memory allocated in the first call to consume_lots_of_memory() will be released before it is called a second time. Ideally the GC would run after the call finished, but it might run half way through the fifth call. So depending on when the GC runs, you could end up consuming more memory than you would expect and possibly even run out of memory.

  • Your function could be modifying global state, and using large amounts of memory that never gets released. Say you have a module level cache, and a function cache_lots_of_objects() called as:

    module_cache = {}
while True:
    cache_lots_of_objects()

    Every call to cache_lots_of_objects() only ever adds to the cache, and the cache just keeps consuming more memory. Even if the GC promptly releases the non-cached objects created in cache_lots_of_objects(), your cache could eventually consume all of your memory.

  • You could be encountering an actual memory leak from Python itself (unlikely but possible), or from a third-party library improperly using the C API, using a leaky C library, or incorrectly interfacing with a C library.

  • One final note about memory usage. Just because Python has freed allocated objects, it does not necessarily mean that the memory will be released from the process and returned to the operating system. The reason has to do with how memory is allocated to a process in chunks (pages). See abarnert's answer to Releasing memory in Python for a better explanation than I can offer.

Upvotes: 3

Related Questions