CODEWITHSUNDEEP
pythonmemory-management
random_user
random_user

Reputation: 848

Delete reference of variable passed to a function in Python

Given the following code:

import numpy as np
import psutil

original = np.ones(2**28) / 2

def func1(x):
    print(1,psutil.virtual_memory().percent)
    x=x**2
    print(2,psutil.virtual_memory().percent)
    func2(x)
 
def func2(x):
    print(3,psutil.virtual_memory().percent)
    y=x**2
    print(4,psutil.virtual_memory().percent)
    del x # this does not lower the memory usage
    print(5,psutil.virtual_memory().percent)
    return

func1(original)

Would it be possible alter func1 in order to lower the memory consumption after del x in func? (since variable passed to func2 is no longer usefull to func1).

For instance, the following does not work:

import numpy as np
import psutil

original = np.ones(2**28) / 2

def func1(x):
    print(1,psutil.virtual_memory().percent)
    x=x**2
    print(2,psutil.virtual_memory().percent)
    x=[x]
    func2(x.pop())
 
def func2(x):
    print(3,psutil.virtual_memory().percent)
    y=x**2
    print(4,psutil.virtual_memory().percent)
    del x # this does not lower the memory usage
    print(5,psutil.virtual_memory().percent)
    return

func1(x)

Note: assume func2 is external code that cannot be changed.

To put in another way, I would like that func1 passed x to func2 and deleted its internal reference to x before the begging of func2.

Upvotes: 0

Views: 777

Answers (2)

user2357112
user2357112

Reputation: 280310

Under the current CPython implementation, when performing a Python function call, the caller retains ownership of all argument references until the call is complete. Rather than transferring ownership, the callee gets new or borrowed references (depending on how the callee is implemented, especially whether it's written in C or Python).

There is no way to transfer ownership to the callee, and no way for the callee to clear the caller's references. Even if you try something like your pop trick to ensure the caller doesn't have a variable referring to the argument, there's still a reference living in the caller's bytecode operand stack, and that reference only gets cleared after the call is done. (You can see this in the CPython source code.)

Your best shot would be to pass the wrapper list to func2 directly, except you said you don't have access to the source code for func2, so you can't modify it to take a list like that.

You're pretty much out of luck.

Upvotes: 1

Paul Becotte
Paul Becotte

Reputation: 9977

Python stores data objects in underlying data structures. If you do

x = 42

There are two things now - there is an "integer" object that contains a value 42, and there is a pointer "x" that points at that integer object.

When you do del x it is the pointer that is getting deleted. Remember, x inside that function is NOT x in any other context. The integer object itself is managed with reference counting under the hood. If there are no other variables pointing to it, then at some point the interpreter will go ahead and delete it.

To try and save memory, imagine your func is taking a small slice from your array and working on it. When its done, nothing else ever needs that slice, so get rid of it, right? There are two possibilities...

  1. You are done with the slice, the last reference is gone... the interpreter will eventually remove the slice object from memory.

  2. You're using numpy (like it looks like you are) and the code is implemented in such a way that it does not actually make a copy, but allows you to operate over the original data structure in place. In this case, the slice is just a pointer to the data rather than the data, so deleting it doesn't get you much.

Either way, you're probably looking at the wrong spot to try and save memory here.

Upvotes: 0

Related Questions