CODEWITHSUNDEEP
python
guorui
guorui

Reputation: 891

Is there an easy way to delete most variables but keep the loaded data for ongoing code?

The situation is, there is a for-loop which runs 100 times, sharing a same loaded data. As each single loop goes, it will create a growing number of variables which occupies a lot of memory resource and then further slows down the running speed. The code can be abstractly written as

data = sio.loadmat('very_big_data_set.mat')
for t in range(100):
    config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
    sess = tf.Session(graph=tf.get_default_graph(), config=config)
    var_1 = ...
    var_2 = ...


    var_10000 = ...
    model = ...

    sess.close()

    # The line below is a solution but it's very inconvinient
    del config, sess, var_1, ..., var_10000, model

My question is if there is a python command I can use to delete all of created variables during the for-loop but keep the loaded data for next loop? I know the method del config, sess, var_1, ..., var_10000, model but it requires me to list all of variables I want to delte. So I am looking for a more easy way, kind of like del all except data.

Upvotes: 1

Views: 84

Answers (2)

Dorian Turba
Dorian Turba

Reputation: 3735

What you need is reduced scope for your variable. Actually, your variables is global, which made the data present in memory until the program ends.

Here is an example of your needs:

def data_valorisation(config, sess):
    # All theses variable will be deleted and recreated each time you call the function
    var_1 = ...
    var_2 = ...
    [...]
    var_10000 = ...
    model = ...

data = sio.loadmat('very_big_data_set.mat')
for t in range(100):
    config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
    sess = tf.Session(graph=tf.get_default_graph(), config=config)
    data_valorisation(config, sess)
    sess.close()

You should always put computation into function and avoid doing it in a global context. For this, you can use function or classes.

Variables and scope

Upvotes: 1

Grismar
Grismar

Reputation: 31319

Generally, if you want to use variables that need to be cleaned up after you're done with it, especially if you're using them repeatedly, writing a function is a good idea:

def do_something(arg):
    # arg gets the same value as x (or points to the same thing)
    y = 10
    # now y exists, we can do do something with y and arg
    r = y + arg
    # now r, y and arg exist
    return r
    # once the function returns, arg no longer exists or references x
    # y and r are no longer referenced either and will no longer exist

for x in range(10)
    print(do_something(x))
    # here, y and r don't exist anymore, because the function has ended
# here, y and r don't exist, but x still does, since a loop variable sticks around
# capture all this in a function and call it and x would be cleaned up as well

Upvotes: 1

Related Questions