CODEWITHSUNDEEP
pythonpython-3.x
sln
sln

Reputation: 210

How to find memory usage with memory_profiler Python?

At the end of this question I found some code. I will put it here for convenience:

import memory_profiler as mp

def fun(n):
    tmp = []
    for i in range(n):
        tmp.extend(list(range(i*i)))
    return "XXXXX"

start_mem = mp.memory_usage(max_usage=True)
res = mp.memory_usage(proc=(fun, [100]), max_usage=True, retval=True) 
print('start mem', start_mem)
print('max mem', res[0][0])
print('used mem', res[0][0]-start_mem)
print('fun output', res[1])

But this doesn't work because res is not a double array, it is float number. Moreover, I don't understand how to check memory usage for many functions. I mean, would something like this work?

import memory_profiler as mp

def fun1(n):
    return "XXXXX"
def fun2(n):
    return "YYYYY"

methods = [
    'fun1(n)',
    'fun2(n)',
    ]

start_mem = mp.memory_usage(max_usage=True)
res = mp.memory_usage(proc=(methods[0], [100]), max_usage=True, retval=True) 
print('start mem', start_mem)
print('max mem', res[0][0])
print('used mem', res[0][0]-start_mem)
print('fun output', res[1])

Upvotes: 4

Views: 13684

Answers (2)

sln
sln

Reputation: 210

I'll try to complement Alok's answer.

memory_usage method takes tuple (method, args, kwargs).

To find memory usage of a method and store value in variable, you can do this:

from memory_profiler import memory_usage


def foo(n):
    t = [list(range(i * i)) for i in range(n)]


if __name__ == '__main__':
    memory = memory_usage((foo, (5,), {}))
    print(memory)

If you have multiple methods, you can put them in list and then iterate over it:

from memory_profiler import memory_usage


def foo(n):
    t = [list(range(i * i)) for i in range(n)]


def bar(n):
    t = [1 for i in range(n)]


if __name__ == '__main__':
    methods = [foo, bar]
    memory = memory_usage((methods[0], (5,), {}))

Don't forget about using memory_usage in if __name__ == '__main__': or in other methods and don't forget about comma in (5,). This code will throw two exceptions because of __main__ and comma:

from memory_profiler import memory_usage


def foo(n):
    t = [list(range(i * i)) for i in range(n)]


def bar(n):
    t = [1 for i in range(n)]


functions = [foo, bar]
memory = memory_usage((functions[0], (5), {}))

Full information about memory_usage you can find in documentation.

Upvotes: 2

Pratap Alok Raj
Pratap Alok Raj

Reputation: 1206

Installing memory_profiler:

pip3 install -U memory_profiler

Sometimes it is useful to have full memory usage reports as a function of time (not line-by-line) of external processes (be it Python scripts or not). In this case, the executable mprof might be useful

To get the line by line profiling, add decorator ‘@profile’ in front of the def function.

from memory_profiler import profile
@profile
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

if __name__ == '__main__':
    my_func()

After this save the .py file and run the below code in terminal.

mprof run perf_test_v1.py

This will show you the memory usage by every line of code. To Plot the memory usage vs time through the below code.

mprof plot

The available commands for mprof are:

mprof run: running an executable, recording memory usage
mprof plot: plotting one the recorded memory usage (by default, the last one)
mprof list: listing all recorded memory usage files in a user-friendly way.
mprof clean: removing all recorded memory usage files.
mprof rm: removing specific recorded memory usage files

This would help you identify the bottlenecks in your codes and helps in optimizing the steps.

To get time-based memory usage

Execute the code passing the option -m memory_profiler to the python interpreter to load the memory_profiler module and print to stdout the line-by-line analysis. If the file name was example.py, this would result in:

$ python -m memory_profiler example.py

Upvotes: 7

Related Questions