CODEWITHSUNDEEP
python-3.xmemory
Trekkie
Trekkie

Reputation: 994

How can I monitor mem usage during a Python function call

I have a Python function for which I would like to monitor memory usage over time, call it foo.

Note that I don't want to use an external profiler... I just want very simple statistics that I can append to a log file.

I would like to use the following function to perform the monitoring in a separate process while the body of foo is executing:

def monitor_mem(monitor_list, tic):
    while process_not_terminated:
        monitor_list.append(psutil.virtual_memory())
        time.sleep(tic)

The foo function would look something like this (note that mp is multiprocessing module):

def foo():
    # Start collecting system memory data
    monitor_list = []
    monitor_process = mp.Process(target=monitor_mem, args=(monitor_list, 0.5))
    monitor_process.start()

    # Do interesting stuff
    print('Doing interesting stuff!')
    time.sleep(5)

    # End collecting system memory data
    monitor_process.terminate()
    monitor_process.join()
    return monitor_list

I've played around with making the loop in monitor_mem a while True. That doesn't seem to work. I feel like I need to be able to send some sort of signal to stop the function... but I'm just not sure. Hopefully one of you all can look at my code example and give me some hints. Thanks!

Upvotes: 0

Views: 144

Answers (1)

zelenyjan
zelenyjan

Reputation: 703

You should use Queue instead of list when you working with threads and processes.

Example with Queue:

from multiprocessing import Process, Queue
from datetime import datetime
from time import sleep


def monitor_mem(monitor_queue):
    while True:
         monitor_queue.put(datetime.now())
         sleep(1)


queue = Queue()
process = Process(target=monitor_mem, args=(queue,))
process.daemon = True
process.start()

print("Doing something")
sleep(5)

process.terminate()
while not queue.empty():
    item = queue.get()
    print(item)

Upvotes: 1

Related Questions