Scenario: I have a python process which runs continuously ( until we stop manually - infinitely ) and collects data by reading certain system files every 1 minute. When it is started it occupies around 25 MB. But day by day the amount of memory occupied is increasing and after 15 days, it is more than 500 MB. I am a python newbie, so unable to find out the problem by going over code manually / using debugger, I am not aware of any tool which can easily spot the issue, please suggest some tool or method which can help me in spotting out the issue. I assume it may be a memory leak issue or a variable which appends data to it again and again, and not at all freeing it up... Please point me to certain references which can help me in understanding these kind of issues in python. Hope the details are clear enough, for more details let me know, i can clarify. Update I was trying to follow this thread Showing the stack trace from a running Python application , but i face the following errors which am not able to completely solve. No symbol "PyEval_EvalFrameEx" in current context. No symbol "PyStringObject" in current context. I searched to find out such a method, to hook into the already running process and get the stack / memory information. But no luck yet, kindly help me.

Reputation: 537

python processes increases memory usage day by day

Scenario:

I have a python process which runs continuously ( until we stop manually - infinitely ) and collects data by reading certain system files every 1 minute. When it is started it occupies around 25 MB. But day by day the amount of memory occupied is increasing and after 15 days, it is more than 500 MB.

I am a python newbie, so unable to find out the problem by going over code manually / using debugger,
I am not aware of any tool which can easily spot the issue, please suggest some tool or method which can help me in spotting out the issue.
I assume it may be a memory leak issue or a variable which appends data to it again and again, and not at all freeing it up... Please point me to certain references which can help me in understanding these kind of issues in python.

Hope the details are clear enough, for more details let me know, i can clarify.

Update

I was trying to follow this thread Showing the stack trace from a running Python application, but i face the following errors which am not able to completely solve.

No symbol "PyEval_EvalFrameEx" in current context.
No symbol "PyStringObject" in current context.

I searched to find out such a method, to hook into the already running process and get the stack / memory information. But no luck yet, kindly help me.

Upvotes: 5

Answers (2)

sam

Reputation: 11

This is a well known problem exhibited by long running python processes. Improvements have been made but have not completely solve the problem. For instance Python uses its own memory management algorithm which may lead to fragmentation of its own heap. The best strategy is to run multiple processes and gracefully restart when each of them hit a high memory watermark. Apache also uses this strategy.

Upvotes: 1

Vinay Sajip

Reputation: 99297

use Dowser to help track memory leaks. It uses CherryPy as a web server and even if you're not developing a Web application, it can provide a view of memory allocations to you from your browser.

See also this post, which you may find of interest. With Dowser, you should be able to see what unexpected objects are hanging around in your process memory.

Upvotes: 3

python processes increases memory usage day by day

Answers (2)

Related Questions