CODEWITHSUNDEEP
pythonmemory-leaksipythonjupyter-notebook
chentingpc
chentingpc

Reputation: 1283

ipython notebook - memories disappear after running some function

After runing some (imported) function that returns a variable, say X. The memory usage increases much more than the size of X. And even I del X or all variables by %reest. The memory usage is still much larger than before running the function. And I can only clear all memory by restarting the notebook.

I am confused, it seems the notebook saves the local variables used only inside a (imported) function? Which seems unnecessary for me. And if this is not a bug, how can I trace those large (local) variables used only in the called function, and clear those memories?

Upvotes: 0

Views: 412

Answers (1)

Peter Rowell
Peter Rowell

Reputation: 17713

As you said in your comment above, you are wondering why the iPython notebook process doesn't release memory back to the OS. The answer is somewhat complicated, and goes to the heart of how memory is managed in both the OS and in the process.

When a process asks for more memory, the OS has to find an appropriately sized chunk to map into the process's address space -- let's say 8KB (although it is probably some large multiple of that). This is always done on page-boundaries because of a number of factors, including how virtual memory works. TLDP has a fairly decent description of how and why this is done: Chapter 3 Memory Management

Meanwhile, the process didn't actually want 8KB for a new object, it only wanted 128B. So it gets the 8KB chunk and adds it to the in-process pool of available memory. Note that this is normally a portion of the process's address space that is contiguous so that the 8KB boundaries of the OS page are invisible to the process.

The in-process memory manager then allocates 128B to be used by whoever asked for it. This happens over and over as objects are created. When they are destroyed, the memory is returned to the in-memory pool of available memory.

So if it's free, why not give it back to the OS? Well, for one thing it's almost certainly not on a page boundary, so the OS wouldn't know what to do with it. It's also almost certain that the freed memory is not at the end of the in-memory pool. So releasing it would create a discontinuity in the pool's address space, which could then become a major PITA for the in-process memory manager to tip-toe around.

In certain circumstances, where you know you need some big chunk of memory, that it will be used in a way that is independent from the rest of the in-process memory management, and that when you're done with it you can give it back to the OS, then it is possible to reduce the process's memory size. This is most often seen when a file is mmap'ed into the process image, worked on, and then unmapped. But that's special case stuff.

Upvotes: 1

Related Questions