Sluggish after usinge 100% RAM

Question

My hardware specs are listed below, though this is probably not important

I use my machine for data processing, which often involves holding large amounts of data in memory (I have made many improvements in my code lately by reading in partial files to decrease memory use, but this remains problematic), with all of my analysis being done in Python 2.7 (numpy/scipy and pandas). When RAM usage == 100% the computer performance becomes expectedly sluggish. What I don't understand is why it remains sluggish (e.g. it takes ~20s to open a ~4kb file) after my code finishes running, memory usage has dropped to ~10-50%, and the CPU sits just above idle. I have even forced all Python processes to close in task manager, which does not help. Furthermore, the sluggishness persists until reboot.

From my hardware status stats I would expect better performance, so I suspect there is something going on behind the scenes. Since this only happens after I push memory usage to 100% (which only happens when I am processing data in Python) I wonder if Python is doing something behind the scenes that I am not aware of (honestly, I cannot image what this would be, but I am truly stumped). Since I only see this in Python I give it a python tag (though I think this is not a python issue).

Any insight will be appreciated. I have searched online, but everything I find relates to "why my computer runs slow when CPU/RAM usage == 100%"-type queries.

OS: Windows 7 Enterprise, 64-bit

Python: 2.7 via Anaconda

CPU: Xeon E5645, six-core

RAM: 12GB

Answer 1

I would have thought that Task Manager, Processes, with appropriate columns selected by View, sorted by clicking on column heading (for Top), would suggest reasons, along with Performance tab info. Then there is Perfmon Resource Monitor and my favourite, logging via Perfmon, Performance, Data Collection Sets, User Defined.

Answer 2

There is a concept called 'system-load'. A system under a large load ( comprising of interrupts, IO wait, context switches, computation etc) takes time to recover. The load persists for some time because unlike core-computation, the other operations that make the system work hard, do not simply stop suddenly.

For example- the script while(true) i++; for example, will consume only cpu, but the script- while(true) print(i++) will push i's value to something of a queue like nature- the stdout may be a monitor, a NW port or a printer queue. Now when you kill your process, what the kernel does is that it frees that process's memory and stop allocating it CPU.

But there is a total 'sin', that stays behind- at different levels and components. These, consume the resources in turn, until they are done with. And because you can only kill your python script and not the n printing processes that you have irreversibly called, the kernel cannot do anything.

But understand that this is just one of the SW case scenarios. There are Paging, Swapping, Thrashing operations (and a lot more) which too make the 'recovery' slower. Recovery from swap takes 1000X the time as memory (RAM) operations. All in all, a system under zero load can handle 10 tasks in x time whereas a system under full load will take 100x time to do the same tasks because each context swich, each housekeeping operation is 'costlier'.

The do-able thing here is to run a utility like top or htop and see how long the high load-average persists.

Answer 3

Once you've hit 100% of your memory, you're going to start hitting the swap on your machine, which uses the disk which is WAYYY slower than memory. After your program is done running, all of the other programs need to re-read thier information back into memory. If the problem doesn't go away after a couple of minutes, then you might have a seperate problem going on.