CODEWITHSUNDEEP
pythonnumpymmapmemory-mapped-fileslarge-data
NPE
NPE

Reputation: 500475

Purging numpy.memmap

Given a numpy.memmap object created with mode='r' (i.e. read-only), is there a way to force it to purge all loaded pages out of physical RAM, without deleting the object itself?

In other words, I'd like the reference to the memmap instance to remain valid, but all physical memory that's being used to cache the on-disk data to be uncommitted. Any views onto to the memmap array must also remain valid.

I am hoping to use this as a diagnostic tool, to help separate "real" memory requirements of a script from "transient" requirements induced by the use of memmap.

I'm using Python 2.7 on RedHat.

Upvotes: 1

Views: 678

Answers (1)

Igor Nazarenko
Igor Nazarenko

Reputation: 2264

If you run "pmap SCRIPT-PID", the "real" memory shows as "[ anon ]" blocks, and all memory-mapped files show up with the file name in the last column.

Purging the pages is possible at C level, if you manage to get ahold of the pointer to the beginning of the mapping and call madvise(ptr, length, MADV_DONTNEED) on it, but it's going to be cludgy.

Upvotes: 2

Related Questions