CODEWITHSUNDEEP
linuxubuntumemorylinux-kernel
max
max

Reputation: 52343

Measuring memory used by a process, accounting for copy-on-write

Linux has copy-on-write, which means that, after a fork, a child process can share the memory with the parent process as long as it doesn't modify it.

Let's say the parent process takes 10 GB of physical RAM. When I fork the process, the physical memory used by the OS doesn't immediately go up by 10 GB (it may go up slightly due to the creation of some administrative structures). This can be confirmed using free shell command. Thus free correctly accounts for CoW.

However, when I ask the OS about the amount of memory used by a specific process (e.g., using top or any C API function I am aware of), it shows that the physical memory used by the child process is 10 GB right away (before it modifies anything). Thus per-process memory tracking functions don't correctly account for CoW.

I am looking for a way to measure per-process memory accounting for CoW. (Going to use it from python, but once I know the relevant C API, I'm fine.)

To clarify: the shared memory used by multiple processes should be allocated, for accounting purposes, to the parent process.

USE CASE:

We're trying to reduce the total memory used by an application. We have very large data structures in the parent process, which are shared with the child processes by simple forking. We don't need to modify those structures in child processes, but modifications to reference counters (in python) causes parts of the memory to be copied. We're trying to minimize the extent to which this happens to preserve physical memory.

RELATED QUESTIONS

https://serverfault.com/questions/676335/how-measure-memory-without-copy-on-write-pages (provides a possible answer)

How to know whether a copy-on-write page is an actual copy? (provides some useful details to create a solution)

Upvotes: 2

Views: 766

Answers (1)

Andrew Henle
Andrew Henle

Reputation: 1

I don't know of any way to solve this outside the kernel - you'd need to go through the virtual-to-physical mappings of every process, then correlate physical mappings between processes while accounting for swapped out memory that didn't have a physical mapping. And by the time you got finished, your answer would no longer be correct.

I know of no OS that provides what you're asking. If it were worth solving I have to think someone would have done so.

Upvotes: 1

Related Questions