CODEWITHSUNDEEP
clinuxout-of-memory
Ælex
Ælex

Reputation: 14869

Linux C app runs out of memory

I am running the word2phrase.c using a very large (45Gb) training set. My PC has 16Gb of physical RAM and 4Gb of swap. I've left it train overnight (second time tbh) and I come back in the morning, to see it was "killed" without further explanation. I sat and watched it die, when my RAM run out.

I set in my /etc/sysctl.conf

vm.oom-kill = 0
vm.overcommit_memory = 2

The actual source code does not appear to write to the file the data, but rather keep it in memory, which is creating the issue.

  1. Is the total memory (RAM + SWAP) used to kill OOM? For example, if I increase my SWAP to 32Gb, will this stop happening?
  2. Can I force this process to use SWAP instead of Physical RAM, at the expense of slower performance?

Upvotes: 1

Views: 189

Answers (2)

t0mm13b
t0mm13b

Reputation: 34592

To answer the first question, yes.

Second question: Can I force this process to use SWAP instead of Physical RAM

linux dictates how the process is running, and allocate the memory appropriately for the process. When the threshold gets reached, linux will use the swap space as a measure.

Increasing swap space may work in this case. Then, again, I do not know how linux will cope with such a large swap, bear in mind, this could decrease performance dramatically.

Best alternative thing to do is split up the 45GB training set to smaller chunks.

Upvotes: 1

nwellnhof
nwellnhof

Reputation: 33658

Q: Is the total memory (RAM + SWAP) used to kill OOM?

Yes.

Q: For example, if I increase my SWAP to 32Gb, will this stop happening?

Yes, if RAM and swap space combined (48 GB) are enough for the process.

Q: Can I force this process to use SWAP instead of Physical RAM, at the expense of slower performance?

This will be managed automatically by the operating system. All you have to do is to increase swap space.

Upvotes: 1

Related Questions