CODEWITHSUNDEEP
dockerout-of-memory
user3675188
user3675188

Reputation: 7409

Any hook for Docker killed by out of memory

I'm running a docker for a daemon job. And the container will be killed every several hours. I'd like to add some hook (callback), such as:

restart the container and then run some commands on the restarted container

Is it possible to do that with Docker?

Otherwise, is there any better approach to detect the behaviour with Python or Ruby?

java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
java cpuset=bcb33ac552c23cfa531814fbc3a64ae5cd8d85aa19245e1560e0ce3e3310c798 mems_allowed=0
CPU: 3 PID: 14182 Comm: java Not tainted 4.1.0-x86_64-linode59 #1
 0000000000000000 ffff8800dc520800 ffffffff8195b396 ffff880002cf5ac0
 ffffffff81955e58 ffff8800a2918c38 ffff8800f43c3e78 0000000000000000
 ffff8800b5f687f0 000000000000000d ffffea0002d7da30 ffff88005bebdec0
Call Trace:
 [<ffffffff8195b396>] ? dump_stack+0x40/0x50
 [<ffffffff81955e58>] ? dump_header+0x7b/0x1fe
 [<ffffffff8119655d>] ? __do_fault+0x3f/0x79
 [<ffffffff811789d6>] ? find_lock_task_mm+0x2c/0x7b
 [<ffffffff81961c55>] ? _raw_spin_unlock_irqrestore+0x2d/0x3e
 [<ffffffff81178dee>] ? oom_kill_process+0xc5/0x387
 [<ffffffff811789d6>] ? find_lock_task_mm+0x2c/0x7b
 [<ffffffff811b76be>] ? mem_cgroup_oom_synchronize+0x3ad/0x4c7
 [<ffffffff811b6c92>] ? mem_cgroup_is_descendant+0x29/0x29
 [<ffffffff811796e7>] ? pagefault_out_of_memory+0x1c/0xc1
 [<ffffffff81963e58>] ? page_fault+0x28/0x30
Task in /docker/bcb33ac552c23cfa531814fbc3a64ae5cd8d85aa19245e1560e0ce3e3310c798 killed as a result of limit of /docker/bcb33ac552c23cfa531814fbc3a64ae5cd8d85aa19245e1560e0ce3e3310c798
memory: usage 524288kB, limit 524288kB, failcnt 14716553
memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
Memory cgroup stats for /docker/bcb33ac552c23cfa531814fbc3a64ae5cd8d85aa19245e1560e0ce3e3310c798: cache:72KB rss:524216KB rss_huge:0KB mapped_file:64KB writeback:0KB inactive_anon:262236KB active_anon:262044KB inactive_file:4KB active_file:4KB unevictable:0KB
[ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[14097]  1000 14097     5215       20      17       3       47             0 entry_point.sh
[14146]     0 14146    11960        0      30       3      101             0 sudo
[14150]  1000 14150     1112        7       8       3       22             0 xvfb-run
[14162]  1000 14162    51929    11220      90       3       95             0 Xvfb
[14169]  1000 14169   658641    18749     120       6        0             0 java
[14184]  1000 14184    28364      555      58       3        0             0 fluxbox
[24639]  1000 24639     5212       59      16       3        0             0 bash
Memory cgroup out of memory: Kill process 14169 (java) score 96 or sacrifice child
Killed process 14169 (java) total-vm:2634564kB, anon-rss:74996kB, file-rss:0kB

Upvotes: 1

Views: 1301

Answers (1)

Nathaniel Waisbrot
Nathaniel Waisbrot

Reputation: 24483

Docker itself doesn't have any such mechanism. All you can do is pass the --restart flag to tell Docker when it should try to bring a failed container back.

However, most places where you want to keep a container up you'll want something more complex than the --restart flag anyway. Once you're using runit or systemd to manage your containers it's easy to add in a little extra shell code to figure out why the last invocation crashed and take some special actions based on that.

Upvotes: 2

Related Questions