CODEWITHSUNDEEP
elixir
William Ross
William Ross

Reputation: 3920

Elixir recovering erl_crash.dump from a k8s pod with memory issues

I have Elixir code running on Kubernetes pods which crashes from what I suspect is a memory issue. Once Elixir crashes the pod restarts and doesn't have the erl_crash.dump anywhere under the /opt/ directory, I suspect because the pod is ephemeral.

The log message I get is,

Crash dump is being written to: erl_crash.dump...done

Is there some way to write this file to an s3 bucket or see what it's contents are to get more granular info on why the crash is occurring?

Upvotes: 0

Views: 660

Answers (2)

Roger Lipscombe
Roger Lipscombe

Reputation: 91925

One option I would consider is mounting some node-local storage into the pod and setting $ERL_CRASH_DUMP to point to it. You can do that in the rel/env.sh.eex file.

Then use a DaemonSet pod to watch the same volume for these files and push them to S3.

Upvotes: 2

zraul
zraul

Reputation: 11

You could attach a persistent volume to the pods - this way you don't lose the file at reboot and you can retrieve it afterwards.

Upvotes: 1

Related Questions