CODEWITHSUNDEEP
fileubuntugoconsul
Maurício Linhares
Maurício Linhares

Reputation: 40333

Go program running out of file handles when there are plenty file handles available

I have a golang program that connects to other services (and a local consul agent) and when talking to the consul agent it's failing with the following error:

Put http://localhost:9501/v1/kv/health_checks/item: dial tcp: lookup localhost: too many open files

And this is all good, I can just check how many open files the system and the program has, right?

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7871
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 16384
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7871
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

And lsof:

lsof | wc -l
5343

This is still way below the limit. Other programs in the system are all good, I can CURL services, I can CURL this consul agent, create files and everything, but the golang program continues to fail.

Any ideas on how I could debug this? Where should I be looking at?

Upvotes: 0

Views: 506

Answers (2)

themihai
themihai

Reputation: 8631

Consider closing the idle connections if you are connecting to different endpoints/services(i.e. if you are running a crawler).

Upvotes: 0

Nick Craig-Wood
Nick Craig-Wood

Reputation: 54079

Any ideas on how I could debug this? Where should I be looking at?

Send the program a SIGQUIT while it is running. This will make a backtrace of all the running go routines. You'll probably find your culprit in there.

My money would be on forgetting to Close() the Body of an http.Response as that is what I do the most often!

Upvotes: 4

Related Questions