Any Java debugging tips for finding the cause of "Too Many Files Open"

Question

I'm developing in a linux environment and the system is intended to run continuously over a long period of time. After an overnight test we see the FileNotFoundException with a message of "Too Many Files Open". We started logging the output of the lsof command at various times in the system to see if we can see what is happening. We noticed lots of unnamed pipes opened. So I figured these were due to File Streams not getting closed. I searched through the source for any *Stream objects used and made sure they were all getting closed in a finally{} block. Are there any other Java object types that I could search for that I might not be closing that would cause all these unnamed pipes to be opened?

Also, my ulimit is 1024 and I also searched for *Writer and made sure those were all closing too.

Answer 1

Other classes that might leak file descriptors are FileChannel and RandomAccessFile - the latter doesn't even seem to have a finalizer, so its leaks might be permanent.

Answer 2

I'm assuming your ulimit is the output of ulimit -n. 1024 is a fairly small number of file descriptors to allow for in a production system. For a debugging step, rather than running lsof at random times and trying to correlate, why not catch the FileNotFound exception and run a Runtime.exec("lsof") and print the output to a log file to get a fairly accurate view of exactly what file descriptors were used when the problem occurred.

Answer 3

YourKit might be worth a look. Its probes are meant to help with this type of problems, although I've never had the occasion to try that functionality myself.