Tomcat too many files open (tomcat-users.xml)

Question

Earlier today I visited a site that I recently launched and the page was just filled with a big ugly jasperexception: java.io.filenotfoundexception: /opt/tomcat/webapps/root/web-inf/lib/jstl-1.2.jar (too many open files). I checked my server logs and tomcat logs and they were full of the same error. I increased tomcat's max open file limit as a temporary solution to keep the site running, but it's quickly filling up again. I've launched other sites with the same methods I used for this one, and never had this problem before. The only thing I did different this time was use Mobirise to make the base layout for the front-end, so I'm not ruling out that some scripts used by Mobirise could be the culprit. But when I get the list of files that tomcat has open, I see that it's got hundreds of copies of tomcat-users.xml open, and it just keeps opening more and more.

...
java    319 tomcat  935r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  936r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  937r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  938r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  939r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  940r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  941r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  942r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  943r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  944r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  945r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  946r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  947r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  948r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  949r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  950r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  951r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  952r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  953r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  954r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  955r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  956r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  957r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  958r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  959r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  960r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  961r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  962r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  963r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  964r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
...

I don't understand why it's doing this. I've been debugging, scanning through my code, and scouring the internet for issues where tomcat keeps opening too many files, but they all just say "increase the max file limit." None of them address the root of the problem or mention anything about tomcat just infinitely opening copies of a single file.

[EDIT]: Here is the full stack trace of the FileNotFoundException:

28-Nov-2018 03:51:33.857 SEVERE [http-nio-8080-exec-5] org.apache.catalina.core.ApplicationDispatcher.invoke Servlet.service() for servlet [jsp] threw exception
 java.io.FileNotFoundException: /opt/tomcat/webapps/ROOT/WEB-INF/lib/jstl-1.2.jar (Too many open files)
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:225)
        at java.util.zip.ZipFile.<init>(ZipFile.java:155)
        at java.util.jar.JarFile.<init>(JarFile.java:166)
        at java.util.jar.JarFile.<init>(JarFile.java:130)
        at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:164)
        at org.apache.tomcat.util.scan.JarFileUrlJar.<init>(JarFileUrlJar.java:65)
        at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
        at org.apache.tomcat.util.descriptor.tld.TldResourcePath.openJar(TldResourcePath.java:134)
        at org.apache.jasper.compiler.TagLibraryInfoImpl.<init>(TagLibraryInfoImpl.java:125)
        at org.apache.jasper.compiler.Parser.parseTaglibDirective(Parser.java:431)
        at org.apache.jasper.compiler.Parser.parseDirective(Parser.java:489)
        at org.apache.jasper.compiler.Parser.parseElements(Parser.java:1445)
        at org.apache.jasper.compiler.Parser.parse(Parser.java:144)
        at org.apache.jasper.compiler.ParserController.doParse(ParserController.java:244)
        at org.apache.jasper.compiler.ParserController.parse(ParserController.java:105)
        at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:206)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:385)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:362)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:346)
        at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:603)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:400)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:386)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:330)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:459)
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)
        at kom.kitten.gingerpage.servlets.HomeServlet.doGet(HomeServlet.java:16)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:791)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

28-Nov-2018 03:51:33.859 SEVERE [http-nio-8080-exec-5] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [kom.kitten.gingerpage.servlets.HomeServlet] in conte$
 java.io.FileNotFoundException: /opt/tomcat/webapps/ROOT/WEB-INF/lib/jstl-1.2.jar (Too many open files)
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:225)
        at java.util.zip.ZipFile.<init>(ZipFile.java:155)
        at java.util.jar.JarFile.<init>(JarFile.java:166)
        at java.util.jar.JarFile.<init>(JarFile.java:130)
        at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:164)
        at org.apache.tomcat.util.scan.JarFileUrlJar.<init>(JarFileUrlJar.java:65)
        at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
        at org.apache.tomcat.util.descriptor.tld.TldResourcePath.openJar(TldResourcePath.java:134)
        at org.apache.jasper.compiler.TagLibraryInfoImpl.<init>(TagLibraryInfoImpl.java:125)
        at org.apache.jasper.compiler.Parser.parseTaglibDirective(Parser.java:431)
        at org.apache.jasper.compiler.Parser.parseDirective(Parser.java:489)
        at org.apache.jasper.compiler.Parser.parseElements(Parser.java:1445)
        at org.apache.jasper.compiler.Parser.parse(Parser.java:144)
        at org.apache.jasper.compiler.ParserController.doParse(ParserController.java:244)
        at org.apache.jasper.compiler.ParserController.parse(ParserController.java:105)
        at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:206)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:385)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:362)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:346)
        at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:603)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:400)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:386)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:330)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:459)
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)
        at kom.kitten.gingerpage.servlets.HomeServlet.doGet(HomeServlet.java:16)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:791)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

This is the same stack trace that it logs every time the error occurs (i.e. every time someone attempts to load a page on the site while there are too many files open in tomcat)

Answer 1

(Posting answer on behalf of the question author, in order to move it from the question edit history into the answer space).

I now believe this issue is being caused by something in the Mobirise assets. I checked the servers of my other websites (ones built without Mobirise), and none of them are having this issue. I checked all of my sites built with Mobirise, and all of them are having this issue. I don't have time right now to pinpoint exactly what it is in Mobirise that's causing this.

I found a hacky workaround: simply deleting tomcat-users.xml and restarting Tomcat fixed the issue. This is not a proper solution, but it's a workaround that works for me because I manage my Tomcat servers through SSH and SFTP rather than using the default Tomcat Manager interface, which I know is probably bad practice because the web app can't be updated without server downtime, but I've had buggy issues with Tomcat Manager in the past.

Answer 2

If you have applications deploy on Tomcat. First, You must make sure that when you open a file you close it explicitly. You can use the following command in order to see the open files in linux:

lsof -p <process Id of Tomcat>

Answer 3

This was caused by a bug in Tomcat 9.0.13 which has been fixed in Tomcat 9.0.14.

Answer 4

I just ran into the same issue after moving an application from a tomcat7 instance to a tomcat 9 instance. I have a feeling this may be a tomcat bug. I removed the tag

<Resource pathname="conf/tomcat-users.xml" 
factory="org.apache.catalina.users.MemoryUserDatabaseFactory" 
type="org.apache.catalina.UserDatabase" 
auth="Container" name="UserDatabase"/>

in the GlobalNamingResources tag in $CATALINA_HOME/conf/server.xml and after a restart the issue resolved itself. So I'm guessing the issue lies with Tomcat's management app.