CODEWITHSUNDEEP
javarestubuntutomcat
Eraldo Forgoli
Eraldo Forgoli

Reputation: 85

Tomcat too many open files error (Ubuntu 18.04)

I have deployed my Java Rest Application in tomcat (ubuntu 18.04). Everything seems fine until this error shows :

java.io.FileNotFoundException: /opt/tomcat/webapps/botiv1/WEB-INF/lib/javax.inject-2.5.0-b42.jar (Too many open files)
    java.base/java.io.RandomAccessFile.open0(Native Method)
    java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
    java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
    java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
    java.base/java.util.zip.ZipFile$Source.<init>(ZipFile.java:1263)
    java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1229)
    java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:727)
    java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:845)
    java.base/java.util.zip.ZipFile.<init>(ZipFile.java:245)
    java.base/java.util.zip.ZipFile.<init>(ZipFile.java:175)
    java.base/java.util.jar.JarFile.<init>(JarFile.java:341)
    java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
    java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
    org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:209)
    org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:149)
    org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:307)
    org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:97)
    org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256)
    org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:281)
    org.apache.catalina.webresources.Cache.getResource(Cache.java:62)
    org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:216)
    org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:225)
    org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2260)
    org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:834)
    org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1297)
    org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1157)
    org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:422)
    org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:285)
    org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
    org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
    org.glassfish.jersey.internal.Errors.process(Errors.java:316)
    org.glassfish.jersey.internal.Errors.process(Errors.java:298)
    org.glassfish.jersey.internal.Errors.process(Errors.java:268)
    org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
    org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
    org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
    org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
    org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
    org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
    org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
    org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    com.ritech.model.CORSFilter.doFilter(CORSFilter.java:59)

I have set the limits to 65000, but this error still shows up after a couple of hours.

/etc/security/limits.conf file

*soft nproc 65535

*hard nproc 65535

*soft nofile 65535

*hard nofile 65535 root soft nproc 65535

root hard nproc 65535

root soft nofile 65535

root hard nofile 65535

webuser hard nofile 64000

Note: This is not a database connections leak, because i have tracked the opened/active connecions, and there are not more than 4 active connections.

Upvotes: 0

Views: 3790

Answers (2)

Eraldo Forgoli
Eraldo Forgoli

Reputation: 85

The problem was with the Tomcat version. Version 9.0.13 would show "Too many open files" error, even when the total amount of opened files was around 5000 out of 100000 (limit). I updated the version to 9.0.14 and everything seems fine now.

Upvotes: 1

k500
k500

Reputation: 31

I assume tomcat service is running under tomcat user. check open files with command lsof -u tomcat | wc -l check maximum number of file handles for the entire system with command: cat /proc/sys/fs/file-max

check the current usage of file handles with command: cat /proc/sys/fs/file-nr

to make the change permanent, add or change the following line in the file /etc/sysctl.conf. This file is used during the boot process. echo "fs.file-max=65536" >> /etc/sysctl.conf (values are valid after reboot)

run command ulimit -a and check number of open files( -n) and max user processes (-u) increase the values if necessary.

e.g. : in /etc/security/limits.conf file 
tomcat              hard    nofile          102400
tomcat              soft    nofile          102400

run command: systemctl status tomcat get tomcat pidand

check cat /proc/'tomcat pid'/limits, if limits are still at default values add below lines to /etc/init.d/tomcat file.

ulimit -Hn 16384

ulimit -Sn 16384

after changing /etc/init.d/tomcat file restart tomcat service toapply changes.

Upvotes: 2

Related Questions