java.net.SocketException: java.net.SocketException: Too many open files

Question

I have a large java application (which is used to generate some type of report) in which below class is used to create datasource.

 import org.apache.log4j.Logger;
 import org.springframework.jdbc.datasource.DriverManagerDataSource;
 import com.mysql.jdbc.Connection;
 public class DatabaseConnection
  {
     private static final Logger LOGGER = Logger.getLogger(DatabaseConnection.class.getName());
    @SuppressWarnings("deprecation")
    public static DriverManagerDataSource jdbcConnection(WebmartConfiguration webmartconnection)
    {
        DriverManagerDataSource dataSource = null;
        try
        {
            dataSource = new DriverManagerDataSource("com.mysql.jdbc.Driver", "jdbc:mysql://" + webmartconnection.getHostname() + ":" + webmartconnection.getPort() + "/" + webmartconnection.getDatabasename() + "", webmartconnection.getUsername(), webmartconnection.getPassword());
    }
    catch (Exception sqle)
    {
        LOGGER.info(sqle);
    }
    return dataSource;
 }
}

and that datasource is passed to many method that are used to execute query using JDBCTEMPLATE's query method. For some time application runs smooth and generate reports but after some time application terminated with following stacktrace.

ERROR [run has started] (DivisionThread.java:217) - Could not get JDBC Connection;      nested exception is com.mysql.jdbc.CommunicationsException:     Communications link failure due to underlying exception:    

** BEGIN NESTED EXCEPTION **    

java.net.SocketException    
MESSAGE: java.net.SocketException: Too many open files  

STACKTRACE: 

java.net.SocketException: java.net.SocketException: Too many open files 
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:156)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:276)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2641)
at com.mysql.jdbc.Connection.<init>(Connection.java:1531)
   at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
   at java.sql.DriverManager.getConnection(DriverManager.java:579)
   at java.sql.DriverManager.getConnection(DriverManager.java:190)
   at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:173)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:164)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:149)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:119)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:674)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:729)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:745)
at com.mpsinsight.reporting.dao.common.CommonUtilReport.getOutFileName(CommonUtilReport.java:1509)
at com.mpsinsight.reporting.bean.common.ActionEnum$9.getparameterType(ActionEnum.java:507)
at com.mpsinsight.reporting.main.common.JasperReport.populateParameters(JasperReport.java:142)
at com.mpsinsight.reporting.main.common.JasperReport.generateReport(JasperReport.java:61)
at com.mpsinsight.reporting.main.common.GenerateReport$4.generateReport(GenerateReport.java:123)
at com.mpsinsight.reporting.main.common.DivisionThread.run(DivisionThread.java:179)


** END NESTED EXCEPTION **  

Example : Below is the one of the method of application that uses datasource.

public String getOutFileName(DriverManagerDataSource datasource, WebmartConfiguration     webmartconnection, String abbrev)
    {
    DriverManagerDataSource dmDatasource = null;
    dmDatasource = datasource;
    if (dmDatasource == null)
    {
        dmDatasource = DatabaseConnection.jdbcConnection(webmartconnection);
    }
    JdbcTemplate jdbcTemplateOb = new JdbcTemplate(dmDatasource);
    String sql = QueryList.Value("outFileNameQuery");

    LOGGER.info("abbrev :::: " + abbrev);
    LOGGER.info("outFileNameQuery :::: " + sql);

    String pathname = (String) jdbcTemplateOb.queryForObject(sql, new Object[] { abbrev }, String.class);

    LOGGER.info("pathname :::: " + pathname);
    return pathname;
}

Could someone explain what could the reason for too many open files error and how can i resolve it.

EDIT : I am working on linux environment. The above error occur only in linux environment.

Answer 1

The reason for a "too many open files" is that the application or system is reaching the limit of allowed file handles.

This problem usually occurs after several concurrent users get a connection to the Server. Java opens many files in order to read in the classes required to run your application. High volume applications can use a lot of file descriptors. This could lead to a lack of new file descriptors. Also, each new socket requires a descriptor. Clients and Servers communicate via TCP sockets. Each browser's http request consumes TCP sockets when a connection is established to a Server.

In order to prevent a file descriptor leak you should ensure that all resources like streams, database connections are explicitly closed as soon as they are no longer needed. Even Java manages resources for you: don't rely on the garbage collector to clean up your used resources. i.e. close all streams either within a finally block or use the Java 7 try with resources syntax.

The reason for the limit is that the operating system needs memory to manage manage each open file, and memory is a limited resource. Different platforms have different limits on the number of files that can be open in a single process at one time.

On Linux you can as root user change the maximum of the open files count per process (via ulimit -n) and per system (e.g. echo 800000 > /proc/sys/fs/file-max) or per user using the /etc/security/limits.conf file. For the last option you may need a special user running your application.

Before you change one or more of these values you should check whether your application has a resource leak by monitoring the open files. One possible option on Linux is to use strace.

strace -e trace=open,close,read,write,connect,accept your-command-here

You'll need to use the -o option to put strace's output somewhere other than the console, if the process can print to stderr. If your process forks, you'll also need -f or -ff

Or if you want to connect you to a running process and you know the process id you can use strace -p $MyProcess.

If you search the web you'll find a solution for monitoring the open file descriptors that matches your needs. For example monitor open process files on linux (real-time)

Answer 2

Check open file limit using below linux command.

ulimit -a 

and you can provide open file limit in below file through root user.

/etc/security/limits.conf 

For example

{username}       soft       nofile   1024
{username}       hard   nofile        65536