Reduce memory usage in java

Question

I have a web service Java program which reads 13,000,000 dates like '08-23-2016 12:54:44' as strings from database. My developing environment is Java 8, MySQL 5.7 and tomcat 8. I have declare a string array String[] data to store it. I use Guice to inject the initial values of data array to empty. However, the memory usage is still huge. This is my code:

String[] data;//size is 1,000,000
void generateDataWrapper(String params) {
        //read over 13000000 dates string
        ResultSet rs = mySQLCon.readData(params);  
        clearData(data);//set to empty string 
        int index = 0;
        while(rs.next()) {
             data[index++] = rs.getString("date");
             if (index == (size - 1)) {//calculate every 1,000,000 total 13 times 
                 //calculate statistics
                 ... 
                 //reset all to empty string
                 clearData(data);    
                 index = 0;
             } 
        }
}
//mySQLCon. readData function
ResultSet readData(String params) {
        try {
             String query = generateQuery(params);
             Statement postStmt = connection.createStatement();
             ResultSet rs = postStmt.executeQuery(query);
        return rs;
        } catch (Exception e) {
        }
        return null;
}

If I call this function once, the memory is reached 12G, If I call it again, the memory goes to 20G, on the third time the memory will goes to 25G and throw a 'java.lang.OutOfMemoryError: GC overhead limit exceeded' error in com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2174)

This is part of the error message:

java.lang.OutOfMemoryError: GC overhead limit exceeded
    com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2174)
    com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1964)
    com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3316)
    com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:463)
    com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3040)
    com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2288)
    com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2681)
    com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2547)
    com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2505)
    com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1370)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    java.lang.reflect.Method.invoke(Unknown Source)

I have changed the garbage collection algorithms to: -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode But it's not helping. I have tried change the data to static variables, still will have this problem. Currently the JVM heap is 8g, the tomcat memory is 24g, however, I don't think increase the memory will solve the problem.

I don't understand why my memory is still increasing every time I call this function, Could someone give me some suggestion?

Answer 1

1. Used resources like a ResultSet have to be closed to release the underlying system-resources. This can be done automatically declaring the resources in a try-block like try (ResultSet resultSet =...).

2. You can try to fetch only a limited number of rows from database when they are requested from ResultSet and not all of them immediately.

3. Objects get eligible for garbage collection when they are not referenced any more. So, your array-object keeps in memory with it's whole size as long as it is referenced. If it's not referenced any more and the VM is running out of memory it will be able to dispose the array-object possibly avoiding an OutOfMemoryError.

4. Unexpectedly high memory usage can be analyzed by creating a heap dump and exploring it in the tool jvisualvm of the JDK.

Answer 2

First, thanks for all your suggestions. I have figured this out by reading from mm759 and realized that I forgot to close the ResultSet after I have done reading. After I add rs.close(), every time it takes the same time to finish, although the memory will reach the maximum memory I set.

Answer 3

Additionally you can change your string array to an long array since strings consume a huge amount of memory. In your case the size of a date string is 38 bytes ( 19 char * 2 bytes ) whereas a long only takes 8 bytes of memory.

long[] data;//size is 1,000,000
void generateDataWrapper(String params) {
    //read over 13000000 dates string
    ResultSet rs = mySQLCon.readData(params);
    clearData(data);//set to empty string
    int index = 0;
    SimpleDateFormat formater =  new SimpleDateFormat("MM-dd-YYYY HH:mm:ss");
    while(rs.next()) {
        try{
            Date date = formater.parse(rs.getString("date"));
            data[index++] = date.getTime();
        }catch(ParseException pe) {
            pe.printStackTrace();
        }
        if (index == (size - 1)) {//calculate every 1,000,000 total 13 times
            //calculate statistics
            ...
            //reset all to empty string
            clearData(data);
            index = 0;
        }
    }
}

Wherever you need your string you can just parse it back with the following

SimpleDateFormat formater =  new SimpleDateFormat("MM-dd-YYYY HH:mm:ss");
Date date = new Date(data[i]);
String dateString = formater.format(date);