chubakur
chubakur

Reputation: 51

Why does this function cause a memory leak?

public void filterLogins() throws IOException, SQLException{
    for(int i = 0; i<200; ++i){
        BufferedReader bufferedReader = new BufferedReader(new FileReader(folder + String.format("\\data\\part-%05d", i)));
        long prev_id = 0;
        boolean contains = false;
        while(bufferedReader.ready()){ //very big file
            String line = bufferedReader.readLine();
            Login login = new Login(line);
            if ( login.userId == prev_id && !contains )
                continue;
            if ( samples.contains(login.userId) ){
                mysql.execute("INSERT INTO ..."); // i think it doesn't matter in this case
                contains = true;
            }else{
                contains = false;
            }
            prev_id = login.userId;
        }
        bufferedReader.close();
        System.out.println((double)i/2.0);
    }
}

This function works long time, because data is more big files. 2 hours ago this is crushess with OutOfMemory Exception

mysql is instance of

public class MySQLHandler {
private Connection connection = null;
MySQLHandler() throws ClassNotFoundException, SQLException{
    try{
        Class.forName("com.mysql.jdbc.Driver");
        connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/database", "root", "");
    }catch (ClassNotFoundException e){
        System.out.println("Failed jdbc driver load.");
        throw e;
    }
}


public void close() throws SQLException{
       if ( connection != null)
           connection.close();
   }
   public boolean execute(String sql) throws SQLException{
       Statement statement = connection.createStatement();
       return statement.execute(sql);
   }
}

Login it is just class with data. (id, name, value, something else).

Upvotes: 0

Views: 337

Answers (2)

Mani
Mani

Reputation: 3364

Seems, you are creating Statement each time and not closing the statement in your loop which will leak memory . close the statement once execute is completed.

 public boolean execute(String sql) throws SQLException{
       Statement statement = connection.createStatement();
       return statement.execute(sql);
   }

Like

 public boolean execute(String sql) throws SQLException{
     Statement statement = null;
    try {
        statement = connection.createStatement();
        return statement.execute(sql);
    }finaly{
         if (statement != null) statement.close();
     }

UPDATE

As @Holger mentioned in comment, if you are using JDK 7 and higher then you can use try-with-resources like below

public boolean execute(String sql) throws SQLException{
       try(Statement s=connection.createStatement()) {
          return s.execute(sql);
       }
 }

Upvotes: 5

Bobz
Bobz

Reputation: 2604

Try increasing the max heap size of your JVM eg:

-Xms<size>        set initial Java heap size
-Xmx<size>        set maximum Java heap size
-Xss<size>        set java thread stack size

java -Xms16m -Xmx64m ClassName

Also optimzie your code and reuse variables and close/cleanup after use. For eg: Move these outside the loop: BufferedReader bufferedReader, String line and Login login

public void filterLogins() throws IOException, SQLException{
    BufferedReader bufferedReader;
    long prev_id;
    boolean contains;
    String line;
    Login login;
    for(int i = 0; i<200; ++i){
        bufferedReader = new BufferedReader(new FileReader(folder + String.format("\\data\\part-%05d", i)));
        prev_id = 0;
        contains = false;
        while(bufferedReader.ready()){ //very big file
            line = bufferedReader.readLine();
            login = new Login(line);
            if ( login.userId == prev_id && !contains )
                continue;
            if ( samples.contains(login.userId) ){
                mysql.execute("INSERT INTO ..."); // i think it doesn't matter in this case
                contains = true;
            }else{
                contains = false;
            }
            prev_id = login.userId;
        }
        bufferedReader.close();
        System.out.println((double)i/2.0);
    }
}

Upvotes: 0

Related Questions