Reputation: 51
public void filterLogins() throws IOException, SQLException{
for(int i = 0; i<200; ++i){
BufferedReader bufferedReader = new BufferedReader(new FileReader(folder + String.format("\\data\\part-%05d", i)));
long prev_id = 0;
boolean contains = false;
while(bufferedReader.ready()){ //very big file
String line = bufferedReader.readLine();
Login login = new Login(line);
if ( login.userId == prev_id && !contains )
continue;
if ( samples.contains(login.userId) ){
mysql.execute("INSERT INTO ..."); // i think it doesn't matter in this case
contains = true;
}else{
contains = false;
}
prev_id = login.userId;
}
bufferedReader.close();
System.out.println((double)i/2.0);
}
}
This function works long time, because data is more big files. 2 hours ago this is crushess with OutOfMemory Exception
mysql is instance of
public class MySQLHandler {
private Connection connection = null;
MySQLHandler() throws ClassNotFoundException, SQLException{
try{
Class.forName("com.mysql.jdbc.Driver");
connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/database", "root", "");
}catch (ClassNotFoundException e){
System.out.println("Failed jdbc driver load.");
throw e;
}
}
public void close() throws SQLException{
if ( connection != null)
connection.close();
}
public boolean execute(String sql) throws SQLException{
Statement statement = connection.createStatement();
return statement.execute(sql);
}
}
Login it is just class with data. (id, name, value, something else).
Upvotes: 0
Views: 337
Reputation: 3364
Seems, you are creating Statement each time and not closing the statement in your loop which will leak memory . close the statement once execute is completed.
public boolean execute(String sql) throws SQLException{
Statement statement = connection.createStatement();
return statement.execute(sql);
}
Like
public boolean execute(String sql) throws SQLException{
Statement statement = null;
try {
statement = connection.createStatement();
return statement.execute(sql);
}finaly{
if (statement != null) statement.close();
}
UPDATE
As @Holger mentioned in comment, if you are using JDK 7 and higher then you can use try-with-resources like below
public boolean execute(String sql) throws SQLException{
try(Statement s=connection.createStatement()) {
return s.execute(sql);
}
}
Upvotes: 5
Reputation: 2604
Try increasing the max heap size of your JVM eg:
-Xms<size> set initial Java heap size
-Xmx<size> set maximum Java heap size
-Xss<size> set java thread stack size
java -Xms16m -Xmx64m ClassName
Also optimzie your code and reuse variables and close/cleanup after use. For eg: Move these outside the loop: BufferedReader bufferedReader, String line and Login login
public void filterLogins() throws IOException, SQLException{
BufferedReader bufferedReader;
long prev_id;
boolean contains;
String line;
Login login;
for(int i = 0; i<200; ++i){
bufferedReader = new BufferedReader(new FileReader(folder + String.format("\\data\\part-%05d", i)));
prev_id = 0;
contains = false;
while(bufferedReader.ready()){ //very big file
line = bufferedReader.readLine();
login = new Login(line);
if ( login.userId == prev_id && !contains )
continue;
if ( samples.contains(login.userId) ){
mysql.execute("INSERT INTO ..."); // i think it doesn't matter in this case
contains = true;
}else{
contains = false;
}
prev_id = login.userId;
}
bufferedReader.close();
System.out.println((double)i/2.0);
}
}
Upvotes: 0