CODEWITHSUNDEEP
javajdbc
Vikas J
Vikas J

Reputation: 887

Java how to retrieve more than 1 million rows in a Resultset

I am executing select query on a table of MYSQL which has 16,213,156 rows and 10 columns. But after connection is established code just executes for few mins and then throws error : Exception in thread "Thread-3" java.lang.OutOfMemoryError: Java heap space

My system configuration is 16 gb RAM, Java 8

I tried setting Jvm Parameters as -Xms4G & -Xmx12G . Also tried setting stmt.setFetchSize(); // to 10,100,1000 still same error

Can we fetch such large number of records using JDBC API am I missing anything ? Any help would be really appreciated.

package com;
import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Date;

import com.opencsv.CSVWriter;

public class Test1 {
    private static Connection conn = null;

    public static void main(String[] args) throws ClassNotFoundException, SQLException, IOException {

        connection();
        retrieve(conn);

    }



    public static void connection()
    {

        try
        {
            Class.forName("com.mysql.jdbc.Driver");
            String url = "<jdbc connection url>";
            conn = DriverManager.getConnection(url, "<username>", "<password>");
            System.out.println("Connection established");       
        }

        catch(Exception e)
        {
            e.printStackTrace();
        }

    }


    public static void retrieve(Connection conn)
    {
        ResultSet rs = null;
        Statement stmt = null;
        try
        {

            stmt = conn.createStatement();
            //  stmt.setFetchSize(100); // 1000, 10
            System.out.println(stmt.getFetchSize());  // By default prints 0
            rs = stmt.executeQuery("SELECT * FROM tablename");
            CSVWriter writer = new CSVWriter(new BufferedWriter(new FileWriter("C:\\finaldata\\test.csv")));
            System.out.println("**** Started writing Data to CSV ****");
            int lines = writer.writeAll(rs, true, false, false);        
            writer.flush();
            writer.close();
            System.out.println("** OpenCSV -Completed writing the resultSet at " +  new Date() + " Number of lines written to the file " + lines);  

        }
        catch(Exception e)
        {
            e.printStackTrace();
        }
    }
}

Upvotes: 7

Views: 20961

Answers (4)

Vikas J
Vikas J

Reputation: 887

@MickMnemonic Thanks for your help this solved the issue.

Setting fetchSize alone might not be enough for the MySQL driver to start streaming the data from the DB instead of loading everything at once. You could try with

stmt = conn.createStatement(ResultSet.TYPE_FORWARD_ONLY, 
ResultSet.CONCUR_READ_ONLY); stmt.setFetchSize(Integer.MIN_VALUE);

Upvotes: 7

Hasitha Nanayakkara
Hasitha Nanayakkara

Reputation: 149

Limit the query, like,

SELECT * FROM message_history limit {start from row no.} , {no. of rows to select}

example- 

SELECT * FROM message_history limit 100000,200000;

will retrieve rows from 100000 to 300000; like this divide into batches.also

PreparedStatement statement = con.prepareStatement(query);
statement.setFetchSize(Integer.MIN_VALUE);
rs = statement.executeQuery();

This method worked for me to retrieve 22 million records.

Upvotes: 0

Fredy Fischer
Fredy Fischer

Reputation: 458

I've run into a similar problem reading some million rows from a MySQL DB. I'm reading with a PreparedStatement called reader. Then just after the PrepareStatement I've reduced the fetchsize to minimum:

PreparedStatement reader = connection.prepareStatement("select....");
reader.setFetchSize(Integer.MIN_VALUE);

and from then on, I've never run into problems.

Upvotes: -1

Gaurav Srivastav
Gaurav Srivastav

Reputation: 2561

If you want to some searching or manipulation try to do at database Layer rather to fetch all of them in Application Layer and then do manipulation/searching. Getting the large set of records is not good practice. Provide the search filter option there so that user can filter the records as per their need, as records grow this really tedious task for you to manage all these.

Upvotes: -1

Related Questions