CODEWITHSUNDEEP
javahibernatebufferedreaderheap-memory
John Doe
John Doe

Reputation: 319

Read large file (Java Heap Space)

I want to read CSV file, create objects from every rows and then save these objects to a database. When i read all lines from my file, and store all objects inside ArrayList i get Java Heap Space Error. I tried to save every record immediately after reading, but then saving records by Hibernate method save() take a lot of time.

I also tried to check size of my arrayList and save data when this size equals 100k (commented part of code).

Question: Is any way to read file partly or better way to store data in Java?

String[] colNames;
String[] values;
String line;
Map<Object1, Object1> newObject1Objects = new HashMap<Object1, Object1>();
Map<Object1, Integer> objIdMap = objDao.createObjIdMap();

    StringBuilder raportBuilder = new StringBuilder();
    Long lineCounter = 1L;
        BufferedReader reader = new BufferedReader(new InputStreamReader(
                new FileInputStream(filename), "UTF-8"));
        colNames = reader.readLine().split(";");
        int columnLength = colNames.length;

        while ((line = reader.readLine()) != null) {
            lineCounter++;
            line = line.replace("\"", "").replace("=", "");
            values = line.split(";", columnLength);

            // Object1
            Object1 object1 = createObject1Object(values);
            if (objIdMap.containsKey(object1)) {
                object1.setObjId(objIdMap.get(object1));
            } else if (newObject1Objects.containsKey(object1)) {
                object1 = newObject1Objects.get(object1);
            } else {
                newObject1Objects.put(object1, object1);
            }

            // ==============================================
            // Object2
            Object2 object2 = createObject2Object(values, object1,
                    lineCounter, raportBuilder);
            listOfObject2.add(object2);
            /*
            logger.error("listOfObject2.size():"+listOfObject2.size());
            if(listOfObject2.size() % 100000 == 0){
                object2Dao.performImportOperation(listOfObject2);
                listOfObject2.clear();
            }
            */
        }
        object2Dao.performImportOperation(listOfObject2);

Upvotes: 0

Views: 574

Answers (1)

Nikita
Nikita

Reputation: 6427

Increase of max heap size won't help you if you want to process really large files. Your friend is batching.

Hibernate doesn’t implicitly employ JDBC batching and each INSERT and UPDATE statement is executed separately. Read "How do you enable batch inserts in hibernate?" to get information on how to enable it.

Pay attention to IDENTITY generators, as it disables batch fetching.

Upvotes: 1

Related Questions