CODEWITHSUNDEEP
javaspringcrashout-of-memory
mirzak
mirzak

Reputation: 1057

Is there a guarantee that Java OutOfMemoryError will be thrown

I have a Java Spring Boot app. Its really big application with many services and it can perform a bunch of tasks. One of the new tasks I am trying to implement is to read some data from Oracle DB and send it via rest to some external app.

Data that is being read is quite big (hard to tell how big, it contains geometry objects), and there are around 1.8 million records to be read.

In order to handle that I am using "keyset pagination" as a way of reading from DB. That means that I get the last read id and then get the next page based on that (e_id > lastReadId). The page size is 100 entities.

On the last run, it got up to page '6672' (667200 entities have been read and sent to an external app). It's worth noting that I do not store any references to those objects, simply fetch a page, store it a list and send it via rest. On the next run, that list is overridden with a new page and so on.

Here is a graph of a number of fetched entities per 3 hours, max being 1030 and min 145 entities.

Fetched entities per 3h

My problem is that the app crashes without any errors. In the logs, I can see that last page has been fetched (in this case it was '6672', sometimes its some other page) and then suddenly there is a log message that is logged when my app is starting.

My first thought was that it runs out of memory and simply crashed. But there is no indication for that. Is it guaranteed that OutOfMemoryError will be thrown at such point? Should I be looking at something else? Maybe I am doing something wrong.

EDIT

I am adding some code for you to see how I perform these actions

// Get first page, last read id is null
List<MyEntity> data = dataService.collectData(pageSize, null);

sendDataToExternalService(data);
while(true) {
    final String lastReadID = data.get(data.size() - 1).getId();
    data = dataService.collectData(pageSize, lastReadID);
    sendDataToExternalService(data);
}

Method sendDataToExternalService looks like this

restTemplate.exchange("some/url/to-external-app", HttpMethod.PUT, new HttpEntity<>(data), List.class);

RestTemplate is org.springframework.web.client.RestTemplate

Upvotes: 0

Views: 300

Answers (2)

mirzak
mirzak

Reputation: 1057

After some profiling using JProfiler I was able to find out that Hibernate would cache everything that was selected since everything was done in single transaction. Adding EntityManager.clear() in a existing while loop solved the problem.

Also it's worth noting that entire process was speeded up significantly.

Upvotes: 0

Sugan
Sugan

Reputation: 455

You can configure the JVM to generate a heap dump when it receives this error.

To configure the JVM to generate a heap dump, add the -XX:+HeapDumpOnOutOfMemoryError option to the Java options and restart the JVM. When the heap space error occurs, the JVM creates a file the size of the configured maximum heap size.

Check this for details https://docs.bmc.com/docs/AtriumOrchestratorPlatform/77/troubleshooting-java-virtual-machine-memory-errors-329147248.html

Upvotes: 1

Related Questions