CODEWITHSUNDEEP
hibernatetomcatgrailsmemory-managementmemory-leaks
Mulone
Mulone

Reputation: 3663

OutOfMemoryError during long batch on Grails/Tomcat

I'm running a long batch on my Grails app. The service contacts a web service and download xml that is store locally in the grails app (in the db). The objects downloaded are quite big and complex, and the app processes them and create local domain objects. I'm using Grails 1.2.2 because I couldn't upgrade my app to the most recent version (I spent a few hours on it and then I threw in the sponge).

It's basically a long for loop with little shared information between the iterations.

It's something like:

while( stillObjectsToDo ){
   def bigObj = myservice.fetchXML
   def localInstance = myservice.processObj( bigObj )
   localInstance.saveEverythingToDB
   clearGORM
}

I run the app on Tomcat, which has been tweaked for increasing the heap size. When I process one single object I've never had an issue. But when I run the full batch (about 1500 big objects), I always get:

codehaus.groovy.grails.web.servlet.mvc.exceptions.ControllerExecutionException: Executing action [runSampleBatch] of controller [semanticopenstreetmap.EngineController]  caused exception: java.lang.OutOfMemoryError: Java heap space
 at java.lang.Thread.run(Thread.java:619)
Caused by: org.codehaus.groovy.runtime.InvokerInvocationException: java.lang.OutOfMemoryError: Java heap space
 ... 1 more
Caused by: java.lang.OutOfMemoryError: Java heap space
 at java.util.Arrays.copyOf(Arrays.java:2882)

It seems some sort of memory leak but I can't locate it. It's really strange because the objects are stored in the db and not used any more in the procedure, so they should just be flushed.

I tried the following solutions to avoid the issue with no luck:

None of this worked effectively. The memory usage keeps growing. I used Virtual VM and I got this graph: http://img12.imageshack.us/img12/5660/memoryleak.png And the classes: http://img263.imageshack.us/img263/331/memoryleakclasses.png

How can I locate and solve this memory leak?

EDIT: Would it make sense to run each object in a separate thread? The memory leak happens in a http-0-x process, how can I inspect this object?

Any hints?

Thanks!

Mulone

Upvotes: 1

Views: 1618

Answers (4)

BungleFeet
BungleFeet

Reputation: 1810

There's a known memory leak in GORM that can't be fixed by clearing the session. Burt Beckwith has written it up here. Basically, the solution is to call DomainClassGrailsPlugin.PROPERTY_INSTANCE_MAP.get().clear()

Upvotes: 1

hvgotcodes
hvgotcodes

Reputation: 120308

You need to attach an external profiler to your app. Here is a good one.

https://visualvm.dev.java.net/

The basic idea is you add some options to your tomcat startup scripts, fire up tomcat and your profiler, and use your app. You will see realtime updates and be able to watch your memory being consumed, freed, and whatnot.

what you should see is a graph that rises, then falls, then rises, then falls, and so on. The key point is that while memory usage will rise periodically, there is no upward trend of the baseline over time. So when the graph falls, over time it falls back to some baseline. If, after the graph falls, you see an upward trend, you need to identify which area of the app is causing it. VisualVM has ways of analyzing the heap so you can see how many objects of each type are present. You should use that tool to figure out which objects are not being collected.

Upvotes: 1

Adam Vandenberg
Adam Vandenberg

Reputation: 20671

This is a potential area of exploration: Troubleshooting Grails/Groovy memory leaks?

Upvotes: 1

duffymo
duffymo

Reputation: 308998

I'd use Visual VM to attach to the process and see what's going on with memory.

I'm wondering if dynamically generated proxies are filling up your perm gen space. Visual VM will give you the evidence you need to see if this is true.

Upvotes: 1

Related Questions