CODEWITHSUNDEEP
javamultithreadinggarbage-collection
Simon Nicholls
Simon Nicholls

Reputation: 93

JAVA Multi-Threading, Memory Leak, Garbage Collector

I'm a bit of a novice when it comes to JAVA applications, but have been involved in developing a fairly complex JAVA(8) app that requires multi-threading. Myself and another developer have kept running into a problem where the app keeps running out of memory after running for a while.

At first we gave the application 64GB of memory, but after a few hours it'd run out of memory, crash and restart. Only to keep doing it over and over. Context; The application takes messages from a messaging system (ActiveMQ) and from the message's meta has to build an XML file by calling various data sources for values. There could be literally millions of messages that need to be processed, so we developed a multi-threading system, each thread deal with a message - and gave the application 40 threads.

However, as it keeps taking messages the overall memory consumption goes up and up over time. I feel like the garbage collector isn't being utilized by us correctly?

So at the moment we have one parent thread:

(new Thread(new ReportMessageConsumer(config, ""))).start();

Then within the ReportMessageConsumer we have X number of threads setup, so this would be 40 in our current setup. So this would be all under this one group. Once the XML has been built and the threads done with how do we effectively kill the thread and enforce the Garbage collector to free that memory, so that we can then create a new clean thread to pick up another message?

Upvotes: 4

Views: 5692

Answers (1)

Stephen C
Stephen C

Reputation: 718698

I feel like the garbage collector isn't being utilized by us correctly?

That is not the problem. The best thing you can do is to let the GC do its thing without any interference. Don't try to force the GC to run. It is rarely helpful, and often bad for performance.

The real problem is that you have a memory leak. It may be happening because you are getting more and more threads ... or it may be something else.

I would recommend the following:

  1. Rewrite you code so that it uses a ExecutorService to manage a bounded pool of threads, and a queue of tasks to be run on those threads. Look at the javadocs for a simple example.

    Using a thread pool is likely to improve your application's overall performance. Creating a thread (i.e. Thread.start()) is rather expensive in Java.

    (And don't shut down the pool as a way to ensure that a batch of work has completed. That is bad for performance. The simple way to do that is to submit the batch using invokeAll; see ExecutorService, how to wait for all tasks to finish.)

  2. If that doesn't cure your leak, then use a memory profiling tool to find out how / why your application is leaking memory. There are lots of StackOverflow Q&A's on how to do this. For example:

Upvotes: 7

Related Questions