CODEWITHSUNDEEP
javamultithreadingmemory-leaks
0xCAFEBABE
0xCAFEBABE

Reputation: 5666

Java: Multithreaded mail application leaks memory

I have a self-written Java application - a little mail monitor. It works with a MySQL database that has a table which is filled somewhere else regularily. It looks through the table and sends mails as the records appear in the table.

My problem is, the application leaks memory. I was quite sure that it didn't, as the scope of everything I use seems to disappear, making all objects used garbage collectable. But after a while (depending on the -Xmx I pass), the application stops with a OutOfHeapSpace error.

I cannot post the whole code, as it is not mine anymore, yet I tried to recreate it using pseudo code.

Main:
  Startup
  Create .lock file (FileChannel)
  Instantiate Main Class

Constructor Main:
  Class.ForName for the MySQL driver
  Read properties file (settings)
  Create connection object (MySQL)
  Fetch unsent mail ids (ArrayList)
  while(true)
    while have more mail ids
      new Thread(Top Mail ID, MySQL Connection object, Sleep Time, Blacklist);
    end while
    if have no more mail ids in ArrayList:
      sleep for a number of seconds (usually 300)
    end if
  end while

Constructor Thread:
  Prepare Statement
  New Thread(this).start();
  Sleep

Thread run():
  Select Record by passed Mail ID
  Extract everything (Sender, Receiver, Subject etc.)
  Check Blacklist, return if matched
  Extract Attachments as blobs

What I have tried so far:

jvisualvm has shown me how the memory changes over time. What I see is a jagged line in the Heap: Allocating and collecting memory happens regularily, however, after the collection there's always a litttle more memory allocated than after the last collection. the number of Threads seem to be fine, it always drops down to the standard number.

The information amount in jvisualvm is too much for me. There are threads listed there that I cannot identify, the threads I create are not listed as my class, so it's difficult to determine exactly what is "my" code.

Can anybody recognize any common mistakes of multithreading in my pseudo code, or recommend any tools that allow me to determine my leaks easier?

Thank you.

Edit 1: Data access via the connection object which is passed around to all threads is synchronized on itself.

Edit 2: I checked the number of mails that are unsendable (as these stay in the database), there are about 10.

Edit 3: I found Eclipse Memory Analyzer, installed it, and it hinted at the problem. It seems that using PreparedStatements while retaining one connection object keeps a HashMap of all PreparedStatements ever run over this connection inside it, therefor adding all data ever selected to it. I relied on the PreparedStatement going out of scope and being collected. I will rewrite and try if this fixes the problem.

Upvotes: 0

Views: 472

Answers (2)

0xCAFEBABE
0xCAFEBABE

Reputation: 5666

If this is of any interest to anybody, I fixed the memory problem.

The application did not actually leak memory. As I passed the same connection object around, it seems that it saved the result of every prepared statement run against it in a large hashmap. Therefor, the memory usage was constantly going up. I found this out by using a heap dump created using jvisualvm and loaded into Eclipse Memory Analyzer.

I rewrote the application to use a ThreadPool (Cached), opened a dedicated connection for every Thread and close the connection at the end of every thread.

Upvotes: 0

Brady
Brady

Reputation: 10357

From your pseudo code, it appears as though you are creating a thread per email to be sent, which doesnt seem so efficient. I know you mentioned that the number of threads is "normal", but the code appears to indicate otherwise. Could you try using a thread pool instead, whereby you have a limited number of thread workers and pass work to them via some sort of job queue.

Do you know if there are some errors that could cause the threads to not complete or leaves them "hanging"? This could be a memory leak, so look into your error handling code.

In the past I have used jprofiler with some success, maybe it would be a useful alternative.

Upvotes: 1

Related Questions