CODEWITHSUNDEEP
javamultithreadingconcurrencyjava.util.concurrent
kachanov
kachanov

Reputation: 2765

Strange behaviour of ExecutorService

I have 5000 similar Callable tasks to be executed in 8 threads of ExecutorService created by Executors.newFixedThreadPool(8). Each task goes to a database to retrieve a lot of data to process.

Everything works fine 99% time, BUT sometimes I see a very strange execution log messages in the log file, when DB is slow or stuck (don't ask why) and 8 currently running tasks are stalled and not finished yet in all 8 threads, ExecutorService starts submitting more tasks to execute one by one!

So the log shows that at some point ExecutorService goes crazy and starts calling call() method of Callable of more and more tasks in the waiting queue without waiting the previous tasks to complete. More and more tasks send requests to DB which finally brings DB to it's knees and Java heap memory becomes exhausted.

It looks like something strange is happening inside ExecutorService or my understanding of the situation is wrong. Has anyone seen anything like that?

My brain stack is overflown

p.s. the is the quote from the Java API:

Executors.newFixedThreadPool(int nThreads)

Creates a thread pool that reuses a fixed number of threads operating off a shared unbounded queue. At any point, at most nThreads threads will be active processing tasks. If additional tasks are submitted when all threads are active, they will wait in the queue until a thread is available. If any thread terminates due to a failure during execution prior to shutdown, a new one will take its place if needed to execute subsequent tasks.

Could this actually happen that my tasks cause thread to die and ExecutorService creates more threads and submits new 8 tasks to them and they die and ExecutorService creates 8 more threads and submits more 8 tasks?

p.s.s.: the whole operation inside call() of Callable is surrounded with try catch so if any exception is happening inside my operation the exception will be captured and logged. None of this is happening. The call is called and just never returns, while next tasks are beign called one by one and never returning and never finishing and never throwing any exceptions.

I suspect my tasks cause threads in thread pool to die. How is it possible to imitate?

Upvotes: 0

Views: 6809

Answers (2)

Sahil Muthoo
Sahil Muthoo

Reputation: 12506

I'll take a shot at guessing as well:

  1. You submit 5000 tasks which involve fetching data from a database.
  2. Soon after, you encounter heavy lock contention on required rows/tables. Maybe external processes are acquiring exclusive locks for writing. Maybe there is a deadlock.
  3. One after another, the tasks block, waiting for a shared/read lock to be granted.
  4. It appears as if all 8 threads are suspended, waiting on I/O.
  5. Soon after, the database/DB driver notices that tasks have been waiting too long for a shared lock. It summarily hands out Lock Wait Timeout exceptions to tasks, in order.
  6. Thus, one after another, tasks fail out of the queue and waiting tasks are pushed into execution, only to fail again.

Do note, that an exception in a Task, will not stop the ExecutorService. It'll just mark that task as done and continue.

See this example:

public class Foo {

    static class Task implements Callable<String> {
        private static AtomicInteger i = new AtomicInteger(1);

        public String call() throws Exception {
            i.incrementAndGet();
            if (i.get() % 2 != 0) {
                throw new RuntimeException("That's odd, I failed.");
            }
            return "I'm done";
        }
    }

    public static void main(String[] args) throws Exception {
        ExecutorService es = Executors.newFixedThreadPool(2);
        List<Future<String>> futures = new ArrayList<Future<String>>();
        for (int i = 0; i < 5; i++) {
            futures.add(es.submit(new Task()));
        }
        for (Future<String> future : futures) {
            try {
                System.out.println(future.get());
            } catch (ExecutionException ee) {
                System.err.println(ee.getCause());
            }
        }
        es.shutdown();
    }
}

Possible output:

I'm done
I'm done
I'm done
java.lang.RuntimeException: That's odd, I failed.
java.lang.RuntimeException: That's odd, I failed.

Upvotes: 3

laher
laher

Reputation: 9110

This is just a guess, (I think a guess is merited, given the lack of code in the question):

ExecutorService.invokeAll(Collection<? extends Callable<T>> tasks) will move on to other tasks if current tasks throw an exception. (are you using invokeAll()? I think submit(Callable<T> task) has the same behaviour, but it isn't clear from the javadoc)

Can you check incase those 'stuck' tasks become Future.isDone() before the subsequent tasks start running? Potentially exceptions are being thrown and not seen in the logs...

From the javadoc:

Note that a completed task could have terminated either normally or by throwing an exception.

http://download.oracle.com/javase/6/docs/api/java/util/concurrent/ExecutorService.html#invokeAll(java.util.Collection%29

If this is the case, you could just catch & log all exceptions inside your Callable.call() method definition.

HTH

Upvotes: 0

Related Questions