CODEWITHSUNDEEP
javamultithreadingconcurrency
Kevin Dolan
Kevin Dolan

Reputation: 5097

Java Concurrency Abstraction for Work Queue

I found myself thinking through solving a concurrency problem too much, which to me indicates there is probably an abstraction to accomplish what I want that I should use instead.

Here is the basic premise:

I figured the following API would be nice:

The hard part was guaranteeing that data doesn't sit in the queue not being worked on in the event that a thread drops off a piece of work around the same time as the work thread is finishing up.

I would assume there exists an abstraction to solve this problem. Does anybody know of any?

Edit:

To be more specific, I am speaking about an incremental search indexer, and the data being added to the queue is additional documents to be processed. When I'm talking about starting and stopping a group of work, I'm referring to building up a caching context and committing an updated index. We can't just run a work thread at all times, because if we never commit, the new index never gets committed, and if we don't close the caching context periodically, it can grow too large.

The goal is that most of the time, only a few pieces of work are processed at a time, but during periods of heavy load, we don't waste too much time doing too many commits.

Upvotes: 2

Views: 823

Answers (3)

java_mouse
java_mouse

Reputation: 2109

"Only one thread can work on the data at a time" -- This line indicates that you need a semaphore to control how many workers(consumers) can work.

For producer side you will need a blockingqueue so that producer will block(wait) if the list is full before adding to the list.

Upvotes: 1

Grundlefleck
Grundlefleck

Reputation: 129217

You may want to check out the Disruptor, an open source project which is all about processing a lot of data with very low latency, in order, sequentially. I have not used it, personally but it sounds about right for your problem.

Upvotes: 0

dMb
dMb

Reputation: 9337

Java has a lot of concurrency abstractions at http://download.oracle.com/javase/1,5.0/docs/api/java/util/concurrent/package-summary.html

Upvotes: 1

Related Questions