CODEWITHSUNDEEP
javamultithreadingqueue
marius
marius

Reputation: 1613

Chained filters with parallelization in Java

I am trying to parse an input into an output by applying a sequence of transformations t0, t1, ..., tn. The execution will be a chain:

input -> t0 -> t1 -> ... -> tn -> output

Some of the transformations should be parallelized, to prevent becoming a bottleneck.

Is there any framework for creating such a process chain in Java? I know how I could do it manually (e.g. Queuing jobs in a processing chain in Java) but I am specifically looking for a framework, since

Upvotes: 0

Views: 74

Answers (3)

edharned
edharned

Reputation: 1904

As Alexei said about advice, let the buyer beware. I just happen to maintain a Fork/Join framework on SourceForge that supports filters. Since the code is free and open-source, you can use it as is or do what you want. TymeacDSE

Upvotes: 1

Alexei Kaigorodov
Alexei Kaigorodov

Reputation: 13535

There exist many different execution models to express such a transformation chain. Transformations can be threads with loops reading from input queues, or objects with a method to handle each incoming message (threads and loops hidden under the hood). Transformation may have single input or several inputs. It may know its successors and send them results directly, or just return a value from method, and a separately described topology takes care of further message routing. Availability of space to hold results may or may not be taken into account when a transformation act is started, and so on. A transformation act may be very short, so overhead for message passing is significant, and message passing should be carefully optimized (see Disruptor), or ordinary linked queue generating wrapper objects for each message would suffice.

It is said, it is hard to give an advice without knowing your requirements. You should explore which models and implementations exist and find out most appropriate for your case. Without knowing details, advices like "I believe framework XYZ is what you are looking for" only describe advisor as a person who hardly knows other frameworks besides XYZ. Nevertheless, I dare to recommend you to look at Dataflow Framework for Java, whose model is reach enough, though has its limitations (e.g. execution nodes may not block).

Upvotes: 1

jtahlborn
jtahlborn

Reputation: 53694

I believe the fork/join framework (in jdk 7) is what you are looking for. it enables specifying task dependencies and splitting jobs into more parallel chunks depending on available resources.

Upvotes: 1

Related Questions