CODEWITHSUNDEEP
disruptor-patternlmax
jamb
jamb

Reputation: 196

LMAX Distruptor Partition and join batch

So currently I have a Executor implementation with blocking queue and the implementation specific is like, I have list of items per request and I divide them into partitions each partition is then computed and finally they are joined to have the final list.

How do I go about implementing it in LMAX? I see that once I have partition and push them into RingBuffer, each partition is treated as separate item so I am custom joining them. something like,

ConcurrentHashMap<Long, LongAdder> map = new ConcurrentHashMap<>();
@Override
public List<SomeTask> score(final List<SomeTask> tasks) {
    long id = tasks.get(0).id;
    map.put(id, new LongAdder());
    for (SomeTask task : tasks) {
        producer.onData(task);
    }
    while (map.get(id).intValue() != tasks.size()) ;
    map.remove(id);
    return tasks;
}

Is there a clean way to do it ? I looked at https://github.com/LMAX-Exchange/disruptor/tree/master/src/test/java/com/lmax/disruptor/example and KeyedBatching specifically but they seem to batch and execute on one thread.

Currently for me each partition takes up around 200ms and I wanted to parallel execute them.

Any help is greatly appreciated.

Upvotes: 1

Views: 175

Answers (1)

Sam Turtel Barker
Sam Turtel Barker

Reputation: 843

I think you should take a look at the worker-pool options and followed by a final event processor which re-combines the shards.

Upvotes: 0

Related Questions