Vert.x: How to process event sequences in whole worker thread pool

Question

My question is:

If I receive multiple event sequences in event loop threads, how could I process every sequence blocking and ordered but different sequences be processed by the whole worker thread pool.

Use case:

I have gRPC client in a vert.x with 4 event loop and 20 worker threads. I start remote calls in 10 threads.

• Every call receives 2 events StreamObserver.onNext and StreamObserver.onCompleted
• On every such event I'd like to start blocking task in such way that for a gRPC call onNext and onClose are ordered, while for different gRPC calls events are executed in different worker threads.

The goal is to distribute call execution to as much worker threads as possible but events for single call to be ordered.

• If I use executeBlocking with ordered true - they will be executed in a worker thread per event thread.
• If false - there will be no order in processing onNext and onClose.

Answer 1

I think it's a good usecase for Kafka.

You can keep working with a worker pool with every thread creating and listening kafka with a consumer. Instead of a worker pool, you can also instanciate multiple verticles in the same java process or even multiple jvm instances (or even containers, why not :p ). I prefer this second approach to handle the dynamic scalability in an event-driven and distributed architecture (I wrote more details about that in this answer).

Either way, all the consumers needs to be instanciate with the same group.id to be a part of the same group of consumers. That way, kafka will be able to assign the partitions of the topic to each group's member (a single partition cannot be read by two members at the same time).

And finally to be able to create "event sequences", you only need to produce the events in the kafka topic with a distribution key that will define your sequence (two messages produced with the same distribution key will be stored in the same partition and will be read by the same consumer of the group).