CODEWITHSUNDEEP
javadistributed-computingignite
Steffen Harbich
Steffen Harbich

Reputation: 2749

Distributed processing with Apache Ignite queues

Consider the following scenario. I have a cluster of Apache Ignite nodes. On each node, work units can appear (e.g. based on input of end users) that need be processed asynchronously. The order of execution is not strictly important, i.e. work units that occurred early in time should be executed before later work units but only approximately.

After some reading, my first thought was to use a distributed queue (collocated because it could be big). Each node would have one worker thread listening to the queue and processing work units once they arrive. However, as far as I understand, queue items are distributed to all cluster nodes then.

So my second thought was: Is there a way to keep work units arising on a node on that node and being processed mainly on that node? I mean, when node local work units are all processed then work units from other nodes are taken etc.

Did you encounter a similar problem and how did you solve it?

EDIT: Work units need to be persistent in my case. That means, if a node is shutdown during execution of a work unit then the work unit must be processed by other node or re-processed on node restart.

Upvotes: 1

Views: 816

Answers (1)

Valentin Kulichenko
Valentin Kulichenko

Reputation: 8390

This sounds like a use case for Compute Grid functionality: https://apacheignite.readme.io/docs/compute-grid

Specifically, you might be interested in job stealing: https://apacheignite.readme.io/docs/load-balancing#job-stealing

Upvotes: 5

Related Questions