CODEWITHSUNDEEP
apache-kafka
Denis V
Denis V

Reputation: 3370

Using Kafka instead of Redis for the queue purposes

I have a small project that uses Redis for the task queue purposes. Here is how it basically works.

I have two components in the system: desktop client (can be more than one) and a server-side app. Server-side app has a pull of tasks for the desktop client(s). When a client comes, the first available task from the pull is given to it. As the task has an id, when the desktop client gets back with the results, the server-side app can recognize the task by its id. Basically, I do the following in Redis:

  1. Keep all the tasks as objects.
  2. Keep queue (pool) of tasks in several lists: queue, provided, processing.

When a task is being provided to the desktop client, I use RPOPLPUSH in Redis to move the id from the queue list to the provided list.

When I get a response from the desktop client, I use LREM for the given task ID from the provided list (if it fails, I got a task that was not provided or was already processed, or just never existed - so, I break the execution). Then I use LPUSH to add the task id to the processing list. Given that I have unique task ids (controlled on the level of my app), I avoid duplicates in the Redis lists.

When the task is finished (the result got from the desktop client is processed and somehow saved), I remove the task from the processing list and delete the task object from Redis.

If anything goes wrong on any step (i.e. the task gets stuck on the processing or provided list), I can move the task back to the queue list and re-process it.

Now, the question: is it somehow possible to do the similar stuff in Apache Kafka? I do not need the exact behavior as in Redis - all I need is to be able to provide a task to the desktop client (it shouldn't be possible to provide the same task twice) and mark/change its state according to the actual processing status (new, provided, processing), so that I could control the process and restore the tasks that were not processed due to some problem. If it's possible, could anyone please describe the applicable workflow?

Upvotes: 4

Views: 1845

Answers (1)

arunvg
arunvg

Reputation: 1269

It is possible for kafka to act as a standard queue. Check the consumer group feature.

If the question is about the appropriateness, please also refer Is Apache Kafka appropriate for use as a task queue?

We are using kafka as a task queue, one of the consideration went in favor of kafka was that it is already in our application ecosystem, found it easier than adding one more component.

Upvotes: 1

Related Questions