CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Saurav Prakash
Saurav Prakash

Reputation: 1949

Implementing throttling on Kafka topic consumer

We are building a system to move data from one point to other in our ecosystem at a controlled and predictable rate. The system basically has a main kafka topic which gets messages of N types. The agent consuming from this topic then puts the message to one of the N worker queue based on message payload topic. These worker queues then have workers actually doing the work.

Now I want to implement control knobs like - pausing/resuming/throttling messages of 1 type from the agent. How could I do this assuming no communication between agent and worker. Should I have a separate "system events" topic for each worker where the agent writes those messages? OR better have N queues at the top level too?

Upvotes: 0

Views: 1403

Answers (1)

Bitswazsky
Bitswazsky

Reputation: 4698

I'd personally prefer to have the pausing/resuming/throttling logic handled by different threads in the agent, which come after the Kafka consumer. This would ensure if you're throttling a particular type, it won't hinder the processing for other types. So, basically the Kafka consumer would delegate the type specific operations to different threads, based on the type.

Upvotes: 1

Related Questions