CODEWITHSUNDEEP
apache-kafka
deitch
deitch

Reputation: 14591

How do I configure a Kafka broker to refuse producers but not consumers?

I have a cluster of 3 brokers that have been running for a long time. I want to drain them down, so that no new messages are accepted, but consumers can read messages off.

The goal is to end up with nothing unconsumed in the brokers, so that I can then replace them. My producers are resilient enough to handle inability to produce for a period of time (local cache, fallback systems, etc.). My lag is fairly low, so it should be very quick.

I hunted through the various configs, but couldn't find a way to block only producers, but not consumers.

For what it is worth, I did try adding new brokers and then reassigning partitions. It takes hours, even for small topics.

Upvotes: 0

Views: 406

Answers (1)

Andrei
Andrei

Reputation: 520

It seems to me that using the recommended method https://kafka.apache.org/documentation/#basic_ops_decommissioning_brokers should be enough (You can of course try to firewall producers, or add ACL on the cluster and ban it, but it seems to me that this is overhead and inappropriate in this case). At least I have never had a problem with it.

If there is a lot of data, the reassign will indeed take quite a long time, but considering your words that you have a fairly small lag, I would suggest the following:

to ensure that the lag was consistently close to zero, reduce the retention of topics (retention.ms for every topic or globally log.retention.hours (here only with restart)), move all partitions to a new cluster (as described above), then return the retention.

Upvotes: 1

Related Questions