CODEWITHSUNDEEP
apache-kafka
Ryn
Ryn

Reputation: 491

What happens when a Kafka Broker runs out of space before the configured Retention time/bytes?

I understand that most systems should have monitoring to make sure that this doesn't happen (and that we should set the retention policies properly), but am just curious what happens if the Kafka Broker does indeed run out of disk space (for example, if we set retention time to 30 days, but the Broker runs out of disk space by the 1st day)?

In a single Broker scenario, does the Broker simply stop receiving any new messages and return an exception to the Producer? Or does it delete old message to make space for the new ones?

In a multi Broker scenario, assuming we have Broker A (leader of the partition but has not more disk space) and Broker B (follower of the partition and still has disk space), will leadership move to Broker B? What happens when both Brokers run out of space? Does it also return an exception to the Producer?

Upvotes: 3

Views: 1250

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191963

Assuming the main data directory is not on a separate volume, the OS processes themselves will start locking up because there's no free space left on the device.

Otherwise, if the log directories are isolated to Kafka, you can expect any producer acks to stop working. I'm not sure if a specific error message is returned to clients, though. From what I remember, the brokers just stop responding to Kafka client requests and we had to SSH to them, stop kafka services, and manually clean up files rather than waiting for retention policies. No, the brokers don't preempt old data in favor of new records

Upvotes: 4

Related Questions