CODEWITHSUNDEEP
apache-kafka
Mangat Rai Modi
Mangat Rai Modi

Reputation: 5706

Kafka: Why broker isn't pull based like consumers

I was reading Kafka docs where it was mentioned that:-

  1. Consumers pulls data from broker by requesting from offset.
  2. Producer pushes messages to broker.

Making Kafka consumers pull based make sense that the consumers can drive the pace and broker can store the data for a really long time.

However with producers being push based, How does Kafka make sure that speed mismatch between producer and kafka won't happen? Also producers don't have persistance by design.This seems to be a bigger problem, when producers and brokers are separated over high latency network(internet).

Upvotes: 0

Views: 812

Answers (1)

Robin Moffatt
Robin Moffatt

Reputation: 32080

As a distributed commit log, Kafka solves exactly this (impedance mismatch). You produce your events at the rate at which they occur into Kafka, and then you consume them at the rate at which your application can. The data is persisted in Kafka regardless. If your application needs to consume at a greater rate, you scale it out and partition your topic and consume in parallel. Because the data is persisted the only factor is how fast you want to consume the data.

Upvotes: 5

Related Questions