CODEWITHSUNDEEP
apache-kafka
Mandroid
Mandroid

Reputation: 7468

Kafka message composition

A kafka message has:

key, value, compression type, headers(key-value pairs,optional), partition+offset, timestamp

Key is hashed to partition to find which partition producer would write to.

Then why do we need partition as part of message.

Also, how does producer know the offset as offset seems more like a property of kafka server? And doesn't it cause coupling between server and producer?

And how would it work if multiple producers are writing to a topic, as offset send by them may clash?

Upvotes: 0

Views: 165

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191671

why do we need partition as part of message.

It's optional for the client to set the record partition. The partition is still needed in the protocol because the key is not hashed server-side, then rerouted.

how does producer know the offset as offset seems more like a property of kafka server?

The producer would need a callback to get the OffsetMetadata, but it's not known when a batch is sent

And doesn't it cause coupling between server and producer?

Yes? It's the Kafka protocol. The consumer is also "coupled" with the server because it must understand how to communicate with it.

multiple producers are writing to a topic, as offset send by them may clash?

If max.inflight.connections is more than 1 and retires are enabled, then yes, batches may get rearranged, but send requests are initially ordered, and clients do not set the record offset, the broker does.

Upvotes: 1

Related Questions