CODEWITHSUNDEEP
javaapache-kafkaspring-kafka
Іван Гладуш
Іван Гладуш

Reputation: 395

How to get message by key from kafka topic

I try to get message by key from kafka. I found the only one solution is to use StateStore but I think it might be not a good idea. How to get message by key from kafka topic? Is it a good idea to use StateStore for this operation?

Upvotes: 23

Views: 46254

Answers (2)

Chris Matta
Chris Matta

Reputation: 3443

Every record written to Kafka can optionally have a key (but it doesn't have to!), the key can be accessed a number of ways:

Console consumer:

$ kafka-console-consumer --bootstrap-server <servername>:9092 --topic topicname --from-beginning --property print.key=true --property key.separator=:

kafkacat:

$ kafkacat -b <servername>:9092 -C -t topicname -o beginning -K :

Java Consumer API:

ConsumerRecord#key()

Kafka isn't indexed by key though, it's indexed by offset, and optionally by timestamp. If you need to lookup a key then you'll need to materialize the data to a system that's been designed to lookup by key: relational database, key value store, or some index. You can do this pretty easily with Kafka Connect, or if you'd like to build it into your service you could use the interactive queries feature of Kafka Streams.

Upvotes: 23

Gary Russell
Gary Russell

Reputation: 174769

You can't "get messages by key from Kafka".

One solution, if practical, would be to have as many partitions as keys and always route messages for a key to the same partition.

Upvotes: 5

Related Questions