CODEWITHSUNDEEP
javaapache-kafka
sushant shekhar
sushant shekhar

Reputation: 51

kafka API for Reading kafka messages in a Group from topic with same partition key

I have 5 million of messages in internal Kafka Topic.

1 million  message with Partition key  -1234-Messge1
1 million  message with Partition key  -2345-Messge2
1 million  message with Partition key  -5678-Messge3
1 million  message with Partition key  -6789-Messge4
1 million  message with Partition key  -6565-Messge5

I have to join messages having same Partitionkey as part of single message and send to an consumer Topic [Eg: For key 1234-Messge1,consumer should receive single message instead of 1 million message]

Is there any Kafka API available at kafka end ,using which I can read all the messages having same Partition key in a group instead of reading single message at a time like traditional spring boot kafka Listener.

Upvotes: 0

Views: 396

Answers (2)

Adam Kotwasinski
Adam Kotwasinski

Reputation: 4564

Not really, as Kafka does not provide key-driven API. In the end (whether you use KafkaConsumer directly or Streams), you are going to read all records from the partitions you want.

If you know what partition(s) your records are in, then you might set up the consumers to read from these partition(s).

However please remember you might have been impacted by situation like increasing number of partition in a topic (what changes a hash function, unless you use stable hashing partitioner).

Upvotes: 1

OneCricketeer
OneCricketeer

Reputation: 191983

In Kafka Streams, you can filter on a specific record key, but this will read all the partitions. You can also groupByKey if you wanted the same logic for all keys, and can query all values from a KTable, for example.

If you already know (or can compute) the topic partition, you can assign (or use @KafkaListener property topicPartitions) to make a Consumer read one-to-many TopicPartition. This will still read all records for that partition, and if you have multiple keys, you would need if (record.key().equals(partitionKey))

Upvotes: 1

Related Questions