CODEWITHSUNDEEP
javaapache-kafka
Joker
Joker

Reputation: 11140

How to verify the size of partition or Topic in Kafka?

I want to verify the partition size before producing the the record in Kafka.

I have a custom partitioned class which gives me exact partition number in which my message is supposed to drop.

Now my requirement is I want to check the size of partition before sending my record.

 List<String> users = userService.findAllUsers();
            for (String user : users) {
            String msg = "Hello " + user;
            //Check size here
          producer.send(new ProducerRecord<String, String>(topic, user, msg), new Callback() {
                    public void onCompletion(RecordMetadata metadata, Exception e) {
                        if (e != null) {
                            e.printStackTrace();
                        }
                    }
                });

Is there any way in kafka I can achieve this ?

Upvotes: 0

Views: 3121

Answers (1)

Adam Kotwasinski
Adam Kotwasinski

Reputation: 4574

Capacity is purely a Kafka-broker level aspect. Basically, if there is disk space in broker's data directory, you can deliver the message. The message are cleaned up by broker based on time and partition size (that's in broker configuration), so if you configure your broker accordingly, you might always have space - the old messages would just get thrown away. It might not suit your business usecase though.

Also, responding to your comment Can we check size of Topic, you can actually check the current size of partition by using beginningOffsets & endOffsets methods in KafkaConsumer. Beware that these methods might block if the partitions do not exist (at least in 0.10.2). e.g. when you request data for partition 4 when topic actually contains 3 partitions.

Kafka 0.11 introduces administrative capabilities in client, however it is still work in progress.

Upvotes: 1

Related Questions