CODEWITHSUNDEEP
apache-kafka
Venkata
Venkata

Reputation: 325

Get the latest offsets in SSL Enabled Kafka via CMD

I have being using the below CMD to get the latest offsets in from a Kafka Queue which has plain-text port open

kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list server:9092 --topic sample_topic --time -1

But, now we only have the SSL port open, so I tried passing the SSL details as a property file 

kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list server:9093 --topic sample_topic --time -1 --consumer-config /path/to/file

Getting the below error - 

Exception in thread "main" joptsimple.UnrecognizedOptionException: consumer-config is not a recognized option

How do I pass the SSL details to this command? These are all the available arguments for kafka-run-class.sh kafka.tools.GetOffsetShell

--broker-list <String: hostname:and port,...,hostname:port>                
--max-wait-ms <Integer: ms>            
--offsets <Integer: count>             
--partitions <String: partition ids>   
--time <Long: timestamp/-1(latest)/-2             
--topic <String: topic>

Upvotes: 15

Views: 5698

Answers (3)

toien
toien

Reputation: 352

Good News!

Starting with kafka-3.x GetOffsetShell can work with SASL. SASL configs are passed by --command-config option.

Look source code on github: GetOffsetShell.scala

Upvotes: 0

kairius
kairius

Reputation: 540

I succeeded to retrieve it on docker image quay.io/strimzi/kafka:0.29.0-kafka-3.2.0 by running:

./bin/kafka-get-offsets.sh --bootstrap-server "$BOOTSTRAPSERVER" --command-config kafkaConfig.properties --topic "$TOPIC" --time -1

where kafkaConfig.properties in my case contain SASL/SSL properties:

security.protocol=SSL
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
ssl.truststore.type=PEM
ssl.truststore.location=/var/kafka-server-ca-cert.pem
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="kafka-user" password="******";

Upvotes: 1

Mickael Maison
Mickael Maison

Reputation: 26895

Unfortunately kafka.tools.GetOffsetShell only supports PLAINTEXT connection. This tools is not used a lot and nobody has bothered updating it.

Depending on your use case, you have a few options:

  • Use the kafka-consumer-groups.sh tool: Assuming you have a consumer group consuming from that topic, this tool display the log end offsets of each partitions

  • Patch kafka.tools.GetOffsetShell: It's realtively easy to add support to secured connections bby reusing logic from the other tool. If you do so, consider sending a patch to Kafka =)

  • Write a tiny tool that calls Consumer.endOffsets()

  • Use kafka.tools.DumpLogSegments: As a last resort this tool can also be used to find the last offset

Upvotes: 16

Related Questions