CODEWITHSUNDEEP
amazon-web-servicesapache-kafkaaws-msk
Vivere
Vivere

Reputation: 2270

AWS MSK Public cluster secured with SASL/SCRAM

I configured an AWS MSK cluster with public access. I created an AWS Secret via Secrets Manager and assigned it to the cluster.

Based on that secret, I managed to publish messages to MSK (I think). However, when I try to read messages from the topic, I get unauthorized errors.

I followed this documentation page, and I ended up with the following settings:

KafkaClient {
   org.apache.kafka.common.security.scram.ScramLoginModule required
   username="<username>"
   password="<password>";
};

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512

Then, I'm using the consumer shell script as follows:

./kafka-console-consumer.sh --bootstrap-server <urls> --topic <topicName> --from-beginning --consumer.config client.properties

And I'm getting the following error:

[2022-06-21 13:34:17,566] WARN [Consumer clientId=consumer-console-consumer-13840-1, groupId=console-consumer-13840] Error while fetching metadata with correlation id 2 : {users=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient) [2022-06-21 13:34:17,568] ERROR [Consumer clientId=consumer-console-consumer-13840-1, groupId=console-consumer-13840] Topic authorization failed for topics [topicName] (org.apache.kafka.clients.Metadata) [2022-06-21 13:34:17,571] ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$) org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [topicName] Processed a total of 0 messages

Am I missing an important aspect of the documentation?

Upvotes: 1

Views: 2474

Answers (1)

Daniel Sinai
Daniel Sinai

Reputation: 135

You need to create acls to read from this topic.

Keep in mind that in order to do that you should Address to the zookkeeper due to limitations in aws msk.

For example:

./bin/kafka-acls.sh --authorizer-properties zookeeper.connect=ZooKeeper-Connection-String --add --allow-principal "User:CN=Distinguished-Name" --operation Read --group=* --topic Topic-Name

Upvotes: 1

Related Questions