CODEWITHSUNDEEP
amazon-web-servicesapache-kafka
Tom
Tom

Reputation: 101

How can I access my AWS MSK managed kafka queue from my local machine and EC2 instances in other regions

I'm setting up a managed kafka queue on AWS MSK. I can't seem to get the security to work when connecting from a local machine and I can't work out if I can use security groups from one region to another.

I've gone through the information on setting up the security groups on the main documentation here. I still can't seem to connect to the broker though. I'm currently using kafka scripts from my local machine in the following way:

bin/kafka-console-producer.sh --broker-list "my-broker-ip:9092" --topic "some-topic"

but keep getting the result 

[2019-01-28 12:06:13,278] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

I've associated my local IP with all the ports on the security group that I set up with my VPC and associated with the kafka queue but it doesn't seem to have helped. I also don't see how I can associate my boxes with the kafka queue as they are in different regions. Is this possible?

I'm currently using kafka scripts from my local machine in the following way:

bin/kafka-console-producer.sh --broker-list "my-broker-ip:9092" --topic "some-topic"

but keep getting the result

[2019-01-28 12:06:13,278] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

I've associated my local IP with all the ports on the security group that I set up with my VPC and associated with the kafka queue but it doesn't seem to have helped. I also don't see how I can associate my boxes with the kafka queue as they are in different regions. Is this possible?

I'm expecting to be able to connect my local producer code the kafka queue and observe the output rather than have it constantly reject the connection.

Upvotes: 9

Views: 29278

Answers (5)

BeeOnRope
BeeOnRope

Reputation: 65006

You can access your brokers from anywhere by setting up public access to your brokers. From that page:

For security reasons, you can't turn on public access while creating an MSK cluster. However, you can update an existing cluster to make it publicly accessible. You can also create a new cluster and then update it to make it publicly accessible.

The page goes on to list other restrictions on public access, but as long as you adhere to those, it is possible.

Upvotes: 0

Kuntal-G
Kuntal-G

Reputation: 2991

As @Robin mentioned, you cannot access MSK directly from a local machine using kafka client or kafka stream. Because the broker url, zookeeper connection string are private ip's of the msk cluster vpc/subnet. To access through kafka client, you need to launch ec2 instance in the same vpc of MsK and execute kafka client(producer/consumer) to access msk cluster.

But you can set up kafka Rest Proxy framework open-sourced by Confluent to acess the MSK cluster from the outside world via rest api. This framework is not a full-fledgeddged kafka client and it doesn't allow all kafka client operations, but you can do some operations on the cluster like: fetching metadata of the cluster, fetching topic information, producing and consuming messages, etc.

I have answered in detail this scenario along with a few other questions related to MSK, refer-

Amazon Managed Streaming for Kafka- MSK features and performance

Upvotes: 5

Robin Moffatt
Robin Moffatt

Reputation: 32110

As far as I know, you have to access your MSK cluster from a client machine on EC2, and cannot do so from a local machine.

Upvotes: 7

hopeIsTheonlyWeapon
hopeIsTheonlyWeapon

Reputation: 567

One solution I can think which we have is using a transit gateway. That way the VPC where AWS MSK resides and the on-premises network where your laptop is present can be interconnected and can be accessed seamlessly.

Also you can take advantage of VPC peering.

Please refer to the below documentation for details.

https://docs.aws.amazon.com/msk/latest/developerguide/client-access.html

Details about Transit gateway

https://docs.aws.amazon.com/vpc/latest/tgw/what-is-transit-gateway.html

Upvotes: 0

David
David

Reputation: 1999

As @Kuntal-G mentioned, the broker URLs etc. are private IPs and cannot be accessed from outside the VPC. However, you can assign ("public") Elastic IPs to the brokers and adjust the security groups to allow traffic to/from the Zookeeper and Kafka ports as explained in more detail here:

https://www.repetitive.it/aws-msk-how-to-expose-the-cluster-on-the-public-network/

Alternative solutions are also mentioned in the official AWS documentation (e.g. using AWS Transit Gateway or REST proxies):

https://docs.aws.amazon.com/msk/latest/developerguide/client-access.html

Upvotes: 2

Related Questions