CODEWITHSUNDEEP
spring-bootapache-kafkaspring-kafka
Arun Kalvichandran
Arun Kalvichandran

Reputation: 31

connect Kafka aws instance from Java API

I was trying to connect kafka aws instance through local Spring Boot API.

I am able to connect it but while listening to the topic, it's throwing an below exception but the new topics were created successfully by Spring Boot API I am unable to publish any message as well. 

java.io.IOException: Can't resolve address: ip-xxx-xx-xx-xx.ec2.internal:9092
at org.apache.kafka.common.network.Selector.doConnect(Selector.java:235) ~[kafka-clients-2.0.1.jar:na]

Caused by: java.nio.channels.UnresolvedAddressException: null
    at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_192]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_192]
    at org.apache.kafka.common.network.Selector.doConnect(Selector.java:233) ~[kafka-clients-2.0.1.jar:na]
    ... 30 common frames omitted

2019-07-17 15:36:13.581  WARN 3709 --- [           main] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=group_id] Error connecting to node ip-172-31-80-50.ec2.internal:9092 (id: 0 rack: null)

I allowed this port as well Custom TCP Rule TCP 2181 0.0.0.0/0 Custom TCP Rule TCP 9092 0.0.0.0/0

server:
  port: 8081
spring:
  kafka:
    consumer:
      bootstrap-servers: xx.xx.xx.xx:9092
      group-id: group_id
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
    producer:
      bootstrap-servers: xx.xx.xx.xx:9092
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer


@KafkaListener(topics = "ConsumerTest", groupId = "group_id")
    public void consume(String message) throws IOException {
        logger.info(String.format("#### -> Consumed message -> %s", message));
    }

Upvotes: 1

Views: 1547

Answers (1)

Robin Moffatt
Robin Moffatt

Reputation: 32140

java.io.IOException: Can't resolve address: ip-xxx-xx-xx-xx.ec2.internal:9092

Error connecting to node ip-172-31-80-50.ec2.internal:9092

When consumers connect to the broker they get back the metadata of the broker for the partition from which they're reading data. What your client is getting back here is the advertised.listener of the Kafka broker. So whilst you connect to the broker on the public address of the broker, it returns to your client the internal address of the machine.

To fix this, you need to set up your listeners correctly on your brokers. See my blog https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details.

Upvotes: 2

Related Questions