CODEWITHSUNDEEP
google-cloud-platformapache-kafka
Arun
Arun

Reputation: 1286

Cannot connect to kafka hosted in cloud from spring boot application

The kafka virual machine address in the Google cloud platform is xx.xx.xxx.xxx.

From my local spring boot app that usually connects to the localhost:9092 of the local machine's kafka server, I changed it to the GCP's virtual machine's ip as xx.xx.xxx.xxx:9092

But the server start up spits out the warning

2020-04-05 15:30:41.356  WARN 7968 --- [| adminclient-4] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-4] Connection to node -1 could not be established. Broker may not be available.

and eventually times out. Should there be a different way to connect to cloud kafka from springboot application?

Upvotes: 0

Views: 2507

Answers (2)

Robin Moffatt
Robin Moffatt

Reputation: 32060

You need to configure your broker on the GCP VM with a correct advertised.listener so that your client receives a correct hostname/IP from it after the initial successful connection.

You can validate the connection and broker metadata provided by the advertised.listeners setting using kafkacat -L

$ kafkacat -b xx.xx.xxx.xxx:9092 -L
Metadata for all topics (from broker -1: xx.xx.xxx.xxx:9092/bootstrap):
1 brokers:
  broker 0 at a.b.c.d:9092

The a.b.c.d. returned should be an IP or hostname that your client can successfully resolve to the broker itself (not a loopback address, internal network IP, etc).

To understand more see my blog https://rmoff.net/2018/08/02/kafka-listeners-explained/

Upvotes: 1

Arun
Arun

Reputation: 1286

The following properties are added

advertised.host.name in server.properties to public IP address, metadata.broker.list in producer.properties to public IP address, host.name to 0.0.0.0.

These property files will be inside config folder. A restart is necessary.

And it solved the problem. For broker/bootstrap.sever, the public IP address is used in the application after adding the above mentioned properties.

Upvotes: 0

Related Questions