CODEWITHSUNDEEP
spring-bootapache-kafkaspring-kafka
intA
intA

Reputation: 2671

How do I connect my sample kafka producer application to a remote kafka server?

https://www.javainuse.com/spring/spring-boot-apache-kafka-hello-world

I followed this example to build a simple Kafka producer for some testing. However, my Kafka instance is not local and is managed by my company's infrastructure team, so it is running on a remote server. I have the address and port it is running on, but I am unsure how to configure my application to connect to it. Do I need to use an application.properties file to define the kafka address pool or something like this?

spring.kafka.bootstrap-servers=[kafka-server-1:port],[kafka-server-2:port]
spring.kafka.consumer.group-id=my-sample-group
spring.kafka.security.protocol=SSL

In the tutorial I was following for the consumer I have something like this in my application.properties but I'm not sure if it's the same for the producer or should be different. The second value here seems to be specific to the consumer and based on the autocomplete in my IDE I'm not seeing a similar value for producers.

Any help is appreciated

I've changed my code around and followed the example at https://docs.spring.io/spring-kafka/reference/html/#spring-boot-producer-app

Now when I try to connect the output is just alternating between WARN messages about the 2 kafka servers in my connection pool:

2021-11-10 11:44:29.609 WARN 11839 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -2 (kafka1.mycompany.com/[some-ip-address]:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.

2021-11-10 11:44:29.798 WARN 11839 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -1 (kafka2.mycompany.com/[some-ip-address]:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.

At the beginning it seems to output some information about the kafka setup. Maybe there is some info there about which additional params I might need in my application.properties to be able to actual connect.

bootstrap.servers = [kafka1.mycompany.com:9092, kafka2.mycompany.com:9092]
    client.dns.lookup = use_all_dns_ips
    client.id = 
    connections.max.idle.ms = 300000
    default.api.timeout.ms = 60000
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 30000
    retries = 2147483647
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = GSSAPI
    security.protocol = SSL
    security.providers = null
    send.buffer.bytes = 131072
    socket.connection.setup.timeout.max.ms = 127000
    socket.connection.setup.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
    ssl.endpoint.identification.algorithm = https
    ssl.engine.factory.class = null
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.certificate.chain = null
    ssl.keystore.key = null
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLSv1.3
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.certificates = null
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS

Upvotes: 1

Views: 3590

Answers (2)

OneCricketeer
OneCricketeer

Reputation: 191681

spring.kafka.bootstrap-servers is the same property for both consumers and producers (and admin clients), yes.

Do I need to use an application.properties file

It's not required for a SpringBoot application, no.

Upvotes: 0

Andreas
Andreas

Reputation: 5309

The tutorial you followed is highly outdated. I would recommend to look at the official docs: https://docs.spring.io/spring-kafka/reference/html/#spring-boot-producer-app. Depending on your setup, the config could just be one bootstrap server.

spring.kafka.bootstrap-servers=kafka.your-company.com:9092

I would ask the Kafka ops team how to connect, as this can get complex with jaas and ssl.

Spring Boot Kafka config settings can be found here: https://docs.spring.io/spring-boot/docs/current/reference/html/application-properties.html#application-properties.integration.spring.kafka.admin.client-id

Upvotes: 1

Related Questions