CODEWITHSUNDEEP
javacassandradatastax-java-driverspring-data-cassandrak8ssandra
Avinash Shetty
Avinash Shetty

Reputation: 35

Java driver keeps reconnecting to old Cassandra pod IPs but pods have new IPs after restart

I have deployed a Java application as a pod in a Kubernetes environment, as well as Cassandra with a 3-node cluster. I am using Cassandra service FQDN to connect to Cassandra. Due to the Cassandra pod restart, all its IPs got changed, and Cassandra client/java is still connecting to older IPs. Below error:

Error while opening new channel ( \
  ConnectionInitException: [s0|connecting...] \
  Protocol initialization request, step 1 (STARTUP {\
    CQL_VERSION=3.0.0, \
    DRIVER_NAME=Apache Cassandra Java Driver, \
    DRIVER_VERSION=4.18.0, \
    CLIENT_ID=c6ddf75c-3ffd-433a-b6ee
  }): failed to send request (java.nio.channels.NotYetConnectedException))

How can I make java application(Using Java 17) connect to latest IPs though we connect through service FQDN?

Java version: 17, cassandra driver version: 4.18

Upvotes: 0

Views: 76

Answers (1)

Erick Ramirez
Erick Ramirez

Reputation: 16353

The default behaviour for the Cassandra Java driver is to resolve the service FQDN (contact point) to IP addresses just once. Those IP addresses get cached by the driver and are used for subsequent connection attempts.

This behaviour is obviously problematic in dynamic environments such as Kubernetes deployments where the IP addresses of the Cassandra nodes change when the underlying pods are restarted/rescheduled.

In Java driver 4.0, a new configuration option was added to override the default behaviour (JAVA-1978) with:

datastax-java-driver {
  advanced {
    resolve-contact-points = false
  }
}

This tells the driver to keep addresses in contact points unresolved. If the driver needs to reconnect to the nodes for whatever reason, it will resolve the service FQDN again so it will pick up the new IP addresses if they've changed. This also means that you won't need to restart your application to force the driver to refresh its copy of the cluster metadata.

For more info on the resolve-contact-points option, see the Cassandra Java driver reference configuration. Cheers!

Upvotes: 1

Related Questions