CODEWITHSUNDEEP
bashspring-bootdockercassandra
Quinten Scheppermans
Quinten Scheppermans

Reputation: 1094

How to check if docker Cassandra instance is ready to take connections

I have two docker instances that I launch with docker-compose. One holds a Cassandra instance One holds a Spring Boot application that tries to connect to that application.

However, the Spring Boot application will always fail, because it's trying to connect to a Cassandra instance that is not ready yet to take connections.

I have tried:

Right now the only thing that really works is using that same bash script and sleep the process for x seconds, then start the jar. This is even more hacky...

Does anyone have an idea on how to solve this?

Thanks!

Upvotes: 2

Views: 2903

Answers (2)

br3nt
br3nt

Reputation: 9586

We have this same problem with trying to instantiate our containerised application in a GitLab pipeline for end to end testing. That job is extremely flakey exactly because of the issue described in the question. It often fails, and we have to re-run the tests and hope for the best. It's hard to believe this is a problem with all the fancy solutions available to us.

We are still trying to find a good solution, however, I have a different approach that may be viable in some situations. Rather than instantiate a new Cassandra instance for every pipeline or job, we could instead connect to a persistent Cassandra instance. That is, each job would use a unique keyspace or set of keyspaces. As the Cassandra instance would be external from the CI/CD environment and is always up and ready to take a connection, the problem is avoided entirely.

There are a few benefits to this approach.

First, our pipeline will be faster as there is one less service that needs to start up before the actual job is run.

Second, we can start treating the environment and the running application separately. We can have separate pipelines: one which builds and test the environment, the other which builds and test the app running within that environment.

Hopefully, this solution is helpful to a subset of problems, or inspires other out-of-the-box answers.

Upvotes: 0

Thomas Herzog
Thomas Herzog

Reputation: 506

Does the spring boot service defined in the docker-compose.yml depends_on the cassandara service? If yes then the service is started only if the cassandra service is ready.

https://docs.docker.com/compose/compose-file/#depends_on

Take a look at this github repository, to find a healthcheck for the cassandra service.

https://github.com/docker-library/healthcheck

CONCLUSION

After some discussion we found out that docker-compose seems not to provide a functionality for waiting until services are up and healthy, such as Kubernetes and Openshift provide (See comments below). They recommend to use wrapper script (docker-entrypoint.sh) which waits for the depending service to come up, which make binaries necessary, the actual service shouldn't use such as the cassandra client binary. Additionally the service depending on cassandra could never get up if cassandra doesn't, which shouldn't happen.

A main thing with microservices is that they have to be resilient for failures and are not supposed to die or not to come up if a depending service is currently not available or unexpectedly disappears. Therefore the microservice should be implemented in a way so that it retries to get connection after startup or an unexpected disappearance. Unexpected is a word actually wrongly used in this context, because you should always expect such issues in a distributed environment, and even with docker-compose you will face issues like that as discussed in this topic.

The following link points to a tutorial which helped to integrate cassandra properly into a spring boot application. It provides a way to implement the retrieval of a cassandra connection with a retry behavior, therefore the service is resilient to a non existing cassandra database and will not fail to start anymore. Hope this helps others as well.

https://dzone.com/articles/containerising-a-spring-data-cassandra-application

Upvotes: 1

Related Questions