Docker-Compose: how to wait for other service to be ready?

Question

I have the following docker-compose, where I need to wait for the service jhipster-registry to be up and accepting connections before starting myprogram-app.

I tried the healtcheck way, following the official doc https://docs.docker.com/compose/compose-file/compose-file-v2/

version: '2.1'
services:
    myprogram-app:
        image: myprogram
        mem_limit: 1024m
        environment:
            - SPRING_PROFILES_ACTIVE=prod,swagger
            - EUREKA_CLIENT_SERVICE_URL_DEFAULTZONE=http://admin:$${jhipster.registry.password}@jhipster-registry:8761/eureka
            - SPRING_CLOUD_CONFIG_URI=http://admin:$${jhipster.registry.password}@jhipster-registry:8761/config
            - SPRING_DATASOURCE_URL=jdbc:postgresql://myprogram-postgresql:5432/myprogram
            - JHIPSTER_SLEEP=0
            - SPRING_DATA_ELASTICSEARCH_CLUSTER_NODES=myprogram-elasticsearch:9300
            - JHIPSTER_REGISTRY_PASSWORD=53bqDrurQAthqrXG
            - EMAIL_USERNAME
            - EMAIL_PASSWORD
        ports:
            - 8080:8080
        networks:
          - backend
        depends_on:
          - jhipster-registry:
              "condition": service_started
          - myprogram-postgresql
          - myprogram-elasticsearch
    myprogram-postgresql:
        image: postgres:9.6.5
        mem_limit: 256m
        environment:
            - POSTGRES_USER=myprogram
            - POSTGRES_PASSWORD=myprogram
        networks:
          - backend
    myprogram-elasticsearch:
        image: elasticsearch:2.4.6
        mem_limit: 512m
        networks:
          - backend
    jhipster-registry:
        extends:
            file: jhipster-registry.yml
            service: jhipster-registry
        mem_limit: 512m
        ports:
            - 8761:8761
        networks:
          - backend
        healthcheck:
          test: "exit 0"
networks:
  backend:
    driver: "bridge"

but I get the following error when running docker-compose up:

ERROR: The Compose file './docker-compose.yml' is invalid because:
services.myprogram-app.depends_on contains {"jhipster-registry": {"condition": "service_started"}}, which is an invalid type, it should be a string

Am I doing something wrong, or this feature is no more supported? How to achieve this sync between services?

Updated version

version: '2.1'
services:
    myprogram-app:
        image: myprogram
        mem_limit: 1024m
        environment:
            - SPRING_PROFILES_ACTIVE=prod,swagger
            - EUREKA_CLIENT_SERVICE_URL_DEFAULTZONE=http://admin:$${jhipster.registry.password}@jhipster-registry:8761/eureka
            - SPRING_CLOUD_CONFIG_URI=http://admin:$${jhipster.registry.password}@jhipster-registry:8761/config
            - SPRING_DATASOURCE_URL=jdbc:postgresql://myprogram-postgresql:5432/myprogram
            - JHIPSTER_SLEEP=0
            - SPRING_DATA_ELASTICSEARCH_CLUSTER_NODES=myprogram-elasticsearch:9300
            - JHIPSTER_REGISTRY_PASSWORD=53bqDrurQAthqrXG
            - EMAIL_USERNAME
            - EMAIL_PASSWORD
        ports:
            - 8080:8080
        networks:
          - backend
        depends_on:
          jhipster-registry:
            condition: service_healthy
          myprogram-postgresql:
            condition: service_started
          myprogram-elasticsearch:
            condition: service_started
        #restart: on-failure
    myprogram-postgresql:
        image: postgres:9.6.5
        mem_limit: 256m
        environment:
            - POSTGRES_USER=myprogram
            - POSTGRES_PASSWORD=tuenemreh
        networks:
          - backend
    myprogram-elasticsearch:
        image: elasticsearch:2.4.6
        mem_limit: 512m
        networks:
          - backend
    jhipster-registry:
        extends:
            file: jhipster-registry.yml
            service: jhipster-registry
        mem_limit: 512m
        ports:
            - 8761:8761
        networks:
          - backend
        healthcheck:
          test: ["CMD", "curl", "-f", "http://jhipster-registry:8761", "|| exit 1"]
          interval: 30s
          retries: 20
          #start_period: 30s
networks:
  backend:
    driver: "bridge"

The updated version gives me a different error,

ERROR: for myprogram-app  Container "8ebca614590c" is unhealthy.
ERROR: Encountered errors while bringing up the project.

saying that the container of jhipster-registry is unhealthy, but it's reachable via browser. How can I fix the command in the healthcheck to make it work?

Answer 1

The documentation suggests that, in Docker Compose version 2 files specifically, depends_on: can be a list of strings, or a mapping where the keys are service names and the values are conditions. For the services where you don't have (or need) health checks, there is a service_started condition.

depends_on:
  # notice: these lines don't start with "-"
  jhipster-registry:
    condition: service_healthy
  myprogram-postgresql:
    condition: service_started
  myprogram-elasticsearch:
    condition: service_started

This syntax also works in version 2 of the Compose tool, which ignores version: and uses the moving Compose Specification file format. If you have the old version 1 of the Compose tool, and your file is declared as version: '3.x', this syntax is not allowed.

Depending on how much control you have over your program and its libraries, it's better still if you can arrange for the service to be able to start without its dependencies necessarily being available (equivalently, to function if its dependencies die while the service is running), and not use the depends_on: option. You might return an HTTP 503 Service Unavailable error if the database is down, for instance. Another strategy that often is helpful is to immediately exit if your dependencies aren't available but use a setting like restart: on-error to ask the orchestrator to restart the service.

Answer 2

With the new docker compose API, we can now use the new --wait option:

docker compose up --wait

If your service has a healthcheck, Docker waits until it has the "healthy" status; otherwise, it waits for the service to be started. That's why it is crucial to have relevant healthchecks for all your services.

Note that this option automatically activates the --detach option.

Check out the documentation here.

Answer 3

After trying several approaches, IMO the simplest and most elegant option is using the jwilder/dockerize (dockerize) utility image with its -wait flag. Here is a simple example where I need a PostgreSQL database to be ready before starting my app:

version: "3.8"
services:
  # Start Postgres.
  db:
    image: postgres

  # Wait for Postgres to be joinable.
  check-db-started: 
    image: jwilder/dockerize:0.6.1
    depends_on:
      - db
    command: 'dockerize -wait=tcp://db:5432'
  
  # Only start myapp once Postgres is joinable.
  myapp:
    image: myapp:latest
    depends_on:
      - check-db-started

Answer 4

Best Approach - Resilient App Starts

While docker does support startup dependencies, they officially recommend updating your app start logic to test for the availability of external dependencies and retry. This has lots of benefits for robust applications that may restart in the wild on the fly in addition to circumventing the race condition in docker compose up

depends_on & service_healthy - Compose 1.27.0⁺

depends_on is back in docker compose v1.27.0+ (was deprecated in v3) in the Compose Specification

Each service should also implement a service_healthy check to be able to report if it's fully setup and ready for downstream dependencies.

version: '3.0'
services:
  php:
    build:
      context: .
      dockerfile: tests/Docker/Dockerfile-PHP
    depends_on:
      redis:
        condition: service_healthy
  redis:
    image: redis
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 1s
      timeout: 3s
      retries: 30

wait-for-it.sh

The recommended approach from docker according to their docs on Control startup and shutdown order in Compose is to download wait-for-it.sh which takes in the domain:port to poll and then executes the next set of commands if successful.

version: "2"
services:
  web:
    build: .
    ports:
      - "80:8000"
    depends_on:
      - "db"
    command: ["./wait-for-it.sh", "db:5432", "--", "python", "app.py"]
  db:
    image: postgres

Note: This requires overriding the startup command of the image, so make sure you know what wanted to pass to maintain parity of the default startup.

Further Reading

• Docker Compose wait for container X before starting Y
• Difference between links and depends_on in docker_compose.yml
• How can I wait for a docker container to be up and running?
• Docker Compose Wait til dependency container is fully up before launching
• depends_on doesn't wait for another service in docker-compose 1.22.0

Answer 5

The best approach I found is to check for the desired port in the entrypoint. There are different ways to do that e.g. wait-for-it but I like to use this solution that is cross-platform between apline and bash images and doesn't download custom scripts from GitHub:

Install netcat-openbsd (works with apt and apk). Then in the entrypoint (works with both #!/bin/bash and #!/bin/sh):

#!/bin/bash

wait_for()
{
  echo "Waiting $1 seconds for $2:$3"
  timeout $1 sh -c 'until nc -z $0 $1; do sleep 0.1; done' $2 $3 || return 1
  echo "$2:$3 available"
}


wait_for 10 db 5432
wait_for 10 redis 6379

You can also make this into a 1-liner if you don't want to print anything.

Answer 6

Update to version 3+.

Please follow the documents from version 3:

There are several things to be aware of when using depends_on:

depends_on does not wait for db and redis to be “ready” before starting web - only until they have been started.
If you need to wait for a service to be ready, see Controlling startup order for more on this problem and strategies for solving it.
Version 3 no longer supports the condition form of depends_on.
The depends_on option is ignored when deploying a stack in swarm mode with a version 3 Compose file.

I would consider using the restart_policy option for configuring your myprogram-app to restart until the jhipster-registry is up and accepting connections:

 restart_policy:
        condition: on-failure
        delay: 3s
        max_attempts: 5
        window: 60s

Answer 7

Although you already got an answer, it should be mentioned that what you are trying to achieve have some nasty risks.

Ideally a service should be self sufficient and smart enough to retry and await for dependencies to be available (before a going down). Otherwise you will be more exposed to one failure propagating to other services. Also consider that a system reboot, unlike a manual start might ignore the dependencies order.

If one service crash causes all your system to go down, you might have a tool to restart everything again, but it would be better having services that resist that case.