CODEWITHSUNDEEP
dockercelery
Always_Beginner
Always_Beginner

Reputation: 2916

HEALTHCHECK of a Docker container running Celery tasks?

I know one of the ways to check health for Docker container is using the commmand 

HEALTHCHECK CMD curl --fail http://localhost:3000/ || exit 1

But in case of workers there is no such URL to hit , How to check the container's health in that case ?

Upvotes: 44

Views: 32438

Answers (6)

pangyuteng
pangyuteng

Reputation: 1839

The below example snippet, derived from that posted by @PunKeel, is applicable for those looking to implement health check in docker-compose.yml which could be used through docker-compose or docker stack deploy.

worker:
    build:
        context: .
        dockerfile: Dockerfile
    image: myimage
    links:
        - rabbitmq
    restart: always
    command: celery worker --hostname=%h --broker=amqp://rabbitmq:5672
    healthcheck:
        test: celery -b amqp://rabbitmq:5672 inspect ping -d celery@$$HOSTNAME
        interval: 30s
        timeout: 10s
        retries: 3

Notice the extra $ in the command, so that $HOSTNAME actually gets passed into the container. I also didn't use the -A flag.

Ideally, rabbitmq should also have its own health check, perhaps with rabbitmqctl status, since docker wouldn't be able to discern if worker is down or the broker is down with celery inspect ping.

-- edit:

please also check out the solution provided by SomeGuyOnAComputer on using kill -0 $PID to check if PID is alive.

Upvotes: 12

chethan
chethan

Reputation: 305

To configure a HEALTHCHECK for a Docker container running Celery tasks and ensure another container depends on its health status:

In your Docker Compose file:

version: "3.8"
services:
  celery:
    # Celery container config
    healthcheck:
      test: celery inspect ping
      interval: 1m
      timeout: 10s
      retries: 10
      start_period: 1m

  dependent_container:
    # Dependent container config
    depends_on:
      celery:
        condition: service_healthy

This sets up a health check for the celery container using celery inspect ping.
The dependent_container depends on celery with condition: service_healthy, ensuring it starts only when celery is healthy.

Upvotes: 1

Taras Mykhalchuk
Taras Mykhalchuk

Reputation: 917

As simple as that:

...
healthcheck:
    test: sh -c 'celery -A your_celery_module inspect ping'
    ...

Your log file will have a similar ping log:

[2023-05-05 11:18:47,309: DEBUG/MainProcess] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': '8195241f-122a-3dc4-9841-739f55804b82'} ticket:bada7609-ce6b-499a-b3c2-c80911e0fe09]

Also, you might want to check it yourself in an already-running container. In that case, it will look something like this:

celery ping output

Upvotes: 0

swimmer
swimmer

Reputation: 3313

Landed on this question looking for a health check for Celery workers as part of an Airflow setup (Airflow 2.3.4, Celery 5.2.7), which I eventually figured out. This is a very specific use case of the original question, but might still be useful for some:

# docker-compose.yml

  worker:
      image: ...
      hostname: local-worker
      entrypoint: airflow celery worker
      ...
      healthcheck:
        test: [ "CMD-SHELL", 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"' ]
        interval: 5s
        timeout: 10s
        retries: 10
      restart: always
      ...

I got inspiration from Airflow's quick-start Docker Compose.

Upvotes: 1

Bildad N. Urandu
Bildad N. Urandu

Reputation: 99

For celery 5.2.3 I used celery -A [celery app name] status for the health check. This is how my docker-compose file looks like

worker:
      build: .
      healthcheck:
        test: celery -A app.celery_app status
        interval: 10s
        timeout: 10s
        retries: 10
      volumes:
          - ./app:/app
      depends_on:
        - broker
        - redis
        - database

Upvotes: 7

punkeel
punkeel

Reputation: 959

The celery inspect ping command comes in handy, as it does a whole trip: it sends a "ping" task on the broker, workers respond and celery fetches the responses.

Assuming your app is named tasks.add, you may ping all your workers:

/app $ celery inspect ping -A tasks.add
-> celery@aa7c21dd0e96: OK
        pong
-> celery@57615db15d80: OK
        pong

With aa7c21dd0e96 being the Docker hostname, and thus available in $HOSTNAME.

To ping a single node, you would have to run:

celery inspect ping -A tasks.add -d celery@$HOSTNAME

Here, d stands for destination.

The line to add to your Dockerfile:

HEALTHCHECK CMD celery inspect ping -A tasks.add -d celery@$HOSTNAME

Sample outputs:

/app $ celery inspect ping -A tasks.add -d fake_node
Error: No nodes replied within time constraint.
/app $ echo $?
69

Unhealthy if the node does not exist or does not reply

/app $ celery inspect ping -A tasks.add -d celery@$HOSTNAME
-> celery@d39b3d31cc13: OK
        pong
/app $ echo $?
0

Healthy when the node replies pong.

/app $ celery inspect ping -d celery@$HOSTNAME
Traceback (most recent call last):
...
    raise socket.error(last_err)
OSError: [Errno 111] Connection refused
/app $ echo $?
1

Unhealthy when the broker is not available - I removed the app, so it tries to connect to a local AMPQ and fails This might not suit your needs, the broker is unhealthy, not the worker.

Upvotes: 44

Related Questions