CODEWITHSUNDEEP
dockerdocker-composerabbitmqcelery
RonZhang724
RonZhang724

Reputation: 575

Deploy containers from different docker-compose.yml

Currently I have a rabbitmq message broker and multiple celery workers that need to be containerized. My problem is, how can I fire up containers using different docker-compose.yml? My goal is to start the rabbitmq once and for all, and never touch it again.

Currently I have a docker-compose.yml for the rabbitmq:

version: '2'
services:
    rabbit:
        hostname: rabbit
        image: rabbitmq:latest
        environment:
            - RABBITMQ_DEFAULT_USER=admin
            - RABBITMQ_DEFAULT_PASS=mypass
        ports:
            - "5672:5672"
        expose:
            - "5672"

And another docker-compose.yml for celery workers:

version: '2'
services:
    worker:
        build:
            context: .
            dockerfile: dockerfile
        volumes:
            - .:/app
        environment:
            - CELERY_BROKER_URL=amqp://admin:mypass@rabbit:5672
        links:
            - rabbit

However, when I do docker-compose up for celery workers, I keep getting the following error:

ERROR/MainProcess] consumer: Cannot connect to 
amqp://admin:**@rabbit:5672//: failed to resolve broker hostname.

Can anyone take a look if there is anything wrong with my code? Thanks.

Upvotes: 1

Views: 667

Answers (2)

Thomasleveil
Thomasleveil

Reputation: 103955

the domain name rabbit in your second docker-compose.yml file does not resolve because there is no service with that name in that docker-compose.yml file.

As stated in the comments, one solution is to put both the rabbit service and worker service in the same docker-compose.yml file. In such a setup, all containers started for those services would join the same docker network and those service names could be resolved to the IP adresses of their containers.

Since having a single docker-compose.yml file is not convenient in your case, you have to find an other way to have the containers originating from different docker-compose.yml files join a same docker network.

To do so, you need to create a dedicated docker network for that purpose:

docker network create rabbitNetwork

Then, in each docker-compose.yml file, you need to refer to this network in the services definitions:

version: '2'
services:
    rabbit:
        hostname: rabbit
        image: rabbitmq:latest
        environment:
            - RABBITMQ_DEFAULT_USER=admin
            - RABBITMQ_DEFAULT_PASS=mypass
        # ports:
        #    - "5672:5672"  # there is no need to publish ports on the docker host anymore
        expose:
            - "5672"
        networks:
            - rabbitNet

networks:
  rabbitNet:
    external:
      name: rabbitNetwork

version: '2'
services:
    worker:
        build:
            context: .
            dockerfile: dockerfile
        volumes:
            - .:/app
        environment:
            - CELERY_BROKER_URL=amqp://admin:mypass@rabbit:5672
        networks:
            - rabbitNet

networks:
  rabbitNet:
    external:
      name: rabbitNetwork

Upvotes: 3

Akash Sharma
Akash Sharma

Reputation: 757

You can use any file as service definition. docker-compose.yml is default file name but any other name can be passed using -f argument.

docker-compose -f rabbit-compose.yml COMMAND

Upvotes: 0

Related Questions