CODEWITHSUNDEEP
dockerdocker-composedockerfile
CvRChameleon
CvRChameleon

Reputation: 421

Docker - Reuse docker-compose configurations for different projects

I have a large Docker project with Dockerfiles for nginx, apache2, varnish, redis configured and working well after weeks of changes and testing.

I am now at a point where I setup the projects to use docker-compose and override.yml files for easy setup:

I am trying to use the same docker-compose setup for multiple projects (websites)

Normal startup (using docker-compose.yml and optional docker-compose.override.yml) docker-compose up -d

Custom startup (using specific docker-compose files) docker-compose -f docker-compose.yml -f custom/docker-compose.website1.yml up -d

Both these methods starts up fine: docker-compose ps

Ignore the fact that they are Exit 0 - I stopped them using docker-compose stop, the containers work fine nginx-proxy /usr/bin/supervisord Exit 0 redis-cache /usr/bin/supervisord Exit 0 varnish-cache /usr/bin/supervisord Exit 0 web-server-apache2 /usr/bin/supervisord Exit 0

Now I want a second project (website) to use the same docker/docker-compose configuration setup:

docker-compose -f docker-compose.yml -f anothercustomfolder/docker-compose.website2.yml up -d

To my surprise docker-compose recreated containers and do not create a new set of containers:

See 'current setup' section for how I setup things.

  Creating network "delete-network-frontend" with the default driver
  Recreating nginx-proxy        ... done
  Recreating varnish-cache      ... done
  Recreating web-server         ... done
  Recreating redis-cache        ... done

When running docker-compose ps in the second setup folder:

Note the names are not the same as above (this is the second test setup)

         Name                       Command          State                Ports
  ------------------------------------------------------------------------------------------------
  nginx-proxy-delete       /usr/bin/supervisord   Up      0.0.0.0:443->443/tcp,
                                                          0.0.0.0:80->80/tcp
  redis-cache-delete       /usr/bin/supervisord   Up      0.0.0.0:6379->6379/tcp
  varnish-cache-delete     /usr/bin/supervisord   Up      0.0.0.0:6081->6081/tcp,
                                                          0.0.0.0:6082->6082/tcp
  web-server-              /usr/bin/supervisord   Up      0.0.0.0:8080->8080/tcp
  apache2-delete

It appears docker-compose did two things : 1. Recreate (replace) the project 1 containers, used the project 1 container names to mention that they were 'recreated', and 2. Remove the project 1 containers, renamed it to project 2 containers.

Current setup

Question

TLDR: How do I use a working docker-compose project on the same host operating system for multiple projects... without it replacing another project's containers.

I want to be able to see (use both) at the same time, and for instance be able to see this: Ignore the fact that the ports are the same here, I am aware they won't run at the same time, I will update the project docker-compose.yml custom files when this works

docker-compose ps

               Name                       Command          State                Ports
  ------------------------------------------------------------------------------------------------
  nginx-proxy              /usr/bin/supervisord   Up      0.0.0.0:443->443/tcp,
                                                          0.0.0.0:80->80/tcp
  redis-cache              /usr/bin/supervisord   Up      0.0.0.0:6379->6379/tcp
  varnish-cache            /usr/bin/supervisord   Up      0.0.0.0:6081->6081/tcp,
                                                          0.0.0.0:6082->6082/tcp
  web-server-              /usr/bin/supervisord   Up      0.0.0.0:8080->8080/tcp
  apache2
  nginx-proxy              /usr/bin/supervisord   Up      0.0.0.0:443->443/tcp,
                                                          0.0.0.0:80->80/tcp
  redis-cache-delete       /usr/bin/supervisord   Up      0.0.0.0:6379->6379/tcp
  varnish-cache-delete     /usr/bin/supervisord   Up      0.0.0.0:6081->6081/tcp,
                                                          0.0.0.0:6082->6082/tcp
  web-server-              /usr/bin/supervisord   Up      0.0.0.0:8080->8080/tcp
  apache2-delete

If anyone asks: Why not just put the websites into the same (one) container??

For the possibility someone might ask this, I know I can add multiple websites into the /etc/apache2/sites-enabled (or nginx) and add custom configuration files using ADD in Dockerfile for each site, but using that method I cannot test different slight setups.

Different setups that can be used by referencing another different image in the 'override docker-compose files'

For instance I can create a Dockerfile that installs all php7.3 libraries required and run Magento 2.3 on it, then have another Dockerfile to test php7.4, and have another to run an older Magento 1 site on a PHP5.6 installation and so on.

Upvotes: 0

Views: 2376

Answers (1)

CvRChameleon
CvRChameleon

Reputation: 421

Thanks to advice from David Maze, I struggled further with configuring the docker-compose setup to work with multiple projects.

Information based on docker-compose v1.25.0 (July 2020)

This discussion is especially important when you want to re-use (persist) your containers (start/stop instead of just up/down - deleting)

As initially pointed out in my question - if you try to create containers using docker-compose up -d there are some pitfalls which the tool simply does not support right at the moment.

Pitfalls

PITFALLS OF CURRENT DOCKER-COMPOSE IMPLEMENTATION:

  1. If you just use overridden docker-compose*.yml with different container_names (per 'project') with files in the same folder

docker-compose up will simply replace existing containers as explained in my question.

  1. You can do the following: docker-compose -p CUSTOM_PROJECT_NAME -f file1.yml -f file2.yml up -d, but:

This on its own is useless - these containers will only work until you want stop them. As soon as you want to do docker-compose start (to restart existing container set) it will simply fail with Error: No containers to start

  1. If you use two different folders with the same docker-compose project (ie cloned project): for instance ./dc-project1 ./dc-project2 but using container_name field inside docker-compose.*.yml file:

When you try to run docker-compose -f f1.yml -f f2.yml up -d inside ./dc-project1 and the same inside ./dc-project2 folder, you will get the following error: You have to remove (or rename) that container to be able to reuse that name.

  1. Similar issues with your Docker network will occur with docker-compose when you use overridden files: Removed most of the custom settings to make the network setting clearer:

Network will be attached correctly from your overridden file on docker-compose up, but as soon as you want to docker-compose start it looks for your default network name: in the default docker-compose.yml or even the docker-compose.override.yml file if it exists. In other words - it ignores your custom docker-compose override files (see example below):

docker-compose.yml:

   networks:
      network_frontend:
          name: stage6-network-frontend

customfolder/docker-compose.custom.yml:

   networks:
      network_frontend:
          name: magento2.3-network-frontend

SOLUTION

Example

Objective : to get docker-compose start/stop to work correctly with multiple setups (aka projects/websites/tools)using the same docker-compose project.

Suppose you have the following docker-compose files:

**Main file: ** docker-compose.yml:

   web_server:
       image: current_timezone/full-supervisord-web-server/php7.3:1.00
       container_name: web-server-apache2
       networks:
         - network_frontend
       build:
         context: "./all-services/"
         dockerfile: ./web-server/Dockerfile.webserver.apache2
         args:
       volumes:
         - website_data:/var/www/html
       ports:
         - "8080:8080"

   networks:
     network_frontend:
       driver: bridge
       ipam:
         driver: default
         config:
           - subnet: 172.100.0.0/16
       name: stage6-network-frontend
       driver_opts:
         # Custom name for host-side network : for instance on Ubuntu : ip addr | ifconfig
  com.docker.network.bridge.name: docker-custom # Seems limit of 15 characters only

and then an override file: customfolder/magento2.override.yml:

     web_server:
       container_name: web-server-apache2-magento2.3.5
       networks:
         - network_frontend
       build:
         args:
       volumes:
         - website_data:/var/www/html
       ports:
         - "8080:8080"

   networks:
     network_frontend:
       driver: bridge
       ipam:
         driver: default
         config:
           - subnet: 172.100.0.0/16
       driver_opts:
         # Custom name for host-side network : for instance on Ubuntu : ip addr | ifconfig
  com.docker.network.bridge.name: d-glo-femag2_35 # Seems limit of 15 characters only
       name: glo-magento2.3-network-frontend

Do the following:

  1. Copy the full Docker project (Dockerfiles/ADDs/docker-compose.yml files etc) into a new seperate folder:

    /var/docker/project1 /var/docker/project2

  2. Make sure that the container_name entries in your override docker-compose.yml are unique between the two projects.

  3. In project project1 run docker-compose -f docker-compose.yml -f customfolder/magento2.override.yml up -d && docker-compose stop, navigate to project2 and do the same.

Using -p flag as David Maze suggested does not work on its own, JSON files are still sourced as ./foldername on docker-compose start/stop

  1. Since networks are having similar issues on start/stop , before you can correctly use your custom name defined in your override file, .... unfortunately you need to update the main base docker-compose.yml to the overridden file!

Extended explanation: There is no way to call the correct custom network name from docker-compose start, so since docker-compose ignores the overridden files on start, you need to make sure to update the base file docker-compose.yml or docker-compose.override.yml has your custom network name.

In case you have not updated the names before using up -d, you will need to replace the content of each /var/lib/docker/containers/*/config.v2.json.

For example you could do this: you have to stop docker first

  sudo service docker stop
  find /var/lib/docker/containers/ -type f -name "config.v2.json" -exec sed -i "s|wrong-network-name|overridden-network-name|g" '{}' \;
  sudo service docker start
  1. IF done correctly, you should have unique container names, and each folder can be accessed separately correctly now without it breaking each other's containers: docker-compose start, docker-compose stop, docker-compose ps

NOTE: You still need to navigate to the seperate folder to run those commands

Upvotes: 1

Related Questions