How to push a Docker Compose stack to a Docker Swarm without uploading containers to a registry?

Question

I've researched around before asking here, but all answers lead me to the same conclusion:

1. Build your Docker Compose stack locally
2. Tag and push the images to a registry (either a private or public one like Docker Hub)
3. Push the stack to the swarm using docker stack deploy --compose-file docker-compose.yml stackdemo
4. From here, the stack picks up and "pulls" the images from the registry and runs the containers

Is there no straightforward way to make the following (I think common sense) scenario work seamlessly?

1. Docker swarm manager has access (SSH keys) to pull the project from Git.
2. It periodically pulls the project and builds it "locally" using docker-compose up
3. When the build succeeds (containers are ready), it pushes the stack to the swarm using docker stack deploy, propagating the images to all worker nodes.
4. In that way, the original "source code" is only known by the Manager Node and only it has direct access to the Git repository.

Maintaining a registry (or paying for a cloud one), seems like a huge disadvantage for using Docker in Swarm Mode.

Side note: I've tried the approach of deploying a registry as a service within the stack and tagging + pushing the images to 127.0.0.1/myimage but that led to a different set of problems of its own - e.g. the fact that worker nodes that do not have an instance of the Registry container running, have no access to pull the image (the registry needs to be replicated to all nodes).

Answer 1

Docker swarm is orchestration & used or intended for managing docker node cluster. When any service is deployed, docker engine can start it on any of the node in the cluster (node that satisfy placement constraints if provided). Now, if one dont have registry, & images are available on node locally, docker cant verify if all nodes will point to same version of the image. Hence, swarm pulls image from registry & then deploys it to nodes.

Having registry also helps in keeping copy of images & registry keeps all version even if images are prune on one or all of docker nodes. One can enable backup of registry & hence there's no chance for loss of any image built & pushed to a registry.

Starting a registry (at least on localhost) is very easy - but that's not what your question.

Coming to your question, you can keep the compose stack file in the same directory where you have Dockerfile & then in the stack compose file you can write service which will get build at the time you deploy the stack:

version: "3.9"

services:
  web:
    image: 127.0.0.1:5000/stackdemo
    build: .
    ports:
      - "8000:8000"

Here,

build: .

this line builds the image with given name tagging to registry on localhost - but is not pushed which needs to be done manually. So, in Dockerfile you can write git pull or git clone & then run command to build your code & so on.

Here's a link which provides simple steps to start a registry & build image on the fly while deploying the stack: https://docs.docker.com/engine/swarm/stack-deploy/

Also, swarm does not works without registry & hence it's not possible to just save & load image & use with swarm orchestration.

Answer 2

Use the docker save and docker load commands to transfer images from your dev machine to all of your swarm machines.