How to run a container from within another container when building a tool/utility

Question

We've begun to use containers when creating tools and utilities so individual developers don't have to bother installing prerequisites. Works great, as in:

docker run --rm -v $PWD/input.yaml:/input.yaml  our.registry.com/dev/toolA /input.yaml

Until toolA needs to call another utility, toolB, that is also wrapped in a container.

Is there a generic way to run a container from within a container, so that it works with docker, podman, or whatever the container-tech-du-jour is without toolA's caller having to add "scary" extra parameters like e.g. --privileged=true or -v /var/run/docker.sock:/var/run/docker.sock?

I'm aware that one can run docker-in-docker, and apparently one can also run podman-in-podman, but that is not recommended. Adding -v /var/run/docker.sock:/var/run/docker.sock is the recommended approach, but that is docker-only, right?

Because otherwise this is a leaky abstraction. Currently I build toolA by including the essential bits of the toolB container in toolA, making toolA sensitive to changes in the implementation of toolB, which I'd like to avoid. Or I could build toolA but only support docker and require users to add the -v /var/run/docker.sock:/var/run/docker.sock parameter :-(.

Is there a more elegant way to call toolBs container from inside toolA's container that works with all common container techs and doesn't require extra parameters when starting toolA?

Answer 1

Passing in -v /var/run/docker.sock:/var/run/docker.sock to the outer container is problematic from a security point of view as it gives the outer container permission to run containers on the host. A malicious outer container could run such containers with bind-mounts so that host directories could be written to.

Running the outer container with Podman as your normal user (rootless) with --privileged is protecting you from that scenario. The outer container will only be able to write to the directories that you pass in as bind-mounts (e.g. with the -v command-line option).

The meaning of the --privileged flag is explained in this Red Hat article.

Update June 2021. It's not necessary anymore to pass the --privileged flag when running Podman in Podman. See my example in https://stackoverflow.com/a/64537135/757777 and the Red Hat blog post How to use Podman inside of a container

Answer 2

Is there a generic way to run a container from within a container [...] without having to add "scary" extra parameters like [...] -v /var/run/docker.sock:/var/run/docker.sock?

No.

(You do have to mount the host's Docker socket; the calling container does not need to be --privileged; you cannot limit what the calling container can do once it has access to the socket, so it can itself launch privileged containers, or launch a container bind-mounting the host's /etc directory, or otherwise do naughty things. I'm not actually sure how this interacts with Podman.)

Especially if the container needs to write back to the host system, it's hard to make the docker run command be usably compact. You'll need a -u option to specify the host user ID, -v options for the Docker socket and the current directory, --rm, and the repository name, all as overhead, every time you run the command. I might prefer using a native command, maybe in a widely-available scripting language like Python or the Bourne shell.

If you still want to run this from a container, you could COPY or otherwise install tool B into the same container, so you can make a normal subprocess invocation to launch it; or break tool B into a library and an application, and include the library in tool A (kind of what you're doing now but more formalized).

Answer 3

The approach to run a container from inside container is unorthodox and it can be avoided by redesigning the flow. It will work, but it will create huge complexity and overhead.

If you need the tool just for the period during the build, you can use multi-stage builds. What you need to do is to create a base container usually called builder that has all the tools that are needed for the build, fetch the project in the base container and run the build. After the build is finished, you will move to the next stage and create a base runner where you will install only the required libraries and modules for the runtime environment and add the resulting files from the builder to the runner container.

In this way, you will have full control over both build-time and run-time environments. In addition, if you have multiple projects, or complex builds, you can use the multi-stage build in combination with buildx to run the build in parallel.