Pass environment variables from docker-compose to container at build stage

Question

I am trying to create configuration for several environments with a single Dockerfile, several docker-compose files and several envoronment_variables files. I need it to use a different file with python dependencies for each environment.

Let's say, we create a web service in two environments: development and production. To achieve this, I create the following file structure:

docker-compose-dev.yml
docker-compose-prod.yml
envs/
  dev.env
  prod.env
web/
  Dockerfile
  requirements_dev.txt 
  requirements_prod.txt

The objective is to instantiate the proper name of a requirements_*.txt file in the Dockerfile during the container's build process. According to the documentation, I tried this naïve approach, which doesn't seem to work:

1. Define the name as an environment variable:
   • envs/dev.env: REQUIREMENTS=requirements_dev.txt
   • envs/prod.env: REQUIREMENTS=requirements_prod.txt

2. Use this environment variable in the Dockerfile:

   FROM python:3.5
   ENV PYTHONUNBUFFERED 1
   ENV APP_ROOT /usr/src/app
   ...
   COPY $REQUIREMENTS $APP_ROOT/
   RUN pip install -r $APP_ROOT/$REQUIREMENTS
   

3. Import the corresponding definition in a docker-compose configuration:

   version: '2'
   
   services:
       web:
           build: ./web
           env_file:
               - envs/dev.env   # in docker-compose-dev.yml
               - envs/prod.env   # in docker-compose-prod.yml
           ...
   

When I run docker-compose up --build docker-compose-dev.yml, the $REQUIREMENTS variable is not defined at the build process.

In the docker-compose documentation there is a note on this:

Note: If your service specifies a build option, variables defined in environment files will not be automatically visible during the build. Use the args sub-option of build to define build-time environment variables.

But to use the ARG option in the Dockerfile, like ARG REQUIREMENTS, one needs a corresponding args option in docker-compose, like this:

services:
    web:
        build:
            context: ./web
            args:
                - REQUIREMENTS

Nevertheless, with this configuration my tests also failed, because, according to the documentation, the value of the REQUIREMENTS variable in Dockerfile is taken from

the environment where Compose is running

Therefore, it takes value from my shell, but not from envs/dev.env.

So my question is:

Is it possible to pass environment variable from env_file in docker-compose to Dockerfile at build time without setting variables locally in my shell?

Answer 1

Running a container with a build: block happens in two separate stages: first Compose builds the image, and then it runs a container based on that image. Nothing outside the build: block affects the build step, and more specifically, environment variables set via env_file: only affect the running container and aren't visible during the build.

Similarly, Compose has two different notions of "environment". As the Compose file itself is processed, before anything is started, Compose takes into consideration environment variables visible to the docker-compose command, any docker-compose --env-file options, and the implicit .env file. When containers are run, each container's environment: and env_file: set the environment for that container specifically.

For a variable to be visible to the build process, you need several steps:

1. The variable itself must be declared as a Dockerfile ARG allowing it to be passed in from outside. For example,

   FROM python:3.5
   WORKDIR /usr/src/app
   ARG REQUIREMENTS
   COPY $REQUIREMENTS ./
   RUN pip install -r $REQUIREMENTS
   

   ARG values are visible in places like COPY commands and RUN instructions, but not to the eventual container CMD. Note that they are embedded in the image and tools may be able to extract them, which means you should avoid passing credentials this way.

2. In the Compose file, declare build: { args: } either setting or passing through the variable. The syntax in the question is correct to pass it through from Compose's environment; repeating that

   services:
     web:
       build:
         context: ./web
         args:
           - REQUIREMENTS
   

3. Make the value visible in Compose's environment when you run it. You can do this with a normal environment variable on the host

   REQUIREMENTS=requirements_dev.txt docker-compose build
   

   but if you want to use the existing environment file, there's a Compose CLI option for it

   docker-compose --env-file envs/dev.env build
   

If this file only contains build-time settings, you don't necessarily need to include it in the Compose env_file:, and you may be able to delete that line or block entirely. Since the file will be loaded into Compose's outer environment, you can also use environment: to selectively pass through values or map them to different names if needed.

Answer 2

Your docker-compose.yml should look like this:

version: '2'
services:
  web:
    build:
      context: ./web
      args:
        REQUIREMENTS: "requirements_dev.txt"

Your Dockerfile should define the build-argument using ARG like this:

FROM python:3.5
ENV PYTHONUNBUFFERED 1
ENV APP_ROOT /usr/src/app
ARG REQUIREMENTS
...
COPY $REQUIREMENTS $APP_ROOT/
RUN pip install -r $APP_ROOT/$REQUIREMENTS

I validated this and created a minified functional demo at Github:

https://github.com/jannikweichert/stackoverflow-41747843