CODEWITHSUNDEEP
dockerfilecommand-substitution
Thomas Sablik
Thomas Sablik

Reputation: 16447

How do I use command substition in Dockerfile

In my Dockerfile I need to use command substition to add some environment variables. I want to set

ENV PYTHONPATH /usr/local/$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')

but it doesn't work. The result is 

foo@bar:~$ echo $PYTHONPATH
/usr/local/$(python3 -c from distutils import sysconfig; print(sysconfig.get_python_lib()))

What's wrong?

Upvotes: 9

Views: 7477

Answers (3)

Kaz
Kaz

Reputation: 58617

What we can do is write a file Dockerfile.in instead of a Dockerfile, and then generate Dockefile from that. To do this, we can use a GNU Makefile like this:

%: %.in
        printf "cat <<__TEMPLATE_END\n" > [email protected]
        printf "# This file is GENERATED from Dockerfile.in!\n" >> [email protected]
        cat $< >> [email protected]
        printf "__TEMPLATE_END\n" >> [email protected]
        bash [email protected] > $@
        rm -f [email protected]

Now when we type make Dockerfile it will generate Dockerfile if it doesn't exist or is older than Dockerfile.in.

Inside Dockerfile.in we can use Bash "here document" syntax: we can freely refer to variables and command substitutions.

Here is an example Dockerfile.in:

FROM ubuntu:20.04

RUN apt-get update && \
    apt-get install -y vim && n\
    useradd -u $UID $USER && \
    usermod -G sudo -a $USER

# Define the command to run when the container starts
USER $UID
CMD ["/bin/bash", "-l"]

We have no issues injecting the $USER and $UID variables so that we create a matching user account in the container.

How the Makefile pattern rule works is that it takes the Dockerfile.in file and adds these lines around it:

cat << __TEMPLATE_END
# This file is GENERATED from Dockerfile.in!
[ ... contents of Dockerfile.in file go here verbatim ...]
__TEMPLATE_END

This is put into a file called Dockerfile.tmp which is executed as a script using bash Dockerfile.tmp, and the output is redirected into Dockerfile.

Problem solved: no messing around with user-hostile ARG, ENV and --build-arg ... nonsense.

Upvotes: 0

wisbucky
wisbucky

Reputation: 37923

You should use ARG if possible. But sometimes you really need to use command substitution for a dynamic variable. As long as you put all the commands in the same RUN statement, then you can still access the value.

RUN foo=$(date) && \
    echo $foo

Upvotes: 3

clarity123
clarity123

Reputation: 2046

What went wrong

The $( ... ) command substitution you attempted is for Bash, whereas the Dockerfile is not Bash. So docker doesn't know what to do with that, it's just plain text to docker, docker just spews out what you wrote as-is.

Recommendation

To avoid hard-coding values into a Dockerfile, and instead, to dynamically change a setting or custom variable as PYTHONPATH during the build, perhaps the ARG ... , --build-arg docker features might be most helpful, in combination with ENV ... to ensure it persists.

Within your Dockerfile:

ARG PYTHON_PATH_ARG

ENV PYTHONPATH ${PYTHON_PATH_ARG}

In Bash where you build your container:

python_path="/usr/local$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')"

docker build --build-arg PYTHON_PATH_ARG=$python_path .

Explanation

According to documentation, ARG:

The ARG instruction defines a variable that users can pass at build-time to the builder with the docker build command using the --build-arg <varname>=<value> flag.

So, in Bash we first:

python_path="/usr/local$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')"
  • $(...) Bash command substitution is used to dynamically put together a Python path value
  • this value is stored temporarily in a Bash variable $python_path for clarity
docker build --build-arg PYTHON_PATH_ARG=$python_path .
  • Bash variable $python_path value is passed to docker's --build-arg PYTHON_PATH_ARG

Within the Dockerfile:

ARG PYTHON_PATH_ARG
  • so PYTHON_PATH_ARG stores the value from --build-arg PYTHON_PATH_ARG...

ARG variables are not equivalent to ENV variables, so we couldn't merely do ARG PYTHONPATH and be done with it. According to documentation about Using arg variables:

ARG variables are not persisted into the built image as ENV variables are.

So finally:

ENV PYTHONPATH ${PYTHON_PATH_ARG}
  • We use Dockerfile's ${...} convention to get the value of PYTHON_PATH_ARG, and save it to your originally named PYTHONPATH environment variable

Differences from original code

You originally wrote:

ENV PYTHONPATH /usr/local/$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')

I re-wrote the Python path finding portion as a Bash command, and tested on my machine:

$ python_path="/usr/local/$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')"

$ echo $python_path
/usr/local//usr/lib/python3/dist-packages

Notice there is a double forward slash ... local//usr ... , not sure if that will break anything for you, depends on how you use it in your code.

Instead, I changed it to:

$ python_path="/usr/local$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')"

Result:

$ echo $python_path
/usr/local/usr/lib/python3/dist-packages

So this new code will have no double forward slashes.

Upvotes: 5

Related Questions