CODEWITHSUNDEEP
dockercopydockerfile
Luis
Luis

Reputation: 1293

Specify multiple files in ARG to COPY in Dockerfile

Consider the following docker build context:

src/
  hi
  there
  bye

and Dockerfile:

FROM ubuntu

RUN mkdir test
COPY src/hi src/there test/

This works just fine but I would like to make the list of files to copy an ARG, something like:

FROM ubuntu

ARG files

RUN mkdir test
COPY ${files} test/

Unfortunately calling with docker build --build-arg files='src/hi src/there' some_path fails because it treats src/hi src/there as a single item. How can I "expand" the files argument into multiple files to copy?

On a whim I tried specifying the files arg multiple times: docker build --build-arg files='src/hi' --build-arg files='src/there' some_path, but this only copies "there".

Upvotes: 7

Views: 3159

Answers (5)

Luis
Luis

Reputation: 1293

I have run into this again years later but now found an adequate solution, assuming the files can be grouped into a fixed set of categories.

The answer is basically combining the answers of Docker COPY files using glob pattern? and https://unix.stackexchange.com/questions/59112/preserve-directory-structure-when-moving-files-using-find

Conceptually, the idea is to separate the groups of files in one stage, then copy each group one by one (or ignoring it).

In code:

# Create dummy stage for splitting up files
FROM ubuntu AS src
RUN mkdir /groups
WORKDIR /groups
RUN mkdir group1 group2
# Makes `group2` the de-facto default, can be any though
COPY src group2
RUN rsync --recursive --remove-source-files --prune-empty-dirs \
    --include='hi' \
    --include='there' \
    --exclude='*' \
    group2/ group1/

# Create the image we actually care about
FROM ubuntu
COPY --from=src /groups/group1 test/
RUN commands requiring only `group1` files
COPY --from=src /groups/group2 test/
RUN commands requiring all files

Upvotes: 1

PeWu
PeWu

Reputation: 641

Although COPY doesn't accept a list of files in a variable, the cp command does. Here is a solution I used.

# Create an image with all files from src/ copied into it.
FROM ubuntu as builder

COPY src/ /src/

# Copy only the files specified in the `ARG` to a separate folder files_to_copy/
ARG files
RUN mkdir /files_to_copy && cp $files /files_to_copy/


# Create new image with files that were copied to files_to_copy/
FROM ubuntu

RUN mkdir src
COPY --from=builder /files_to_copy/* src/

Build like this:

docker build . --build-arg files='src/hi src/there'

Upvotes: 0

Ilyan
Ilyan

Reputation: 315

As a possible workaround, you can try to use .dockerignore file

*
!src/hi
!src/there

with

COPY . test/

https://docs.docker.com/engine/reference/builder/#dockerignore-file

Finally, you may want to specify which files to include in the context, rather than which to exclude. To achieve this, specify * as the first pattern, followed by one or more ! exception patterns.

Update:

Using wildcards in .dockerignore is limited because of bug: https://github.com/moby/moby/issues/30018

Upvotes: 1

Shaqil Ismail
Shaqil Ismail

Reputation: 1959

From the Docker documentation, for your information.

If you have multiple Dockerfile steps that use different files from your context, COPY them individually, rather than all at once. This ensures that each step’s build cache is only invalidated (forcing the step to be re-run) if the specifically required files change.

Upvotes: 0

VonC
VonC

Reputation: 1325137

because it treats src/hi src/there as a single item.
How can I "expand" the files argument into multiple files to copy?

That seems unlikely considering the Dockerfile format mentions, regarding arguments:

whitespace in instruction arguments, such as the commands following RUN, are preserved

And that is not limited to RUN.

COPY, however, also includes:

Each <src> may contain wildcards and matching will be done using Go’s filepath.Match rules.

It would not work in your case.

Remain the multi-stage builds approach

  • COPY src (the all folder)

  • Remove everything you don't want:

    RUN find pip ${files} -maxdepth 1 -mindepth 1 -print | xargs rm -rf

  • Build your actual image based on the resulting state of the first image.

You can pass as one argument the folders you want to preserve

docker build --build-arg="files=! -path "src/hi" ! -path "src/there"" .

See an example in "Docker COPY files using glob pattern?".

Upvotes: 2

Related Questions