CODEWITHSUNDEEP
dockerdocker-composedockerfile
OzrenTkalcecKrznaric
OzrenTkalcecKrznaric

Reputation: 5646

Docker, docker-compose and copying files between projects

I have the following setup.

Folder structure

solution-root
    ├── docker-compose.yml
    ├── project1
    │   ├── Dockerfile
    │   ├── sub1
    │   │   ├── ...loads of stuff...
    │   ├── sub2
    │   │   ├── ...more stuff...
    ├── project2
    │   ├── Dockerfile
    │   ├── sub1
    │   │   ├── ...more stuff...
    │   ├── sub2
    │   │   ├── ...even more stuff...
    ├── project-db
    │   ├── Dockerfile

docker-compose.yml

version: '3'

services:
  project1:
    build:
      context: ./project1
      dockerfile: Dockerfile
    ...
  project2:
    build:
      context: ./project2
      dockerfile: Dockerfile
    ...
  project-db:
    build:
      context: ./project-db
      dockerfile: Dockerfile
    ...
...

project-db/Dockerfile

FROM mysql:5.7

COPY ../project1/app/seeders /seeders/
COPY ../project2/app/seeders /seeders/

Obviously, I want to copy files from another sibling folder because this project-db needs them.

So, when I run docker-compose build I am presented with this error:

Service 'project-db' failed to build: COPY failed: Forbidden path outside the build context: ../project1/app/seeders

Ok, I get it, context does not allow me to level up. Let's move context to root then and then run project/Dockerfile from there.

docker-compose.yml

  project-db:
    build:
      context: .
      dockerfile: ./project-db/Dockerfile
    ...

Now we can copy files that we need.

project-db/Dockerfile

COPY project1/app/seeders /seeders/
COPY project2/app/seeders /seeders/

And now all is well(ish) with docker-compose build. BUT there is a problem - building project-db lasts quite some time. And that means every time it's being run. I guess that it's due to the fact that now the context of the project-db is the entire folder structure.

So, I tried with .dockerignore to filter out unneeded folders:

.dockerignore

project3
project3/**
project4
project4/**
project5
project5/**
...

But nothing removes that lag.

I can't get this to work properly. Also - I can't fiddle with the internal structures of the existing projects.

What is wrong here?

Upvotes: 3

Views: 4317

Answers (1)

agentsmith
agentsmith

Reputation: 1326

As the author correctly pointed out, volumes are used for persisting data. Here I want to show two solutions, on how to use them for sharing data between container. This solution is far from being perfect!

Solution 1

Downsides

First, I want to point out the downsides of this solution.

  • You need to clean up the volumes. Volumes only get populated with a container's content at first creation. See here for an explanation. Because of that, docker-compose down -v must be done, if some files in the project-directories have changed.
  • An alternative way to docker-compose down -v`` is to manually delete the named volumes using docker volume rm ``.
  • If you don't not want this, you can temporally COPY the files to a folder (which is not a mounted volume, e. g. /tmp). Using an entrypoint script you can than copy the files to its intended position (e. g. /home/developer/). See Solution 2 for this.

My setup: Folder-Structure

My folder structure looks similar to yours:

├── docker-compose.yaml
├── project1
│   ├── Dockerfile
│   ├── entrypoint.sh
│   ├── sub1
│   │   ├── testfile_project_1_1.txt
│   │   └── testfile_project_1_2.txt
│   └── sub2
│       └── testfile_project_1_3.txt
├── project2
│   ├── Dockerfile
│   ├── entrypoint.sh
│   ├── sub1
│   │   └── testfile_project_2_1.txt
│   └── sub2
│       ├── testfile_project_2_2.txt
│       ├── testfile_project_2_3.txt
│       ├── testfile_project_2_4.txt
│       └── testfile_project_2_5.txt
└── project-db
    ├── Dockerfile
    └── entrypoint.sh

Sources

docker-compose.yaml

version: "3.8"
services:
  first-service:
    build: 
      context: ./project1
      dockerfile: Dockerfile
    volumes:       
      - data-first-service:/home/developer/

  second-service:
    build: 
      context: ./project2
      dockerfile: Dockerfile
    volumes:       
      - data-second-service:/home/developer/

  databse-service:
    build: 
      context: ./project-db
      dockerfile: Dockerfile
    volumes:       
      - data-first-service:/home/developer/project1/
      - data-second-service:/home/developer/project2/
    depends_on: 
      - first-service
      - second-service
 
volumes: 
  data-first-service:  
  data-second-service:

Dockerfile(s)

They are pretty much the same. The dockerfile for the* db-service* only copies it's entrypoint-script. The part with the sudoers is here, because this is my default testing image. I just included it to make clear which permissions my user has and to make passwordless sudo with regular user possible. It is not mandatory.

FROM ubuntu:latest
# We need some tools
RUN apt-get update && apt-get install -y sudo
# We want to have another user than `root`
## USER SETUP 
RUN adduser developer
# We want to have passwordless sudo access
RUN \
    sed -i /etc/sudoers -re 's/^%sudo.*/%sudo ALL=(ALL:ALL) NOPASSWD: ALL/g' && \
    sed -i /etc/sudoers -re 's/^root.*/root ALL=(ALL:ALL) NOPASSWD: ALL/g' && \
    sed -i /etc/sudoers -re 's/^#includedir.*/## **Removed the include directive** ##"/g' && \
    echo "developer ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers;  su - developer -c id

# Run now with user developer
USER developer
COPY sub1 /home/developer/sub1
COPY sub2 /home/developer/sub2
RUN ls -l

ADD ./entrypoint.sh /entrypoint.sh
RUN sudo chmod +x /entrypoint.sh
ENTRYPOINT [ "/entrypoint.sh" ]

entrypoint.sh

The entrypoints for the two projects are not special, they just contain a simple ls -l to /home/developer. The entrypoint for the* db-service* just shows a tree output (see Results):

#!/bin/bash
cd ~
echo "db service - You are here: ${PWD} "
tree --du -shaC | grep -Ev '(  *[^ ]* ){5}\['

Result

As you can see, the project-db container now contains the files from the other two projects.

databse-service_1  | .
databse-service_1  | |-- [ 220]  .bash_logout
databse-service_1  | |-- [3.7K]  .bashrc
databse-service_1  | |-- [ 807]  .profile
databse-service_1  | |-- [ 17K]  project1
databse-service_1  | |   |-- [ 220]  .bash_logout
databse-service_1  | |   |-- [3.7K]  .bashrc
databse-service_1  | |   |-- [ 807]  .profile
databse-service_1  | |   |-- [4.0K]  sub1
databse-service_1  | |   |   |-- [   0]  testfile_project_1_1.txt
databse-service_1  | |   |   `-- [   0]  testfile_project_1_2.txt
databse-service_1  | |   `-- [4.0K]  sub2
databse-service_1  | |       `-- [   0]  testfile_project_1_3.txt
databse-service_1  | `-- [ 17K]  project2
databse-service_1  |     |-- [ 220]  .bash_logout
databse-service_1  |     |-- [3.7K]  .bashrc
databse-service_1  |     |-- [ 807]  .profile
databse-service_1  |     |-- [4.0K]  sub1
databse-service_1  |     |   `-- [   0]  testfile_project_2_1.txt
databse-service_1  |     `-- [4.0K]  sub2
databse-service_1  |         |-- [   0]  testfile_project_2_2.txt
databse-service_1  |         |-- [   0]  testfile_project_2_3.txt
databse-service_1  |         |-- [   0]  testfile_project_2_4.txt
databse-service_1  |         `-- [   0]  testfile_project_2_5.txt
databse-service_1  | 
databse-service_1  |   42K used in 6 directories, 17 files

How to use

As said, this method has some downside. You need to class a docker-compose down in order to make this solution work. So, the workflow looks similar to this: docker-compose build && docker-compose up. If you change a file in one of the project-directories or if you want to update the content ud must call docker-compose down -v, otherwise it will still reuse the pre-populated content from the old volumes.

Solution 2

Basically, it’s the same as Solution 1. The difference is, that the “project-containers” first cop the sources to a temporal location and after the container has been started (and the volume is mounted) to the path where the volume is mounted.

Dockerfile

Just minor changed for this solution

[...]
# Run now with user developer
USER developer
COPY sub1 /tmp/sub1
COPY sub2 /tmp/sub2
RUN ls -l

ADD ./entrypoint.sh /entrypoint.sh
RUN sudo chmod +x /entrypoint.sh
ENTRYPOINT [ "/entrypoint.sh" ]

and the entrypoint looks loke this

#!/bin/bash
mv /tmp/sub1 /home/developer/sub1
mv /tmp/sub2 /home/developer/sub1

# Then do your stuff

Upvotes: 1

Related Questions