CODEWITHSUNDEEP
docker
Crystal
Crystal

Reputation: 29458

build context for docker image very large

I have created a couple different directories on my host machine as I try to learn about Docker just to keep my dockerfiles organized. My Dockerfile I just ran looks like this:

FROM crystal/centos
MAINTAINER crystal

ADD ./rpms/test.rpm ./rpms/ 
RUN yum -y --nogpgcheck localinstall /rpms/test.rpm

My actual rpm is only 1 GB. But when I try to do sudo docker build -t="crystal/test" ., I get sending build context to Docker daemon 3.5 GB. Is there something else that I'm unaware of as you continue to build Docker images? Is my memory accumulating as I build more images in my other directories on my host machine?

Upvotes: 317

Views: 220937

Answers (13)

NoodleArms
NoodleArms

Reputation: 1

Make sure your context is the path to your request directory and not all your PC disk. while using docker compose:

...
build:
  context: ${HOME}/<path to request directory>
  dockerfile: ${HOME}/<path to Dockerfile>
...

Upvotes: 0

samneric
samneric

Reputation: 3218

An option to troubleshoot large context problems caused by inadequate .dockerignore exclusions is to use this command:

rg -uuu --ignore-file .dockerignore --files --sort path .

Which uses this tool: https://github.com/BurntSushi/ripgrep

It works really well.

Upvotes: 22

kaznovac
kaznovac

Reputation: 1572

The great tool to inspect the built docker image is dive https://github.com/wagoodman/dive

Here're the instructions from the official readme:

To analyze a Docker image simply run dive with an image tag/id/digest:

dive <your-image-tag>

or if you want to build your image then jump straight into analyzing it:

dive build -t <some-tag> .

Upvotes: 3

Alexander Salykin
Alexander Salykin

Reputation: 163

Just to summarize what you can do if your Docker image build context too large:

  • make sure you don't have unused files in the context (unused files - those that still untouchable during image building);
  • add unused files and/or directories to .dockerignore (as mentioned here);
  • make sure you specify the right folder as an image build context (as mentioned here);
  • try to use BuildKit with DOCKER_BUILDKIT=1 (as mentioned here);
  • try to split your project into smaller parts to optimize content of build context;
  • if you have a complex project you may want to manually collect all necessary files in separate folder before build and use it as a build context for a specific image.

Upvotes: 5

Thomas Orozco
Thomas Orozco

Reputation: 55197

The Docker client sends the entire "build context" to the Docker daemon. That build context (by default) is the entire directory the Dockerfile is in (so, the entire rpms tree).

You can setup a .dockerignore file to get Docker to ignore some files. You might want to experiment with it.

Alternatively, you can move your rpms folder one directory level above your Dockerfile, and only symlink test.rpm into the Dockerfile's directory.

You’ll often want to add the .git folder to the .dockerignore which was the cause of a 150MB -> 5GB difference for some users in the comments here.

Upvotes: 544

Bernard Nongpoh
Bernard Nongpoh

Reputation: 1058

For NodeJS Application, add a .dockerignore file your root project directory and inside the .dockerignore file add the following

node_modules
dist

Upvotes: 6

Shashikant Pandit
Shashikant Pandit

Reputation: 2824

if you are creating image and getting message sending build context to docker daemon which is taking log time to copy,

then add .dockerignore file. it should include the files or directory which does not need to be copied.

Upvotes: 2

Christian.D
Christian.D

Reputation: 949

If you want to be in full control of your build context you could also build the container completely without any context and COPY relevant data into the container afterwards.

docker build - < Dockerfile

One downside of this would be that with this approach you can only ADD things in the dockerfile referencing to a remote URL, and not files from your local host.

See https://docs.docker.com/engine/reference/commandline/build/#build-with--

Upvotes: 4

Andriy Berestovskyy
Andriy Berestovskyy

Reputation: 8534

Update 2019

Starting from Docker v18.06 there is an option to use a new image builder called Build Kit.

It's pre-bundled with the Docker, no need to install anything. It's backward compatible with the Dockerfile syntax, no need to change the Dockerfile.

Legacy Docker Build vs New Docker BuildKit

Here is an example of building an image with a huge unused file in the build directory:

Legacy Docker Build:

$ time docker image build --no-cache .
Sending build context to Docker daemon  4.315GB
[...]
Successfully built c9ec5d33e12e

real    0m51.035s
user    0m7.189s
sys 0m10.712s

New Docker BuildKit:

$ time DOCKER_BUILDKIT=1 docker image build --no-cache .
[+] Building 0.1s (5/5) FINISHED                                                
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 37B                                        0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
[...]
 => => writing image sha256:ba5bca3a525ac97573b2e1d3cb936ad50cf8129eedfa9  0.0s

real    0m0.166s
user    0m0.034s
sys 0m0.026s

The only change is the DOCKER_BUILDKIT=1 environment variable, the difference in time is huge.

.dockerignore File

Please note, that the .dockerignore file is still valid and useful. Some Dockerfile commands like COPY . . will still take into account the .dockerignore rules. But the side files in the build directory (not referenced in the Dockerfile) are not getting copied anymore as a "build context" by the BuildKit.

Upvotes: 109

Lu&#237;s Bianchin
Lu&#237;s Bianchin

Reputation: 2486

If you have a .dockerignore file and build context is still large, you can check what is being sent to the docker build context using The Silver Searcher:

ag --path-to-ignore .dockerignore --files-with-matches

Note that some ** patterns might not work properly.

See this Github issue for additional comments: https://github.com/moby/moby/issues/16056

Upvotes: 23

BravoRomeo23
BravoRomeo23

Reputation: 11

I had the same issue as FreeStyler. However I was building from a directory one up from my context. So the -f arguments were correct the context was incorrect.

project 
|
-------docker-dir

Building from the docker-dir the following was fine

docker build -t br_base:0.1 .

Building from the dock-dir the the build context changed. Therefore I needed to change the context in the command. Context is given by the '.' in the command above.

The the new command from the project directory should be

docker build -t br_base:0.1 ./base

Context here is given by the './base'

Upvotes: 1

FreeStyler
FreeStyler

Reputation: 317

In my case that was when i execute with wrong -f arguments - without path to directory where located Dockerfile

docker build --no-cache -t nginx5 -f /home/DF/Dockerfile /home/DF/ - right

docker build --no-cache -t nginx5 -f /home/DF/Dockerfile - wrong

Upvotes: 7

Emad
Emad

Reputation: 4200

I fixed it by moving my Dockerfile and docker-compose.yml into a subfolder and it worked great. Apparently docker sends the current folder to the daemon and my folder was 9 gigs.

Upvotes: 28

Related Questions