how to merge Docker's layers of image and slim down the image file

Question

docker image inspect <name> gives me 16GB

and about 20 layers

When I am logged as root, this du -hs / show me just 2GB

FYI, there are already very multi-lines RUN commands in Dockerfile.

can I squash all layers into one layer without touching Dockerfile, rebuilding etc?

or possibly by adding extra action to Dockerfile which clear/improve caching

Dockerfile is

FROM heroku/heroku:18

ENV PYENV_ROOT="/pyenv"
ENV PATH="/pyenv/shims:/pyenv/bin:$PATH"
ENV PYTHON_VERSION 3.5.6
ENV GPG_KEY <value>
ENV PYTHONUNBUFFERED 1
ENV TERM xterm
ENV EDITOR vim

RUN apt-get update && apt-get install -y \
    build-essential \
    gdal-bin \
    binutils \
    iputils-ping \
    libjpeg8 \
    libproj-dev \
    libjpeg8-dev \
    libtiff-dev \
    zlib1g-dev \
    libfreetype6-dev \
    liblcms2-dev \
    libxml2-dev \
    libxslt1-dev \
    libssl-dev \
    libncurses5-dev \
    virtualenv \
    python-pip \
    python3-pip \
    python-dev \
    libmysqlclient-dev \
    mysql-client-5.7 \
    libpq-dev \
    libcurl4-gnutls-dev \
    libgnutls28-dev \
    libbz2-dev \
    tig \
    git \
    vim \
    nano \
    tmux \
    tmuxinator \
    fish \
    sudo \
    libnet-ifconfig-wrapper-perl \
    ruby \
    libssl-dev \
    nodejs \
    strace \
    tcpdump \
    # npm & grunt
    && curl -L https://npmjs.com/install.sh | sh \
    && npm install -g grunt-cli grunt \
    # ruby & foreman
    && gem install foreman \
    # installing pyenv
    && curl https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash

COPY . /app
COPY ./requirements /requirements
COPY ./requirements.txt /requirements.txt
COPY ./docker/docker_compose/django/foreman.sh /foreman.sh
COPY ./docker/docker_compose/django/Procfile /Procfile
COPY ./docker/docker_compose/django/entrypoint.sh /entrypoint.sh

# ADD sudoer user django with password django
RUN groupadd -r django -g 1000 && \
    useradd -ms /usr/bin/fish -p $(openssl passwd -1 django) --uid 1000 --gid 1000 -r -g django django && \
    usermod -a -G sudo django && \
    chown -R django:django /app 

COPY --chown=django:django ./docker/docker_compose/django/fish /home/django/.config/fish
COPY --chown=django:django ./docker/docker_compose/django/tmuxinator /home/django/.tmuxinator
COPY ./docker/docker_compose/django/fish /root/.config/fish

WORKDIR /app

RUN sed -i 's/\r//' /entrypoint.sh \
    && sed -i 's/\r//' /foreman.sh \
    && chmod +x /entrypoint.sh \
    && chown django /entrypoint.sh \
    && chmod +x /foreman.sh \
    && chown django /foreman.sh \
    && chown -R django:django /home/django/ \
    && pyenv install ${PYTHON_VERSION%%} \
    && mkdir -p /app/log \
    && pyenv global ${PYTHON_VERSION%%} \
    && pyenv rehash \
    && ${PYENV_ROOT%%}/versions/${PYTHON_VERSION%%}/bin/pip install -U pip \
    && ${PYENV_ROOT%%}/versions/${PYTHON_VERSION%%}/bin/pip install -r /requirements.txt \
    && chown -R django:django /pyenv/ \
    && ${PYENV_ROOT%%}/versions/${PYTHON_VERSION%%}/bin/pip install -r /requirements/dev_requirements.txt

# this user receives ENVs from the top
USER django

ENTRYPOINT ["/entrypoint.sh"]

---

What I've tried so far:

The --squash option from experimental mode of docker build is rather not for me. That Dockerfile is one of more Dockerfiles inside docker-compose.

I've also checked this: https://github.com/jwilder/docker-squash

but seems docker load cannot load a squashed image. also, that squash gives me 8GB (still far away from expected ~2GB)

docker save <image_id> | docker-squash -t latest_tiny | docker load

---

update after answers:

when I've added this:

&& apt-get autoremove \             # ? to consider
&& apt-get clean \                  # ? to consider
&& rm -rf /var/lib/apt/lists/*

to apt-get and --no-cache-dir to each pip, the result was 72GB (yes, even much more - docker images shows 36GB before pip command, and 72GB as final size).

my working directory is clear (regarding COPY). du -hs / (as a root) still has 2GB. And all images were removed before rebuilding.

Following the @Mihai approach, I was able to slim down the image from 16GB to 9GB.

Answer 1

To squash a (Docker) container image, without re-building the image or manipulating the original Dockerfile,

You can extend from your image and squash it:

docker build --squash -t your_image_squashed - <<< "FROM your_image"

As --squash is obsolete or maybe even removed in future versions, you can also do:

docker build -t your_image_squashed - <<< "FROM scratch
COPY --from=your_image / /"

Answer 2

It's very easy, just use

docker commit YOUR_CONTAINER_ID NEW_IMAGE_ID

The docker will throw away the intermediate layers, you lost history but the size is small

Answer 3

Definitely add && rm -rf /var/lib/apt/lists/* on the end of your main run command, like Mihai said. Another thing that may help (depending on how big your dependencies are) is installing with pip using the --no-cache-dir option . Also, make sure you understand build context and consider using either a .dockerignore or sending the context to another directory (totally depends on how you're directory is setup)

I've also had luck exploring an image using dive. Honestly this looks like a pretty big image so not sure how much you're going to be able to get it down

Answer 4

There is a simple trick to get rid of the intermediate layers. It will bring down the size as well but with how much depends on how it was built.

Create a Dockerfile like this:

FROM your_image as initial

FROM your_image_base

COPY --from=initial / /

your_image_base should be something like 'alpine' - so the smallest image from which your image and its parents descend from.

Now build the image and check the history and size:

docker build -t your-image:2.0 .
docker image history your-image:2.0
docker image ls

This way you do create a new Dockerfile (if that is acceptable for your process) without touching the initial Dockerfile.

Let me know if this solves your issue.

UPDATE AFTER SEEING THE Dockerfile:

maybe I miss it but I don't see you cleaning up the apt-get cache after you perform the installations. Your big RUN command should end with "&& rm -rf /var/lib/apt/lists/*" on the same line so that it doesn't store the whole cache on the layer.