Build of Dockerfile hangs or taking long?

Question

I have this Dockerfile:

FROM fedora:28

# create new user
RUN adduser --home /home/rpi-user rpi-user

# install things needed for yocto
RUN dnf -y install make wget bzip2 python unzip perl patch \
     diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath \
     ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue socat \
     findutils which SDL-devel xterm cpio file hostname rpcgen

# run the commands that follow as rpi-user
USER rpi-user

# clone yocto/poky and raspberrypi layer
RUN git clone git://git.yoctoproject.org/poky /home/rpi-user/poky
RUN cd /home/rpi-user/poky; git clone https://github.com/agherzan/meta-raspberrypi.git

# set up oe build environment and add meta-raspberrypi to bblayers.conf
# modify machine name and add options to local.conf
WORKDIR /home/rpi-user/poky
ENV LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8
COPY local.conf.append /tmp/
RUN ls -l
RUN . ./oe-init-build-env rpi-build; \
        cd conf; perl -0777 -pi -e 's/(BBLAYERS \?= "Provided more updates.[^"]*)"/\1\/home\/rpi-user\/poky\/meta-raspberrypi \\\n  "/g' bblayers.conf; \
        sed -i 's/\(MACHINE ??= \).*$/\1"raspberrypi3"/g' local.conf; \
        cat /tmp/local.conf.append >> local.conf; \
        cd ../; bitbake -DDDv rpi-basic-image;
RUN echo "[!] Build complete."

Doing a docker build -t bla/rpi . results in my terminal hanging with (the last 6 lines):

NOTE: Running noexec task 3294 of 3294 (/home/rpi-user/poky/meta-raspberrypi/recipes-core/images/rpi-basic-image.bb:do_build)
DEBUG: Teardown for bitbake-worker
DEBUG: Teardown for bitbake-worker
NOTE: Tasks Summary: Attempted 3294 tasks of which 5 didn't need to be rerun and all succeeded.

Summary: There were 5 WARNING messages shown.

Running the exact same commands directly on a terminal on the same machine, I am able to build an image without a hang. Only when using docker (and only this specific Dockerfile) does my build hang. I am almost certain it is memory related.

I usually use a script to remove any docker left overs which basically consists of a docker rmi <images>, docker rm <containers> and docker system prune --volumes -f, here it is if interested. Using said script did not fix my issue (removing of containers, images and volumes did not fix the hang). I then removed /var/lib/docker and reset docker daemon, but still no luck.

Any recommendations?

Update

Looking at docker stats while docker was building my Dockerfile, I saw that my cpu usage percentage was constantly over %100 (like in the thousands!). A dmesg -w shows messages like this:

[ 4409.558822] mce: CPU1: Core temperature above threshold, cpu clock throttled (total events = 3471)
[ 4409.558823] mce: CPU7: Core temperature above threshold, cpu clock throttled (total events = 3471)
[ 4409.558825] mce: CPU0: Package temperature above threshold, cpu clock throttled (total events = 4603)
[ 4409.558826] mce: CPU6: Package temperature above threshold, cpu clock throttled (total events = 4603)

What is even more interesting is that with docker stats running, when the docker build hangs, the entry for that docker/image in docker stats goes away, and dmesg shows:

[ 4701.032315] docker0: port 1(veth0ef04d2) entered disabled state
[ 4701.032481] veth2578e04: renamed from eth0
[ 4701.130810] docker0: port 1(veth0ef04d2) entered disabled state
[ 4701.135113] device veth0ef04d2 left promiscuous mode
[ 4701.135137] docker0: port 1(veth0ef04d2) entered disabled state

Which I assume is what I should see upon the exiting of a docker build? Running a ps aux | grep -i docker, I still see the process running:

<user>    6281  0.9  0.1 1961860 27800 pts/1   Sl+  09:17   0:35 docker -D build -t bam/rpi .

So it seems docker thinks the build is done (docker stats removes the respective entry) but linux still sees the docker build running (ps aux still shows the process). What the?

I am not able to get more "deets" from bitbake, unless there is a flag I am missing. docker stats did not show any memory issues as I was expecting.

Wow, apparently a docker ps -a shows the container as exited also (in agreement with docker stats):

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS               NAMES
3ea12a369089        22be583f7349        "/bin/sh -c '. ./oe-…"   About an hour ago   Exited (0) 13 minutes ago                       nifty_gauss

I thought it couldn't get more interesting, but it did. In my ps aux | grep -i docker, I see an interesting process:

root      7775 19.2  1.1 2170612 181216 ?      Sl   10:13   4:04 docker-untar /var/lib/docker/overlay2/f3aade16d0d44ef61037ca5fc7ac0690f66b1894578141add2197fe7f8cdd402/diff

What is that? Is it untaring something? That can take a while. Maybe I should wait some more for this hang (I will). This process' stat column is constantly changing from Rl to Sl and back. Weird.

So I waited. The docker-untar process is now gone, but the build is still hanging. Now when I do a docker ps -a, i see this:

3ea12a369089        22be583f7349        "/bin/sh -c '. ./oe-…"   2 hours ago         Removal In Progress                           nifty_gauss

Interesting, will wait some more.

The waiting paid off. The docker image has, rather clumsily, successfully built.

Why is it taking so long using docker (about x4 longer than running on the same machine without docker) to build an arm based binary?

Answer 1

The reason my docker build was taking much more time than executing the commands on the host directly is because after the RUN that executes the bitbake command, I am not cleaning up my layer!

As you may know, each RUN command is a layer, and each layer is auto-cached by docker to make use of later if no changes occurred. That means docker has to tar, remove and cache my layer including all the stuff that was cloned and all the outputs/artifacts of running bitbake. So rather than have docker struggle with all this unwanted stuff, I removed it manually:

RUN . ./oe-init-build-env rpi-build && \
        cd conf && perl -0777 -pi -e 's/(BBLAYERS \?= "[^"]*)"/\1\/home\/rpi-user\/poky\/meta-raspberrypi \\\n  "/g' bblayers.conf && \
        sed -i 's/\(MACHINE ??= \).*$/\1"raspberrypi3"/g' local.conf && \
        cat /tmp/local.conf.append >> local.conf && \
        cd ../ && bitbake -DDDDvvv rpi-basic-image && mv tmp/deploy /home/rpi-user/ && rm -rf *

Note the last two bash commands in this RUN (the mv and the rm).

One could also remove the clones.