CODEWITHSUNDEEP
pythondockerpipdockerfile
nmoses
nmoses

Reputation: 63

Docker build fails if pip is installed and updated to 10.0.1 in a single RUN section

How would you explain docker build failure with Dockerfile1, and it's success with Dockerfile2 (see below).

1) 

// Dockerfile1

FROM ubuntu:16.04

RUN apt-get -y update && \
    apt-get -y install python-pip python-dev build-essential && \
    pip install --upgrade pip && \
    pip install --upgrade virtualenv

docker build . fails with the following err

Collecting pip
Downloading 
https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl (1.3MB)
Installing collected packages: pip
Found existing installation: pip 8.1.1
Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside 
environment /usr
Successfully installed pip-10.0.1
Traceback (most recent call last):
File "/usr/bin/pip", line 9, in <module>
   from pip import main
ImportError: cannot import name main
The command '/bin/sh -c apt-get -y update &&     apt-get -y install 
python-pip python-dev build-essential &&     pip install --upgrade pip &&     pip install --upgrade virtualenv &&     virtualenv /venv' returned a non-zero code: 1

However, it succeeds if we split it into two RUN.

2) 

// Dockerfile2

FROM ubuntu:16.04

RUN apt-get -y update && \
    apt-get -y install python-pip python-dev build-essential && \
    pip install --upgrade pip

RUN pip install --upgrade virtualenv

The installation failure for pip is related to this reported issue. So my questions:

  1. Why does docker build fail in the first case? If we just run those command in bash, there wont be any error.
  2. Why does docker build succeed in the second case? How is it related to layering concept in docker?
  3. Why specifying pip version in Dockerfile1 (i.e. pip install --upgrade pip=0.9.3) solves the problem too?

Update (May 6, 2018):

I've figured out the issue. What happens here is as below:

  1. apt-get -y install python-pip installs an old version of pip whose shim script import pip's main directly.

  2. pip install --upgrade pip installs pip 10.0.1 and moves main into an internal directory _internal. It adds its shim script to PATH.

  3. Calling pip fails as it still calls the old shim script as it's path is cached. Running hash -d pip in between fixes the issue.

So apparently, splitting install and update into two RUN sections has similar effect as hash -d pip. Workarounds (also suggested by Andriy Maletsky) are 1) pin pip update to 9.0.3, or 2) install (latest) pip from source in the first place, or 3) use hash -r in between, or 4) use another RUN command for later use of pip.

Upvotes: 0

Views: 1461

Answers (1)

Andrii Maletskyi
Andrii Maletskyi

Reputation: 2232

The problem is that pip executable (/usr/bin/pip) breaks while updating pip from version 9 to version 10.

Possible solutions:
1. Do not update and use pip v9
2. Do not use apt-get to install pip. Download it manually.

Why does docker build fail in the first case? If we just run those command in bash, there wont be any error.

No, there will be an error. I ran those commands inside docker run --rm -it ubuntu:16.04 bash and got it.

Why does docker build succeed in the second case? How is it related to layering concept in docker?

I believe you made a mistake somewhere in second RUN and it's silencing an error (in that place which you didn't provide). For example, this will work (because ; used instead of && and execution doesn't break after bad command):

RUN pip install --upgrade virtualenv && \
virtualenv /venv; source /venv/bin/activate

Why specifying pip version in Dockerfile1 (i.e. pip install --upgrade pip=0.9.3) solves the problem too?

Because this pip bug appeared in version 10.

P.S. You should not update or manually change files you added to your system via apt-get (you are doing this via pip install --upgrade pip).

Upvotes: 2

Related Questions