CODEWITHSUNDEEP
pythondockeranacondaconda
DCS
DCS

Reputation: 3394

Docker and Conda: Differences when building the same container on Mac and on Ubuntu

I'm using to Docker to build a Python container with the intention of having a reproducible environment on several machines, which are a bunch of development Macbooks and several AWS EC2 servers.

The container is based on continuumio/miniconda3, i.e. Dockerfile starts with 

FROM continuumio/miniconda3

A few days ago on Ubuntu the conda install and conda upgrade commands in the Docker file complained about a new Conda version (4.11) being available:

==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.4.11

  Please update conda by running

  $ conda update -n base conda

If I ignore this, package installations quit with an error:

Downloading and Extracting Packages
The command '/bin/sh -c conda install -y pandas=0.22.0 matplotlib 
scikit-learn=0.19.1 pathos lazy openpyxl pytables dill pydro psycopg2 
sqlalchemy pyarrow arrow-cpp parquet-cpp scipy tensorflow keras 
xgboost' returned a non-zero code: 1

When I add this conda update... to the Docker file, things work again.

What's really annoying, however, is that the update that makes things run in Ubuntu does not work on Mac Docker. I get the following error:

CondaEnvironmentNotFoundError: Could not find environment: base .
You can list all discoverable environments with `conda info --envs`.

Note that I get this error when I docker build the same Docker file that works on the Ubuntu machine, which kind of ruins the whole point about using Docker in the first place. On the Mac, the old version of the file (without conda update -n base conda) still runs fine and installs all packages.

Docker / Conda experts, any ideas?

Edit: Here's the full Dockerfile (the one that works in Ubuntu):

# Use an official Python runtime as a parent image
FROM continuumio/miniconda3

WORKDIR /app/dev/predictive.analytics

RUN apt-get update; \
apt-get install -y gcc tmux htop

RUN conda update -y -n base conda

RUN conda config --add channels babbel; \
conda config --add channels conda-forge; 

RUN conda install -y pandas=0.22.0 matplotlib scikit-learn=0.19.1 pathos lazy openpyxl pytables dill pydro psycopg2 sqlalchemy pyarrow arrow-cpp parquet-cpp scipy tensorflow keras xgboost
RUN pip install recordclass sultan
RUN conda upgrade -y python 

ENV DATA_DIR /host/data
ENV PYTHONPATH /host/predictive.analytics/python
ENV PATH="/host/predictive.analytics:${PATH}"

Upvotes: 2

Views: 2056

Answers (1)

Lie Ryan
Lie Ryan

Reputation: 64913

Perhaps you're using an outdated miniconda on one of the build machine, try doing docker build --pull --no-cache.

Docker doesn't necessarily pull the latest image from the repository, so unless you do a --pull, it is possible that some of your machines may be starting the build with outdated base image.

Upvotes: 2

Related Questions