CODEWITHSUNDEEP
dockeranacondadockerfile
sanjayr
sanjayr

Reputation: 1949

Dockerfile: How to source Anaconda

I am trying to set up an Anaconda environment with AWS Batch

Here is a snippet from my Dockerfile

#: Download Anaconda
COPY Anaconda3-2019.10-Linux-x86_64.sh /setup/
RUN bash Anaconda3-2019.10-Linux-x86_64.sh -b -p /home/ec2-user/anaconda3 
RUN echo 'export PATH=$PATH:/home/ec2-user/anaconda3/bin' >>~/.bashrc \
    && /bin/bash -c "source ~/.bashrc"
ENV PATH $PATH:/home/ec2-user/anaconda3/bin

# Create the environment:
COPY environment.yml .
RUN conda env create -f environment.yml

# Make RUN commands use the new environment:
SHELL ["conda", "run", "-n", "solver_env", "/bin/bash", "-c"]

When I access my container, and do some simple tests I get an error that

conda: command not found

Even though I have sourced the path to my bashrc file.

Also when trying to access my bashrc file I get

cat ~.bashrc: No such file or directory

and

cat: //.bashrc: No such file or directory

Any ideas on where I may be going wrong/where to check?

EDIT:

Last line in my Dockerfile to activate the venv and kick off the script is:

ENTRYPOINT ["conda", "run", "-n", "solver_env", "/bin/bash", "/usr/local/bin/fetch_and_run.sh"]

Once I have the image built and deployed to AWS ECR, I kick off a Batch job which essentially runs this shell script:

#!/bin/bash
date
echo "Args: $@"
env
echo "script_path: $1"
echo "script_name: $2"
echo "path_prefix: $3"
echo "jobID: $AWS_BATCH_JOB_ID"
echo "jobQueue: $AWS_BATCH_JQ_NAME"
echo "computeEnvironment: $AWS_BATCH_CE_NAME"
   
mkdir /tmp/scripts/
aws s3 cp $1 /tmp/scripts/$2
python3 /tmp/scripts/${@:2}

Upvotes: 0

Views: 1233

Answers (3)

Konrad Botor
Konrad Botor

Reputation: 5063

You seem to have forgotten a slash in your command:

cat ~.bashrc: No such file or directory

It should be:

cat ~/.bashrc

Also everything you do with .bashrc in your Dockerfile is pointless. Not only, as @Itamar states in his answer is .bashrc only executed in interactive non-login shells, but every RUN command uses its own Bash process and you even use a separate process to source it manually, which essentially means you start Bash, source .bashrc and then discard that instance of Bash, discarding all changes with it.

I have answered a similar question on SuperUser and the solution applies here as well. It's actually simpler, since you want to use the Docker image with Docker and not with Singularity and you want to run is as root.

COPY Anaconda3-2019.10-Linux-x86_64.sh /setup/Anaconda3-2019.10-Linux-x86_64.sh
COPY environment.yml /setup/environment.yml
RUN bash /setup/Anaconda3-2019.10-Linux-x86_64.sh -b -p /home/ec2-user/anaconda3 && \
    rm /setup/Anaconda3-2019.10-Linux-x86_64.sh && \
    ln -s /home/ec2-user/anaconda3/bin/conda /usr/bin/conda && \
    conda env create -f /setup/environment.yml && \
    rm /setup/environment.yml

CMD ["conda", "run", "-n", "solver_env", "/bin/bash"]

If you want to run some more conda commands after creating the environment you should append them to the existing RUN instruction, like so:

COPY Anaconda3-2019.10-Linux-x86_64.sh /setup/Anaconda3-2019.10-Linux-x86_64.sh
COPY environment.yml /setup/environment.yml
RUN bash /setup/Anaconda3-2019.10-Linux-x86_64.sh -b -p /home/ec2-user/anaconda3 && \
    rm /setup/Anaconda3-2019.10-Linux-x86_64.sh && \
    ln -s /home/ec2-user/anaconda3/bin/conda /usr/bin/conda && \
    conda env create -f /setup/environment.yml && \
    rm /setup/environment.yml && \
    conda run -n solver_env python -V

CMD ["conda", "run", "-n", "solver_env", "/bin/bash"]

Upvotes: 2

Itamar Turner-Trauring
Itamar Turner-Trauring

Reputation: 3900

.bashrc is only used in interactive shells. Try adding -i to the ENTRYPOINT, before the -c.

Upvotes: 1

jkr
jkr

Reputation: 19310

Instead of modifying SHELL, you should instead prepend the environment's bin directory to PATH.

COPY Anaconda3-2019.10-Linux-x86_64.sh /setup/
RUN bash Anaconda3-2019.10-Linux-x86_64.sh -b -p /home/ec2-user/anaconda3 
RUN echo 'export PATH=$PATH:/home/ec2-user/anaconda3/bin' >>~/.bashrc \
    && /bin/bash -c "source ~/.bashrc"
ENV PATH=$PATH:/home/ec2-user/anaconda3/bin

# Create the environment:
COPY environment.yml .
RUN conda env create -f environment.yml
# Activate the environment.
ENV PATH=/home/ec2-user/anaconda3/envs/solver_env/bin:$PATH

Upvotes: 1

Related Questions