CODEWITHSUNDEEP
pythonmakefileanacondaconda
DataOrc
DataOrc

Reputation: 839

Makefile can't use `conda activate`

I need to activate a conda environment in my makefile in order to run some python scripts, however, whenever I try to run conda activate env_name, I get the following message:

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. If your shell is Bash or a Bourne variant, enable conda for the current user with

$ echo ". /Users/MY_USERNAME/anaconda3/etc/profile.d/conda.sh" >> ~/.bash_profile

or, for all users, enable conda with

$ sudo ln -s /Users/MY_USERNAME/anaconda3/etc/profile.d/conda.sh /etc/profile.d/conda.sh

The options above will permanently enable the 'conda' command, but they do NOT put conda's base (root) environment on PATH. To do so, run

$ conda activate

in your terminal, or to put the base environment on PATH permanently, run

$ echo "conda activate" >> ~/.bash_profile

Previous to conda 4.4, the recommended way to activate conda was to modify PATH in your ~/.bash_profile file. You should manually remove the line that looks like

export PATH="/Users/MY_USERNAME/anaconda3/bin:$PATH"

^^^ The above line should NO LONGER be in your ~/.bash_profile file! ^^^

I've tried changing the shell for the makefile by adding SHELL := /bin/zsh at the top, but this doesn't fix the problem. Additionally, I need this makefile to be able to run using whatever the default shell is for the computer (some of my teammates use zsh, others use bash). It seems like no matter what I do, I can't get conda activate to work in the makefile.

What can I do to get it to work, or is this impossible?

Upvotes: 26

Views: 15470

Answers (7)

Antonio
Antonio

Reputation: 61

Summing up all the answers, I came out to the simple set-up in the Makefile executin python file in conda virtual env:

SHELL:=/bin/bash # change shell to bash for conda

setup: # conda setup virtenv and python run
    source ~/.bash_profile && \
    conda activate my_conda_venv && \
    python my_script.py

where the content of .bash_profile file is updated automatically after conda was installed

cat ~/.bash_profile

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/opt/homebrew/Caskroom/miniconda/base/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/opt/homebrew/Caskroom/miniconda/base/etc/profile.d/conda.sh" ]; then
        . "/opt/homebrew/Caskroom/miniconda/base/etc/profile.d/conda.sh"
    else
        export PATH="/opt/homebrew/Caskroom/miniconda/base/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

Upvotes: 0

Amir Pourmand
Amir Pourmand

Reputation: 629

This is also a minimal working solution in bash. I have tested this one in Ubuntu.

Note that use of .ONESHELL: is actually mandatory. If you ignore that flag, every command in the shell would be executed in a separate shell. In that case you would need to append all you scripts with a semicolon to make a big one-line command!

.ONESHELL:

SHELL = /bin/bash
CONDA_ACTIVATE = source $$(conda info --base)/etc/profile.d/conda.sh ; conda activate ; conda activate

train:
        $(CONDA_ACTIVATE) <<put_your_env_name_here>>
        # other stuff

Upvotes: 2

indrapaul824
indrapaul824

Reputation: 121

Wonderful solution by @Traveler. But keeping in mind the suggestions by both @Felipe Maza & @Tomasz Bartkowiak, adding .ONESHELL: before the recipe solves all the errors.

I use a Linux system with zsh as my shell and the current latest conda version: 4.12.0

Therefore, the recipe will be:

.ONESHELL:

SHELL = /bin/zsh
CONDA_ACTIVATE = source $$(conda info --base)/etc/profile.d/conda.sh ; conda activate ; conda activate

# Create conda env from env.yml and compile and install exact pip packages
conda-pip:
    conda env update --prune -f env.yml
    $(CONDA_ACTIVATE) <<env_name>>
    pip-compile requirements/req.in
    pip-sync requirements/req.txt

conda-pip does the following:

  1. Create a conda environment based on an env.yml created beforehand.
  2. Activate the conda environment by replacing <<env_name>> with the environment name that is set in the env.yml.
  3. Use pip-compile and pip-sync to compile and install exact pip packages to the conda environment.

Note: You still need to activate the conda environment to use it in the terminal after this because conda-pip started and ended in a separate sub-shell.

You can check more about pip-compile and pip-sync here: pip-tools

Upvotes: 8

Tomasz Bartkowiak
Tomasz Bartkowiak

Reputation: 15058

One important thing you need to remember is that Makefile will execute each line of the recipe in a separate sub-shell so e.g. exporting PATH in one line will not affect what the command in the next Makefile line can see! Please refer to the Makefile manual - Recipe Execution:

When it is time to execute recipes to update a target, they are executed by invoking a new sub-shell for each line of the recipe, unless the .ONESHELL special target is in effect (see Using One Shell) (In practice, make may take shortcuts that do not affect the results.)

Please note: this implies that setting shell variables and invoking shell commands such as cd that set a context local to each process will not affect the following lines in the recipe.

Upvotes: 3

Felipe Maza
Felipe Maza

Reputation: 101

You should use .ONESHELL: directive at beginning of script. This run all in the same shell.

Upvotes: 10

Traveler
Traveler

Reputation: 1108

After a bit of searching around, I came up with adding this pattern to my Makefile to make conda activate work. Others may be able to simplify.

# Need to specify bash in order for conda activate to work.
SHELL=/bin/bash
# Note that the extra activate is needed to ensure that the activate floats env to the front of PATH
CONDA_ACTIVATE=source $$(conda info --base)/etc/profile.d/conda.sh ; conda activate ; conda activate

py3build:
    ($(CONDA_ACTIVATE) py3.6 ; python setup.py build )

Upvotes: 34

Maxim Egorushkin
Maxim Egorushkin

Reputation: 136515

conda activate among other things set environment variables. However, GNU Make invokes each line of recipe in a newly spawned shell. You would need to invoke conda activate in each line of the recipe.

A better way is for makefile to do conda activate and then re-run itself in the newly activated environment and only then build your targets.

Upvotes: 0

Related Questions