CODEWITHSUNDEEP
pythonanacondaconda
Alex I
Alex I

Reputation: 20327

How to constrain package versions in conda?

I'm working in a conda environment with a bunch of packages preinstalled (conda list has 360 packages, a lot of ML tools and some bioconda). I sometimes need to add a package; however, I find that conda install newpackage is often extremely slow (hours or days spent at "Solving environment"), and if it ever finishes, it often suggests updates to packages I really don't want to touch.

I'd like to pin the current versions of some of the core packages (python, numpy, scipy, etc) so that modifying them isn't even considered possible by conda. This is both for speed, and to avoid any unintentional updates. If that means a certain package I want to try adding can't be installed, that's okay! I'd much rather have a quick answer like "newpackage conflicts with your version of numpy" than no answer. I can then decide if I want to ignore the conflict; make a simple environment just for this one package; conda build locally, or whatever.

How do I do that?

See also: https://www.anaconda.com/blog/understanding-and-improving-condas-performance (which didn't solve the problem)

Upvotes: 1

Views: 3757

Answers (1)

merv
merv

Reputation: 77117

Package Pinning

Packages can be pinned to specific versions on a per-environment basis. See the documentation on package pinning. For example, suppose we want to pin numpy and scipy to the exact versions we currently have in an env called foo. We could process the output of conda list to match the expected syntax of the Conda pinning specification:

conda activate foo
conda list "^(numpy|scipy)$" | tail -n+4 | awk '{ print $1 " ==" $2 }' > $CONDA_PREFIX/conda-meta/pinned

A few things to note here:

  • conda list takes a regex: use that to your advantage
  • tail is just to skip the header
  • this depends on being in the activated env to define $CONDA_PREFIX
  • this overwrites any existing pinned file

Freeze Installed

A less labor intensive means of keeping everything constant is to use the --freeze-installed flag. However, in more recent versions of Conda this is used by default in the first round of solving. So really all this flag does now is to skip the second round of solving that allows for packages that are not part of the explicit specifications to be updated.

PyPI Packages

PyPI packages installed by Pip require some additional configuration to get Conda to pin them. Specifically, one needs the following syntax in the envs/<env>/conda-meta/pinned file:

numpy=1.21.4=pypi*

That is, indicate that pypi should be in the build string. And for this to be respected, one must enable Pip interoperability and allow for flexible channel prioritization:

## settings only for this environment
conda activate foo
conda config --env --set pip_interop_enabled True
conda config --env --set channel_priority flexible

Enabling this tells Conda to regard PyPI packages as valid substitutes for solving dependencies.

Personally, I would be careful using this since Conda is a general package manager and sometimes packages installed via Pip (such as yaml or wget) do not correspond to the Conda package by the same name. Hence, why I use --env in the example to only enable this configuration setting within that foo environment.

Upvotes: 4

Related Questions