CODEWITHSUNDEEP
pythoncondasdlcmamba
Sean Blanton
Sean Blanton

Reputation: 41

conda environments in team and shared Linux environment - best practices?

Conda is great, but it assumes that people code completely independently much like the complete lack of namespace hierarchy in public python modules. Looking for best practices with 6 people all developing in largely the same conda environment. Let's pretend there won't be exceptions. No containers, which would make this easier.

I want to ask others what tips other have? Any other best practices anyone can recommend?

Here's what we are working on currently:

  1. Having a "source" environment.yml and then having all developers work from an exported snapshot yml (unless they are developing the conda environment itself) so they are all developing in the exact same environment. you don't want surprises when someone's app goes to prod and you find out the environment it was developed under. This is just standard best practice in every language.

  2. Deploying shared conda environments to QA and Prod. Conda environments take up several GB of space, which is a minor concern, but it also takes a lot of time to install. It's a lot of machinery to install for every application update. Yes, need to be concerned about versioning that conda environment, but it is a less a problem to have 2 or 3 versions in use at the same time. And yes, there are occasionally bugs in conda/mamba. It's a pain to work around them

Upvotes: 2

Views: 334

Answers (1)

Robert Alexander
Robert Alexander

Reputation: 1181

Been using conda for a couple of years but lately abandoned it after it put my team into deadlock situations a couple of times.

First suggestion I'd give, if you want to remain in the conda system, is at least to switch to mamba, since conda at times can be exceedingly slow.

You did not specify what environment/languages you need to manage.

For my team it's 95% python, and we switched to poetry which we find a lot faster, and makes it easier to manage a consistent environment via its TOML config. We use it also to separate the productions project dependencies from the development env, where the latter is equipped with many linters/testers etc used in either local (eg pre-commit) or remote (eg github actions) workflows.

Of course if you'd need to install non python stuff that is not on PyPI, either conda or your platform's package managers (eg homebrew for Mac, apt for Debian Linux and derivatives) will be necessary.

Hope any of this is useful.

Upvotes: 2

Related Questions