How to create and activate specific Conda Environment on Dataproc with Jupyter Notebook Initialised

Question

Once you have the Jupyter notebook installed using the Google Initialisation script is it possible to create and activate a conda environment on all the worker nodes?

Do I need to ssh onto each worker and create the same environment or is there a way to accomplish the same via a notebook if the number of modules to add is small?

Answer 1

Once you have a cluster up, there is not a great way to re-configure all workers.

If you create a new cluster the conda initialization action might be helpful.