how to specify python==3.6.8 for PyTorch Estimator (conda_packages not sufficient)

Question

I need to run my python script under Azure Machine Learning, using python=3.6.8 (not the default 3.6.2). I am using the AML "PyTorch()" Estimator, setting the "conda_packages" arg to ["python==3.6.8"].

I am relying on this doc page for the PyTorch Estimator:

https://learn.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.dnn.pytorch?view=azure-ml-py

When my script runs, I print out "sys.version" and see that it is still set to python 3.6.2:

python: 3.6.2 | packaged by conda-forge | (default, Jul 23 2017, 22:59:30) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]

I expected to see python 3.6.8, since I specified that in the PyTorch Estimator's conda_packages arg.

I also tried moving the "python==3.6.8" from conda_packages to pip_packages, but received an error saying pip could not locate that package.

FYI, I have another package specified in pip_packages, and that does get installed correctly during this process. It seems like the value of the "conda_packages" arg is not being used (I can find no mention of a conda or python install error in the AML logs for my job).

Answer 1

one other option is specifying a conda dependency file conda_dependencies_file_path with the right python version. the below docs outlines detailed documentation on how to do that. once you specify conda_depencies_file_path, it overrides pip_packages, and conda_packages so I recommend putting all your packages in the conda dependency file

https://learn.microsoft.com/en-us/python/api/azureml-core/azureml.core.conda_dependencies.condadependencies?view=azure-ml-py

Answer 2

When submitting a training job, Azure ML runs your script in a conda environment within a Docker container. The PyTorch containers have the following dependencies installed.

Dependencies
PyTorch 1.0/1.1/1.2 Python 3.6.2 CUDA (GPU image only) 10.0 cuDNN (GPU image only) 7.6.3 NCCL (GPU image only) 2.4.8 azureml-defaults Latest OpenMpi 3.1.2 horovod 0.16.1 miniconda 4.5.11 torch 1.0/1.1/1.2 torchvision 0.2.1 git 2.7.4 tensorboard 1.14 future 0.17.1 The Docker images extend Ubuntu 16.04.

To install additional dependencies, you can either use the pip_packages or conda_packages parameter. Or, you can specify the pip_requirements_file or conda_dependencies_file parameter. Alternatively, you can build your own image, and pass the custom_docker_image parameter to the estimator constructor.

I would suggest you updating the conda_dependencies_file parameter to see if it works or not.

Reference: https://learn.microsoft.com/en-us/azure/machine-learning/service/how-to-train-pytorch#create-a-pytorch-estimator