When using Azure CLI to create an ML environment it is not always built

Question

I am using Azure CLI as follows to create an ML environment:

az ml environment create --name $(AML_ENVIRONMENT_NAME) --version $(AML_ENVIRONMENT_VERSION) --resource-group $(RESOURCE_GROUP) --workspace-name $(WORKSPACE_NAME)  --image $(AML_ENVIRONMENT_BASE_IMAGE) --conda-file $(AML_ENVIRONMENT_CONDA_SPEC)

This works fine, except two problems:

1. If conda specification is not changed from the previous one, the image ("environment") is not built. The new environment is yet visible in ML Studio "Environments" view, but it says "Not built".

2. Even when the image is built, the Environment class seems not to use it, but a new build is started. The code that I use to get the environment is as follows:

from azureml.core import Experiment
env = Environment.get(ws, name=env_name, version=env_version)

So, this call does give me the env, but makes it by triggering a new build with a version name like "Autosave_2022-xx-xxT...". Then it takes half an hour or so to create that new image. Why it cannot just use the image that is already built? If I print the env it prints the wanted env name and version (e.g. 3), but when I look at the pipeline job, the actual environment version is that "Autosave..." one.

This "Autosave"-strangeness started to happen when I started to create environment with Azure CLI. Before that, I created the environment with Environment.from_docker_image() call and never had this problem.

See the screenshots below.

Answer 1

Reproduced the issue and didn’t find any error from my end. Check the procedure followed

az account set --subscription <subscription ID>
az configure --defaults workspace=<AzureML workspace name> group=<resource group>

there are two types of environments in azure ML. one is custom environment and other is curated environment.

Curated are having predefined environments and having the properties of different frameworks.

To create custom environment, we need to create using “az ml environment create name”

Curated a backed by the cache docker images, which regularly updates the frameworks and libraries supporting to the applications running on it. Use “az ml environments list” to get the list of curated environments. Create an YAML file and upload in the asserts folder in local environment of workspace

$schema: https://azuremlschemas.azureedge.net/latest/environment.schema.json
name: docker-image-example
image: pytorch/pytorch:latest
description: Environment created from a Docker image.

az ml environment create --file assets/environment/conda.yml

the above syntax will create the yaml file

az ml environment create --name $(AML_ENVIRONMENT_NAME) --version $(AML_ENVIRONMENT_VERSION) --resource-group $(RESOURCE_GROUP) --workspace-name $(WORKSPACE_NAME)  --image $(AML_ENVIRONMENT_BASE_IMAGE) --conda-file $(AML_ENVIRONMENT_CONDA_SPEC)

The above are the curated environments automatically created

This is the option which we will get to create the custom environment with docker image and conda environment. This will reflect with the created environment using CLI

The URL of the docker image to be created is mentioned like above image

The Version number is created as normal for this reproducing of the environment. Not autosave extension.