CODEWITHSUNDEEP
azure-machine-learning-service
AlanBB
AlanBB

Reputation: 91

Unable to build local AMLS environment with private wheel

I am trying to write a small program using the AzureML Python SDK (v1.0.85) to register an Environment in AMLS and use that definition to construct a local Conda environment when experiments are being run (for a pre-trained model). The code works fine for simple scenarios where all dependencies are loaded from Conda/ public PyPI, but when I introduce a private dependency (e.g. a utils library) I am getting a InternalServerError with the message "Error getting recipe specifications".

The code I am using to register the environment is (after having authenticated to Azure and connected to our workspace):

environment_name = config['environment']['name']
py_version = "3.7"
conda_packages = ["pip"]
pip_packages = ["azureml-defaults"]
private_packages = ["./env-wheels/utils-0.0.3-py3-none-any.whl"]

print(f"Creating environment with name {environment_name}")
environment = Environment(name=environment_name)
conda_deps = CondaDependencies()

print(f"Adding Python version: {py_version}")
conda_deps.set_python_version(py_version)

for conda_pkg in conda_packages:
    print(f"Adding Conda denpendency: {conda_pkg}")
    conda_deps.add_conda_package(conda_pkg)

for pip_pkg in pip_packages:
    print(f"Adding Pip dependency: {pip_pkg}")
    conda_deps.add_pip_package(pip_pkg)

for private_pkg in private_packages:
    print(f"Uploading private wheel from {private_pkg}")
    private_pkg_url = Environment.add_private_pip_wheel(workspace=ws, file_path=Path(private_pkg).absolute(), exist_ok=True)
    print(f"Adding private Pip dependency: {private_pkg_url}")
    conda_deps.add_pip_package(private_pkg_url)

environment.python.conda_dependencies = conda_deps
environment.register(workspace=ws)

And the code I am using to create the local Conda environment is:

amls_environment = Environment.get(ws, name=environment_name, version=environment_version)

print(f"Building environment...")
amls_environment.build_local(workspace=ws)

The exact error message being returned when build_local(...) is called is:

Traceback (most recent call last):
  File "C:\Anaconda\envs\AMLSExperiment\lib\site-packages\azureml\core\environment.py", line 814, in build_local
    raise error
  File "C:\Anaconda\envs\AMLSExperiment\lib\site-packages\azureml\core\environment.py", line 807, in build_local
    recipe = environment_client._get_recipe_for_build(name=self.name, version=self.version, **payload)
  File "C:\Anaconda\envs\AMLSExperiment\lib\site-packages\azureml\_restclient\environment_client.py", line 171, in _get_recipe_for_build
    raise Exception(message)
Exception: Error getting recipe specifications. Code: 500
: {
  "error": {
    "code": "ServiceError",
    "message": "InternalServerError",
    "detailsUri": null,
    "target": null,
    "details": [],
    "innerError": null,
    "debugInfo": null
  },
  "correlation": {
    "operation": "15043e1469e85a4c96a3c18c45a2af67",
    "request": "19231be75a2b8192"
  },
  "environment": "westeurope",
  "location": "westeurope",
  "time": "2020-02-28T09:38:47.8900715+00:00"
}

Process finished with exit code 1

Has anyone seen this error before or able to provide some guidance around what the issue may be?

Upvotes: 0

Views: 324

Answers (2)

AlanBB
AlanBB

Reputation: 91

The issue was with out firewall blocking the required requests between AMLS and the storage container (I presume to get the environment definitions/ private wheels).

We resolved this by updating the firewall with appropriate ALLOW rules for the AMLS service to contact and read from the attached storage container.

Upvotes: 1

Anders Swanson
Anders Swanson

Reputation: 3961

Assuming that you'd like to run in the script on a remote compute, then my suggestion would be to pass the environment you just "got". to a RunConfiguration, then pass that to an ScriptRunConfig, Estimator, or a PythonScriptStep

from azureml.core import ScriptRunConfig
from azureml.core.runconfig import DEFAULT_CPU_IMAGE

src = ScriptRunConfig(source_directory=project_folder, script='train.py')

# Set compute target to the one created in previous step
src.run_config.target = cpu_cluster.name

# Set environment
amls_environment = Environment.get(ws, name=environment_name, version=environment_version)
src.run_config.environment = amls_environment

run = experiment.submit(config=src)
run

Check out the rest of the notebook here.

If you're looking for a local run this notebook might help.

Upvotes: 0

Related Questions