Azure Machine Learning Vector Index - Why does deployment validation fail?

Question

I'm trying to create a FAISS vector index using Azure Machine Learning's web-based UI. I go through the required steps as per the documentation, and I choose a running compute instance for processing.

The first time I run it, it always fails at the step of "validate_deployments_job" with azure.core.exceptions.ClientAuthenticationError: No token received.:

Where am I supposed to provide said token?

If I go into the job details and run the job again, the same "validate_deployments_job" will still fail, however, the vector index will get created and become usable. I understand this functionality is in preview, but I'm trying to understand what I'm missing here. The logs (see below) imply that there's some kind of an authentication issue going on, but I am logged in with access to all the required components, and when creating the job, there's no requirement to provide any kinds of tokens.

The logs/full trace show the following:

[2023-09-23 16:50:09] INFO     azureml.rag.validate_deployments.validate_deployments - ActivityStarted, validate_deployments (activity.py:108)
[2023-09-23 16:50:09] INFO     azureml.rag.validate_deployments.validate_deployments - [Validate Deployments]: Validating Embeddings model using AOAI (validate_deployments.py:552)
Using llm_config: {
  "type": "azure_open_ai",
  "model_name": "gpt-35-turbo",
  "deployment_name": null,
  "temperature": 0.0,
  "max_tokens": 1500
}
[2023-09-23 16:50:09] INFO     azureml.rag.validate_deployments.validate_deployments - [Validate Deployments]: Received and parsed arguments for validating deployments in RAG.  Starting validation now... (validate_deployments.py:309)
[2023-09-23 16:50:09] INFO     azureml.rag.connections - Getting workspace connection: Default_AzureOpenAI (connections.py:131)
Embedding deployment name: text-embedding-ada-002
Embedding model name: text-embedding-ada-002
[2023-09-23 16:50:10] INFO     azureml.rag.validate_deployments.validate_deployments - [Validate Deployments]: Completion model using Default AOAI connection, parsing ResourceId (validate_deployments.py:389)
Using workspace connection key for OpenAI embeddings
[2023-09-23 16:50:10] INFO     azureml.rag.validate_deployments.validate_deployments - [Validate Deployments]: Embedding workspace connection retrieved and params populated successfully... (validate_deployments.py:400)
[2023-09-23 16:50:10] INFO     azureml.rag.validate_deployments.validate_deployments - [Validate Deployments]: Starting to poll on deployment status validation. (validate_deployments.py:264)
[2023-09-23 16:50:10] INFO     azureml.rag.validate_deployments.validate_deployments - [Validate Deployments]: Default AOAI resource detected. Performing control plane validations now... (validate_deployments.py:174)
_AzureMLOnBehalfOfCredential.get_token failed: No token received.
Using User Identity for authentication
[2023-09-23 16:50:10] ERROR    azureml.rag.validate_deployments.validate_deployments - ValidationFailed: validate_deployments failed with exception: Traceback (most recent call last):
  File "validate_deployments.py", line 585, in main_wrapper
    main(parser_args, activity_logger)
  File "validate_deployments.py", line 554, in main
    validate_aoai_deployments(parser_args, check_aoai_completion, check_aoai_embeddings, activity_logger)
  File "validate_deployments.py", line 415, in validate_aoai_deployments
    poll_on_deployment(completion_params,
  File "validate_deployments.py", line 266, in poll_on_deployment
    polling2.poll(
  File "/azureml-envs/rag/lib/python3.8/site-packages/polling2.py", line 193, in poll
    val = target(*args, **kwargs)
  File "validate_deployments.py", line 267, in <lambda>
    lambda: check_deployment_status(completion_params, "llm", activity_logger) and check_deployment_status(
  File "validate_deployments.py", line 181, in check_deployment_status
    return validate_and_create_default_aoai_resource(ws, model_params, activity_logger)
  File "validate_deployments.py", line 109, in validate_and_create_default_aoai_resource
    response = client.deployments.get(
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/mgmt/cognitiveservices/operations/_deployments_operations.py", line 348, in get
    pipeline_response: PipelineResponse = self._client._pipeline.run(  # pylint: disable=protected-access
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 230, in run
    return first_node.send(pipeline_request)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 86, in send
    response = self.next.send(request)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 86, in send
    response = self.next.send(request)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 86, in send
    response = self.next.send(request)
  [Previous line repeated 2 more times]
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/mgmt/core/policies/_base.py", line 46, in send
    response = self.next.send(request)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/policies/_redirect.py", line 197, in send
    response = self.next.send(request)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/policies/_retry.py", line 531, in send
    response = self.next.send(request)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/policies/_authentication.py", line 124, in send
    self.on_request(request)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/policies/_authentication.py", line 99, in on_request
    self._token = self._credential.get_token(*self._scopes)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/ai/ml/identity/_credentials/aml_on_behalf_of.py", line 36, in get_token
    return self._credential.get_token(*scopes, **kwargs)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/ai/ml/identity/_internal/managed_identity_base.py", line 51, in get_token
    return super(ManagedIdentityBase, self).get_token(*scopes, **kwargs)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/ai/ml/identity/_internal/get_token_mixin.py", line 83, in get_token
    token = self._request_token(*scopes, **kwargs)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/ai/ml/identity/_internal/managed_identity_base.py", line 61, in _request_token
    return cast(ManagedIdentityClient, self._client).request_token(*scopes, **kwargs)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/ai/ml/identity/_internal/managed_identity_client.py", line 127, in request_token
    token = self._process_response(response, request_time)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/ai/ml/identity/_internal/managed_identity_client.py", line 64, in _process_response
    raise ClientAuthenticationError(message="No token received.", response=response.http_response)
azure.core.exceptions.ClientAuthenticationError: No token received.
 (validate_deployments.py:588)
[2023-09-23 16:50:10] ERROR    azureml.rag.validate_deployments.validate_deployments - ActivityCompleted: Activity=validate_deployments, HowEnded=Failure, Duration=759.58 [ms], Exception=ClientAuthenticationError (activity.py:127)
Traceback (most recent call last):
  File "validate_deployments.py", line 616, in <module>
    main_wrapper(parser_args, logger)
  File "validate_deployments.py", line 585, in main_wrapper
    main(parser_args, activity_logger)
  File "validate_deployments.py", line 554, in main
    validate_aoai_deployments(parser_args, check_aoai_completion, check_aoai_embeddings, activity_logger)
  File "validate_deployments.py", line 415, in validate_aoai_deployments
    poll_on_deployment(completion_params,
  File "validate_deployments.py", line 266, in poll_on_deployment
    polling2.poll(
  File "/azureml-envs/rag/lib/python3.8/site-packages/polling2.py", line 193, in poll
    val = target(*args, **kwargs)
  File "validate_deployments.py", line 267, in <lambda>
    lambda: check_deployment_status(completion_params, "llm", activity_logger) and check_deployment_status(
  File "validate_deployments.py", line 181, in check_deployment_status
    return validate_and_create_default_aoai_resource(ws, model_params, activity_logger)
  File "validate_deployments.py", line 109, in validate_and_create_default_aoai_resource
    response = client.deployments.get(
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/mgmt/cognitiveservices/operations/_deployments_operations.py", line 348, in get
    pipeline_response: PipelineResponse = self._client._pipeline.run(  # pylint: disable=protected-access
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 230, in run
    return first_node.send(pipeline_request)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 86, in send
    response = self.next.send(request)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 86, in send
    response = self.next.send(request)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 86, in send
    response = self.next.send(request)
  [Previous line repeated 2 more times]
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/mgmt/core/policies/_base.py", line 46, in send
    response = self.next.send(request)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/policies/_redirect.py", line 197, in send
    response = self.next.send(request)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/policies/_retry.py", line 531, in send
    response = self.next.send(request)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/policies/_authentication.py", line 124, in send
    self.on_request(request)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/core/pipeline/policies/_authentication.py", line 99, in on_request
    self._token = self._credential.get_token(*self._scopes)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/ai/ml/identity/_credentials/aml_on_behalf_of.py", line 36, in get_token
    return self._credential.get_token(*scopes, **kwargs)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/ai/ml/identity/_internal/managed_identity_base.py", line 51, in get_token
    return super(ManagedIdentityBase, self).get_token(*scopes, **kwargs)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/ai/ml/identity/_internal/get_token_mixin.py", line 83, in get_token
    token = self._request_token(*scopes, **kwargs)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/ai/ml/identity/_internal/managed_identity_base.py", line 61, in _request_token
    return cast(ManagedIdentityClient, self._client).request_token(*scopes, **kwargs)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/ai/ml/identity/_internal/managed_identity_client.py", line 127, in request_token
    token = self._process_response(response, request_time)
  File "/azureml-envs/rag/lib/python3.8/site-packages/azure/ai/ml/identity/_internal/managed_identity_client.py", line 64, in _process_response
    raise ClientAuthenticationError(message="No token received.", response=response.http_response)
azure.core.exceptions.ClientAuthenticationError: No token received.

Answer 1

Try again but with a different compute type. I had first attempted connecting to an existing Compute Instance (a slow/cheap one) and encountered the same error as you. Once I switched over to a Serverless instance it's come up ✅