How to deploy custom Huggingface models on Azure ML

Question

Hugging Face models have easy one-click deployment via model catalog. Still, some models such as facebook/audiogen-medium, are not available on the Azure model catalog and do not have Deploy button on Hugging Face.

I followed these official tutorials for deploying custom models:
Deploy a model as an online endpoint
Deploy and score a machine learning model by using an online endpoint
but I could not find a tutorial to deploy other Hugging Face models on Azure ML using Azure Python sdk v2.

Since I'm using audiocraft package which lets me use the model like this:

from audiocraft.models import AudioGen
    
model = AudioGen.get_pretrained("facebook/audiogen-medium")
model.set_generation_params(duration=5)  # generate 5 seconds.
descriptions = ['dog barking', 'sirene of an emergency vehicle', 'footsteps in a corridor']
wav = model.generate(descriptions)  # generates 3 samples.

I have an error when I want to register the model, I don't exactly know what should I use for the path attribute of the Model, the local path of the downloaded Hugging Face model?

model = Model(
    name="audiogen",
    version="1",
    type=AssetTypes.CUSTOM_MODEL,
    path="./score.py",
    description="facebook/audiogen model for sound effects generation",
)

# Register the model
ml_client.models.create_or_update(model)

---> 10 ml_client.models.create_or_update(model)

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/azure/ai/ml/_telemetry/activity.py:289, in monitor_with_activity.<locals>.monitor.<locals>.wrapper(*args, **kwargs)
    285     with tracer.span():
    286         with log_activity(
    287             logger.package_logger, activity_name or f.__name__, activity_type, custom_dimensions
    288         ):
--> 289             return f(*args, **kwargs)
    290 elif hasattr(logger, "package_logger"):
    291     with log_activity(logger.package_logger, activity_name or f.__name__, activity_type, custom_dimensions):

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/azure/ai/ml/operations/_model_operations.py:158, in ModelOperations.create_or_update(self, model)
    151     raise ValidationException(
    152         message=msg,
    153         no_personal_data_message=msg,
    154         target=ErrorTarget.MODEL,
    155         error_category=ErrorCategory.USER_ERROR,
    156     )
    157 if model.name is not None:
--> 158     model_properties = self._get_model_properties(model.name)
    159     if model_properties is not None and _is_evaluator(model_properties) != _is_evaluator(model.properties):
    160         if _is_evaluator(model.properties):

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/azure/ai/ml/operations/_model_operations.py:815, in ModelOperations._get_model_properties(self, name, version, label)
    813     if version or label:
    814         return self.get(name, version, label).properties
--> 815     return self._get_latest_version(name).properties
    816 except (ResourceNotFoundError, ValidationException):
    817     return None

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/azure/ai/ml/operations/_model_operations.py:620, in ModelOperations._get_latest_version(self, name)
    615 def _get_latest_version(self, name: str) -> Model:
    616     """Returns the latest version of the asset with the given name.
    617 
    618     Latest is defined as the most recently created, not the most recently updated.
    619     """
--> 620     result = _get_latest(
    621         name,
    622         self._model_versions_operation,
    623         self._resource_group_name,
    624         self._workspace_name,
    625         self._registry_name,
    626     )
    627     return Model._from_rest_object(result)

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/azure/ai/ml/_utils/_asset_utils.py:853, in _get_latest(asset_name, version_operation, resource_group_name, workspace_name, registry_name, order_by, **kwargs)
    833 result = (
    834     version_operation.list(
    835         name=asset_name,
   (...)
    850     )
    851 )
    852 try:
--> 853     latest = result.next()
    854 except StopIteration:
    855     latest = None

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/azure/core/paging.py:123, in ItemPaged.__next__(self)
    121 if self._page_iterator is None:
    122     self._page_iterator = itertools.chain.from_iterable(self.by_page())
--> 123 return next(self._page_iterator)

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/azure/core/paging.py:75, in PageIterator.__next__(self)
     73     raise StopIteration("End of paging")
     74 try:
---> 75     self._response = self._get_next(self.continuation_token)
     76 except AzureError as error:
     77     if not error.continuation_token:

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/azure/ai/ml/_restclient/v2023_08_01_preview/operations/_model_versions_operations.py:427, in ModelVersionsOperations.list.<locals>.get_next(next_link)
    426 def get_next(next_link=None):
--> 427     request = prepare_request(next_link)
    429     pipeline_response = self._client._pipeline.run(  # pylint: disable=protected-access
    430         request,
    431         stream=False,
    432         **kwargs
    433     )
    434     response = pipeline_response.http_response

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/azure/ai/ml/_restclient/v2023_08_01_preview/operations/_model_versions_operations.py:371, in ModelVersionsOperations.list.<locals>.prepare_request(next_link)
    368 def prepare_request(next_link=None):
    369     if not next_link:
--> 371         request = build_list_request(
    372             subscription_id=self._config.subscription_id,
    373             resource_group_name=resource_group_name,
    374             workspace_name=workspace_name,
    375             name=name,
    376             api_version=api_version,
    377             skip=skip,
    378             order_by=order_by,
    379             top=top,
    380             version=version,
    381             description=description,
    382             offset=offset,
    383             tags=tags,
    384             properties=properties,
    385             feed=feed,
    386             list_view_type=list_view_type,
    387             stage=stage,
    388             template_url=self.list.metadata['url'],
    389         )
    390         request = _convert_request(request)
    391         request.url = self._client.format_url(request.url)

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/azure/ai/ml/_restclient/v2023_08_01_preview/operations/_model_versions_operations.py:63, in build_list_request(subscription_id, resource_group_name, workspace_name, name, **kwargs)
     58 # Construct URL
     59 _url = kwargs.pop("template_url", "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.MachineLearningServices/workspaces/{workspaceName}/models/{name}/versions")  # pylint: disable=line-too-long
     60 path_format_arguments = {
     61     "subscriptionId": _SERIALIZER.url("subscription_id", subscription_id, 'str', min_length=1),
     62     "resourceGroupName": _SERIALIZER.url("resource_group_name", resource_group_name, 'str', max_length=90, min_length=1),
---> 63     "workspaceName": _SERIALIZER.url("workspace_name", workspace_name, 'str', pattern=r'^[a-zA-Z0-9][a-zA-Z0-9_-]{2,32}$'),
     64     "name": _SERIALIZER.url("name", name, 'str'),
     65 }
     67 _url = _format_url_section(_url, **path_format_arguments)
     69 # Construct parameters

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/msrest/serialization.py:652, in Serializer.url(self, name, data, data_type, **kwargs)
    650 data = self._http_component_validation(data, data_type, name, **kwargs)
    651 try:
--> 652     output = self.serialize_data(data, data_type, **kwargs)
    653     if data_type == 'bool':
    654         output = json.dumps(output)

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/msrest/serialization.py:760, in Serializer.serialize_data(self, data, data_type, **kwargs)
    749 """Serialize generic data according to supplied data type.
    750 
    751 :param data: The data to be serialized.
   (...)
    757 :raises: SerializationError if serialization fails.
    758 """
    759 if data is None:
--> 760     raise ValueError("No value for given attribute")
    762 try:
    763     if data_type in self.basic_types.values():

ValueError: No value for given attribute

or just the score.py script which is necassary and is as follows (I used this for now and got the error above):

import json
import os
from audiocraft.models import AudioGen
from audiocraft.data.audio import audio_write
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient

def init():
    global model, blob_service_client, container_name
    model = AudioGen.get_pretrained('facebook/audiogen-medium')
    model.set_generation_params(duration=5)

    # Initialize Azure Blob Storage client
    connect_str = os.getenv('AZURE_STORAGE_CONNECTION_STRING')
    blob_service_client = BlobServiceClient.from_connection_string(connect_str)
    container_name = 'audiogen_generated_files'

def run(payload: str):
    data = json.loads(payload)
    theme = data["theme"]
    prompts = [f'{theme}, slow speed', f'{theme}, fast speed', f'{theme} door closing', f'{theme} starting']
    wavs = model.generate(prompts)
    
    urls = []

    for idx, one_wav in enumerate(wavs):
        file_name = f'{theme}_{idx}.wav'
        local_file_path = f'/tmp/{file_name}'
        audio_write(local_file_path, one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
        
        blob_client = blob_service_client.get_blob_client(container=container_name, blob=file_name)
        with open(local_file_path, "rb") as data:
            blob_client.upload_blob(data, overwrite=True)
        urls.append(blob_client.url)
        os.remove(local_file_path)

    result = {"urls": urls}
    return result

This is how I create the endpoint and deployment in the next steps:

endpoint = ManagedOnlineEndpoint(
    name="audiogen-endpoint-" + str(uuid.uuid4())[:8],
    description="audiogen inference endpoint",
    auth_mode="key"
)
endpoint = ml_client.online_endpoints.begin_create_or_update(endpoint).result()

deployement = ManagedOnlineDeployment(
    name="audiogen-deployment-mo",
    endpoint_name=endpoint.name,
    model=ml_client.models.get(name=model.name, version="1"),
    instance_count=1,
)
deployement = ml_client.begin_create_or_update(deployement).result()

I want to know the proper way to deploy the model, also am I doing the other steps correcly?

Answer 1

For Model function you have to give file path or folder path where you model is there.

If it is custom model then give file path directly like "path_to_model/model.pkl"

If it is MLflow model then give the folder path.

Below are different path you can give.

1. A local file ("path_to_model/model.pkl")
2. Mlflow model ("mlflow_folder") folder having MLflow file.
3. Cloud path ("azureml://subscriptions/XXX/resourceGroups/XX/workspaces/XX/datastores/workspaceblobstore/paths/model.pkl")
4. From a run ("azureml://jobs/{job_name}/outputs/artifacts/paths/model/)

Refer this notebook for more information.

Since you are using the hugging face model save it in a folder and pass that folder to register the model as AssetTypes.CUSTOM_MODEL.

After registering, in you scoring script instead of loading it every time through hugging face hub, load it the model from local file. It will be in model folder, like below.

model_path = os.path.join(os.getenv("AZUREML_MODEL_DIR"),"model/")
AudioGen.get_pretrained(model_path )

next, continue with your prediction in scoring script.

You can also register .bin model in azure ml portal selecting the framework as shown below.

next, select the framework and register.