CODEWITHSUNDEEP
pythonazureazure-blob-storageazure-iot-hubazure-iot-sdk
Locks
Locks

Reputation: 61

Python function to upload files to storage failing with 403 TooManyDevicesError

I have a python app running on an embedded linux device. The app runs async code that connects an iothub device client. I have a class called device_client_wrapper that uses a semaphore to ensure that only one function calls device client methods at a time, as that was causing me issues before.

I just added a new function to this class which is used to upload files from the device to azure blob storage. It was working fine for a while, and often works the first time, but then if I execute it too many times I get the following error:

ERROR upload_file Exception: Unexpected failure
ERROR Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 33, in handle_result
    return await callback.completion()
  File "/usr/lib/python3.10/site-packages/azure/iot/device/common/async_adapter.py", line 91, in completion
    return await self.future
azure.iot.device.exceptions.ServiceError: HTTP operation returned: 403 TooManyDevicesError(Error: Forbidden)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/cogo_ig60/cloud/device_client_wrapper.py", line 99, in upload_file
    blob_info = await self.device_client.get_storage_info_for_blob(blob_name)
  File "/usr/lib/python3.10/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 571, in get_storage_info_for_blob
    storage_info = await handle_result(callback)
  File "/usr/lib/python3.10/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 57, in handle_result
    raise exceptions.ClientError("Unexpected failure") from e
azure.iot.device.exceptions.ClientError: Unexpected failure

This is the function causing the error:

async def upload_file(self, path_to_file: str, blob_name: str):
    """Upload a file to Azure Blob Storage."""
    try:
        async with self.semaphore:
            blob_info = await self.device_client.get_storage_info_for_blob(blob_name)
            sas_url = "https://{}/{}/{}{}".format(
                blob_info["hostName"],
                blob_info["containerName"], 
                blob_info["blobName"],
                blob_info["sasToken"]
            )
            logger.info(
                f"Uploading file {path_to_file} to blob storage as {blob_name}")
            with BlobClient.from_blob_url(sas_url) as blob_client:
                with open(path_to_file, "rb") as f:
                    result = blob_client.upload_blob(f, overwrite=True)
                    return (True, result)
    except FileNotFoundError as e:
        # catch file not found and add an HTTP status code to return in notification to IoT Hub
        logger.error(f'upload_file FileNotFoundError: {e}')
        logger.error(traceback.format_exc())
        e.status_code = 404
        return (False, e)
    except AzureError as e:
        # catch Azure errors that might result from the upload operation
        logger.error(f'upload_file AzureError: {e}')
        logger.error(traceback.format_exc())
        return (False, e)
    except Exception as e:
        # catch all other errors
        logger.error(f'upload_file Exception: {e}')
        logger.error(traceback.format_exc())
        return (False, e)

It seems to be an issue with device_client.get_storage_info_for_blob. I know there are a max of 10 concurrent file uploads per device, but I am uploading 7 fiiles, then they finish uploading successfully before I try again. Even so, with the semaphore it should prevent any concurrent file uploads. That is unless I'm missing something about how my code is working.

Is something not being released properly, or not able to complete properly?

This is python 3.10.4 btw.

Upvotes: 0

Views: 293

Answers (2)

Nayan Goswami
Nayan Goswami

Reputation: 81

device_client.notify_blob_upload_status(blob_info['correlationId'], True, 200, f'OK: {path_to_file}')

Above line is compulsory. The documentation mentions that this is for notifications of file upload. But even if you keep file upload notifications off on IoTHub file upload settings, we have to still acknowledge the file upload.

Upvotes: 0

LeelaRajesh_Sayana
LeelaRajesh_Sayana

Reputation: 650

Found additional information indicating the potential cause of this issue. Posting it here incase other community members encounter this behavior.

Each device is limited to 10 concurrent active file uploads at a time. In addition to that there is additional throttling applied to IoT Hub based on the SKU and the number of units available. For example, S1 tier has a limit of 1.67 file upload initiations/sec/unit. Suppose you have an S1 tier of units, you would hit the throttling limit at 6.67 file uploads per second.

Please refer the following documentation for reference.

Upvotes: 0

Related Questions