CODEWITHSUNDEEP
azureazure-storageazure-blob-storage
Justin Lim
Justin Lim

Reputation: 351

Monitoring azure storage bandwidth usage for each of my users

I have an application that will give SAS tokens to registered users to upload and download directly from Azure storage. What is the best way to monitor the bandwidth for each of my users?

I have taken a look at the logs that Azure provides, and looking at things, a possible way is to bind the SAS token to the user's ip address, and then check the logs for the <requester-ip-address> field. However, this way doesn't seem so concrete as multiple users might have the same ip address.

Also, I see a <request-id-header> field which, according to the documentation, is an opaque value that uniquely identifies the request. However I don't see any way of getting the header value if the user directly communicates with azure storage.

Thanks.

Upvotes: 3

Views: 1298

Answers (2)

sguler
sguler

Reputation: 286

A different approach could be:

  • Using Copy Blob API: create a temporary file with user's username and generate a SAS token for user to download the file
  • Remove the file after the expiry of the SAS token
  • Check for the analytic logs in the Storage account; whether the blob was accessed, and whether the download was successful, and if not downloaded bytes
  • If user failed to download the whole blob; you will see SASNetworkError in the log
  • In the event of SASNetworkError; the log will contain the amount of data that was downloaded by the user (https://learn.microsoft.com/en-us/rest/api/storageservices/storage-analytics-log-format response-packet-size). This will tell you the percentage of the download that completed right before the error.

Upvotes: 1

Tom Sun
Tom Sun

Reputation: 24569

What is the best way to monitor the bandwidth for each of my users?

Based on my experience, we have no suitable way to do that. It seems that we couldn't get the bandwidth value. If you worry about the limition of azure storage, we could get the Azure Storage limitation from the azure official document.

Maximum Request Rate per storage account Blobs: 20,000 requests per second2 for blobs of any valid size (capped only by the account's ingress/egress limits)

Target throughput for single blob Up to 60 MB per second, or up to 500 requests per second

Update:

We could according Azure price calculator to adopt appropriate strategy to limit the cost for user.

enter image description here

Take General-purpose Storage Accounts for example, the cost depends on Storage transactions and Capacity. If we want to limit the cost for the user. On my opinion, we could limit the size*transaction for the user.

  • Create relationship for SAS tokens and user

  • Get the request uri and request-content-length from the log and we also count the request for user

  • Use logic to control the size*transaction for user.

About Storage Analytics Log Format please refer to azure document.

Other related link:

Understanding Windows Azure Storage Billing – Bandwidth, Transactions, and Capacity

Upvotes: 0

Related Questions