Access denied to storage account from Azure Data Factory

Question

My goal is to run an exe file stored in a private Azure Blob container.

The exe is simple : it creates a text file, write the current datetime in it, and then push it to the private Azure Blob container.

This has to be sent from Azure Data Factory. To do this, here is my environment :

• Azure Data Factory running with the simple pipeline : https://i.sstatic.net/txQ9r.png

• Private storage account with the following configuration : https://i.sstatic.net/SJrGX.png

• A linked service connected to the storage account : https://i.sstatic.net/8xW5l.png

• A private managed virtual network approved : https://i.sstatic.net/G2DH3.png

• A linked service connected to an Azure Batch : https://i.sstatic.net/Yaq6C.png

• A batch account linked to the right storage account

• A pool running on this batch account

Two things that I need to add in context :

• When I set the storage account to public, it works and I find the text file in my blob storage. So the process works well, but there is a security issue somewhere I can't find.
• All the resources (ADF, Blob storage, Batch account) used have a role has contributor/owner of the blob with a managed identity.

Here is the error I get when I set the storage account to private :

{
   "errorCategory":0,
   "code":"BlobAccessDenied",
   "message":"Access for one of the specified Azure Blob(s) is denied",
   "details":[
      {
         "Name":"BlobSource",
         "Value":"https://XXXXXXXXXXXXXXXXX/testv2.exe?sv=2018-03-28&sr=b&sig=XXXXXXXXXXXXXXXXXX&sp=r"
      },
      {
         "Name":"FilePath",
         "Value":"D:\\batch\\tasks\\workitems\\XXXXXXXXXXX\\job-1\\XXXXXXXXXXXXXXXXXXXXXXXX\\testv2.exe"
      }
   ]
}

Thank you for your help!

Answer 1

Solution found Azure community support :

Check Subnet information under Network Configuration from the Azure portal > Batch Account > Pool > Properties. Take note and write the information down.

Navigate to the storage account, and select Networking. In the Firewalls and virtual networks setting, select Enable from selected virtual networks and IP addresses for Public network access. Add the Batch pool's subnet in the firewall allowlist.

If the subnet doesn't enable the service endpoint, when you select it, a notification will be displayed as follows:

The following networks don't have service endpoints enabled for 'Microsoft.Storage'. Enabling access will take up to 15 minutes to complete. After starting this operation, it is safe to leave and return later if you don't wish to wait.

Therefore, before you add the subnet, check it in the Batch virtual network to see if the service endpoint for the storage account is enabled.

After you complete the configurations above, the Batch nodes in the pool can access the storage account successfully.