Reputation: 1
I’ve a continuous problem with my ADF pipeline - I’m trying to run a "copy-data" pipeline and encounter access and permission errors with my specific resources. My source is a Microsoft 365 Table connector (I’m retrieving some columns from my organization mails) My sink is a storage account. In my SA access control properties, I’ve given my app appropriate IAM role with the all the necessary permissions so it can access the storage account and write the data successfully. Now since I don’t want my storage to be public, I chose the public network access to be "Enabled from selected virtual networks and IP addresses".
First try: Since ADF is a resource instance in my Azure subscription, I’ve specified my instance to have access to my storage account based on its system-assigned managed identity (Microsoft.DataFactory/factories) and configured all propely (see https://roshan-vin4u.medium.com/authenticate-azure-data-factory-with-azure-data-lake-gen-2-using-managed-identities-3663f1449440). But when I ran the pipeline it failed, claiming I can’t use system-assigned managed identity with Microsoft 365 connector.
Second try:
I tried to access my storage account with private link using ADF PE, so I've created private endpoint and configured it all properly (see https://learn.microsoft.com/en-us/answers/questions/635312/connect-data-factory-to-azure-storage-wiht-private).
First I configured the service endpoint to be the storage's dfs url and run the pipeline, got the error:
"ErrorCode=UserErrorOffice365DataLoaderError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Office365 data loading failed to execute. office365LoadErrorType: PermanentError ...Failure happened on 'Sink' side. ErrorCode=AdlsGen2ForbiddenError"
Then I configured it to be the storage's blob url and got the next error "the remote server return an error: (403) .... Unable to create Azure blob container"
Conclusions I've also tested it and got same results when disabling public network access. My conclusion is that the runtime did access the storage account via the private endpoint but failed (either due to some misconfiguration or some functionality problem). The weirdest issue is when I tried using a simple ADF copy-data pipeline from one storage account to another using PE for my linked-service and integration runtime, it did ran successfully.
What can be the problem? How do I solve this issue?
Upvotes: 0
Views: 352
Reputation: 5317
To copy and transform data from Microsoft 365 (Office 365) into Azure, you need to complete the following prerequisite steps:
You should use service principal authentication instead of system assigned authentication as shown below:
You should use service principal authentication for blob storage also Refer here for how to set up service principal authentication for Azure Blob Storage. In the New Linked Service window, enter "AzureStorageLinkedService" as name, select "Service Principal" from the dropdown list of authentication methods, fill in the Service Endpoint, Tenant, Service principal ID, and Service principal key, then select Save to deploy the linked service.
Then you will be able to copy from Microsoft 365 to blob storage successfully without any error. For more information you can refer below documents:
Upvotes: 0