Azure Data Factory - CRM (OData) Connector

Question

I have an Azure Data Factory for Data extraction from OnPremise CRM. I am running into an issue with one of the Data entities where the Pipeline runs for close to 8 hours and throws this below exception. I know it's not an issue with authentication as I am able to get the other entities without any issues. I tried to change the parallelCopies to 18 and DIUs but when I trigger the Pipeline it sticks to Parallel Copies of '1', DIUs of 4 and eventually fails. Appreciate any inputs.

Operation on target XXXX failed: Failure happened on 'Source' side. ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Upload file failed at path XXXXXXX,Source=Microsoft.DataTransfer.Common,''Type=System.NotSupportedException,Message=The authentication endpoint Kerberos was not found on the configured Secure Token Service!,Source=Microsoft.Xrm.Sdk,'

Answer 1

I ran into something similar when using CRM as a sink; any upsert activities would fail very near exactly 60 minutes. The error I observed in the Azure Data Factory activity was:

'Type=System.NotSupportedException,Message=The authentication endpoint Kerberos was not found on the configured Secure Token Service!,Source=Microsoft.Xrm.Sdk,'

This post helped me find what to change in ADFS. I ran Get-ADFSRelyingPartyTrust and reviewed the TokenLifetime property, which happened to be 0. Apparently tokens last 60 minutes when the configuration is 0.

The following PowerShell increased the timeout, and I confirmed upsert activities no longer fail when exceeding 60 minutes.

Set-ADFSRelyingPartyTrust –TargetName "<RelyingPartyTrust>" –TokenLifetime <timeout in minutes>

Answer 2

It turned out to be a time out setting on the ADFS, once the time out is increased the job ran successfully.