CODEWITHSUNDEEP
azure-data-factory
Andrew Buttigieg
Andrew Buttigieg

Reputation: 71

Azure Data Factory WebHook Fails after 1 minute

We have a few Azure Functions that calls an API endpoint that takes >230 seconds (the maximum runtime for Azure Function call from ADF). The work around we found was to use the Webhook activity and using the callBackUri. But for whatever reason, the webhook always fails at 00:01:01 with a BadRequest Error:

BadRequestError:
BadRequestError

If the function completes within that minute, the callback is working correctly and runs fine.

The WebHook's Timeout is set to 10 minutes (00:10:00), but after 1 minute it will raise a BadRequest error. The Azure function continues to run in the background and will successfully complete it's task, but my pipeline is now broken and not continue to the next step.

I cannot use Durable Azure Functions, as that is not yet supported in Python Azure Functions.

Upvotes: 4

Views: 4136

Answers (2)

bit
bit

Reputation: 4487

I managed to use a work around in the Azure Function itself by using -Timeout param. Details here: Using Azure Data Factory's Web Hook Activity to invoke/poll Azure Powershell Function App times out after one minute

Sample running code below:

$Body = @{
            callbackUri = $Request.Body.callBackUri;
        } | ConvertTo-Json
try{
    # This API will be responsible for issuing the call back after it has finished the long running process
    $output = Invoke-RestMethod -Method Post -Body $Body -Uri $funcapp2Url -ContentType 'application/json' -TimeoutSec 1
}
catch {
    Write-Output $_
}

# Return HTTP 202 immediately
Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{
    StatusCode = [HttpStatusCode]::Accepted
    Body = "Wait for callback"
}) -Clobber

Upvotes: 1

Andrew Buttigieg
Andrew Buttigieg

Reputation: 71

After some further invetigation, the 1 min timeout error is expected. Reading the docs, for long running calls, the activity expects a 202 (Accepted) responce within the minute.

https://learn.microsoft.com/en-us/azure/data-factory/control-flow-webhook-activity#additional-notes

Details on the Asynchronous Request-Reply pattern (and sample c# code) is avalable here: https://learn.microsoft.com/en-us/azure/architecture/patterns/async-request-reply

Upvotes: 3

Related Questions