CODEWITHSUNDEEP
stored-procedurestriggersazure-functionssnowflake-cloud-data-platform
Bilal Shafqat
Bilal Shafqat

Reputation: 821

Calling azure function to call a procedure in snowflake to load data causes timeout in consumption plan, is there another way to achieve this?

I want to trigger a procedure in snowflake warehouse to load file from azure blob storage, for that I have implemented snowflake connector as an azure function and it is running on consumption plan (dynamic). But consumption plan has a default timeout of 5mins and max timeout can be of 10mins. But my data is like 50 GB and it takes like 20mins with medium size snowflake cluster. So is there any other way to achieve this?

Upvotes: 1

Views: 460

Answers (1)

suziki
suziki

Reputation: 14080

If you want to get rid of this limitation, you have multiple solutions.

First, you can design a timetrigger to wake up the function before it times out. This timetrigger is periodic, and its period should be less than the timeout of your function.

Second, because the timeout limit comes from the service plan, you can change your service plan to complete your idea.

In a serverless Consumption plan, the valid range is from 1 second to 10 minutes, and the default value is 5 minutes.

In the Premium plan, the valid range is from 1 second to 60 minutes, and the default value is 30 minutes.

In a Dedicated (App Service) plan, there is no overall limit, and the default value is 30 minutes. A value of -1 indicates unbounded execution, but keeping a fixed upper bound is recommended.

Related documents:https://learn.microsoft.com/en-us/azure/azure-functions/functions-host-json#functiontimeout

Third, use durable functions. Under the consumption plan, ordinary out-of-the-box functions run for up to 10 minutes. But if you use durable functions, there is no such restriction at all. It also introduces support for stateful execution, which means that subsequent calls to the same function can share local variables and static members. This is an extension of the normal out-of-the-box functional model, and it requires some additional boilerplate code to make all functions work as expected.

more details about durable functions:https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=csharp

Upvotes: 1

Related Questions