CODEWITHSUNDEEP
c#azure-functionsorchestrationazure-durable-functions
Jeppe
Jeppe

Reputation: 1575

Azure Durable orchestration function triggering twice

I am trying to implement an Azure Durable Function workflow.

Every 6 minutes I have an Azure TimerTrigger function calls an Azure Orchestration Function (OrchestrationTrigger), that in turn starts a number of activity functions (ActivityTrigger).

Sometimes, however, the Orchestration function gets called twice in the span of a few seconds! This is a big problem as my activity functions are not idempotent!

Below is how my code gets called.

TimerTriggered Function: 

[FunctionName("StartupFunc")]
public static async Task Run([TimerTrigger("0 */6 * * * *", RunOnStartup = true, UseMonitor = false)]TimerInfo myStartTimer, [OrchestrationClient] DurableOrchestrationClient orchestrationClient, TraceWriter log)
{
    List<OrchestrationModel> ExportModels = await getData();

    string id = await orchestrationClient.StartNewAsync("OrchestratorFunc", ExportModels);
}

Orchestration Function:

[FunctionName("OrchestratorFunc")]
public static async Task<string> TransformOrchestration([OrchestrationTrigger] DurableOrchestrationContext context, TraceWriter log)
{
    var dataList = context.GetInput<List<OrchestrationModel>>();
    var tasks = new List<Task>();

    foreach (var data in dataList)
    {
        tasks.Add(context.CallActivityAsync<string>("TransformToSql", new TransformModel(data));
    }
    await Task.WhenAll(tasks);
}

Activity function:

[FunctionName("TransformToSql")]
[public static async Task<string> RunTransformation([ActivityTrigger] DurableActivityContext context, TraceWriter log)
{
    TransformModel = context.GetInput<TransformModel>();

    //Do some work with TransformModel
}

Upvotes: 7

Views: 11259

Answers (3)

Adrian Stanisławski
Adrian Stanisławski

Reputation: 755

It may not be connected but... If you are debugging your durable function code and using emulator storage - stopping the execution in the middle will still save the state of the flow. During the next debugging session when you start your orchestrator, the previous run will also be restarted to continue. This may look like a double execution when in reality, those are two separate triggers.

Upvotes: 0

Marc
Marc

Reputation: 1898

The orchestration function will run more frequently as it is replayed by the Durable Function framework. Do you see that your activity functions are also triggered again? If so then this is indeed a strange behavior.

Durable Functions uses storage queues and tables to control the flow and capture state of the orchestration.

Each time the orchestration function is triggered (by receiving a message from the control queue) it will replay the orchestration. You can check this in code by using the IsReplaying property on the DurableOrchestrationContext. 

if (!context.IsReplaying)
{
   // this code will only run during the first execution of the workflow
}

Don't use the IsReplaying to determine to run the activities, but use it for logging/debugging purposes.

When an activity function is reached a message is put on the work-item queue. The Durable Functions framework will then look into the history table to determine if this activity function has already been run before (with the given parameters). If it is, then it won't run the activity function again and it will continue with the rest of the orchestration function.

The checkpointing and replay functionality of Durable Functions only works when the orchestration code is deterministic. So never use decisions based on DateTime or random numbers.

More info: https://learn.microsoft.com/en-us/azure/azure-functions/durable-functions-checkpointing-and-replay

Upvotes: 8

kamil-mrzyglod
kamil-mrzyglod

Reputation: 5008

This behaviour is perfectly fine - this is how Durable Functions works by design.

You have the following orchestration:

[FunctionName("OrchestratorFunc")]
public static async Task<string> TransformOrchestration([OrchestrationTrigger] DurableOrchestrationContext context, TraceWriter log)
{
    var dataList = context.GetInput<List<OrchestrationModel>>();
    var tasks = new List<Task>();

    foreach (var data in dataList)
    {
        tasks.Add(context.CallActivityAsync<string>("TransformToSql", new TransformModel(data));
    }
    await Task.WhenAll(tasks);
}

When an activity is called, the flow returns to a concept called Dispatcher - it is an internal being of Durable Functions, responsible for maintaining the flow of your orchestration. As it awaits until a task is finished, an orchestration is deallocated temporarily. Once a task is completed, the whole orchestration is replayed until the next await happens.

The important thing there is, though an orchestration is replayed, an activity is not called once more - its result is fetched from the storage and used. The make things more explicit, please consider the following example:

[FunctionName("OrchestratorFunc")]
public static async Task<string> TransformOrchestration([OrchestrationTrigger] DurableOrchestrationContext context, TraceWriter log)
{
    var dataList = context.GetInput<List<OrchestrationModel>>();
    var tasks = new List<Task>();

    foreach (var data in dataList)
    {
        await context.CallActivityAsync<string>("TransformToSql1", new TransformModel(data);
        await context.CallActivityAsync<string>("TransformToSql2", new TransformModel(data);
    }
}

When TransformToSql1 is awaited, the orchestration is deallocated and the whole flow waits until this activity is finished. Then the orchestration is replayed - it once more awaits TransformToSql1 but since the result of it is saved, it just goes back to the orchestration and awaits TransformToSql2 - then the process is repeated.

Upvotes: 11

Related Questions