CODEWITHSUNDEEP
cadence-workflow
GKang
GKang

Reputation: 11

How to handling activity worker failure in Cadence

I am exploring Cadence and have a question on failure recovery. I understand that workflows are fault tolerant (workflow history is maintained), in case of workflow worker failure. I couldn’t find the same guarantees for activity worker. Example: say an activity makes RPC call to service A, which changes some remote object state; now, let’s assume that the call succeeded but activity worker is lost before notifying Cadence service. In this case, would Cadence schedule the activity again on a new worker?

I understand that the above may not be a problem if Service A is idempotent. What are the recommendation of handling above scenario in Cadence, if Service A is not idempotent.

Upvotes: 1

Views: 867

Answers (1)

Maxim Fateev
Maxim Fateev

Reputation: 6890

Cadence by default doesn't retry activities. So in the scenario of the activity worker failure the workflow is going to get a timeout error and can handle it accordingly to its business logic. For non idempotent activities it is usually done by running compensating activities.

Cadence also supports automatic retries for idempotent activities. It is done by providing a retry policy when invoking an activity.

Upvotes: 0

Related Questions