What are different services in Cadence and how history service works as a core workflow engine?

Question

This doc shows the basic architecture of Cadence workflow service: it has three different micro services: FE, MS and HS. So what they are and how they work together?

Answer 1

My Systems@Scale talk explains the overall architecture of Temporal as well as the reasons for such architecture choices.

Answer 2

• FE represents frontend service. Most users only need to know this as Cadence service. It’s mainly for routing requests to the right history service(HS) or matching service(MS)
• Matching service is for matching workflow/activity tasks to workflow/activity workers. Matching got the tasks from history service. If workers are active the task will be matched immediately , It’s called “sync match”. If workers are not available, matching will persist into database and then reload the tasks when workers are back(called “async match”)
• History service does the core logic of cadence workflow engine. It’s the most complicated service compared to others. The rest of the post will explain how it works with other services as the core service of workflow engine.
• There is another internal service “worker service” missing in the graph. Usually we called it “sys worker service “ to be different from client workflow worker and activity worker. The service implements the system workflow to maintain and Cadence service itself— yes , Cadence uses Cadence to implement itself. Examples are : archival workflow which archive history when the feature is enabled; parent close policy workflow— it help close the childWF when parent workflow are closed; batcher workflow which provides the feature to do batch operations like signaling/terminate: etc

So let’s take an example of a helloworld workflow to understand how history service works with others.

• Golang Helloworld Sample
• Java Helloworld Sample

In happy case(no timeout) , this helloworld workflow will make two decisions:

• As the first decision , it schedules an activity
• As the second decision, it completed the workflow

func helloWorldWorkflow(ctx workflow.Context, name string) error {
    ao := workflow.ActivityOptions{
        ...
    }
    ctx = workflow.WithActivityOptions(ctx, ao)
    var helloworldResult string

        // on executing this line, worker will respond decision result of scheduling the helloworld activity, and then `Get` will block until activity is completed by activity worker         
    err := workflow.ExecuteActivity(ctx, helloWorldActivity, name).Get(ctx, &helloworldResult)
    if err != nil {
        return err
    }

        // on executing this line, worker will respond decision result of completing the workflow 
    return nil
}

Here is the timeline of how history service works with others. Here uses bracket to tell history event is written : [HistoryEventName]

• When history service received a start workflow request , it persists the workflow entity and history into database, along with a decision task (and some timeout tasks, we will talk about it in the future) into a task queue. [WorkflowExecutionStartedEvent] [DecisionTaskScheduledEvent]
• The decision task is also called “transfer decision task” internally. (In metrics there is a “TransferActiveTaskDecision” counter increases for it). Because it will be transferred to matching service. ("Active" means it's on active side in global domain, we will cover this in future posts)
• A transfer queue processor keeps polling transfer task queue. Upon getting the transfer decision task, it pushes the task to matching engine.
• Assuming there is an active workflow worker polling(calling “PollForDecisionTask API) for the hello world workflow task. Then worker will receive the task and processing it. [DecisionTaskStartedEvent]
• As the result of the decision task, an activity is scheduled. A “RespondDecisionTaskCompleted” API is called to history service with the results.
• History processes the request— it writes down the decision [DecisionTaskCompletedEvent][ActivityTaskScheduledEvent]
• History service also write down an transfer activity task into transfer task queue
• Transfer task queue processor will poll the queue and got the task. It will push the activity task to matching engine.
• Again, assuming an activity worker is active — calling “PollForActivityTask” API for hello world activity. The worker will got the task and processes it. [ActivityTaskStartedEvent]
• The helloworld activity is simple and worker will finish it very fast. It will return the result by calling “RespondActivityTaskCompleted” API request
• History will process the request. It write down the history event and schedules another transfer decision task. [ActivityTaskCompletedEvent] [DecisionTaskScheduledEvent]
• Repeat the decision task processing again — transfer task queue will push it to matching service and then workflow worker will get the decision task from matching engine. [DecisionTaskStartedEvent]
• As the result of the decision task, helloworld workflow will make a decision of completing the workflow by calling "RespondDecisionTaskComplete" API.
• When history service process the request, it writes down history event and complete the workflow. [DecisionTaskCompletedEvent][WorkflowExecutionCompletedEvent]

This is the sample history of the helloworld workflow. It is exactly the same as above.

• WorkflowExecutionStartedEvent
• DecisionTaskScheduledEvent
• DecisionTaskStartedEvent
• DecisionTaskCompletedEvent
• ActivityTaskScheduledEvent
• ActivityTaskStartedEvent
• ActivityTaskCompletedEvent
• DecisionTaskScheduledEvent
• DecisionTaskStartedEvent
• DecisionTaskCompletedEvent
• WorkflowExecutionCompletedEvent