CODEWITHSUNDEEP
stateapache-flinkstatefulflink-statefun
Arif Ezberci
Arif Ezberci

Reputation: 111

Stateful Functions Fault Tolerant Message Distribution in Apache Flink

I'm trying to implement messaging scenario using apache flink stateful functions.

By design I need to calculate some statistics from incoming messages and store them in the states. After that scenario functions will access these states and messages and run business rules on them. But we may have dozens of scenarios per message and each of them should run exactly once.

the code is more or less as follows

@Override
    public void configure(MatchBinder binder) {
        binder
            .predicate(Transaction.class,this::updateTransactionStatAndSendToScenatioManager)
}

    private void updateTransactionStatAndSendToScenatioManager(Context context, Transaction transaction){
        // state update
        context.send(FnScenarioManager.TYPE,  String.valueOf(transaction.id()) , transaction);
    }

FnScenarioManager:

@Override
    public void configure(MatchBinder binder) {
    binder
        .predicate(Transaction.class,this::runTransactionScenarios);
}


private void runTransactionScenarios(Context context, Transaction transaction){
   context.send(Scenario1.TYPE,String.valueOf(transaction.id()),transaction);
   context.send(Scenario2.TYPE,String.valueOf(transaction.id()),transaction);
   context.send(Scenario3.TYPE,String.valueOf(transaction.id()),transaction);
   ...
   context.send(ScenarioN.TYPE,String.valueOf(transaction.id()),transaction);
}

My question is what happens if cluster crash in the middle of runTransactionScenarios ?

Upvotes: 2

Views: 212

Answers (1)

Seth
Seth

Reputation: 345

Stateful Functions (and Apache Flink in general) supports exactly-once state semantics. What this means is that in the case of failure, the runtime will consistently roll back both state and messages in such a way as to simulate completely failure-free execution.

This means messages may be replayed but the internal state will be rolled back to the point in time before the message was received. So long as your business rules only modify statefun state and interact with the outside world through an egress, you can treat the system as having exactly once properties.

Upvotes: 1

Related Questions