How to rollback MicroServices

Question

I have doubt related to MicroServices. Suppose there are 5 Micro-Services, lets say M1, M2, M3, M3, M4 and M5. There are 4 databases which are connected/accessed by 4 micro-services. For example, M2 connected to MySQL, M3 connected to Cassandra, M4 connected to MongoDB and M5 connected to Oracle.

Now

Step-1: M1 making a call to M2 to update some user data in MySQL and it updated successfully then finally it got success response from M2

Step-2: M1 making a call to M3 to update some data in Cassandra and it updated successfully then finally it got success response from M3

Step-3: M1 making a call to M4 to update some data in MongoDB and it failed due to some DB server problem or any other problem.

Here my requirement is, I want to rollback DB changes that happened to previous micro-services(M2 and M3)

What should we need to do to achieve this kind of rollback scenario?

Answer 1

This is a typical case of distributed transaction. Regardless of the fact that you use separate technology for your database or the same on different server you perform an operation which is transactional. In order to handle a rollback on that type of transaction you can not relay on the database technology mechanism for transactions and rollbacks. You have to do it on your own.

Saga Pattern

Common solution for distributed transaction scenarios in micro-service architecture is the Saga pattern. Distributed sagas is a pattern for managing failures in scenarios as the one that you have described.

Saga are created based on business process for example "Buy a Product in online shop". This process can involve multiple actions on multiple micro-services. Saga will control and manage this process execution and if one of the steps fail it will trigger actions to revert the actions done before the failing action.

There are multiple ways to implement sagas. It depends on your architecture and the way your micro-services communicate with each other. Do you use Commands and/or Events?

---

Example

"Buy a Product in online shop" business process. Lets say this business process has 3 simple steps done by 3 different micro-services:

• Action 1 - Reserve Product in products-inventory-micro-service
• Action 2 - Validate payment in payment-micro-service
• Action 3 - Order a product in orders-micro-service

Using Events:

You can publish events to perform some action(or actions) and if one of the action fails you can publish a revert(or delete) event for that event. For the above business process lets say the 1. Action succeeded and the 2. Action failed. In this case in order to rollback the 1. Action you would publish an event like "RemoveReservationFromProduct" in order to remove the reservation and revert the state back to the state as it was before the transaction for that Business process started. This event would be picked up by a event handler which would go and revert that state in your database. Since it is an event you can implement retry mechanism for failures or just reapply it later if there is some bug in the code.

Using commands:

If you have direct calls to your micro-services as commands using some kind of rest api you could run some delete or update endpoints to revert the changes that you have done. For the above business process lets say the 1. Action succeeded and the 2. Action failed. In this case in order to rollback the 1. Action you would call the delete api to delete the reservation for a particular product in order to remove the reservation and revert the state back to the state as it was before the transaction for that Business process started.

You can take a look at this example how to implement the Saga pattern.

Answer 2

to answer your question lets add some business requirements

Case 1. M1 is doing all interaction with other microservices based on an event recieved like Order Placed

Now in this case M2 ... M5 update ,

requirement 1: if all of them are independent of each other.

first create 5 event from one event and then

in such a case you could add this event in a table mark this event as unprocessed and some timer reads unprocessed event and tries to do all the tasks in a Idempotent way, also you could have reporting if such tasks are failing and your team can check them and manually resolve them.

(you could implement a similar logic by using a failover queue - which sends the same event back to the original queue after some time)

requirement 2: if all are not independent

use a single event and still the same solution.

in the above solution the main benefit is even if your system restart in between the transactions you will alwayss eventually have the consistent system

Case 2. if the M1 api is invoked and M1 needs to do all tasks from multiple microservice and then give response to user.

we could create a started event in M1 microservice DB (sync_event_table) try to do update in all microservice after all complete , update the sync event table with completed

for those cases which are not completed - run a timer which checks for job which are not completed for > X min and then do the undo actions or whatever required,.

Essence:

So if you see all solutions suggests your system to turn all the diff. microservice update

by creating a job

checking job status

writing a undo/redo job feature

Answer 3

From what I understand, a Saga is what you are looking for. The idea is to provide for every state altering operation an undo-operation, that has to be called if things went bad down stream.

Answer 4

You can make sure that you have @Transactional enabled in this entire sequence of Invocation.

1. Consider invocation of all microservices from M1 as single transaction.
2. Expose a rollback in following way:
   • While updating DB in M2, M3 and M4, place the values in Spring cache as well along with DB.
   • Upon invoking /rollback in M2, M3 or M4, get the values from Spring Cache and undo them from DB.
3. In the fallbackMethod of hysterix command, when M1 replies with error or some default output, invoke /rollback of other services.

This may not be a perfect solution, as it introduces another fault point as /rollback handling, but fastest one that can be implemented.