How do you solve concurrency issues when horizontally scaling microservices that are sharing a DB?

Question

Let's assume that you are running two instances of a Tickets Service sharing a MongoDB database.

We receive two updates on the same ticket from a client that will be handled concurrently by the Tickets Service instances thanks to load balancing. For some reason, the request made after the other one updates the ticket before the other could even reach the database thus breaking the original order of the requests. So, the request that should have been handled before is discarded (for example) when adopting an Optimistic Concurrency Control versioning system and creates an inconsistency that could potentially be dangerous for some kind data (like account balances).

How do you solve this kind of problem and guarantee the correct ordering and consistency?

Answer 1

In my case the two requests should be saved into the database with the same order made by the client

The almost only way to somewhat guarantee this I can think of is to guarantee that user can uses only single client instance (for example via blocking login which will validate there are no other active sessions) at a time and make that client instance assign the ordering to the send requests (i.e. atomically increment and use some counter client side which will be then used server side for correct ordering), which obviously not something you usually want in some accounting system (for example if you are writing some banking system you usually do not want to prevent user from using ATM while using mobile bank client).

however due to the fact that the Tickets Service have two instances, the order is being altered.

If we are talking about using such communication channels like HTTP, TCP/IP, etc. you can't actually guarantee the ordering even if you have a single service instance because the order can be scrambled long before the requests hit your server (due to transport specifics or even client machine CPU scheduling, in theory), not to mention that single instance still usually processes requests in parallel (though here could be nuances) and is susceptible in general to the same issues but on smaller scale (and can use some other tools for synchronization).

Consider the case when an user wants to deposit and withdraw from his account balance, you must respect the order of the requests made by the client.

Actually you kind of don't. You must respect business rules like you can't withdraw more than there is on account balance + allowed overdraft. If user sends withdraw request which will overdraft the balance over the limit before the deposit request is processed or even acknowledged then it is kind of client problem and the withdraw attempt should be retried.

What (I would argue) you actually want/need is to guarantee that your two instances will not perform "unatomic"/non-synchronized updates on the same data. Usually it is handled via transactions on the database side with appropriate isolation levels. Another approach can be to test for optimistic concurrency violation approach, i.e. (for most relational databases) you can just use query looking something like the following:

update Ticket 
  set Version = new_unique_id -- for example guid, or next id from sequence 
      , ... -- rest of the update
where Id = ... and Version = current_unique_id

And then check if the returned number of updated rows is equal to 1.

TL;DR

There are cases when you can require strict client ordering but in majority of ("enterprise") systems (at least which I have encountered) do not, usually you care about enforcing some ordering on server and there are several techniques for that which you can use depending on your infrastructure.

Answer 2

If you use the linearizable read concern with the majority write concern, the threads performing queries will appear to be executing queries as if a single thread executed the operations. If you use the replaceOne query, it looks like it should be possible to implement optimistic concurrency control as in Guru's answer: read a document with linearizable read concern (include a version in the document, e.g. "version": 42), construct a new document with an incremented version (e.g. "version": 43), and use replaceOne with majority write concern to ensure that the document will only be updated to this version if a majority of nodes still see the previous version. If the write fails because a majority of nodes have a later version, re-read the document.

The Mongo docs note that this will generally not be as performant as other strategies, especially in the case where you have to attempt things multiple times. If a fairly high level of concurrency is expected, it may be worth having your microservice instances form a stateful cluster among themselves and allocate responsibility for operating on a particular ticket or account amongst them. Since one instance is responsible for a given ticket/account/etc., local pessimistic concurrency control can be used to effectively make the operations (including the "modify" part of "read-modify-write") against Mongo on a given document be executed by a single thread: this also can let you safely cache values in the instance's memory (turning "read-modify-write-read-modify-write..." into "read-modify-write-modify-write..."): depending on how frequently the modifications are being attempted, this may be a major win or a lot of extra complexity. It's especially common in actor style approaches where there's tooling to simplify a lot of this (e.g. Service Weaver for golang, Akka.Net, Akka on the JVM, Orleans, or Erlang/Elixir... disclaimer: my employer maintains one of those projects).