CODEWITHSUNDEEP
node.jsmongodbherokuconcurrencymongodb-atlas
Luc Chevalier
Luc Chevalier

Reputation: 33

How to manage concurrent writes to a large (5mb) MongoDB document with Node JS

I built an app that manages sports tournaments using MongoDB, Mongoose on NodeJS. I'd like to know if I am using the best solution to handle multiple concurrent writes to a large document (5Mb) in rapid succession.

Each "Event" (tournament) is a single document that contains a list of teams. There is a maximum number of teams that can register to each Event. So normally, when a team registers, my Node JS server will load the event, check if the max number of teams has not been reached, add the team to sub-documents and save the Event.

The problem is that some tournaments make players frantic to get a spot and you can have 60 teams complete their registration in the opening seconds which would cause concurrency errors.

For example, if 2 teams click on "save" at the same time, 2 threads (requests) will open on the NodeJS server, both threads will load identical copies of the event, modify them and save two different versions of the document over one another. Obviously, you will get a version error for one of the two threads. Now imagine 60 teams registering within the same second.

The second problem is that the Event document is quite large. Let's be dramatic and say it's 5Mb in size (rare but possible). If I have to load, modify, write 5 megs per registration, the registration system is going to grind to a halt (since my MongoDB is on a different server.)

So I need to know if I built the right solution and if you guys foresee problems with this.

On my node server, I built a Singleton class (accessible to all requests) to manage access to documents. So if a request comes along and asks for Document X, the singleton returns a Promise to the request which will be resolved once this document becomes available to edit. The singleton then turns around, loads the document and grants access to the first request by resolving it's promise. When the request is done editing this document, it tells the singleton that it's done. The singleton then checks if there is queue of other requests waiting to edit this document (other teams that want to register). If so, it does NOT save the document but rather resolves the next promise, allowing the next request to edit the document.

When the last request has finished editing the document and there are no more requests in the queue, the singleton saves the document and clears it from memory.

So in short, the singleton allows the system to load the document once, allow modifications from multiple requests and then saves the document at the end of the rush. This is especially useful since the document is rather large (up to 5mb) and minimizes the number of read/writes to the MongoDB server. The other use is that if we're accepting 50 teams and we get 55 requests wanting to append their teams, the last 5 requests in the queue will take into account that the live document has reached it's team limit and return a "sorry we're full" response.

Is this the best way to manage concurrent writes to a large document?

Upvotes: 0

Views: 128

Answers (1)

D. SM
D. SM

Reputation: 14520

MongoDB provides a multitude of update operators that you should be using on the specific fields instead of modifying the entire document in your application. For example, for adding to arrays use https://docs.mongodb.com/manual/reference/operator/update/push/.

This way you 1) will only be sending the changed data on each write and 2) avoid racing yourself and clobbering your other changes.

This doesn't help you with the time it takes the server to rewrite that 5 mb document each time it's modified - split the document up to fix this (if you find it to be an issue).

Upvotes: 1

Related Questions