CODEWITHSUNDEEP
node.jsmongodbexpress
Alexander Gerasimov
Alexander Gerasimov

Reputation: 303

NodeJS sharding architecture with many MondoDB databases approaches

We have architecture problem on our project. This project requires sharding, as soon as we need almost unlimited scalability for the part of services. 小urrently we use Node.js + MongoDb (Mongoose) and MySQL (TypeORM). Data is separated by databases through the simple 'DB Locator'. So node process needs connections to a lot of DBs (up to 1000).

Requests example:

We tried to implement it in two ways:

  1. Create connection for each request, close it on response. Problems:
    • we can't use connection after response (it's the main problem, because sometimes we need some asynchronous actions);
    • it works slower;
  2. Keep all connections opened. Problems:
    • reach simultaneous connections limit or some another limits;
    • memory leaks.

Which way is better? How to avoid described problems? Maybe there is a better solution? Solution #1 perfectly worked for us on php as it runs single process on request and easily drops connections on process end. As we know, Express is pure JS code running in v8 and is not process based. It would be great to close non-used connections automatically but can't find options to do that.

Upvotes: 1

Views: 1075

Answers (1)

Seitbekir Seidametov
Seitbekir Seidametov

Reputation: 567

The short answer: stop using of MongoDB with Mongoose 馃槒

Longer answer:

MongoDB is document-oriented DBMS. The main usage case is when you have some not pretty structured data that you have to store, but you don't need to use too much. There is lazy indexing, dynamic typing and many more things that not allow you to use it as RDBMS, but it is great as a storage of logs or any serialized data.

The worth part here is Mongoose. This is the library that makes you feel like your trashbox is wonderful world with relations, virtual fields and many things that should not to be in DODBMS. Also, there is a lot of legacy code from previous versions that also make some troubles with connections management.

You already use TypeORM that may works instead Mongoose. With some restrictions, for sure. It works exactly same way as MySQL connection management.

Here is some more data: https://github.com/typeorm/typeorm/blob/master/docs/mongodb.md#defining-entities-and-columns

In this case you may use you TypeORM Repository as transparent client that will init connections and close it or keep it alive on demand.

Upvotes: 1

Related Questions