CODEWITHSUNDEEP
node.jsmariadbsequelize.js
Patrick
Patrick

Reputation: 41

Fetching 3.6 million records with Sequelize crashes Node, MariaDB Connector works. Any idea why?

as the title already says, I'm trying to run a raw SELECT query that results in 3.6 records. When I use the MariaDB Connector (https://github.com/mariadb-corporation/mariadb-connector-nodejs) it works just fine and is done in ~2 minutes. But Sequelize takes much longer and in the end, Node crashes after 20 to 30 minutes.

Do you guys have any idea how to fix this?

Thank you very much and have a great day!

Take care,

Patrick

Upvotes: 0

Views: 1140

Answers (2)

Rick James
Rick James

Reputation: 142518

Redesign your app to do more of the work in the database. Downloading 3.6M rows is poor design for any application.

If you would like to explain what you are doing with that many rows, maybe we can help create a workaround. (And it may even work faster than 2 minutes!)

Upvotes: 0

Orelsanpls
Orelsanpls

Reputation: 23565

When you perform your request, sequelize will perform a SELECT on the underlying database.

Then two thing will happend consecutively :

  • MariaDB will load all the data matching your criteria

  • MariaDB will send all the data to sequelize, that will :

    • Overload your app memory (all the data will be stored into node.js memory)
    • Crash sequelize because it is not made to handle that much data

When you perform request on huge dataset, use cursors. With cursors, MariaDB will load all the data but then, sequelize will get the data by group (For example, sequelize load 100 data, you treat it, then it load 100 data again, which means that at top you will have loaded 100 data on your node.js memory).

https://github.com/Kaltsoon/sequelize-cursor-pagination

https://mariadb.com/kb/en/cursor-overview/

Upvotes: 2

Related Questions