CODEWITHSUNDEEP
node.jsmongodbmongoosesequelize.js
Matt
Matt

Reputation: 650

JavaScript heap out of memory while updating the mongodb

I am trying to synchronize data between to data stores, the source is mssql and the destination is MongoDB. In this syncing process I am getting a memory heap error. I am not sure why this happens and I am fully aware that the following code may not be the best, but for now I am just trying to understand why the allocation error is coming.

I am compiling my code with babel, in development I am just using babel-node.

try {
  const response = await sqlDataStore.findAll({
    attributes: ['id', 'Name'],
  });
  /* eslint no-restricted-syntax: 0 */
  for (const item of response) {
    /* eslint no-await-in-loop: 0 */
    await this.Model.updateOne({}, item, { upsert: true });
  }
} catch (err) {
  console.log(err);
}

If I understand correctly the heap error is caused by the for loop, so that would mean that every await statement is cached in the memory. I would have expected that every await statement is cleared from the memory because I am not assigning it to any variable.

Updated:

Gladly I found already a solution due to another post: Bulk upsert in MongoDB using mongoose

My Code:

  const response = await sqlDataStore.findAll({
    attributes: ['id', 'Name'],
  });

  const bulkUpdate = response.map(doc => ({
    updateOne: {
      filter: { _id: doc.id },
      update: doc.dataValues,
      upsert: true,
    },
  }));

  this.Model.collection.bulkWrite(bulkUpdate);

If someone is using this solution it should be kept in mind that this also could crash for lots amount of data. The solution provided in the other posts suggests that the data should be processed in buckets of 1000 until every document is updated/inserted.

Just for interest and technical understanding I would appreciate an explanation of what exactly I did wrong in my first code.

Upvotes: 3

Views: 1552

Answers (1)

Atul Agrawal
Atul Agrawal

Reputation: 1520

You are getting this because your function call stack is not getting free since they are waiting for other calls to complete its execution.

Since all the call stacks blocked your stack memory and hence after some executions you are getting out of memory exception.

Check this link: https://eslint.org/docs/rules/no-await-in-loop

As you can see your await call blocked in memory for other await to finish and they are returning you the value in one go which is bad in your code.

It is actually you are making a sync call and each sync call waiting for other sync calls to finish and finally, your sync calls get pile up in stack memory and you are getting the exception .

Upvotes: 2

Related Questions