CODEWITHSUNDEEP
javascriptnode.jscronschedule
Takezo4
Takezo4

Reputation: 1

Node.js cron task. Multiprocessing or parallel?

There is a MongoDB with 100k records and there is a cron task (using the node-cron module) that runs every hour to process these 100k records. The processing itself takes one hour.

Could you advise on how to split (or parallelize) this process so that the processing could be done every 15 minutes instead of every hour?

In theory, there could be multiple processes running in parallel, each handling 10k records.

What would you recommend using or doing?

The processing itself involves making API requests to retrieve data for each record and, if necessary, updating the record in the database.

cron.schedule('0 * * * *', async () => {
    /* Every hour sync user data */
    const data = await User.find()

    for (const userIdx in data) {
            const user = data[userIdx]
        let userNfts = await getUserNfts(user.address)
        const isSyncedC = await syncNFTS(user, userNfts) // GET request here for every user to third party
            if (isSyncedC) await user.save()
    }
})

I tried to run 5 same tasks with parts of data (10k for each) but sometimes cron runs only first, other are ignored..

Upvotes: 0

Views: 18

Answers (0)

Related Questions