CODEWITHSUNDEEP
node.js
Manoj Thakur
Manoj Thakur

Reputation: 75

Handling of large array of items in nodejs

I have array of more then 100 items. Each item hits a url. After getting response from the specific url of the item I do insert option in the database.

If I use async await, then I have to wait till the 100 items get processd. But if I do asynchronoulsy, then I got response but as asynchronus behavior the data in db will not be visible untill all process get completed.

Example 1: using async await

var loc = async(locations){
    return new Promise((resolve,reject) => {

        try{
            for(let item of locations){
                var response = await html(location.url);
                await insertDB(response);
            }
        }catch(e){
            console.log('error : ',e);
        }

        resolve('All operation completed successfully');
    })
}

loc(locations);

locations is array of items In this I will get response only when all the requests get completed.

Example 2: asynchronously

var loc = (locations){
    return new Promise((resolve,reject) => {

        locations.forEach(location => {
            html(location.url,(res) => {
                insertDB(response,(rows) => {

                });
            });
        })
        resolve('All operation completed successfully');
    })
}

loc(locations)

Here I will immediately get the response, but the process on hitting and insertion will be processing in the background.

Considering example 1, is it possible that we can divide the looping request into child process so that make it execute fast in nodejs or any other way to do it?

Upvotes: 1

Views: 533

Answers (1)

LHJ
LHJ

Reputation: 692

Is this what you want? It will work like your 2nd solution, but it will be resolved after every insert complete

var loc = (locations){
    return new Promise((resolve,reject) => {
        const promises = locations.map(async (location) => {
            try { 
                var response = await html(location.url);
                await insertDB(response);
            } catch(e) {
                console.log('error : ',e);
            }
        })
        await Promise.all(promises);
        resolve('All operation completed successfully');
    })
}

Should have difference. Can you do me a favor to test the performance time.
solution1() as your first solution function
solution2() as my solution function

console.time('solution1');
await solution1(locations);
console.timeEnd('solution1');

console.time('solution2');
await solution2(locations);
console.timeEnd('solution2');

assume insertDB cost 100ms ~ 200ms, avg = 150ms
Solution 1:
run loop 1, wait insertDB -> run loop 2, wait insertDB
time = 150ms + 150ms = 300ms
Solution 2:
run loop 1, without wait insertDB -> run loop 2, without wait insertDB
till all insertDB complete, then resolve
time = 150ms
because all insertDB start without waiting another insertDB
credit: https://stackoverflow.com/a/56993392/13703967

test: https://codepen.io/hjian0329/pen/oNxdPoZ
result

Upvotes: 1

Related Questions