nodejs express search in request parallel single response

Question

i need to query multiple pages from another api and return a object from the page if a specified value matches.

i guess the problem is that the loop is done asychron because i always get "not found" and later i get "Cannot set headers after they are sent to the client" if the loop found the object.

solved this by calling it recursive but i need more speed because there are many pages with many entries. if possible requests should run parallel but not "found should" be called after all loops finished

router.post('/search', function (req, res) {
    var foundObj = false;
    for (var page = 1; page < req.body.cubesize; page++) {
        request({
            method: 'GET',
            uri: 'http://localhost:8080/api/v1/getpage/json/' + page
        },
            function (error, response, body) {
                if (!error) {
                    var result = JSON.parse(body);
                    for (var obj in result) {
                        console.log(result[obj]);
                        if (result[obj].hasOwnProperty(req.body.field)) {
                            if (result[obj][req.body.field] == req.body.value) {
                                foundObj = true;
                                return res.status(200).send(result[obj]);
                            }
                        }
                    }
                }
            });
    }
    if(!foundObj){
        return res.status(404).send("not found");
    }
});

anyone got an idea how to fast loop all pages with all entries but wait for calling not found?

Answer 1

As long as you have a res.send() inside a for loop and at least two matches occurs, two (at least) res.send() calls will be executed and an error will rise.

How to run in parallel ?

router.post('/search', function (req, res) {
     
    const callApi = (page) => new Promise( (resolve, reject) => {
        request({
            method: 'GET',
            uri: `http://localhost:8080/api/v1/getpage/json/${page}`,
        }, 
            function (error, response, body) {
                if (error) reject(null)
                let result = JSON.parse(body);
                for (var obj in result) {
                    console.log(result[obj]);
                    if (result[obj].hasOwnProperty(req.body.field)) {
                        if (result[obj][req.body.field] == req.body.value) 
                            return resolve(result[obj]);
                        
                    }
                }
                return reject(null);
            }
        });
    });

    const promisesArr = [];
    for ( let page = 1; page < req.body.cubesize; page++) {
        promisesArr.push(callApi(page))
    }


    Promise.allSettled(promisesArr).then((resArr)=>{

        const resolvedArray = resArr.filter(val => !!val);

        if (resolvedArray.length === 0) return res.status(404).send("not found");

        if (resolvedArray.length === 1) 
            return res.status(200).send(resolvedArr[0][obj])

       if (resolvedArray.length > 1) 
            return res.status(500).send("Too many matches")
            // It is not clear to me in your code what you need to do in case more than one resolves
    });

});

Some explanation about the code.

• The idea is to promisify request and run in parallel
• To run in parallel, Promise object allows four methods: Promise.all, Promise.race and Promise.allSettled and Promise.any. The two last ones, Promise.allSettled and Promise.any are not fully compatible, so keep this in mind.
• Once you have the array and run in parallel, Promise.all and Promise.allSettled returns an array of results. The array is filtered and if some value matchs, it response that, otherwise, response 404.

Further information about promises will be required to select the right one for your specific case. You can found about it here[https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise]

Unfortunately my code is not tested, so please review it and refactor to adapt to your specific case.