Looping & Nesting Promises

Question

I am using Request-Promise and cheerio to scrape some website data, essentially I am trying to achieve the following:

1. Create An Empty Array
2. Login
3. Get some info from one page and push objects into the array
4. Get some info from another page and push objects into the array
5. For each now object in the array, I need to:
   • Go to the URL stored within that object {link: "some url", items: []}
   • loop through all the items found within that link, and push this to the items array within the iterated object as such: {link: "some url", items: [{item},{item}]}.
6. Access the completed orderArray, which should spit out something like this:

{link: "some url", items: [{item},{item}]},
{link: "some url", items: [{item},{item}]},
{link: "some url", items: [{item},{item}]}

Step 6 is where I'm running into issues, I don't know how to do this without nesting the promise inside a for loop as per my code below which then starts getting nasty. Can I be pointed in the right direction here?

Here is my current code:

    let orderArray = [];

    rp.post(login)

    .then(function(res1){

        // Login & Set Cookies
        cookieJar = res1.headers['set-cookie'];

        return rp(getOpenOrders);

    })

    .then(function($){

        // Get Some Info from getOpenOrders

        orderArray.push({info});

        return rp(getShippedOrders);

    })

    .then(function($){

        // Get Some Info from getShippedOrders

        orderArray.push({info});

        return orderArray;

    })

    .then(function($){

        // Loop through each object in the orderArray
        for (i = 0,; i < orderArray.length; i++){

            rp(orderArray[I].link)

            .then(function($){

            //Get length of items on page
            let itemsOnPage = $('tbody tr').length;

            //Get some more details for each object
            for (j = 0,; j < items.length; j++) {
                    let moreinfo = {…};
                    orderArray.items.push(moreinfo);
            }

          }
        }

        return orderArray;

    })

    .then(function($){

        // Log finished Array
        console.log(orderArray);

    })

    .catch(function(err){
        console.log(err);
    })

    };

Answer 1

The easiest and cleanest way is using async/await. That code won't run in parallel though (Unless we await Promise.all)

.then(async() => {

    // Loop through each object in the orderArray
    for(let i = 0; i < orderArray.length; i++) {

        // Some may argue no await inside loop...
        // We wait for rp to resolve, it looks like
        // synchronous code so it's easy to understand
        const $ = await rp(orderArray[i].link);

        let items = $('tbody tr');

        for(const item of items) {
            let moreinfo = {};
            orderArray[i].items.push(moreinfo);
        }

    }

    return orderArray;
})

You can also use Promise.all to send all requests in parallel and handle the results when they all finish.

.then(() => {

    // Loop through each object in the orderArray

    const requests = [];

    for(const order of orderArray) {
        // Push each promise
        requests.push(
            rp(order.link)
        );
    }

    // This will resolve when every request finishes
    // If one fails, it will reject, going to `.catch`
    return Promise.all(requests); 
})
.then(results => {

    // Results is an array containing each individual request result.

    results.forEach(($, i) => {

        //Get length of items on page
        let items = $('tbody tr');

        //Get some more details for each object
        for(const item of items) {
            let moreinfo = {};
            orderArray[i].items.push(moreinfo);
        }

    })

    return orderArray;

});

I assume rp resolves a cheerio object, if it doesn't please let me know.

I can't test it because I don't have your full code, but it should work.