Array of filtered axios results from paginated API is empty

Question

In my code below I get an empty array on my console.log(response) but the console.log(filterdIds) inside the getIds function is showing my desired data. I think my resolve is not right.

Note that I run do..while once for testing. The API is paged. If the records are from yesterday it will keep going, if not then the do..while is stopped.

Can somebody point me to the right direction?

const axios = require("axios");

function getToken() {
    // Get the token
}

function getIds(jwt) {
    return new Promise((resolve) => {
        let pageNumber = 1;
        const filterdIds = [];

        const config = {
            //Config stuff
        };

        do {
            axios(config)
                .then((response) => {
                    response.forEach(element => {
                        //Some logic, if true then:
                        filterdIds.push(element.id);
                        console.log(filterdIds);
                    });
                })
                .catch(error => {
                    console.log(error);
                });
        } while (pageNumber != 1)
        resolve(filterdIds);
    });
}


getToken()
    .then(token => {
        return token;
    })
    .then(jwt => {
        return getIds(jwt);
    })
    .then(response => {
        console.log(response);
    })
    .catch(error => {
        console.log(error);
    });

I'm also not sure where to put the reject inside the getIds function because of the do..while.

ggorlen · Accepted Answer

The fundamental problem is that resolve(filterdIds); runs synchronously before the requests fire, so it's guaranteed to be empty.

Promise.all or Promise.allSettled can help if you know how many pages you want up front (or if you're using a chunk size to make multiple requests--more on that later). These methods run in parallel. Here's a runnable proof-of-concept example:

const pages = 10; // some page value you're using to run your loop

axios
  .get("https://httpbin.org") // some initial request like getToken
  .then(response => // response has the token, ignored for simplicity
    Promise.all(
      Array(pages).fill().map((_, i) => // make an array of request promisess
        axios.get(`https://jsonplaceholder.typicode.com/comments?postId=${i + 1}`)
      )
    )
  )
  .then(responses => {
    // perform your filter/reduce on the response data
    const results = responses.flatMap(response =>
      response.data
        .filter(e => e.id % 2 === 0) // some silly filter
        .map(({id, name}) => ({id, name}))
    );
    
    // use the results
    console.log(results);
  })
  .catch(err => console.error(err))
;

The network tab shows the requests happening in parallel:

If the number of pages is unknown and you intend to fire requests one at a time until your API informs you of the end of the pages, a sequential loop is slow but can be used. Async/await is cleaner for this strategy:

(async () => {
  // like getToken; should handle err
  const tokenStub = await axios.get("https://httpbin.org");
  
  const results = [];
  
  // page += 10 to make the snippet run faster; you'd probably use page++
  for (let page = 1;; page += 10) {
    try {
      const url = `https://jsonplaceholder.typicode.com/comments?postId=${page}`;
      const response = await axios.get(url);
      
      // check whatever condition your API sends to tell you no more pages
      if (response.data.length === 0) { 
        break;
      }
      
      for (const comment of response.data) {
        if (comment.id % 2 === 0) { // some silly filter
          const {name, id} = comment;
          results.push({name, id});
        }
      }
    }
    catch (err) { // hit the end of the pages or some other error
      break;
    }
  }
  
  // use the results
  console.log(results);
})();

Here's the sequential request waterfall:

A task queue or chunked loop can be used if you want to increase parallelization. p-limit is a useful library for limiting your Promise.all work. A chunked loop would combine the two techniques to request n records at a time and check each result in the chunk for the termination condition. Here's a simple example that strips out the filtering operation, which is sort of incidental to the asynchronous request issue and can be done synchronously after the responses arrive:

(async () => {  
  const results = [];
  const chunk = 5;
  
  for (let page = 1;; page += chunk) {
    try {
      const responses = await Promise.all(
        Array(chunk).fill().map((_, i) => 
          axios.get(`https://jsonplaceholder.typicode.com/comments?postId=${page + i}`)
        )
      );
      
      for (const response of responses) {      
        for (const comment of response.data) {
          const {name, id} = comment;
          results.push({name, id});
        }
      }
      
      // check end condition
      if (responses.some(e => e.data.length === 0)) { 
        break;
      }
    }
    catch (err) {
      break;
    }
  }
  
  // use the results
  console.log(results);
})();

(above image is an except of the 100 requests, but the chunk size of 5 at once is visible)

Note that these snippets are proofs-of-concept and could stand to be less indiscriminate with catching errors, ensure all throws are caught, etc. When breaking it into sub-functions, make sure to .then and await all promises in the caller--don't try to turn it into synchronous code.

Array of filtered axios results from paginated API is empty

Answers (2)

Related Questions