How to make multiple paginated GET requests to an API in while loop with Node.js when you don't know the page count?

Question

I am working with a REST JSON API which provides no way to query what is the total number of pages or entries. But I need to make multiple requests to the API in a loop in order to get all the data available.

After searching through many stackoverflow questions, the closest working solution I could find successfully makes multiple requests, but still requires you to know what the last page is:

This works:

const async = require("async");
const request = require("request");

let page = 1;
let last_page = 20;
let json;

async.whilst(function () {
    // condition here
  return page <= last_page
},
function (next) {
  request(`https://driftrock-dev-test-2.herokuapp.com/purchases?${page}&per_page=20`, function (error, response, body) {
    if (!error && response.statusCode == 200) {
    json = JSON.parse(body);
        console.log(json.data);      
        console.log(page);
    }
    page++;
    next();
  });
},
function (err) {
  // All things are done!
});

I have tried to adapt it slightly to fit my requirements of not knowing the last page (below), but I can't figure out how to get the logic right or how to solve the asynchronous problem of getting the json variable as undefined. I need to get the value of the json.data array to determine the length of the data array which contains all the object data from the API response.

This doesn't work - returns undefined:

const async = require("async");
const request = require("request");

let page = 1;
let json;

async.whilst(function () {
    // condition here
  json.data.length !== 0
},
function (next) {
  request(`https://driftrock-dev-test-2.herokuapp.com/purchases?${page}&per_page=20`, function (error, response, body) {
    if (!error && response.statusCode == 200) {
    json = JSON.parse(body);
        console.log(json.data);      
        console.log(page);
    }
    page++;
    next();
  });
},
function (err) {
  // All things are done!
});

I've been working on this problem for quite a long time now, so any help would be greatly appreciated! Thanks!

Answer 1

There are a few problems with your logic. First, you are not returning the validation in your test function on the whilst. But even if you did, then you are testing against an undefined variable. So that validation will fail and exit your code before the first iteration.

let oldPage = 1;
let nextPage = 2;
let json;

async.whilst(function () {
    // Check that oldPage is less than newPage
  return oldPage < nextPage;
},
function (next) {
  request(`https://driftrock-dev-test-2.herokuapp.com/purchases?${oldPage}&per_page=20`, function (error, response, body) {
    if (!error && response.statusCode == 200) {
    json = JSON.parse(body);
        console.log(json.data);      
        console.log(oldPage);
    }
    if (json.data.length) {
      // When the json has no more data loaded, nextPage will stop 
      // incrementing hence become equal to oldPage and return 
      // false in the test function.
      nextPage++;
    }
    oldPage++;
    next();
  });
},
function (err) {
  // All things are done!
});

This way you can see the moment you don't have a new page to show anymore.

Answer 2

As I said, I'm not a Node.js user so not familiar with the libraries you're using, but this is at least a general idea...

function getPage(page) {
  request(`https://driftrock-dev-test-2.herokuapp.com/purchases?${page}&per_page=20`, function (error, response, body) {
    if (!error && response.statusCode == 200) {

        // do whatever you do with the current page here.

        if (json.data.length == 20) {
            request(`https://driftrock-dev-test-2.herokuapp.com/purchases?${page + 1}&per_page=20`, function (error, response, body) {
                var json = JSON.parse(body);
                if (json.data.length == 0) {
                    // the next page has no items - disable the next button here
                }
            });
        }
        else {
            // this page has < 20 items - disable the next button here
        }
    }
  });
}

You call that method and it will get the page specified by the parameter, and also check to see if it's the last page and disable the next button (once you add the code to do that bit).