Recursive JavaScript Promises (auto paging an API)

Question

I have a paged API that I need to automatically fetch each page of results.

I have constructed the following recursive promise chain, which actually gives me the desired output. I have renamed the traditional resolve to outer to try and wrap my head around it a little more but I am still confused at how passing the outer function into the nested .then is the magic sauce that is allowing this to work?

I thought I knew Promises well but obviously not!

  protected execute<T>(httpOptions): Promise<AxiosResponse | T> {
    const reqObject: HttpConfig = {
      ...httpOptions,
      baseURL: this.config.baseUri,
      headers: { Authorization: `Bearer ${this.config.apiKey}` },
    };

    return new Promise((outer, reject) => {
      axios
        .request<T>(reqObject)
        .then((res) => {
          if (res.headers.link) {
            const pagingObject: PagingObject = this.getPagingObject(res);
            this.pagedListResponses = this.pagedListResponses.concat(res.data);
            if (pagingObject.last) {
              this.execute<T>({
                url: pagingObject.next,
              }).then(outer);
            } else {
              outer(this.pagedListResponses as any);
            }
          } else {
            outer("No paging header found");
          }
        })
        .catch((err) => reject(err));
    });
  }

Answer 1

TL:DR

As you add more and more Promises via recursion they pile up like the skin of an onion. When each is resolved and .then() is called, the Promises are removed from the inside out until there are none left.

Promise Chaining

When a Promise object is created within the Resolve handler of another Promise both Promises are settled in an inside-out order. Additionally, when chaining is used through the Promise instance methods (.then(), .catch(), .always(), etc), the chaining methods take priority and are executed before the outermost Promise object resolves.

Perhaps better explained here.

Your code creates the Axios Promise within the Resolve handler of the outer construced Promise Object. The AxiosPromise's .then() will execute prior to the outer Promise finally settling. After that happens the result is passed through the outer Promise object with no modification or processing. Basically a no-op.

That is why the wrapper Promise (explicit-construction anti-pattern) is unecessary and discouraged - it is a waste of time and resources that provides no benefit in this code.

With recursion in the mix, Promise objects just keep getting piled on.

So (for now) a reference to that outer Promise object is returned from the .execute() method. But when/how is it settled (resolved or rejected)?

1. The Axios Promise resolves with AJAX results
   • If this is a recursive call, this is an inner Promise (even though created with .then(outer) which is more confusing in this case)
2. The outer most .then() is called with the AJAX results (or rejected with reason)
3. The instance variable pagedListResponses is updated with the results
4. if res.headers.link == true && pageObject.last == true
   • Recurse & start back at 1 <- pending with unsettled Promise(s)
5. else if res.headers.link == true && pageObject.last == false
   • resolve the outer most Promise with the pageListResponses <- completely settled
6. else if res.headers.link == false && pageObject.last == false
   • resolve the outer most Promise with "No paging header found" <- completely settled
7. Then then() method attached to the initial call to execute() is called with the pageListResponses
   • ex. this.execute({...}).then(pageList=>doSomethingWithPageOfResults());

So chaining, using .then() midstream, allows us to do some data processing (as you have done) prior to returning an eventually settled Promise result.

In the recursive code:

this.execute<T>({
    url: pagingObject.next,
}).then(outer);

The .then() call here simply adds a new inner Promise to the chain and, as you realize, is exactly the same as writing:

.then(result=>outer(result)); Reference

Async/Await

Finally, using Async / Await is recommended. It is strongly suggested to Rewrite Promise code with Async/Await for complex (or some say any) scenarios. Although still asychronous, this makes reading and rationalizing about the code sequence much easier.

Your code rewritten to take advantage of Async/Await:

 protected async execute<T>(httpOptions): Promise<T> {
    const reqObject = {
      ...httpOptions,
      baseURL: this.config.baseUri,
      headers: {Authorization: `Bearer ${this.config.apiKey}`},
    };
    try {
      const res: AxiosResponse<T> = await axios.request<T>(reqObject);
      if (res) {
        const pagingObject: boolean = this.getPagingObject(res);
        this.pagedListResponses = this.pagedListResponses.concat(res.data);
        if (pagingObject) {
          return this.execute<T>({ url: 'https://jsonplaceholder.typicode.com/users/1/todos'});
        } else {
          return this.pagedListResponses as any;
        }
      } else {
        return "No paging header found" as any;
      }
    } catch (err) {
      throw new Error('Unable to complete request for data.');
    }
  }

Here, async directly indicates "This method returns a Promise" (as typed). The await keyword is used in code only once to wait for the resolution of the AJAX request. That statement returns the actual result (as typed) as opposed to a Promise object. What happens with a network error? Note the use of try/catch here. The catch block handles any reject in the nest/chain. You might notice the recursive call has no await keyword. It isn't necessary in this case since it is a Promise that just passes along its result. (it's OK to add await here but it has no real benefit).

This code is used the same way from the outside: this.execute({...}).then(pageList=>doSomethingWithPageOfResults());

Let me know what gaps you may still have.

Answer 2

Well when you call this.execute, it returns a Promise<AxiosResponse | T>.

So by calling .then(outer), the response from the recursive execute function is being used to resolve your outer promise.

It's basically the same as this.execute(...).then(response => outer(response)), if that makes it any clearer.