What is the best way to implement client-side paginated API calls with Vertx?

Question

I have a Vert.x web service that needs to make a series of paginated calls to an external API. The external service implements pagination by including a "next" field in each response -- which is a direct link to the next page of data -- as well as a count of the total number of pages required to fetch all data. Here's an example response:

"pagination": {
  "count": 1000,
  "totalPages": 112,
  "next": "https://some-host.com?next=some-long-alphanumeric-hash"
},
"data": [ ... ]

After making the first API call, I know the total number of follow-up calls (in this example, 111) as well as the URL to fetch the next page of data. In a synchronous environment, I could just do something like this:

Collection aggregatedResults;
int count = 0;
String nextUrl = "";
while (count <= total pages) {
   make next request
   add the chunk of data from this response to the collection
   store the next URL in local variable
   increment count
}

My strategy with Vertx is to use Futures to represent the result of individual calls, and then chain them together with CompositeFuture.all(). This is roughly what I have so far (some code omitted to save space):

private String nextUrl; // global String

doFirstCall(client).setHandler(async -> {
    if (async.failed()) {
      // blah
    } else {
        Response response = async.result();
        int totalPages = response.getTotalPages();
        next = response.getNext();

        List<Future> paginatedFutures = IntStream
                .range(0, totalPages - 1)
                .mapToObj(i -> {
                    Promise<Response> promise = Promise.promise();
                    doIndividualPaginatedCall(client, next)
                            .setHandler(call -> {
                                if (call.succeeded()) {
                                    Response chunk = call.result();
                                    next = chunk.getNext(); // store the next URL in global string so it can be accessed within the loop
                                    promise.complete(chunk);
                                } else {
                                    promise.fail(call.cause());
                                }
                            });
                    return promise.future();
                })
                .collect(Collectors.toList());

        CompositeFuture.all(paginatedFutures).setHandler(all -> {
            if (all.succeeded()) {
                // Do something with the aggregated responses
            }
        });
    }
});

When I run this code, the first call always succeeds and I store the "next" URL successfully. Then, every subsequent paginated call I make is to the same URL I got from the first call and I see logs like this:

Call succeeded. i: 16, next: https://blah.com/blah?filter=next(DnF1ZXJ5VGhlbkZldGNoBQAAAAAAlMYVFjdaM2ducHBaVGJHeWV5ZjRzNGRQMXcAAAAAAJTGNhYzcWlRTDEyeVJZS05PeV84QkJlLTVnAAAAAACUxjYWa3UzUkx1MXZURG1Pc2E5WGt5RG9pdwAAAAAAlMY2FnY4TVhXajlqUmMtWEQwWU1naGZFN3cAAAAAAJTGVxZCWWFUV19XR1RXQ05DRkI0NGw4M0xB)
Call succeeded. i: 17, next: https://blah.com/blah?filter=next(DnF1ZXJ5VGhlbkZldGNoBQAAAAAAlMYVFjdaM2ducHBaVGJHeWV5ZjRzNGRQMXcAAAAAAJTGNhYzcWlRTDEyeVJZS05PeV84QkJlLTVnAAAAAACUxjYWa3UzUkx1MXZURG1Pc2E5WGt5RG9pdwAAAAAAlMY2FnY4TVhXajlqUmMtWEQwWU1naGZFN3cAAAAAAJTGVxZCWWFUV19XR1RXQ05DRkI0NGw4M0xB)
Call succeeded. i: 18, next: https://blah.com/blah?filter=next(DnF1ZXJ5VGhlbkZldGNoBQAAAAAAlMYVFjdaM2ducHBaVGJHeWV5ZjRzNGRQMXcAAAAAAJTGNhYzcWlRTDEyeVJZS05PeV84QkJlLTVnAAAAAACUxjYWa3UzUkx1MXZURG1Pc2E5WGt5RG9pdwAAAAAAlMY2FnY4TVhXajlqUmMtWEQwWU1naGZFN3cAAAAAAJTGVxZCWWFUV19XR1RXQ05DRkI0NGw4M0xB)

TLDR: How can I execute a series of paginated API calls, where the URL changes between each call and isn't known until the previous call finishes executing? I've tried using CompositeFuture.join, but same effect. I know that for sequential composition, you're supposed to use compose(), but how can I compose an unknown number of function calls?

Answer 1

It turns out I was misunderstanding the API I'm connecting to in this question, and the "next" field does not change between calls. Therefore, this question reduces to "How do I implement async client-side pagination in Vertx, where I do know the URL prior to each paginated call?". I'm accepting Alexey's answer because it answered the original question, and posting the rough code I used below in case this helps anyone with the same use case:

// start()
doFirstCall(client).setHandler(async -> {
  if (async.succeeded()) {
    Response response = async.result();
    final int totalPages = response.totalPages();
    final String next = response.next();

    // Fire off 'totalPages' async calls and wait for them to come back
    List<Future> paginatedFutures = IntStream
      .range(0, totalPages)
      .mapToObj(i -> {
         Promise<Response> promise = Promise.promise();
         doPaginatedCall(client).setHandler(call -> {
            if (call.succeeded()) {
              promise.complete(call.result());
            }
         });
         return promise.future();
      }).collect(Collectors.toList());

   // Wait for all HTTP calls to come back before continuing
   CompositeFuture.join(paginatedFutures).setHandler(all -> {
      if (all.succeeded()) {
         // Do something with all of the aggregated calls
      }
  });
  }
});

private Future<Response> doFirstCall(WebClient client) {
   Promise<Response> promise = Promise.promise();

   // If call succeeded, promise.complete(response), otherwise fail

   return promise.future();
}

private Future<Response> doPaginatedCall(WebClient client, String nextUrl) {
   Promise<Response> promise = Promise.promise();

   // If call succeeded, promise.complete(response), otherwise fail

   return promise.future();
}

Answer 2

You're trying to mutate next

if (call.succeeded()) {
    Response chunk = call.result();
    next = chunk.getNext(); // store the next URL in global string so it can be accessed within the loop
    promise.complete(chunk);
}

But you're actually reusing the same value you got the first time:

next = response.getNext();

That's because all your calls are invoked long before even one of them will return.

Since you cannot know the next value before the previous call returns, you'll have to implement it in a recursive manner, and drop the map:

doIndividualPaginatedCall(client, next)
   .setHandler(call -> {
        if (call.succeeded()) {
            Response chunk = call.result();
            next = chunk.getNext(); // store the next URL in global string so it can be accessed within the loop
            promise.complete(chunk);
            doIndividualPaginatedCall(client, next);
       } else {
           promise.fail(call.cause());
       }
  });

Please note that I didn't actually compile your code, so there may be more changes you'll have to do for it to actually work.