CODEWITHSUNDEEP
javaspring-boothttp-headersspring-webflux
G. Blandin
G. Blandin

Reputation: 73

Repeat a WebClient get with new parameters

I am developing a Spring WebFlux application and I have a web adapter that I use to call the external api methods that I use. One of the apis has a paginated result using a link header with a rel="next". I need to call this api until this next link does not exist but I am unsure how to achieve this. Below is the basic call that I am currently using:

public Flux<ItemDto> getAllItems(){
   return webClient.get() //The headers and base url for the api are defined in the constructor
        .uri("/api/items?limit=200") //limit is the number of items returned with 200 being the maximum
        .retrieve()
        .bodyToFlux(Map.class)
        .map(ItemConverter::mapValueToItemDto);//This is just a conversion method that handles mapping
}

What I am needing is to be able to repeat this call but add a request param that is based on part of the link header with the "next" rel value.

I've seen mention of using expand or repeat but I am not sure how exactly to use them. I know using exchange is necessary to grab the clientResponse so I can get the headers.

This question is probably decently vague so I can provide any clarification if needed.

Thanks!

UPDATE

I've been attempting various things and found something that almost works.

public Flux<ItemDto> getAllItems(){
   return recursiveMethod("")//Start with the first page basically
      .flatMap(clientResponse ->
           clientResponse.bodyToFlux(Map.class)
                .map(ItemConverter::mapValueToItemDto));
}

private Flux<ClientResponse> recursiveMethod(String after){
   return webClient.get()
            .uri("/api/items?limit=3" + after) //The test list I am using is only 9 items so I'm using 3 for good results
            .exchange()
            .expand(clientResponse -> {
                List<String> links = clientResponse.headers().asHttpHeaders().getValuesAsList("LINK");
                if(links.stream().anyMatch(link->link.contains("rel=\"next\""))){
                    for (String link : links){
                        if (link.contains("rel=\"next\"")){
                            return recursiveMethod("&" + link.substring(link.indexOf("after="), link.indexOf("&"))); 
                            //The header link has limit and after switched, that's why this looks a little odd
                        }
                    }
                }
                return Flux.empty();
            });
}

The only issue is that it ends up repeating the last "page", eg: Item 1, Item 2, Item 3, Item 4, Item 5, Item 6, Item 7, Item 8, Item 9, Item 7, Item 8, Item 9. Not sure whether this is due to the recursion I've set up or because of Flux or because I am possibly misusing the expand method.

Upvotes: 2

Views: 2607

Answers (1)

G. Blandin
G. Blandin

Reputation: 73

After lots of trial and error and finding solutions to specific parts of this, I found the solution that worked for me.

public Flux<ItemDto> getAllItems() {
    webClient.get()
            .uri("/api/items?limit=1")//Used one to test
            .exchange()
            .expand(clientResponse -> {
                List<String> links = clientResponse.headers().asHttpHeaders().getValuesAsList("LINK");
                if(links.stream().anyMatch(link->link.contains("rel=\"next\""))){
                    for (String link : links){
                        if (link.contains("rel=\"next\"")){
                            return webClient.get()
                                    .uri("/api/items?limit=1&" + link.substring(link.indexOf("after="), link.indexOf("&")))
                                    .exchange();
                        }
                    }
                }
                return Flux.empty();
            })
            .flatMap(clientResponse ->
                    clientResponse.bodyToFlux(Map.class)
                            .map(ItemConverter::mapValueToItemDto));
}

No need for any recursion. Just more appropriately using expand. Now, part of this (see below) can actually be broken off into its own method, but since it's only a few lines I opted not to do this.

webClient.get()
   .uri("/api/items?limit=1" + after)//This after bit would be like what was passed as an argument before
   .exchange();

Upvotes: 4

Related Questions