CODEWITHSUNDEEP
javaspringspring-bootspring-integration
Ali
Ali

Reputation: 450

Spring boot integration url pagination solution

I using spring (boot) integration in order to connect to a restful WebService and retrieve data from that, Here are some part of my codes: 

@Bean
public PollerMetadata downloadTrigger()
{
    PeriodicTrigger trigger = new PeriodicTrigger(config.getDownloadInterval());
    trigger.setFixedRate(true);
    PollerMetadata pollerMetadata = new PollerMetadata();
    pollerMetadata.setTrigger(trigger);
    pollerMetadata.setMaxMessagesPerPoll(1);
    return pollerMetadata;
}

@Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller()
{
    PeriodicTrigger trigger = new PeriodicTrigger(10);
    trigger.setFixedRate(true);
    PollerMetadata pollerMetadata = new PollerMetadata();
    pollerMetadata.setTrigger(trigger);
    return pollerMetadata;
}

@InboundChannelAdapter(value = "channel1", poller = @Poller("downloadTrigger"))
public ResponseEntity<AppsItem[]> download()
{
    String url = "https://example.com/?page{pageNumber}";
    try
    {
        SSLUtil.turnOffSslChecking();
    }
    catch (Exception e)
    {

    }

    HttpHeaders requestHeaders = new HttpHeaders();
    requestHeaders.setAccept(Collections.singletonList(new MediaType("application","json")));
    HttpEntity<?> requestEntity = new HttpEntity<Object>(requestHeaders);
    template.getMessageConverters().add(new MappingJackson2HttpMessageConverter());


    ResponseEntity<AppsItem[]> responseEntity = template.exchange(url, HttpMethod.GET, requestEntity, AppsItem[].class, 10, 1);


    return responseEntity;
}

@Splitter(inputChannel = "channel1", outputChannel = "channel2")
public List<AppsItem> scrape(ResponseEntity<AppsItem[]> payload)
{

    //do something;
}

As you can see, I have a url that accept a variable with pageNum name, i want to increase page number while the response body be empty, I don't know how can i implement it with spring integration.But let me explain what it need: 

1 - Poller calling inbound method once peer hour
2 - Inbound method start calling WebService from page 1 to page n.
3 - Inbound method should pass data to next channel before fetching next page

I need something like below:

    @InboundChannelAdapter(value = "channel1", poller = @Poller("downloadTrigger"))
public ResponseEntity<AppsItem[]> download()
{
    String url = "https://example.com/?page{pageNumber}";
    try
    {
        SSLUtil.turnOffSslChecking();
    }
    catch (Exception e)
    {

    }

    HttpHeaders requestHeaders = new HttpHeaders();
    requestHeaders.setAccept(Collections.singletonList(new MediaType("application","json")));
    HttpEntity<?> requestEntity = new HttpEntity<Object>(requestHeaders);
    template.getMessageConverters().add(new MappingJackson2HttpMessageConverter());


    ResponseEntity<AppsItem[]> responseEntity;
        int i = 0;
        do
        {
            i++;
            responseEntity = template.exchange(url, HttpMethod.GET, requestEntity, AppsItem[].class, 10, i);
            scraper.parse(responseEntity);
            LOG.info("entry={}", "");
            //Send response to the next channel in this loop 
        }
        while (responseEntity.hasBody());


    return responseEntity;
}

Upvotes: 0

Views: 223

Answers (1)

Gary Russell
Gary Russell

Reputation: 174504

Remove setMaxMessagesPerPoll(1) from downloadTrigger and the poller (on each poll) will keep calling the download() until it returns null.

Increment the page count on each call and reset it when no more pages exist; the poller will next run after the interval.

Upvotes: 1

Related Questions