CODEWITHSUNDEEP
javaspringpaginationspring-batchbatch-processing
g0c00l.g33k
g0c00l.g33k

Reputation: 2618

Spring batch using AbstractPaginatedDataItemReader for paginated API call

I am trying to call a paginated API eg. Search API from AbstractPaginatedDataItemReader. I want to keep calling this API till it doesn't have any more data for a page, I am trying to continue the chunk after every page and it seems the batch doesn't get past page 1, here is the code and configuration I am using

Launch context as below

<batch:job id="fileupload">
    <batch:step id="readApi">
        <batch:tasklet>
            <batch:chunk reader="readPaginatedApi" processor="processApiResults"
                         writer="emailItemWriter" commit-interval="10"/>
        </batch:tasklet>
        <batch:next on="NEXT_PAGE" to="readPaginatedApi"/>
        <batch:end on="END" />
    </batch:step>
</batch:job>

And here is the reader snippet

@Component("readPaginatedApi")
@Scope("step")
public class ReadPaginatedApi extends AbstractPaginatedDataItemReader<SearchResponse> {

@BeforeStep
public void beforeStep(StepExecution stepExecution) {
    this.setName("READER");
    this.setExecutionContextName("READER");
    
    String pageSizeString = stepExecution.getJobParameters().getString("page_size");
    if (StringUtils.isNotBlank(pageSizeString) && NumberUtils.isParsable(pageSizeString)) {
        try {
            pageSize = Integer.parseInt(pageSizeString);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    String pageString = stepExecution.getJobParameters().getString("page");
    if (StringUtils.isNotBlank(pageString) && NumberUtils.isParsable(pageString)) {
        try {
            page = Integer.parseInt(pageString);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

@Override
protected Iterator<Payee> doPageRead() {
    //Call API 
    //Return iterator of results or empty iterator
}

@AfterStep
public ExitStatus afterStep(StepExecution stepExecution) {
    AtomicInteger pageAtomicInteger = new AtomicInteger(page);
    SearchResponse searchResponse = //call service, get response
    if (searchResponse != null && CollectionUtils.isNotEmpty(searchResponse.getItems())) {
        pageAtomicInteger.set(page + 1);
        return new ExitStatus("NEXT_PAGE", String.format("page %d", page));
    }
    return new ExitStatus("END", String.format("page %d", page));
}

}

What am I missing here? How can I make this work? Is this the right approach for this case?Appreciate any help on this

Upvotes: 4

Views: 2770

Answers (2)

Mahmoud Ben Hassine
Mahmoud Ben Hassine

Reputation: 31600

batch:next, batch:end, etc are used to define the execution flow of the steps of your job. Those are not intended to iterate over all pages of a paging item reader, they are used at a higher level.

What you need to do is extend AbstractPaginatedDataItemReader and implement doPageRead. Your implementation should maintain the state of which page is currently being read, the list of items, etc.

Upvotes: 4

Kavithakaran Kanapathippillai
Kavithakaran Kanapathippillai

Reputation: 8203

Looking at the equivalent java config and the signature of on and to method, to accepts a Flow, Step or JobExecutionDecider . So I think you need to replace

<batch:next on="NEXT_PAGE" to="readPaginatedApi"/>

    with 

<batch:next on="NEXT_PAGE" to="readApi"/>

Upvotes: 0

Related Questions