Spring Batch MultiResourceItemReader re-reading first resource

Question

I am writing a Spring Batch application using Spring Boot 1.5, following are my classes : -

CustomMultiResourceItemReader.java

@StepScoped
@Component
public class CustomMultiResourceItemReader
        extends MultiResourceItemReader<MyDTO> {

    public MultiResourceXmlItemReader(
            @NonNull final MyResourceAwareItemReader itemReader,
            @NonNull final ApplicationContext ctx)
            throws IOException {
        setResources(
                ctx.getResources(
                        String.format(
                                "file:%s/*.xml", "~/data"))); // gives me a Resource[] array fine
        setDelegate(itemReader);
    }

    @PreDestroy
    void destroy() {
        close();
    }
}

MyResourceAwareItemReader.java

@RequiredArgsConstructor
@StepScope
@Component
@Slf4j
public class MyResourceAwareItemReader
        implements ResourceAwareItemReaderItemStream<MyDTO> {
    private static final String RESOURCE_NAME_KEY = "RESOURCE_NAME_KEY";
    @NonNull private final Unmarshaller unmarshaller; // JaxB Unmarshaller
    private Resource resource;

    @Override
    public void setResource(Resource resource) {
        this.resource = resource; // **gets called only once**
    }

    @Override
    public MyDTO read() throws Exception {
        final MyDTO dto = (MyDTO) unmarshaller.unmarshal(resource.getFile()); // Standard JaxB unmarshalling.
        return dto;
    }

    @Override
    public void open(ExecutionContext executionContext) throws ItemStreamException {
        if (executionContext.containsKey(RESOURCE_NAME_KEY)) {

        } else if (resource != null) {
            executionContext.put(RESOURCE_NAME_KEY, resource.getFilename());
        }
    }

    @Override
    public void update(ExecutionContext executionContext) throws ItemStreamException {
        if (resource != null) executionContext.put(RESOURCE_NAME_KEY, resource.getFilename());
    }

    @Override
    public void close() throws ItemStreamException {}
}

The problem is the setResource method in the delegate reader (MyResourceAwareItemReader.java) gets called only once at the beginning; while the read method gets called multiple times, as a result I read the same item multiple times, instead of reading the next item as expected.

I have also browsed the source code of MultiResouceItemReader in Spring Batch, it seems like, the read method of the delegate class is supposed to return null after each item is read, I can clearly see my code doesnt seem to do that.

I am bit lost how to make this work. Any help is much appreciated

Answer 1

Looking further into it, ItemReader documentation, clearly details that reader must return null at the end of the input data set. So basically I implemented my ItemReader with a boolean flag as follows: -

@RequiredArgsConstructor
@StepScope
@Component
@Slf4j
public class MyResourceAwareItemReader
        implements ResourceAwareItemReaderItemStream<MyDTO> {
    private static final String RESOURCE_NAME_KEY = "RESOURCE_NAME_KEY";
    @NonNull private final Unmarshaller unmarshaller; // JaxB Unmarshaller
    private Resource resource;
    private boolean isResourceRead;

    @Override
    public void setResource(Resource resource) {
        this.resource = resource;
        isResourceRead = false;
    }

    @Override
    public MyDTO read() throws Exception {
        if(isResourceRead == true) return null;

        final MyDTO dto = (MyDTO) unmarshaller.unmarshal(resource.getFile());
        isResourceRead = true;
        return dto;
    }

    @Override
    public void open(ExecutionContext executionContext) throws ItemStreamException {
        if (executionContext.containsKey(RESOURCE_NAME_KEY)) {

        } else if (resource != null) {
            executionContext.put(RESOURCE_NAME_KEY, resource.getFilename());
        }
    }

    @Override
    public void update(ExecutionContext executionContext) throws ItemStreamException {
        if (resource != null) executionContext.put(RESOURCE_NAME_KEY, resource.getFilename());
    }

    @Override
    public void close() throws ItemStreamException {}
}

Answer 2

MultiResourceItemReader does not returns null each time. If there are no more resources to read the it returns NULL otherwise it returns the next resources to the delegate that means – Your actual reader

I can see problem in your read() method . you are not moving to next file. As you are implementing you own MultiResourceItemReader It’s your responsibility to move to next resources item.

This is how it is implanted in MultiResourceItemReader . You will need your own similar implementation.

private T readNextItem() throws Exception {

        T item = delegate.read();

        while (item == null) {

            currentResource++;

            if (currentResource >= resources.length) {
                return null;
            }

            delegate.close();
            delegate.setResource(resources[currentResource]);
            delegate.open(new ExecutionContext());

            item = delegate.read();
        }

        return item;
    }

You need to maintain index of resources array . Please check implementation of MultiResourceItemReader. You need to do exactly similar way