CODEWITHSUNDEEP
javaspringcsvspring-batch
membersound
membersound

Reputation: 86747

How to read csv lines chunked by id-column with Spring-Batch?

I'm using Spring-Batch to read a csv file, format the content and write it to a database like:

StepBuilder<T, T> builder = stepBuilderFactory.get("step")
    .<T, T>chunk(100)
    .reader(flatFileItemReader)
    .processor(processor)
    .writer(jpaItemWriter);

The csv contains an ID column. How can I modify the reader to base the chunks on that ID? Examle:

#id, #value
1, first
1000, second
1001, second
1005, second

In this case the chunk would only read the first line, then commit, and then continue.

Is that possible to apply chunking by a value in the file?

Upvotes: 2

Views: 3221

Answers (1)

Luca Basso Ricci
Luca Basso Ricci

Reputation: 18403

I did the same using a custom CompletionPolicy and a PeekableItemReader.
The idea behind the code is to peek next item, perfrom next element read and check from value change.
When a value change happens return true from CompletionPolicy.isComplete().

Important: this policy must be registered as step listener!

public class BreakKeyCompletionPolicy extends CompletionPolicySupport{
    private BreakKeyCompletionContext cc;
    private PeekableItemReader<Object> reader;
    // Strategy used to check for value break
    private BreakKeyStrategy<Object> strategy;

    public void setReader(PeekableItemReader<Object> forseeingReader){
        this.reader = forseeingReader;
    }

    @Override
    public boolean isComplete(RepeatContext context){
        return this.cc.isComplete();
    }

    @Override
    public RepeatContext start(RepeatContext context)   {
        context.setAttribute("current", null);
        this.cc = new BreakKeyCompletionContext(context);
        return cc;
    }
    /** Context contains current element ("current" property" and manage next element.
     * Null next element is treated as a key break
     */
    protected class BreakKeyCompletionContext extends RepeatContextSupport {
        public BreakKeyCompletionContext(RepeatContext context)     {
            super(context);
        }
        public boolean isComplete(){
            final Object next;
            try{
                next = reader.peek();
            }
            catch (Exception e){
                throw new NonTransientResourceException("Unable to peek", e);
            }
            if (null == next){
                return true;
            }
            return strategy.isKeyBreak(this.getAttribute("current"), next);
        }
    }

    @AfterRead
    public void afterRead(Object item){
        this.cc.setAttribute("current", item);
    }
}

Upvotes: 6

Related Questions