CODEWITHSUNDEEP
spring-batch
Pipkin
Pipkin

Reputation: 109

Handle errors and control breaks with Spring Batch

I have almost completed my work with Spring Batch, it's working but then I have problems to handle the errors. I'll make a simple example:

I read one flat file, that I (later) map with 3 variables:

ID CODE   NAME
AAA3333333Alex
AAA3333333Mark
BBB4444444Paul

I want the reader to read the flat file with a control break (I don't know if it's the right term in english, in italian it's something like "key break"): I read the elements with the same ID and CODE and only when the key changes return them to the reader: 

while ((line = (Person) peek()) != null) {  //while there are elements to read
 if (line.getId().equals(prevElement.getId()) && line.getCode().equals(prevElement.getCode())
    //do something
}

This works fine: when the ID or CODE changes I return the elements to the writer. To make this work I had to set the commit-interval to 1 from the application-context. The thing is that in the worst case, if the elements are different for each line, I commit every single element and it becomes all very very slow.

So I said: let's put an outer control. Instead of returning the elements to the writer each time the key changes, I put them in a list, and then I return the list every 200 key changes (like a...handmade commit-interval):

while (controlBreakCount < 200 &&) {
  while (!exit && (line = (Person) peek()) != null) {  //while there are elements to read
    if (line.getId().equals(prevElement.getId()) && line.getCode().equals(prevElement.getCode())
        //do something
    else { //if the key changes
        //there is a controlBreakCount++;  to increase the count
      //add the elements to a list
    }    
  }
}
return the list

and this works too (the real code has more controls, but this was to explain in a simple way). The problem comes here: how to handle the errors with the listener in this case. With the outer while I have put (the one with the controlBreakCount), if even one of the 200 elements has an error all the elements currently in the list go to the listener and so it's very difficult to recognize the element with the error.

I guess my solution is not the best way to handle the "control break", but I can't find really much about this (and I'm not very pro with Spring Batch)...may I have some help?

Thank you very much :)

Upvotes: 0

Views: 1035

Answers (1)

Luca Basso Ricci
Luca Basso Ricci

Reputation: 18393

The 'do something' in your code is suspect... :/ Which operations do you do on the element just read? In reader you should only aggregate items and pass them to processor or writer.
IMHO you have to set a commit-interval greater than 1 to make process faster and try to group your data with same ID+CODE key in your own custom reader and THEN pass to writer in this way:

class PersonList {
   String id;
   String code;
   List<Person> persons = new ArrayList<Person>();
}

when you find a keybreak (ID+CODE differs from previous) you have to create a new PersonList object and until ID+CODE are the same of previous line add to current PersonList.person list.
If your problem is about manage single line error with my solution you you can manage in your reader and, if you want, skip items with same ID+CODE in your reader as well.
Your reader must change its signature to MyPersonItemReader<PersonList> and your writer will write PersonList objects, but you have under your control the object (with ID+CODE).

I hope I had understood correctly your problem.

Upvotes: 2

Related Questions