CODEWITHSUNDEEP
javajsonspringspring-batch
ulquiorra
ulquiorra

Reputation: 945

read complex json file with spring batch

i have a complex json file( with nested json arrays) structure like this: 

{"persons":[ 
{"id":"1", "firstName": "X", "lastName": "X", "infos": [{"address":[{"city": "X", "country": "X"}]}]},
{"id":"2", "firstName": "Y", "lastName": "Y", "infos": [{"address":[{"city": "Y", "country": "Y"}]}]}
]}

I want to read each line ( one person) separately

So my spring batch configuration is like this

<bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader"
    scope="step">
    <property name="resource" value="#{jobParameters[file]}" />
    <property name="recordSeparatorPolicy" ref="recordPolicy" />
    <property name="lineMapper" ref="lineMapper" />
</bean>

<bean id="lineMapper"
    class="com.batchs.personJob.PersonLineMapper">
    <property name="delegate" ref="lineMapperType" />
</bean>

<bean id="lineMapperType"
    class="org.springframework.batch.item.file.mapping.JsonLineMapper" />

<bean id="recordPolicy"
    class="org.springframework.batch.item.file.separator.JsonRecordSeparatorPolicy" />

The mapper class looks like

    public class PersonLineMapper implements LineMapper<Person> {
    private JsonLineMapper delegate;

    public mapLine(String line, int lineNumber) throws Exception {
        Map<String, Object> personAsMap = delegate.mapLine(line, lineNumber);
        Person person = new Person();
        // map fields
        return person ;
    }

    public void setDelegate(JsonLineMapper delegate) {
        this.delegate = delegate;
    }
}

The problem is than the reader reads only one line (so one commit) because he reads the persons array in my json file like a whole line but i want read instead line per line ( one person at time)

How to do this ?

I have tried with a simple json file like this :

  { "id": "1",
      "firstName": "X",
      "lastName": "X"}
  { "id": "2",
      "firstName": "Y",
      "lastName": "Y"}

And it works good ... I read each person one by one

Thank you very much

Upvotes: 2

Views: 6974

Answers (2)

John Thompson
John Thompson

Reputation: 524

An alternative could be to run the file through a transform step to flatten the object structure, so you just have an array of People to deal with. Which should be fine, unless you're dealing with large files.

Upvotes: 1

Michael Minella
Michael Minella

Reputation: 21473

Unfortunately, you have two options:

  1. Write your own record separator that ignores the wrapper persons element/line.
  2. Edit the file in a step before reading it to remove that wrapper line using something like a sed command.

If you choose option 2, you can execute it via the SystemCommandTasklet in the step before the one you actually process it in.

Upvotes: 3

Related Questions