Spring Batch Json custom ItemWriter

Question

I want to write json format files from a database. I have this prototype of ItemWriter implementation, very simple.

import java.util.ArrayList;
import java.util.List;

import org.springframework.batch.core.ExitStatus;
import org.springframework.batch.core.StepExecution;
import org.springframework.batch.core.StepExecutionListener;
import org.springframework.batch.item.ItemWriter;
import org.springframework.core.io.Resource;

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;

public class CustomItemWriter<T> implements ItemWriter<T>, StepExecutionListener {
    private Gson gson;
    private Resource resource;
    private boolean shouldDeleteIfExists = true;
    private List<T> allItems = new ArrayList<T>();

    @Override
    public void write(List<? extends T> items) throws Exception {
        System.out.println("this is the begin " + items.size());
        allItems.addAll(items);
    }

    public Resource getResource() {
        return resource;
    }

    public void setResource(Resource resource) {
        this.resource = resource;
    }

    public boolean isShouldDeleteIfExists() {
        return shouldDeleteIfExists;
    }

    public void setShouldDeleteIfExists(boolean shouldDeleteIfExists) {
        this.shouldDeleteIfExists = shouldDeleteIfExists;
    }

    @Override
    public ExitStatus afterStep(StepExecution arg0) {
        //write ALL to the output file
        System.out.println(gson.toJson(allItems)); 
        return null;
    }

    @Override
    public void beforeStep(StepExecution arg0) {
        gson = new GsonBuilder().setDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'").disableHtmlEscaping().create();
    }
}

The problem solved here, is that the write method sends to the output file each commitInterval a JSON array, I just wanted a unique JSON array in my file.

Implementing StepExecutionListener after the step runs; I can send to the output file the entire array and transform it to JSON, and write it to the output file (fine!).

My question is, Is this the correct way to do that? I think in the benefit of write to the file each commitInterval, but I'm not sure with my solution. It works, but I don't want to solve this problem and to provake another.

Answer 1

Thanks for a Solution .

In Xml you can add this like that .

    <property name="resource" value="file:opt/output.json" />  <!--  #{jobParameters['input.file.name']} -->
    <property name="shouldDeleteIfExists" value="true" />

    <property name="lineAggregator">
        <bean
            class="com.package.JsonLineAggregator">
        </bean>
    </property>

</bean> 

Answer 2

You want to actually flush the write to your file with each chunk otherwise you'll lose restartability.

Assuming you're okay with one JSON object per line, I would probably just use a FlatFileItemWriter in combination with a custom LineAggregator that converts each object to a JSON string. Something along these lines:

public class JsonLineAggregator<T> implements LineAggregator<T>, StepExecutionListener {

    private Gson gson = new Gson();
    private boolean isFirstObject = true;

    @Override
    public String aggregate(final T item) {
        if (isFirstObject) {
            isFirstObject = false;
            return "[" + gson.toJson(item);
        }
        return "," + gson.toJson(item);
    }

    public void setGson(final Gson gson) {
        this.gson = gson;
    }

    @Override
    public void beforeStep(final StepExecution stepExecution) {
        if (stepExecution.getExecutionContext().containsKey("isFirstObject")) {
            isFirstObject = Boolean.parseBoolean(stepExecution.getExecutionContext().getString("isFirstObject"));
        }
    }

    @Override
    public ExitStatus afterStep(final StepExecution stepExecution) {
        stepExecution.getExecutionContext().putString("isFirstObject", Boolean.toString(isFirstObject));
        return null;
    }
}

EDIT: Updated the LineAggregator implementation above to demonstrate how you'd make it output something that looked like an JSON list.

Note that you'd also want to register a FlatFileFooterCallback to your FlatFileItemWriter that added the final "]".

public class JsonFlatFileFooterCallback implements FlatFileFooterCallback {

    @Override
    public void writeFooter(final Writer writer) throws IOException {
        writer.write("]");
    }
}