CODEWITHSUNDEEP
springcsvspring-batch
TheCyberXP
TheCyberXP

Reputation: 915

Spring Batch ~ Dynamic commit interval or a custom completion policy

What I have?

Spring Integration that watch recursively a folder for new CSV's files; and send them back to Spring batch.

The job: read the CSV file; in the processor, I modify some data in the items; then I use a custom writer to save my data on the DB.

Problem?

In fact that I have dynamic number of CSV beeing send to the batch. I want that my job commit interval will be based on the number of items (lines) present in the CSV's file. In other way, I don't want to commit my data in every fixed number of item, but every end of file. Exemple: CSV 1 have 200 Lines, I want to process all the lines, writes them, commit, close the transaction then read the next CSV.

I have two idea, but I didn't know whoch is the perfect and how to implement it:

  1. Get from the reader the number of lines in my CSV and send it to my commit interval using a job parameter argument like so #{jobParameters['commit.interval.value']}
  2. Implement a Custom Completion Policy to replace my commit inteval, how to implement isComplete() Do you have any exemples? Github project?

But before all that, how can I get the number of items?

Could any one helps me? a code sample maybe?

Thank you in advance.

Upvotes: 1

Views: 2128

Answers (1)

TheCyberXP
TheCyberXP

Reputation: 915

No answer, but I found a solution

I'm using a Dynamic commit interval instead of a completion policy.

With Spring batch integration, I can use a transformer to send my file to the batch, for that I have a custom class FileMessageToJobRequest in that one I added this function that helps me to get the count lines

public static int countLines(String filename) throws IOException {
    InputStream is = new BufferedInputStream(new FileInputStream(filename));
    try {
        byte[] c = new byte[1024];
        int count = 0;
        int readChars = 0;
        boolean empty = true;
        while ((readChars = is.read(c)) != -1) {
            empty = false;
            for (int i = 0; i < readChars; ++i) {
                if (c[i] == '\n') {
                    ++count;
                }
            }
        }
        return (count == 0 && !empty) ? 1 : count;
    } finally {
        is.close();
    }
}

and this one to send parameters

@Transformer
public JobLaunchRequest toRequest(Message<File> message) throws IOException{

    JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();

    jobParametersBuilder.addString("commit.interval", Integer.toString(countLines(message.getPayload().getAbsolutePath())));

    return new JobLaunchRequest(job, jobParametersBuilder.toJobParameters());
}

and in my job context, I just added this commit-interval="#{jobParameters['commit.interval']}"

Hope it help someone in need ;)

Upvotes: 4

Related Questions