CODEWITHSUNDEEP
javaspringspring-batch
user1673567
user1673567

Reputation: 108

Multiple reader/processor/writer in spring batch

I am new to Spring batch and I have a peculiar problem. I want to get results from a 3 different jpa queries with JpaPagingItemReader and process them individually and write them into one consolidated XML file using StaxEventItemWriter.

For eg the resultant XML would look like,

<root>
 <query1>
 ...
 </query1>
 <query2>
 ...
 </query2>
 <query3>
 ...
 </query3>
</root>

Please let me know how to achieve this?

Also, I currently implemented my configurer with one query but the reader/writer is also quite slow. It took around 59 minutes to generate file of 20MB as I am running it in single threaded environment as of now as opposed to multithreaded env. If there are any other suggestions around it, please do let me know. Thanks.

EDIT: I tried following this approach: Created 3 different steps and added 1 reader, processor, writer in each of them but the problem I am facing now is writer is not able to write in the same file or append to it. This is written in StaxEventItemWriter class: FileUtils.setUpOutputFile(file, restarted, false, overwriteOutput); Here 3rd argument append is false by default.

Upvotes: 4

Views: 1054

Answers (3)

Saurabh Chawla
Saurabh Chawla

Reputation: 86

The second approach to your question seems the right direction, you could create 3 different readers/processors/writers and create your custom writer which should extend AbstractFileItemWriter in which setAppend is allowed. Also, I have seen that xmlWriter writes faster xmls than StaxEventItemWriter but there is some trade off in writing boiler plate code.

Upvotes: 2

Binu
Binu

Reputation: 784

use JVisualVm to monitor the bottlenecks inside your application. Since you said it is taking 59 minutes to create file of 20MB, you will get better insights of where you are getting performance hits.

VisualVm tutorial

Open visualvm connect your application => sampler => cpu => CPU Samples. Take snapshot at various times and analyse where is it taking much time. By checking this only you will get enough data for optimisation.

Note: JvisualVm comes under oracle jdk 8 distribution. you can simply type jvisualvm on command prompt/terminal. if not download from here

Upvotes: 0

Emerson Farrugia
Emerson Farrugia

Reputation: 11353

One option off the top of my head is to

  1. create a StaxEventItemWriter
  2. create 3 instances of a step that has a JpaPagingItemReader and writes the corresponding <queryX>...</queryX> section to the shared writer
  3. write the <root> and </root> tags in a JobExecutionListener, so the steps don't care about the envelope

There are other considerations here, like whether it's always 3 files, etc. but the general idea is to separate concerns between processors, step, job, tasks, and listeners to make each perform a clear piece of work.

Upvotes: 0

Related Questions