CODEWITHSUNDEEP
spring-batchpartitioning
Prav
Prav

Reputation: 1

Spring Batch Partitiioning DBtoFile Java Configuration Example

I am currently working on Spring Boot and Spring Batch application to read 200,000 records from Database, process it and generate XML output.

I wrote single threaded Spring Batch program which uses JDBCPagingItemReader to read batch of 10K records from Database and StaxEventItemReader to generate this output. Total process is taking 30 minutes. I am wanting to enhance this program by using Spring Batch local Partitioning. Could anyone share Java configuration code to do this task of Spring Batch partitioning which will split processing into multi thread + multi files.. I tried to multi thread java configuration but StaxEventItemReader is single thread so it didn't work. Only way I see is Partition.

Appreciate help.

Upvotes: 0

Views: 594

Answers (1)

Michael Minella
Michael Minella

Reputation: 21463

You are correct that partitioning is the way to approach this problem. I don't have an example of JDBC to XML of how to configure a partitioned batch job, but I do have one that is CSV to JDBC in which you should be able to just replace the ItemReader and ItemWriter with the ones you need (JdbcPagingItemReader and StaxEventItemWriter respectively). This example actually uses Spring Cloud Task to launch the workers as remote processes, but if you replace the partitionHandler with the TaskExecutorPartitionHandler (instead of the DeployerPartitionHandler as configured), that would execute the partitions internally as threads.

https://github.com/mminella/S3JDBC

Upvotes: 1

Related Questions