CODEWITHSUNDEEP
spring-batch
user3379502
user3379502

Reputation: 223

getting data from DB in spring batch and store in memory

In the spring batch program, I am reading the records from a file and comparing with the DB if the data say column1 from file is already exists in table1.

Table1 is fairly small and static. Is there a way I can get all the data from table1 and store it in memory in the spring batch code? Right now for every record in the file, the select query is hitting the DB.

The file is having 3 columns delimited with "|".

The file I am reading is having on an average 12 million records and it is taking around 5 hours to complete the job.

Upvotes: 1

Views: 1933

Answers (3)

Srinivas Kasarla
Srinivas Kasarla

Reputation: 131

Load static table in JobExecutionListener.beforeJob(-) and keep this in jobContext and you can access through multiple steps using 'Late Binding of Job and Step Attributes'. You may refer 5.4 section of this link http://docs.spring.io/spring-batch/reference/html/configureStep.html

Upvotes: 0

Luca Basso Ricci
Luca Basso Ricci

Reputation: 18383

Preload in memory using a StepExecutionListener.beforeStep (or @BeforeStep).
Using this trick data will be loaded once before step execution.
This also works for step restarting.

Upvotes: 1

Michael Minella
Michael Minella

Reputation: 21453

I'd use caching like a standard web app. Add service caching using Spring's caching abstractions and that should take care of it IMHO.

Upvotes: 1

Related Questions