CODEWITHSUNDEEP
apache-beamapache-beam-io
Poornima Jasti
Poornima Jasti

Reputation: 11

Considering total max records from the user and processing it based on the batch size in apache beam

I am trying to read the records from the source based on the count of total max records to be processed which should be given by the user.

Eg: Total Records in the source table is 1 million Total Max records to process are 100K

I need to process those 100k records only from source. I have gone through JDBC IO library classes to check if I have any option to implement it like there is an option to set the batch size, but I have found none.

PS: I want to implement it IO level, Not by adding limit to query

Upvotes: 0

Views: 297

Answers (2)

Tanveer Uddin
Tanveer Uddin

Reputation: 1525

you can use the withQuery to specify the query with the number of records to read e.g. .withQuery("select id,name from Person limit 1000"). You can also parameterize the number of records using JdbcIO.StatementPreparator. The example in the doc may help.

EDIT Another option is to use withFetchSize

Upvotes: 0

Poornima Jasti
Poornima Jasti

Reputation: 11

I was able to do it using with setMaxRows by turning off the auto-commit for JDBC IO

Upvotes: 1

Related Questions