CODEWITHSUNDEEP
pythonsalesforce
Sarath Baby
Sarath Baby

Reputation: 59

Salesforce API Python - fetching large amount of data in batches

Is there any way I can fetch millions of records from a salesforce DB.The problem is, unlike in Oracle, we cannot use offset-limit loop by ordering the table and fetching each partition. The maximum offset that can be applied is 2000 is salesforce. Due to this I cannot skip more than 2000 rows which means I cannot fetch rows above that limit.

Upvotes: 0

Views: 3893

Answers (1)

eyescream
eyescream

Reputation: 19612

You absolutely can. Which API do you use and do you issue raw HTTP requests or do you use a Salesforce Python library.

Have you seen How to handle an offset greater than 2000 on SOQL without sorting by ID or date using the Salesforce Rest API and Unable to fetch complete records from Salesforce using Python

You can even use bulk API v2 to pull data in 10K chunks: https://developer.salesforce.com/docs/atlas.en-us.api_bulk_v2.meta/api_bulk_v2/queries.htm

It's implemented in simple-salesforce as

query = 'SELECT Id, Name FROM Account LIMIT 10'
sf.bulk.Account.query(query)

Upvotes: 2

Related Questions