CODEWITHSUNDEEP
python-3.xmultithreadingmultiprocessingbatch-processing
Javeed
Javeed

Reputation: 11

How can i implement batch processing for api calls in python?

The API requests that i make as shown below generate me about 100 responses a request. I have about 121000 responses to fetch and load it into the database. Here is the catch, while making the api request, i have to pass the next_cursor parameter so that i can fetch the next 100. My Aim is batch 50 requests at once and then load all of it into a database. How can i do that?

while True:
    if resp['meta']['next_cursor']!=None:  
            
            url = "https://somecompany.gorgias.com/api/tickets?limit=100&order_by=updated_datetime%3Adesc&cursor={}".format(resp['meta']['next_cursor'])
            response = requests.get(url, headers=headers)
            resp=json.loads(response.text)

            response_data.append(resp)

as you can see, the limit for one request is 100, so for 50 requests i can get 5000 responses.

Upvotes: 1

Views: 579

Answers (0)

Related Questions