CODEWITHSUNDEEP
python
Felício
Felício

Reputation: 163

Timming processing time to deal with requests per minute limits when interacting with 3rd party API

So, the 3rd party charges differently for certain limits values. Let's just say I'm cheap and don't wanna pay the full rate for unlimited access.

The approach I've used is rather simple, I just call sleep for 3 seconds after processing each request (paid for the 20 calls per minute plan).

Here is some clipped code for illustration:

import requests
import sqlite3
from time import sleep

timetosleep = 3
conn = sqlite3.connection('db.db')
c = conn.cursor()
keys = [
    "A BIG LIST, SOMETHING LIKE len == 80K" #working with 500 at a time
]

def treatResponse(response):
    #series of validations, some regex and stuff
    returns "A TUPLE WITH DATA TO THE SQLITE DB"

for key in keys:
    response = requests.get("API ADDRESS AND KEYS", {"auth":"auth"})
    #if response is GREAT! yadayada
    c.execute(
           """INSERT INTO tbl(field1, field2) values(?, ?)""",
              treatResponse(response.json()))
    conn.commit()
    sleep(timetosleep)

As the function used to treat the response is rather expensive, with a bunch of sequential regex calls, and the server may get a long time (relative to the limit/minute) to respond, it may be an overkill to wait 3 full seconds (or even to wait at all).

Does anyone has a solution for timing each iteration? If this operation is done neatly, it could be used to reduce the sleep timer and improve performance by a lot. TIA

Upvotes: 1

Views: 640

Answers (2)

Woohoojin
Woohoojin

Reputation: 724

Since requests.get will force python to wait for its' completion, you can accomplish this by recording the time before you make the response, and checking how much time has passed before you sleep:

import time
# Code omitted for brevity
for key in keys:
    start = time.time() # Start time

    ################ TIME THIS BLOCK ################
    response = requests.get("API ADDRESS AND KEYS", {"auth:auth"})
    #if response is GREAT! yadayada
    c.execute(
       """INSERT INTO tbl(field1, field2) values(?, ?)""",
          treatResponse(response.json()))
    conn.commit()
    #################################################

    end = time.time()
    print(end - start) # Time in seconds that have passed
    if end - start < 3 :
        sleep(3 - (end - start))

Upvotes: 1

C.Nivs
C.Nivs

Reputation: 13106

Maybe set your for loop to iterate with the max threshold rather than explicitly wait:

for key in keys:
    start = time.time()
    response = requests.get("API ADDRESS AND KEYS", {"auth:auth"})
    #if response is GREAT! yadayada
    c.execute(
           """INSERT INTO tbl(field1, field2) values(?, ?)""",
              treatResponse(response.json()))
    conn.commit()

    timetosleep = 3. - (time.time()-start)
    if timetosleep<0:
        continue
    sleep(timetosleep)

That way, if your functions take more than 3 seconds, it doesn't sleep, otherwise it will wait the remaining time

Upvotes: 1

Related Questions