CODEWITHSUNDEEP
pythonpython-2.7python-3.x
nish
nish

Reputation: 7280

How to make only "n" API calls until processed by the server

I have two large lists of same size. I am making subprocess calls with ith element of both lists as arguments:

for index in range(len(skuList)):
    sku = skuList[index]
    qty = qtyList[index]
    cmd1 = 'php -f ReviseItem.php %s %s' % (sku,qty)
    print cmd1
    args1 = shlex.split(cmd1)
    p=subprocess.Popen(args1)
    p.wait()

I am actually making an API call in the ReviseItem script. But, the API server is such that I can make a maximum of 15 requests and throttled for next 2 minutes. How can I modify my code to satisfy this. Is there a way to stop the execution for 2 minutes after 15 subprocesses are spawned?

EDIT: I am still getting throttled after waiting for 2 minutes after 15 request. It seems that I'm throttled until 15 requests are processed by the server, as mentioned here and not after making the requests.

EDIT2: if I get throttled and the request does not get processed, I get the following message (among others) in the standard output:

<ErrorResponse xmlns="http://mws.amazonaws.com/doc/2009-01-01/">
  <Error>
    <Type></Type>
    <Code>RequestThrottled</Code>
    <Message>Request is throttled</Message>
  </Error>
  <RequestID>3ca74b8b-cdc2-47df-b48b-043eb5411cda</RequestID>
</ErrorResponse>

Also, I can make another API call to find out the request ID of the last processed request. Again, Request ID is in the standard output of the calls(both the original and this call) among other messages

Upvotes: 2

Views: 133

Answers (3)

cmd
cmd

Reputation: 5830

Assuming they process them in order. You can keep a list of outstanding requests ids, then when you are just short of full, stop and make calls to get the last processed id until you can remove any. Remove those ids and continue.

Upvotes: 0

hjpotter92
hjpotter92

Reputation: 80629

Use a global variable which increments and checks if it reached the count of 15. use os.sleep or sys.sleep (whichever library is applicable in python). A pseudo code would be:

myCount = 0
for index <loop>
  myCounter++
  if myCounter % 15 == 0:
    os.sleep( 2 * 60 ) # or 2000 * 60

Finding the libraries, I see that it is time.sleep(). The code would become:

import time
myCounter = 0
for index in range(len(skuList)):
    myCounter += 1
    if myCounter % 15 == 0:  time.sleep( 2 * 60 )
    sku = skuList[index]
    qty = qtyList[index]
    cmd1 = 'php -f ReviseItem.php %s %s' % (sku,qty)
    print cmd1
    args1 = shlex.split(cmd1)
    p=subprocess.Popen(args1)
    p.wait()

Upvotes: 1

Sneftel
Sneftel

Reputation: 41454

Sure -- just keep a list of timestamps from the last 15 times you spawned a subprocess. Before spawning a new subprocess, if there's 15 timestamps in the list, check whether the oldest timestamp is at least two minutes old; if it isn't, wait for the appropriate amount of time until it is. Then add the new timestamp, and pop off the oldest one if there's more than 15.

Upvotes: 1

Related Questions