Rate exceeded in throttling is thrown on DescribeJobFlows

Question

Several days ago all went well, but now, not just one, but three of our servers are throwing these exceptions:

{ "Error" => { "Message" => "Rate exceeded", "Code" => "Throttling", "Type" => "Sender" } }

This happens when retrieving data from Amazon, while calling DescribeJobFlows on Amazon::Coral::ElasticMapReduceClient, and shows as NFO Exception Retriable invalid response.

Answer 1

Every AWS API has dynamic throttling protection. It's kinda sad but best way it to retry/sleep between API calls :(

There is a configuration for the boto3 client to change max_retries before throwing the exception. There is also the general reference on how to use retries.

Good luck!

Answer 2

You should add some retry logic to your code. See this other Stackoverflow question for some ways of implementing that in Python:

Need advice to add exponenital back-off logic in AWS API in python