CODEWITHSUNDEEP
google-apigoogle-drive-apigoogle-api-python-client
Carlo Pires
Carlo Pires

Reputation: 4926

Can't find a way to deal with Google Drive API 403 Rate Limit Exceeded

I have a huge amount of users and files in a Google Drive domain. +100k users, +10M of files. I need to fetch all the permissions for these files every month.

Each user have files owned by themselves, and files shared by other domain users and/or external users (users that don't belong to the domain). Most of the files are owned by domain users. There is more than 7 millions of unique files owned by domain users.

My app is a backend app, which runs with a token granted by the domain admin user.

I think that doing batch requests is the best way to do this. Then, I configured my app to 1000 requests per user, in google developer console.

I tried the following cases:

  1. 1000 requests per batch, up to 1000 per user -> lots of user rate limits
  2. 1000 requests per batch, up to 100 per user -> lots of rate limit errors
  3. 100 requests per batch, up to 100 per user -> lots of rate limit errors
  4. 100 requests per batch, up to 50 per user -> lots of rate limits errors
  5. 100 requests per batch, up to 10 per user -> not errors anymore

I'm using quotaUser parameter to uniquely identify each user in batch requests.

I checked my app to confirm that each batch was not going to google out of its time. I checked also to see if each batch have no more than the limit of file_id configured to fetch. Everything was right.

I also wait each batch to finish before sending the next one.

Every time I see a 403 Rate Limit Exceeded, I do an exponential backoff. Sometimes I have to retry after 9 steps, which is 2**9 seconds waiting.

So, I can't see the point of Google Drive API limits. I'm sure my app is doing everything right, but I can't increase the limits to fetch more permissions per second.

Upvotes: 1

Views: 993

Answers (1)

Darkcylde
Darkcylde

Reputation: 869

I believe that this is a limit that google sets to stop people spamming the service and tying it up. It doesn't have anything to do with your app itself but is set on the Google server side. If the Google server receives over a particular number of requests within a certain time this is the error you get. There is nothing you can do in your app to overcome this. You can talk to Google about it and usually paying for Google licenses ect can allow you much higher limits before being restricted.

Upvotes: 0

Related Questions