CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-storage
Ehsan Kia
Ehsan Kia

Reputation: 1544

Google Cloud Storage UI shows "Internal error encountered", API calls give Fatal Error

For the past month, something on my Google Cloud project has been running out of quota and breaking, but I cannot figure out what nor how to fix it. My project has billing enabled, and the quota page does not show any quotas that are near or past their limits.

Trying to fetch data from GCS throws: FatalError: Expect status [200] from Google Storage. But got status 429.

Looking into it, 429 means "too many requests", but I'm not sure why that's an issue. Isn't GCloud supposed to scale and charge me for my requests? The Google Cloud interface is also broken, when I try browsing my files, it just shows a red "Internal error encountered" and doesn't let me download my files from GCS.

It seems to fix at midnight when the quotas reset, but again, I don't see any quotas that are near or past their limits, so I'm not sure if this is some secret hidden quota I don't have access to, nor can I pay for.

Upvotes: 0

Views: 1762

Answers (1)

Ehsan Kia
Ehsan Kia

Reputation: 1544

The way I managed to fix this is by

  1. Enabling Cloud Audit Logs (IAM -> Audit Logs -> Google Cloud Storage -> enable all 3)
  2. Wait a day
  3. Go to logging and view all storage accesses

Turns out one of my apps was misbehaving and querying GCS thousands of times per day. Fixing that bug resolves the quota issue.

I still have no idea:

  1. What quota limit was being hit (it did not show anywhere)
  2. Why it was hitting a limit instead of charging my billing
  3. Why it broke in such strange ways instead of just telling me

But yes, some secret underlying API was out of quota because one of my apps was accidentally hitting GCS too much.

Upvotes: 1

Related Questions