Severe GAE performance degradation all of sudden

Question

We've seen calls start to take 10x to 100x longer as of a specific point in time (right about 2012-02-18 04:00:00). For example, our longest call that normally can take up to 5 seconds, is now timing out at a minute. I'm pretty sure it's not a result of a code or configuration change made on our part. All of our API calls seem to be impacted. Using AppStats, there seems to be no glaring culprit. (Unfortunately, I don't have a saved graph before Friday to compare to.) We do many gets/fetches, and some writing, against the high-replication datastore -- both seem impacted.

Any insight?

Answer 1

Best to file a production issue so it can be investigated.