CODEWITHSUNDEEP
google-app-enginegoogle-cloud-platform
Kidogo
Kidogo

Reputation: 171

Google App Engine - Sudden increase in latency

We have seen a sudden latency increase in our application on Google App Engine latency within the past few hours. The logs show that requests fail with message "Request was aborted after waiting too long to attempt to service your request.", with no stack-trace or any other relevant information. Users get an empty page with message "Rate exceeded.". No changes have been done to the application that correlate to this spike in latency.

The application is therefore down, with no information from app engine that can help point to the source of the latency.

We have filed a issue in the issuer tracker, no luck in getting response yet.

Does anyone have ideas on what we could do to deal with this kind of situation?

Screenshot showing increased lattency

Update

The problem went away after 3 hours as suddenly as it came, and without any intervention on our part. Since there is consensus on min_idle_instances, we have decided to leave all the setting as they have always been so that we can see if this ever happens again. If it does happen, we will have an opportunity to test this by making the suggested changes, and post an update here.

Here is a screen shot for the entire incident: Screen shot of the entire incident

Upvotes: 1

Views: 666

Answers (1)

Stefan G.
Stefan G.

Reputation: 938

The comment that @Parth Mehta added is useful and it made me think of what could be causing your issues.

I'm thinking that maybe your increased latency is due to not having idle instances ready for the requests as they increase and come in, so when requests increase a bit is taken until the new instances are ready and there might be your latency cause.

Setting enough min_idle_instances might alleviate the 500's as they would be warm and ready for the requests.

If this doesn't solve your issue I would recommend creating a case with GCP Support and we will surely be able to assist you more.

Try it and let us know.

Upvotes: 1

Related Questions