CODEWITHSUNDEEP
google-app-enginegoogle-cloud-platform
Diane Kaplan
Diane Kaplan

Reputation: 1708

Frequent restarts on GAE application flex environment

I have a GAE application that's set up as a flexible instance, which is expected to be restarted on a weekly basis (and a continually unhealthy instance can be restarted): https://cloud.google.com/appengine/docs/flexible/java/how-instances-are-managed

However, we're seeing this restart ("npm run build" command) several times per week! For example in the past three weeks we've had 9 restarts, and I've confirmed that the log entries leading up are successful 200 responses (no sign of trouble)- all for the active version serving traffic (and not for the other versions that are stopped).

Has anyone seen this symptom before or know of something else that can cause frequent restarts?

Let me know if any other info would be helpful.

Upvotes: 1

Views: 929

Answers (1)

llompalles
llompalles

Reputation: 3166

An instance restart in the Google App Engine flexible environment can occur for several reasons:

According to the GAE documentation, there is no guarantee that an instance runs indefinitely, it can be restarted due to hardware maintenance, software updates or unforeseen issues. Besides that, as you stated, all instances are restarted on a weekly basis.

An instance can also be restarted if it fails to respond to a specified number of consecutive health check requests.

In case that you observe a unusual number of restarts I recommend you to open a ticket in Google Cloud Platform Support. They have internal tools that are able to check what is going on in the instance and figure out why the restarts are happening.

@DianeKaplan's comment:

Contacting GCP support has given me some a few helpful nuggets so far:

  • The automatic weekly restart of an instance due to maintenance can occur around different times (so it may only be 5 days since the last one, for example)
  • our deployments (which result in new GAE versions) make Google Builds
  • In some cases, a VM was being created overnight and then immediately deleted, where it didn't look like autoscaling was needed. Still looking into this, but was pointed towards the Google Cloud Console section Home > Activity as a good place to find clues

Upvotes: 1

Related Questions