Slow initial connection to Google Cloud App Engine

Question

Initial connections to my website are extremely slow (100+ seconds). How can I diagnose the issue?

Using the Chrome dev tool network tab, I see that the issue is "initial connection" and not things like SSL or Waiting/TTFB.

This only happens for the first page visit to the website for a given device; after the first page loads, everything on the website is very fast. This consistently happens for new devices, on the same device if I don't visit the website for X days, and on the same device if I clear the cache and browsing history.

The website is a Django app is hosted using Google Cloud App Engine with 2 flexible instances.

User traffic to the website is low, so I doubt the issue is load balancing or traffic spikes.

Thanks!

Answer 1

I had this exact issue. it was killing our load performance once we switched to a load balancer.

What is ended up being was the instance group port setting. We're obviously using ssl certs for the site but I had indicated port 80 and 443.

Once I removed port 80 from the instance group that the load balancer refers to, it loaded all the pages immediately.

Answer 2

I resolved this issue by removing and recreating custom domain settings for my App Engine project, and removing and recreating the corresponding DNS records in domains.google, following these instructions: https://cloud.google.com/appengine/docs/standard/python/mapping-custom-domains

I'm still not sure what the underlying issue was, but this fixed it. Hope this can help anyone encountering a similar issue.

Answer 3

Yesterday I tried opening the page and I noticed 1.8min to load the main page and 2.1min to make a search, later attempts were faster as you mentioned. I also tried accessing the page today and it loaded quite fast.

From my understanding the high latency of the first connection might be related to session handling, database connections, ssl certificates problem, huge amount of uncached data, expensive operations run before the server sends the response. It's nearly impossible for us to determine it without access to your code, logs and database configurations.

As for how to narrow down the issue I might suggest the following:

• Examine your logs for possible causes.
• Add timeit logging interleaved with each of the statements that handle the requests and watch for bottlenecks or long-running code.
• Deploy the same endpoint without logos, images and other data that would be browser-cached and see if it makes a difference.
• Create a hello-world simple endpoint and check it's latency. Keep slowly evolving the endpoint to resemble your actual handling code with hopes on finding what's the issue.

Answer 4

If only the first connection is slow, it might be because the instance is starting and you do not have minimum idle instances and warmup requests enabled. This configuration will make you have instances ready for taking traffic and the latency will be slower in the first connection.

As it is stated in the documentation:

If you set a minimum number of idle instances, pending latency will have less effect on your application's performance. Because App Engine keeps idle instances in reserve, it is unlikely that requests will enter the pending queue except in exceptionally high load spikes. You will need to test your application and expected traffic volume to determine the ideal number of instances to keep in reserve.

Also you can find more information about warmup requests in this documentation about Configuring Warmup Requests to Improve Performance