CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-cdn
bogumbiker
bogumbiker

Reputation: 527

High latency from Google CDN - how to troubleshoot it?

We are trying to find out why there is a high latency from Google CDN.

Our site is behind Google'a http_load_balancer with CDN turned on.

For example, by inspecting sampe GET request for a jpg file (43Kb), we can see from http_load_balancer logs that around 30% of such requests have httpRequest.latency > 1 second, and a lot are taking much longer like several or hundreds of seconds....

This is just by looking at 24h log sample (around 6K the same requests).

The httpRequest.cacheLookup and httpRequest.cacheHit for all of those requests are true. Also jsonpayload_type_loadbalancerlogentry.statusdetails is response_from_cache and jsonpayload_type_loadbalancerlogentry.cacheid values shows correct region.

When doing the same GET request manually in the browser we are getting expected results with TTFB around 15-20ms.

Any idea where to look for a clue?

Upvotes: 0

Views: 1702

Answers (2)

Oscar DP
Oscar DP

Reputation: 26

Latency can be introduced:

  • Between the original client and the load balancer. You can see the latency of that segment with the metric https/frontend_tcp_rtt.
  • Between the load balancer and the backend instance. Which can be reviewed with the metric https/backend_latencies (this metric also includes the app processing time in your backend).
  • By the software running on the instance itself. To investigate this I would check the access/error logs on the backend instance software and resource utilization of the VM instance.

Further information about metrics description on the GCP load balancer metrics doc.

httpRequest.latency log field description: "The request processing latency on the server, from the time the request was received until the response was sent."

Upvotes: 0

elithrar
elithrar

Reputation: 24300

The httpRequest.latency field measures the entire download duration, and is directly impacted by slow clients - e.g. a mobile device on a congested network or throttled data plan.

You can check this by looking at the frontend_tcp_rtt metric (which is the RTT between the client and Cloud CDN) in Cloud Monitoring, as well as the average, median and 90th percentile total_latencies, where the slow clients will show up as outliers: https://cloud.google.com/load-balancing/docs/https/https-logging-monitoring#monitoring_metrics_fors

You may find that slow clients are from a specific group of client_country values.

Upvotes: 1

Related Questions