Stackdriver stopped logging from GKE Containers

Question

Logs from Spring Boot applications deployed to GKE stopped showing up in Stackdriver Logging after February 2nd 2020. What happened around that time is that Stackdriver moved to a new UI, more integrated with the GCP console - could that have anything to do with it?

I do have other projects in GKE, such as a Node.js based backend, where logging to Stackdriver has continued without interruption, but there is just silence from the Spring Boot apps:

If I select "Kubernetes Container" instead of "GKE Container" in the GCP console at "Stackdriver Logging -> Logs Viewer" I do see some log statements, specifically errors like:

WARNING: You do not appear to have access to project [my-project] or it does not exist.

and

Error while fetching metric descriptors for kube-proxy: Get https://monitoring.googleapis.com/v3/projects/my-project/metricDescriptors?...

and

Error while sending request to Stackdriver Post https://monitoring.googleapis.com/v3/projects/my-project/timeSeries?...

OK, so that seems to start explaining the problem but I haven't been changing any IAM permissions, and when comparing those to the ones in the project hosting the Node.js GKE deployments which continue logging fine, they seem to be the same.

Should I be changing some permissions in the project hosting the Spring Boot GKE deployments, to get rid of those Stackdriver errors? What IAM member affects those? What roles would be required?

Answer 1

Turns out that the GKE cluster had Legacy Stackdriver Logging and Legacy Stackdriver Monitoring enabled:

and the problem was solved by setting those attributes to disabled and configuring the Stackdriver Kubernetes Engine Monitoring attribute:

But why the Stackdriver Logging continues uninterrupted for the Node.js applications, with the legacy options enabled, is still a mystery to me.