Is it normal to wait 18 hours for a log message to show up?

Question

Is this an issue everyone just has to put up with? It takes log messages at least 12-18 hours to appear in Cloud Logging. We never have any idea what our Google VMs are doing. Most of the time we never see our log messages at all, just millions of lines of this stuff:

2021-11-09T18:52:38.887509624Z Unable to export to Monitoring service because: GaxError RPC failed, caused by 3:Field timeSeries[0].points[0].interval.end_time had an invalid value of "2021-10-30T16:09:38.770613-07:00": Data points cannot be written more than 24h in the past.. debug_error_string:{"created":"@1636483958.886243136","description":"Error received from peer ipv4:172.217.11.202:443","file":"src/core/lib/surface/call.cc","file_line":1062,"grpc_message":"Field timeSeries[0].points[0].interval.end_time had an invalid value of "2021-10-30T16:09:38.770613-07:00": Data points cannot be written more than 24h in the past.","grpc_status":3}
 A 
2021-11-09T18:52:38.903761Z 2021-11-09T18:52:38.903512137Z container kill 714e6b93c8c67b5a54569aed3bd4a5985591600230af8ed860b5789e686e77aa (image=gcr.io/stackdriver-agents/stackdriver-logging-agent:1.8.9, name=stackdriver-logging-agent, signal=23) A 
2021-11-09T18:52:38.917711Z 2021-11-09T18:52:38.917505171Z container kill 714e6b93c8c67b5a54569aed3bd4a5985591600230af8ed860b5789e686e77aa (image=gcr.io/stackdriver-agents/stackdriver-logging-agent:1.8.9, name=stackdriver-logging-agent, signal=23) A 
2021-11-09T18:53:38.890751337Z Unable to export to Monitoring service because: GaxError RPC failed, caused by 3:Field timeSeries[0].points[0].interval.end_time had an invalid value of "2021-10-30T16:09:38.770613-07:00": Data points cannot be written more than 24h in the past.. debug_error_string:{"created":"@1636484018.889384555","description":"Error received from peer ipv4:172.217.11.202:443","file":"src/core/lib/surface/call.cc","file_line":1062,"grpc_message":"Field timeSeries[0].points[0].interval.end_time had an invalid value of "2021-10-30T16:09:38.770613-07:00": Data points cannot be written more than 24h in the past.","grpc_status":3}
 A 
2021-11-09T18:54:38.879268968Z Unable to export to Monitoring service because: GaxError RPC failed, caused by 3:Field timeSeries[0].points[0].interval.end_time had an invalid value of "2021-10-30T16:09:38.770613-07:00": Data points cannot be written more than 24h in the past.. debug_error_string:{"created":"@1636484078.878076512","description":"Error received from peer ipv4:172.217.11.202:443","file":"src/core/lib/surface/call.cc","file_line":1062,"grpc_message":"Field timeSeries[0].points[0].interval.end_time had an invalid value of "2021-10-30T16:09:38.770613-07:00": Data points cannot be written more than 24h in the past.","grpc_status":3}
 A 
2021-11-09T18:55:38.889891450Z Unable to export to Monitoring service because: GaxError RPC failed, caused by 3:Field timeSeries[0].points[0].interval.end_time had an invalid value of "2021-10-30T16:09:38.770613-07:00": Data points cannot be written more than 24h in the past.. debug_error_string:{"created":"@1636484138.888728523","description":"Error received from peer ipv4:172.217.11.202:443","file":"src/core/lib/surface/call.cc","file_line":1062,"grpc_message":"Field timeSeries[0].points[0].interval.end_time had an invalid value of "2021-10-30T16:09:38.770613-07:00": Data points cannot be written more than 24h in the past.","grpc_status":3}

If our log messages do appear, they're always vastly outnumbered by these error messages.

Google's troubleshooting instructions (https://cloud.google.com/logging/docs/agent/logging/troubleshooting) are useless; right away I get "sudo: service: command not found" using their standard "container-optimized" VM image.

Answer 1

Please check this.

They mention of a fix in next release (mentioned 4 days ago) and workaround is to use ops-agent.