CODEWITHSUNDEEP
loggingtracestackdrivergoogle-cloud-stackdrivergoogle-cloud-run
teaMonkeyFruit
teaMonkeyFruit

Reputation: 573

Google StackDriver: How do I correlate (and nest) other logs with trace logs?

Please help, I've been unsuccessful for 3 days :/

I'm deploying a handful of services to Google Cloud Run, and would like to trace requests and correlate logs to those requests as the requests hop through the services. StackDriver has a great platform for this in "StackDriver Trace". I'm using the Google.Cloud.Diagnostics library in my asp.net core web api's. I am able to successfully trace requests, and start spans, and see the spans nested inside the trace in StackDriver Trace Timeline. However I'm stumped on how to get my logs to correlate with the traces and spans?

The StackDriver docs state that writing "special fields" to the LogEntry object will be recognized by the Logging API:

Special fields in structured payloads

When the Logging agent receives a structured log record, it treats the following fields specially, allowing you to set specific fields in the LogEntry object that get written to the Logging API.

All (special fields) are stripped from the payload if present.

It specifically says this about the trace field:

The value of this field should be formatted as projects/[PROJECT-ID]/traces/[TRACE-ID], so it can be used by the Logs Viewer and the Trace Viewer to group log entries and display them in line with traces.

source: https://cloud.google.com/logging/docs/agent/configuration#special-fields

I've tried many different things to my structured JSON, but the Stackdriver Trace View doesn't recognize my logs and nest them inside the request trace. Currently I am including both the trace and spanId in the JSON.

Here's one of my logs that StackDriver Trace won't nest in the trace:

{
 insertId: "5da6c3a200j0923bx23x2"  
 jsonPayload: {
  ConnectionId: "0HLQI121N94JM"   
  CorrelationId: null   
  RequestId: "0HLQI121N94JM:00000001"   
  RequestPath: "/graphql"   
  message: "Getting Collection of type: FakeItem in GetCollectionOfItemsAsync"   
  messageTemplate: "Getting Collection of type: FakeItem in GetCollectionOfItemsAsync"   
  spanId: "4560986706170855936"   
  timestamp: "2019-10-16T07:15:46.8713740Z"   
  trace: "projects/myProject/traces/04b4a840df0289bb9fddcd62235d3ee4"   
 }
 labels: {
  instanceId: "00bf4bf02d34e072dc1c49x8dj943x4b5609mubm0409u566ad08acf6283d2b5135651fd8f2633e7b06e7dde4b96cfddbf5373a642da0b65fb21cf87a5aad"   
 }
 logName: "projects/myProject/logs/run.googleapis.com%2Fstdout"  
 receiveTimestamp: "2019-10-16T07:15:47.113845061Z"  
 resource: {
  labels: {
   configuration_name: "baseproject-graphql"    
   location: "us-central1"    
   project_id: "myProject"    
   revision_name: "baseproject-graphql-vhglp"    
   service_name: "baseproject-graphql"    
  }
  type: "cloud_run_revision"   
 }
 timestamp: "2019-10-16T07:15:46.871489Z"  
}

Here's a log thats generated by gcp that does get recognized:

{
 httpRequest: {
  latency: "0.026068056s"   
  protocol: "HTTP/1.1"   
  remoteIp: "73.158.189.48"   
  requestMethod: "POST"   
  requestSize: "1950"   
  requestUrl: "https://baseproject-api.myUrl.com/graphql"   
  responseSize: "2768"   
  serverIp: "152.289.4.125"   
  status: 200   
  userAgent: "PostmanRuntime/7.18.0"   
 }
 insertId: "5da6c3a8j90kjo9db8346"  
 labels: {
  instanceId: "00bf4bf02d34e072dc1cfda1073f2f5ec6888d75e1d75f26259006ad08acf6283d2b5135651fd8f26398n9hu0h9h09gm08g76f67f567fb21cf87a5aad"   
 }
 logName: "projects/myProject/logs/run.googleapis.com%2Frequests"  
 receiveTimestamp: "2019-10-16T07:15:47.207098181Z"  
 resource: {
  labels: {
   configuration_name: "baseproject-graphql"    
   location: "us-central1"    
   project_id: "myProject"    
   revision_name: "baseproject-graphql-vhglp"    
   service_name: "baseproject-graphql"    
  }
  type: "cloud_run_revision"   
 }
 severity: "INFO"  
 timestamp: "2019-10-16T07:15:46.877387Z"  
 trace: "projects/myProject/traces/04b4a840df0289bb9fddcd62235d3ee4"  
}

Any thoughts???

Upvotes: 3

Views: 2463

Answers (1)

Grayside
Grayside

Reputation: 4194

In order to correlate your logs with the Cloud Run request log, you need to populate the property logging.googleapis.com/trace, Cloud Run uses this as the trace property with Stackdriver Logging.

You can read more details in the Cloud Run logging docs and review the structured logging Node.js sample.

The relevant slice of code from the sample:

const traceHeader = req.header('X-Cloud-Trace-Context');
if (traceHeader && project) {
  const [trace] = traceHeader.split('/');
  globalLogFields[
    'logging.googleapis.com/trace'
  ] = `projects/${project}/traces/${trace}`;
}

Where the "globalLogFields" object is merged with the object for each individual log entry as part of the code.

Upvotes: 4

Related Questions