CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquerystackdriver
Tyler Treat
Tyler Treat

Reputation: 15008

Exporting Stackdriver traces to BigQuery

I'm wondering if there is a good way to export spans from Google Stackdriver to BigQuery for better analysis of traces?

The only potential solutions I'm seeing currently are writing to the trace and BigQuery APIs individually or querying the trace API on an ad hoc basis.

The first isn't great because it would require a pretty big change to the application code (I currently just use OpenCensus with Stackdriver exporter to transparently write traces to Stackdriver). The second isn't great because it's a lot of lift to query the API for spans and write them to BigQuery and it has to be done on an ad hoc basis.

A sink similar to log exporting would be great.

Upvotes: 0

Views: 471

Answers (3)

Sneha Mule
Sneha Mule

Reputation: 715

You exporting logs to big-query , you have to create table in big-query and add data in bigquery.

By default in GCP, all logs go via stack driver.

To export logs in the big query from the stackdriver , you have to create Logger Sink using code or GCP logging UI

Then create a Sink, add a filter. https://cloud.google.com/logging/docs/export/configure_export_v2

enter image description here

hen add logs to stack driver using code

public static void writeLog(Severity severity, String logName, Map<String, 
 String> jsonMap) {
 List<Map<String, String>> maps = limitMap(jsonMap);
 for (Map<String, String> map : maps) {
     LogEntry logEntry = LogEntry.newBuilder(Payload.JsonPayload.of(map))
         .setSeverity(severity)
         .setLogName(logName)
         .setResource(monitoredResource)
         .build();
   logging.write(Collections.singleton(logEntry));
}

}

    private static MonitoredResource monitoredResource = 
    MonitoredResource.newBuilder("global")
       .addLabel("project_id", logging.getOptions().getProjectId())
       .build();

Upvotes: 0

Arcosphere
Arcosphere

Reputation: 36

Unfortunately, at this moment there is no way to Exporting Stackdriver traces to BigQuery.

I noticed that exactly the same feature was already asked to be implemented on GCP side. GCP product team are already aware about this feature request and considering to implement this feature.

Please note that the Feature Requests are usually not resolved immediately as it depends on how many users are demanding the same feature. All communication regarding this feature request will be done inside public issue tracker 1, also you can 'star' it to acknowledge that you are interested in it, but keep in mind that there is no exact ETA.

Upvotes: 1

guillaume blaquiere
guillaume blaquiere

Reputation: 75900

Yes. It's a best practice that recommend you.

  1. the analysis is better, your logs are partitioned and the queries efficient.
  2. the log format don't change. The values that you log can , but not your query structure
  3. The logs have a limited retention period in stackdriver. With bigquery, you keep all the time that you want
  4. It's free! At least, the sink process. You have to pay storage and bigquery processing

I have 3 advices:

  • think to purge your logs for reducing storage cost. However, data older than 90 days are cheaper.
  • before setting up a sink, select only the relevant logs entry that you want to save to bigquery.
  • don't forget the partition time, logs can be rapidly huge, and uncontrolled query expensive.

Bonus: if you have to comply to RGPD and you have personal data in logs, be sure to list the process in your RGPD log book.

Upvotes: 0

Related Questions