Google Stackdrive custom metrics - data retention period

Question

I'm using GCP Stackdrive custom metrics and created few dashboard graphs to show the traffic on the system. The problem is that the graph system is keeping the data for few weeks - not forever.

From Stackdrive documentation:

See Quotas and limits for limits on the number of custom metrics and the number of active time series, and for the data retention period. If you wish to keep your metric data beyond the retention period, you must manually copy the data to another location, such as Cloud Storage or BigQuery.

Let's decide to work with Cloud Storage as a container to store data for the long term.

Questions:

1. How does this "manual data copy" is working? Just write the same data into two places (Google storage and Stackdrive)?
2. How the stackdrive is connecting the storage and generating graph of it?

Answer 1

I'll suggest this guide on creating a pipeline to export metrics to BigQuery for long-term storage and analytics. https://cloud.google.com/solutions/stackdriver-monitoring-metric-export

Answer 2

You can use Stackdriver's Logs Export feature to export your logs into either of three sinks, Google Cloud Storage, BigQuery or Pub/Sub topic. Here are the instructions on how to export stackdriver logs. You are not writing logs in two places in real-time but exporting logs based on the filters you set.

One thing to keep in mind is you will not be able to use stackdriver graphs or alerting tools with the exported logs.

Answer 3

In addition, if you export logs into bigquery, you can plug a Datastudio graphe to see your metrics.

You can also do this with Cloud Storage export but it's less immediate and less handy