CODEWITHSUNDEEP
google-bigquerygoogle-cloud-platformgoogle-cloud-stackdriver
Graham Polley
Graham Polley

Reputation: 14791

How do I coalesce Stackdriver logs/sinks into one BigQuery project/dataset?

Setting up Stackdriver log sinks to BigQuery is straightforward.

However, I have lots of projects, and instead of each export sink going to its corresponding project, I'd like to coalesce the logs from all my projects to one dedicated project.

The configuration in the Stackdriver sink config doesn't appear to let me select a different project to send the logs to.

enter image description here

How do I select a different project/dataset?

Upvotes: 2

Views: 973

Answers (3)

Pradeep Bhadani
Pradeep Bhadani

Reputation: 4741

You can configure Bigquery Audit sink at Org level using gcloud command as below

gcloud logging sinks create my-org-bq-sink bigquery.googleapis.com/projects/my-gcp-project/datasets/my_bq_audit_dataset --include-children --organization=MY_ORG_ID

Hope this helps.

Upvotes: 0

Summit Raj
Summit Raj

Reputation: 890

If all of your projects are under a Cloud Folder or under the same organization, you can also create an Aggregated Export as documented in https://cloud.google.com/logging/docs/export/aggregated_exports.

The aggregated sink lives in the folder or org, and exports all logs of children projects to the same destination.

Upvotes: 1

Graham Polley
Graham Polley

Reputation: 14791

You need to select the 'Custom destination' option. This will allow you to plug in a different project/dataset. The default datasets you see are always tied to whichever project you've currently got selected, so you can't select a different project/dataset.

Instead:

  1. Select 'Custom Destination'
  2. Add the destination using this format: bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET_ID]
  3. Give the sink writer editor permissions on the said project/dataset in step 2.

See more here.

Upvotes: 4

Related Questions