CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquery
Tanveer Uddin
Tanveer Uddin

Reputation: 1525

BigQuery - Project, DataSet setup best practices

I am setting up BigQuery for a greenfield GCP implementation. I am wondering if there is any best practice available regarding the project, dataset organisation e.g whether should I create a single project with different datasets for all sources/layers for raw, processed, datamart layers? Or different projects for different uses cases and access patterns?

Option 1:

Project

|_ Dataset_RAW

|_ Dataset_Processes

|_ Dataset_Datamart_Finance

|_ Dataset_Datamart_Marketing

Option 2:

Project RAW:

|_ Dataset_Source_A

|_ Dataset_Source_A

Project Processes:

|_ Dataset_Standardized

Project Finance:

|_ Dataset_Finance_DataMart

Project Marketing:

|_ Dataset_Marketing_DataMart

I suppose, it's a broad question and depends a lot on company objective. But I am yet curious if there is any guidelines available based on different scenarios.

Upvotes: 1

Views: 558

Answers (1)

guillaume blaquiere
guillaume blaquiere

Reputation: 75735

There is 2 things to know:

  1. If you avoid all primitive roles on the project (owner, editor, viewer), the access right can be managed at the dataset level. By the way, putting data in one project or another one has no impact
  2. The billing is separate in 2: the cost for the storage and the cost for the queries. If you separate the project like the option 2, and if, for example, the projects belong to folders for a re-bill of subsidiaries (like in my company), you can have project which only host the data (and pay almost nothing) and the project that query the data. However, you can also manage this with label on dataset if you prefer

Another consideration: If you want to secure your data with a VPC Service Control, it's interesting to store the sensitive data in a specific project (that you want to protect with VPC SC)

As you see, all depend of your organisation, your strategy and your wishes. My advice, is to reproduce the real team organisation, in project organisation. You have 3 different teams? Configure 3 projects, each team is responsible of its project.

Upvotes: 2

Related Questions