Reputation: 51
I am building data pipelines (ETL) using Python and BigQuery. My repository is safely stored on a GitHub-like service and the pipeline will be built to a Docker container that is later run on a Kubernetes cluster.
The pattern is always the same for the data pipelines:
dataset.table
names.I let the Python BigQuery Client
object to manage the project id as when using the private key to generate the client, the project-id
becomes an attribute of the Client, so I don't need to worry to much about that. The private key is safely passed to the container as a secret.
My question is if that building my pipelines this way poses a security risk?
Another thing I've tried is creating environment variables for the destination table names. But it seems to me this may add some unnecessary complexity and obscurity to the code.
Upvotes: 0
Views: 51