CODEWITHSUNDEEP
google-cloud-dataflowapache-beam
Rich
Rich

Reputation: 1195

How to provide GCP Beam/Dataflow secrets (host/port, username, password) when consumer does not expect a ValueProvider?

What is the best way (considering both security and template re-use) to provide secrets like a host/port and external service username/password when the consumer doesn't expect or support a ValueProvider? By consumer I mean what is typically an IO class, such as MongoDbIO, ElasticsearchIO, etc.

I've already seen and successfully used this method: https://henrysuryawirawan.com/posts/dataflow-secret-manager/ but in that example, and some apache-provided IOs, the IO accepts the parameters in a ValueProvider. The apache ElasticsearchIO does not, neither does the MongoDbIO. The RedisIO does (via the RedisConnectionConfiguration). SpannerIO does.

So if the IO we want to use does not, is providing the config statically at template compile time the only option, and how is this typically done? Providing them dynamically seems safer security-wise, especially through something like Secret Manager.

Upvotes: 1

Views: 261

Answers (1)

Bruno Volpato
Bruno Volpato

Reputation: 1428

Yes, SecretManager / KMS are likely the best ways to deal here. But unfortunately, your options are limited if ValueProviders are not allowed.

You may use Flex Templates to work around the need of ValueProviders. We have a probably useful example in DataflowTemplates: KMSUtils.java, which expands options using KMS during runtime.

Upvotes: 3

Related Questions