CODEWITHSUNDEEP
javagoogle-cloud-dataflowapache-beam
Nikhil Suthar
Nikhil Suthar

Reputation: 2431

How to pass Runtime query to SqlTransform in apache beam?

I want to pass the Query dynamically while running the Dataflow job. I am using SQLTransform which works fine when I pass Query within code.

My use case requires passing the Query at Runtime, is it possible with SqlTransform in Apache Beam?

This works if I hard-code it in code.

String PQuery = "SELECT col1, max(col2) as max_watermark FROM PCOLLECTION GROUP BY col1";
PCollection<Row> rows1 =   rows.apply(SqlTransform.query(PQuery));

But with valueProvider input, it gives compile time error.

PCollection<Row> rows1 =   rows.apply(SqlTransform.query(options.getQuery()))

Error

The method query(String) in the type SqlTransform is not applicable for the arguments (ValueProvider<String>)

Upvotes: 1

Views: 214

Answers (2)

robertwb
robertwb

Reputation: 5104

You should use FlexTemplates which allow dynamic graph construction (such as SqlTransform) uses based on template parameters.

Upvotes: 1

Mazlum Tosun
Mazlum Tosun

Reputation: 6572

To solve your issue, you need to get the value inside the ValueProvider :

PCollection<Row> rows1 = rows.apply(SqlTransform.query(options.getQuery().get()))

The query method takes a String as parameter, that's why you need to get the String value of the ValueProvider option.

Upvotes: 1

Related Questions