CODEWITHSUNDEEP
javagoogle-bigquerygoogle-cloud-dataflowapache-beam
Alexander Chandra
Alexander Chandra

Reputation: 659

Dataflow Pipeline - Using dynamic param or query

I am trying to create a dataflow pipeline template, which required me to read data from bigquery. So what i need is to make my query dynamic using like Instant.now() but it seems the query is locked when creating the template

  Some Code HERE
  Some Code HERE
  Some Code HERE
 
  pipeline.apply("ReadFromBigQuery",
      BigQueryIO.read(new DataTransformer(MyCustomObject.getQuery()))
          .fromQuery(spec.getQuery())
          .usingStandardSql()
          .withQueryLocation("US")
          .withoutValidation()
  ).apply("do Something 1",
      Combine.globally(new CombineIterableAccumulatorFn<MyCustomObject2>())
  ).apply("do Something 2",
      ParDo.of(new SendToKenshoo(param, param2)
  );

My query is like this

SELECT * FROM `my-project-id.my-dataset.my-view` where PARTITIONTIME between TIMESTAMP('@currentDate') and  TIMESTAMP('@tomorrowDate')

need to replace that @currentDate and @tomorrowDate using Instant.now() or any time function please give me some example

note : i need to change the date on the code instead on query level like this

SELECT * FROM `my-project-id.my-dataset.my-view` where PARTITIONTIME between DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY) and  CURRENT_DATE()

Upvotes: 0

Views: 400

Answers (1)

fferngar
fferngar

Reputation: 29

I'm not sure how you're sending those parameters to the query (via value provider, etc). However, I wouldn't recommend using templates for that because you need dynamic inputs. If you want to do that, I would use Flex Templates: https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates

Upvotes: 1

Related Questions