CODEWITHSUNDEEP
google-cloud-platformetlgoogle-cloud-data-fusioncdap
RaptorX
RaptorX

Reputation: 139

Cloud data fusion iterating over same pipeline

Usecase to execute-

  1. Excel with multiple tabs uploaded to cloud storage
  2. Cloud function trigger calls cloud datafusion pipeline
  3. Pipeline reads the file uses wrangler to read the individual sheets and write to separate tables as per the sheet

Though I am using params to define the sheetname in wrangler(as in usage for usable pipelines), I am unable to figure out how to either iterate to run for each tab, or use cloud function to span 20 calls to same pipeline with parameter as each tab name. I do not think running parallel paths in same pipeline is going to help or is a good solution. Any help appreciated.

Upvotes: 0

Views: 532

Answers (1)

RaptorX
RaptorX

Reputation: 139

To run parallel pipelines, we can iterate through API call using cloud functions. Though I am not sure how many instances are allowed in a single call to CDF, I could run three parallel runs of same pipeline.

Upvotes: 1

Related Questions