CODEWITHSUNDEEP
azureazure-data-factorygoogle-cloud-dataflow
Serdia
Serdia

Reputation: 4418

How to modify file after SInk transformation was used already in Azure dataflow?

Using dataflow

  1. connect to source
  2. use Sink. combine all excel files into a single .csv
  3. Need to modify newly created .csv file

How can I now modify newly created csv file and delete rows with NULL value?

Do I need to create a new(second) Source?

If I create a second source will it be executed in sequence or in parallel?

enter image description here

Upvotes: 0

Views: 45

Answers (1)

Daredevil
Daredevil

Reputation: 1615

After writing the combined data to a CSV to Delete rows with NULL values, You can use the Filter transformation. If, for example, you want to remove rows where a column named "ColumnName" has a NULL value, you'd add a Filter transformation with the condition: isNotNull(ColumnName).

You do not need a second source unless you're planning to read from a different location. If your aim is just to remove NULL rows from the combined CSV, then no need for a second source. The flow can handle it in a single run.

Mapping Data Flows in Azure Data Factory are executed in sequence, based on the order of transformations you set up. If you had two distinct sources, they could potentially read in parallel, but the subsequent transformations would be in sequence based on your flow design.

Upvotes: 1

Related Questions