CODEWITHSUNDEEP
azure-data-factory
jidl
jidl

Reputation: 295

Splitting a Data Factor Pipeline based on the presence of a column

I'm creating a pipeline to ingest a series of csvs into an Azure SQL Database

The CSVs come from a piece of medical software named SystmOne, the CSVs are either a full dataset or a delta dataset. The only difference in their schemas is the presence of an additional column: RemovedData.

The presence of this column will require an additional step in the pipeline (deleting any row from the database with RemovedData == true).

Is there a way in ADF or (ADF with Data Flow Preview) to query the file for the presence of a column and split the pipeline based on the result?

I have no control over the intial output of the file.

Upvotes: 0

Views: 253

Answers (1)

DraganB
DraganB

Reputation: 1138

You can check the number of columns in your source dataset, with getMetadataActivty columnCount property, and then with If Activity do what you want.

GetMetadataActivty

Expression in IF activity: @equals(activity('YourGet Metadata').output.columnCount,numberOfColumns)

And then based on true or false you choose your dataset with propper schema.

Upvotes: 1

Related Questions