CODEWITHSUNDEEP
azure-data-factory
SmallFry
SmallFry

Reputation: 153

Azure Data Flows - can column names be set based on a non-header row's values

If I have an input text file and don't import the schema, because the header row is in not in the first row, is it possible to somehow detect the schema from somewhere in the file other than the header row i.e. a file with no header row but a row that could be used as a header further down the file:

<No Header Row>
123 01/01/2020 NULL
345 01/01/2021 01/01/2022
456 01/01/2019 NULL
AccNo StartDate EndDate
678 01/01/2021 01/02/2022

Could be transformed to and sunk in this schema after further transformations to match the output schema:

AccNo StartDate EndDate DateofDeath
123 01/01/2020 NULL NULL
345 01/01/2021 01/01/2022 NULL
456 01/01/2019 NULL NULL
678 01/01/2021 01/02/2022 NULL

I can't define an input schema as it needs to be flexible to allow for changes and multiple files (with different columns) will use the same process. I will though know the output schema.

Upvotes: 0

Views: 532

Answers (1)

Abhishek Khandave
Abhishek Khandave

Reputation: 3230

You can utilize the skipLineCount property available in Copy activity source settings.

  1. Select firstRowAsHeader property in dataset connection settings

  2. Copy activity source settings, set skipLineCount = 4 this will skip the first 4 rows

  3. In the mapping section do import schemas.

Above approach can be used, but you will not get first 4 records.

You can also refer this article

Upvotes: 1

Related Questions