CODEWITHSUNDEEP
azure-data-factory
Shivam Yadav
Shivam Yadav

Reputation: 23

How to add header to file in Azure Data Factory

I am storing the header in a CSV file and concatenating it with the data file using mapping data flow. I am using union Activity to combine these two files. While combining the header file and data file, I can see the data but header data is not at the top. It's randomly present in the sink file.

How can I make the header at top ?

1,Company_Reference_ID,PH2_TIC,USD,PH2_Actuals,2021-03-12,PH2_X_VB_V3,85738,V3Premium
1,Company_Reference_ID,PH2_TIC,USD,PH2_Actuals,2021-03-12,PH2_X_VB_V3,85738,V3Premium Journey,CompanyRerenceIDType,CompaReferenceID,Currecy,Ledgerype,Accountinate,Journaource 1,Company_Reference_ID,PH2_TIC,USD,PH2_Actuals,2021-03-12,PH2_X_VB_V3,85738,V3Commission

Upvotes: 1

Views: 2806

Answers (2)

Joseph Xu
Joseph Xu

Reputation: 6083

Update:
My debug result is as follows, I think it is what you want: enter image description here

I created a simple test to merge two csv files. One header.csv and another vlaues.csv.
As @Mark Kromer MSFT said we can use Surrogate Key and then sort these rows.The Row_No of heard.csv will start from 1 and values.csv will start from 2.

  1. Set header source to the header.csv and don't select First row as header. enter image description here

  2. Set header source to the values.csv and don't select First row as header. enter image description here

  3. At SurrogateKey1 activity , enter Row_No as Key column and 1 as Start value. enter image description here

  4. At SurrogateKey2 activity , enter Row_No as Key column and 2 as Start value. enter image description here

  5. Then we can uion SurrogateKey1 stream and SurrogateKey2 stream at Union1 activity. enter image description here

  6. Then we can sort these rows by Row_No at Sort1 activity. enter image description here

  7. We can use Select1 activity to filter Row_No column. enter image description here

  8. I think it is what you want: enter image description here

Upvotes: 1

Mark Kromer MSFT
Mark Kromer MSFT

Reputation: 3838

For now, you would need to use a Surrogate Key for the different streams and make sure that the header row has 1 for the surrogate key value and sort by that column.

We are working on a feature for adding a header to the delimited text sink as a property in the data flow Sink. That will make it much easier and should light-up in the UI soon.

Upvotes: 1

Related Questions