CODEWITHSUNDEEP
azure-data-factory
Harry Leboeuf
Harry Leboeuf

Reputation: 745

Is there a way to SkipLinesAtEnd in a TextFormat Azure Data Factory

We receive Text files from a external partner. They claim to be csv but have some difficult pre-header and footers.

In a ADF TextFormat I can use "skipLineCount": 6, But at the end i'm running in troubles ...

Any suggestions ?

Can't find something like SkipLinesAtEnd ....

This is the Sample

TITLE : Liste de NID_C_BG_NPIG configuré.
FILE NAME : Ines_bcn_npig_net_f.csv
CREATION DATE : 09/10/2019 23:18:43
ENVIRONMENT : Production 12c
<Begin of file>
"NID_C";"NID_BG";"N_PIG"
"253";"0";"0"
"253";"0";"1"
"253";"1";"0"
"253";"1";"1"
"253";"2";"0"
"253";"2";"1"
"253";"3";"0"
<End of file>

Upvotes: 1

Views: 206

Answers (2)

Mark Kromer MSFT
Mark Kromer MSFT

Reputation: 3838

Jay & Joel are correct in pointing you toward Data Flows to solve this problem. Use Copy Activity in ADF for data movement-focused operations and Data Flows for data transformation.

You'll find the price for data movement similar to that of data transformation.

I would solve this in Data Flow and use a Filter transformation to filter out any row that has the string "" in it.

Should not need an Alter Row in this case. HTH!!

Upvotes: 0

Jay Gong
Jay Gong

Reputation: 23792

It seems that you are using skipLineCount setting in Data Flow.No feature like skipLinesAtEnd in ADF.

You could follow suggestion mentioned by @Joel that using Alter Row.

enter image description here

However,based on the official document,it only supports database mode sink.

enter image description here

So,if you are limited by that,i would suggest you parse the file first before copy job.For example,add an Azure Function Activity to cut the extra rows if you know the specific location of header and foot.Inside the Azure Function,just use the code to alter the file.

Upvotes: 1

Related Questions