CODEWITHSUNDEEP
azureazure-data-factory
Sameera
Sameera

Reputation: 1

ADF: Dataflow sink activity file format for adls

I wanted to copy multiple table data from Azure SQL database to ADLS gen2. I created a pipeline which take table names as dynamic input values. later i used dataflow activity which copies the data to adls. I used sink type as delta. Now few of my table data are getting copied to adls properly with snappy.parquet format but few are giving error as column names are invalid for delta format.

How can we deal with this error and get data copied from all tables?

Also for knowledge wanted to know that does file formats for the files generated at destination folder in adls are by default parquet file? Or is there any option to change that?

Upvotes: 0

Views: 874

Answers (1)

Kiran-MSFT
Kiran-MSFT

Reputation: 234

Delta format is parquet underneath. You cannot use characters like " ,;{}()\n\t=" and have to replace that with _ or another character.

Dataflow has easy ways to rename column names in derive or select transforms.

Upvotes: 0

Related Questions