CODEWITHSUNDEEP
azure-data-factory
PyRar
PyRar

Reputation: 549

Data Factory Dataflow One Partition from Sink Fails to bring data

I have a dataflow inside data factory with a sink that is ADLS gen 2. Basically, I'm trying to move some parquet files from one zone to another. The problem comes when regarding the option chosed in the 'Optimize' tab, I'm missing some data at the end of the flow. Did somebody else encountered this?

Upvotes: 0

Views: 509

Answers (1)

Kiran-MSFT
Kiran-MSFT

Reputation: 234

Partitions in parquet have nothing to do partitions in dataflow. The partitions are formed by splitting files. If you want the same number of files you should use the "Column to store file name" feature on both source and sink and create as many files as source.

Upvotes: 0

Related Questions