CODEWITHSUNDEEP
azure-data-factory
Bob5421
Bob5421

Reputation: 9073

Difference between DataFlow and Pipelines

I do not understand the difference between dataflow and pipeline in Azure Data Factory.

I have read and see DataFlow can Transform Data without writing any line of code.

But I have made a pipeline and this is exactly the same thing.

Thanks

Upvotes: 17

Views: 19641

Answers (2)

Jay Gong
Jay Gong

Reputation: 23782

Firstly, dataflow activity need to be executed in the pipeline. So I suspect that you are talking about the copy activity and dataflow activity as both of them are used for transferring data from source to sink.

I have read and see DataFlow can Transform Data without writing any line of code.

Your could see the overview of Data Flow. Data flow allows data engineers to develop graphical data transformation logic without writing code. All data transfer steps are based on visual interfaces.

I have made a pipeline and this is exactly the same thing.

Copy activity could be used for data transmission. However, it has many limitations with column mapping. So,if you just need simple and pure data transmission, Copy Activity could be used. In order to further meet the personalized needs, you could find many built-in features in the Data Flow Activity. For example, Derived column, Aggregate,Sort etc.

Upvotes: 4

Joel Cochran
Joel Cochran

Reputation: 7728

A Pipeline is an orchestrator and does not transform data. It manages a series of one or more activities, such as Copy Data or Execute Stored Procedure. Data Flow is one of these activity types and is very different from a Pipeline.

Data Flow performs row and column level transformations, such as parsing values, calculations, adding/renaming/deleting columns, even adding or removing rows. At runtime a Data Flow is executed in a Spark environment, not the Data Factory execution runtime.

A Pipeline can run without a Data Flow, but a Data Flow cannot run without a Pipeline.

Upvotes: 28

Related Questions