CODEWITHSUNDEEP
azure-data-factory
arghtype
arghtype

Reputation: 4544

Two pipelines writing into single dataset in AzureDataFactory

I'm trying to point two different copy activity pipelines into single output dataset. All pipelines and dataset have frequency/availability set to Day. I've tried configuring pipeline1 as "style": "StartOfInterval" and pipeline2 as "style": "EndOfInterval". But with that setup I'm getting error on publish:

The Activity schedule does not match the schedule of the output Dataset. Activity: 'MyCopyActivity'. Dataset: 'MyDataset'.","code":"ActivityDataSetSchedulerMismatch"

As a workaround I could create two different datasets, and point them to the same resource.

Is it possible to achieve this with single output dataset?

Upvotes: 0

Views: 1281

Answers (3)

mjonsing
mjonsing

Reputation: 74

No it is not possible to use a single output dataset in two copy activities. You need to create two datasets and point them to the same resource.

Upvotes: 0

Dhanraj Pervaram
Dhanraj Pervaram

Reputation: 1

Both pipeline and output datasets availability/scheduling properties should be same in all the cases. In you case, you have different "style" for Pipelines, but you are referring single output dataset which has only one style(default is Endofinterval).

For One pipeline it will match, but for other pipeline it will throw error.

To overcome this, you have to create two output datasets with same linked service. Don't forget to match the "style" of OutputDatasets with corresponding pipelines

Upvotes: 0

Alex KeySmith
Alex KeySmith

Reputation: 17101

If the reason is to merge multiple inputs into one output.

You could instead have a single copy activity pipeline have two separate inputs.

The data set inputs could have different availability windows and then the copy activity could combine them into one output dataset.

Upvotes: 1

Related Questions