CODEWITHSUNDEEP
azureazure-data-factory
user6140587
user6140587

Reputation: 81

Azure: Waiting Dataset dependencies in manual created pipelines

I've created a few simple pipelines in data factory with different kinds of linked services and datasets. Manually, by writting jsons. But none works.

Pipeline created by wizard works and have Ready status. All manual pipelines have status Waiting: Dataset dependencies. What does it mean and how can I fix it?

Upvotes: 6

Views: 4556

Answers (2)

Kez Nwichi
Kez Nwichi

Reputation: 141

Also, need to set external property of upstream dataset to true

    "external": true,

Upvotes: 11

Kyle Hale
Kyle Hale

Reputation: 8120

From the documentation:

Some scenarios require the ability to produce output at a frequency different than frequencies of one or more inputs. Data factory supports modeling these scenarios.

. . .

The output slice for every day depends on 24 hourly slices from input dataset. Data factory computes these dependencies automatically by figuring out the input data slices that fall in the same time period as the output slice to be produced. If any of the .. input slices is not available (due to processing happening in an activity upstream that produces the that slice for example) data factory will wait for the input slice to be ready before kicking off the daily activity run.

So you're doing something where your input data slices aren't available to the data factory. Unfortunately, without you posting your pipeline JSON, it's hard to diagnose the issue.

Upvotes: 4

Related Questions