CODEWITHSUNDEEP
powershellazure-data-factory
baatchen
baatchen

Reputation: 489

How to check current status of a Data Factory pipelinerun with just Pipeline name?

Is it possible to check the current status of a Azure Data Factory pipelinerun with just Pipeline name either through Powershell or API ?

I have seen that you can use Get-AzDataFactoryV2PipelineRun but that requires you to have the pipelinerunid.

My goal is to build a script that will first check if a pipeline run is running and if not trigger it. I want to avoid the script to trigger a pipelinerun so that there will be multiple pipeline runs running at the same time.

Upvotes: 2

Views: 3973

Answers (2)

Abhishek Tiwari
Abhishek Tiwari

Reputation: 11

Using .Net Core SDK

ServiceClientCredentials cred = new TokenCredentials(accessToken);
using (var client = new DataFactoryManagementClient(cred) { SubscriptionId = subscriptionId })
{
    RunQueryFilter filter1 = new RunQueryFilter("PipelineName", "Equals", new List<string> { "ActualPipelineName" });
    RunQueryFilter filter2 = new RunQueryFilter("Status", "Equals", new List<string> { "Queued" });
    DateTime before = DateTime.UtcNow;
    DateTime after = before.AddHours(-4);
    RunFilterParameters param = new RunFilterParameters(after, before, null, new List<RunQueryFilter> { filter1, filter2 }, null);
    PipelineRunsQueryResponse pipelineResponse = client.PipelineRuns.QueryByFactory(resourceGroupName, azureDataFactoryName, param);
    int? QueuedPipelines = pipelineResponse?.Value?.Count;
}

In filter 1, you can use "In" operator to query for more than one PipelineName.

In filter2, you can use any valid status of ADF (Success, InProgress, Queued etc.)

This same thing can also be achieved using REST API.

P.S- you might need to use Continuation token if the count of pipelines is more than 100 (for that particular status).

Upvotes: 1

Martin Esteban Zurita
Martin Esteban Zurita

Reputation: 3209

I've done it with PowerShell, although I'm sure there are better ways, this is the solution I've come up with.

It basically gets all the pipeline run ids from a week ago to this day, iterates through each one and if it finds one that is "InProgress", it will not do anything. If no pipeline run id is in that state, then execute it.

You obviously need to be authenticated previously to run this:

$before = get-date
$after = (get-date).AddDays(-7)
$runIds = Get-AzDataFactoryV2PipelineRun -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName -LastUpdatedAfter $after -LastUpdatedBefore 
$before
$shouldRun = 1

$runIds | ForEach-Object {
    ## Check for all statuses
    if($_.Status -eq "InProgress"){
        $shouldRun = 0
    }
}

if($shouldRun){
    ## Logic to run pipeline, this is just an example.
    Invoke-AzDataFactoryV2Pipeline -PipelineName $PipelineName -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName
}

Upvotes: 2

Related Questions