CODEWITHSUNDEEP
powershellazureazure-powershellazure-data-factory
Joseph
Joseph

Reputation: 570

Get Status of Pipeline in Azure Data Factory

I am using Set-AzureRmDataFactoryPipelineActivePeriod to schedule the pipeline via PowerShell. However I am trying to get hold of Azure Commandlet that can give me the completion status of the pipeline once scheduled OR the code should run into some loop till the time the pipeline has not completed.

Upvotes: 1

Views: 6902

Answers (3)

Antonia
Antonia

Reputation: 288

ADF v2

For v2 of Azure DataFactory, you can use Get-AzureRmDataFactoryV2PipelineRun cmdlet to get pipeline runs from some time period. Parameters you need to set are ResourceGroupName, DataFactoryName, LastUpdatedAfter and -LastUpdatedBefore and optionally PipelineName.

For example, if you wanted to get pipeline runs from the last hour, for your pipeline my-pipeline in my-adf DataFactory in resource group my-rg, you would execute something like 

Get-AzureRmDataFactoryV2PipelineRun -ResourceGroupName "my-rg" `
        -DataFactoryName "my-adf" `
        -PipelineName "my-pipeline" `
        -LastUpdatedAfter (Get-Date).AddHours(-1) `
        -LastUpdatedBefore (Get-Date).AddHours(1)

For more information, here's the cmdlet documentation: https://learn.microsoft.com/en-us/powershell/module/azurerm.datafactories/get-azurermdatafactoryv2pipelinerun?view=azurermps-4.4.1

ADF v1

If you are using ADF v1, you can invoke Get-AzureRmDataFactoryRun cmdlet to get pipeline status.However, since in DataFactory v1 output dataset slice is what runs the schedule for pipeline execution, you need to pass your output dataset name as well. Below is an example how you would invoke that cmdlet to get slices from the past hour

Get-AzureRmDataFactoryRun -ResourceGroupName "my-rg" `
        -DataFactoryName "my-adf" `
        -DatasetName "my-dataset" `
        -StartDateTime (Get-Date).AddHours(-1)

For more information, here's the cmdlet documentation: https://learn.microsoft.com/en-us/powershell/module/azurerm.datafactories/get-azurermdatafactoryrun?view=azurermps-5.0.0

Upvotes: 1

Mark Kromer MSFT
Mark Kromer MSFT

Reputation: 3838

The RunID comes from every pipeline run. With PowerShell, you can Invoke a pipeline and capture the RunID from that cmdlet:

https://learn.microsoft.com/en-us/powershell/module/azurerm.datafactoryv2/Invoke-AzureRmDataFactoryV2Pipeline?view=azurermps-4.4.1

There is an example of setting $RunID in this PS script: https://learn.microsoft.com/en-us/azure/data-factory/scripts/incremental-copy-powershell?toc=%2fpowershell%2fmodule%2ftoc.json

Upvotes: 0

Joseph
Joseph

Reputation: 570

Got below code, but from where do we get the $Runid?

while ($True) {
    $run = Get-AzureRmDataFactoryV2PipelineRun -ResourceGroupName $resourceGroupName -DataFactoryName $DataFactoryName -PipelineRunId $runId

    if ($run) {
        if ($run.Status -ne 'InProgress') {
            Write-Host "Pipeline run finished. The status is: " $run.Status -foregroundcolor "Yellow"
            $run
            break
        }
        Write-Host  "Pipeline is running...status: InProgress" -foregroundcolor "Yellow"
    }

    Start-Sleep -Seconds 30
}

Upvotes: 0

Related Questions