CODEWITHSUNDEEP
jsonpowershellazure-data-factory
DavidP
DavidP

Reputation: 623

How can you get valid JSON for Azure Data Factory V2 Datasets/Pipelines using PowerShell?

I have some pipelines in Azure Data Factory V2 that I need to update. Specifically I need to update the schema of the datasets. I initially created them using the user interface. Since the number of columns is very large, the user interface has become very slow for working on them, so I'd like to switch to using PowerShell.

I put together a script that I am able to use to update them from a JSON file using the Set-AzureRmDataFactoryV2Dataset and Set-AzureRmDataFactoryV2Pipeline cmdlets. I'd like to also be able to generate JSON files with the current configurations using PowerShell. I've tried using the matching Get-AzureRmDataFactoryV2Dataset and Get-AzureRmDataFactoryV2Pipeline cmdlets which have been useful for getting general information about them, but I haven't been able to go from the PSDataSet object returned to a valid JSON file. Below are the commands that I've been trying, but it ends up giving me a bunch of empty JSON arrays for the schema.

$dtSrc = Get-AzureRmDataFactoryV2Dataset -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -Name $dtSrcName
ConvertTo-Json $dtSrc.Properties -Depth 100 | Out-File "$dtSrcName.json" -Width 1000000

Edit: Based on the answer by Wang Zhang, I've edited my ConvertTo-Json statement as shown below. The output looks much better now (no empty arrays), but it still doesn't match the Azure Dataset JSON schema.

$dtSrc = Get-AzureRmDataFactoryV2Dataset -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -Name $dtSrcName
ConvertTo-Json $dtSrc -Depth 1 | Out-File "$dtSrcName.json" -Width 1000000

Upvotes: 0

Views: 1616

Answers (2)

fgbaezp
fgbaezp

Reputation: 476

You could just download the ARM templates for your ADF (they use the json format) edit them and upload them again, either by using the UI, az-cli or even powershell.

Or even better, link a git repository to the ADF and then just clone the repo locally, make your changes and publish them to ADF by doing a push to the specified production branch (normally master).

Upvotes: 1

Wang Zhang
Wang Zhang

Reputation: 327

would you please tell the reason that you set "-Depth -100" in the second Command? Seems remove it and modify the second command as the following works well:

ConvertTo-Json $dtSrc.Properties | Out-File "$dtSrcName.json" -Width 1000000

Please have a try.

Upvotes: 1

Related Questions