CODEWITHSUNDEEP
azure-pipelinesazure-resource-managerazure-data-factory
Rob Bowman
Rob Bowman

Reputation: 8731

Data Factory Deploy Managed Private Endpoint. Error: Invalid payload

I have been using the new ADF CI/CD process as described here: ms doc. This worked well until I secured the linked services through managed private endpoints.

A build pipeline generates an ARM template and parameters file based what what is deployed to the data factory in my "Dev" environment. The template and parameters file are then published from the build and made available to the release pipeline. At this point, the generated parameters just contains placeholder values.

The release pipeline executes the ARM template, taking template values from the "Override template parameters" text box:

enter image description here

My problem is, when this runs I get the following error from the resource group deployment:

"Invalid resource request. Resource type: 'ManagedPrivateEndpoint', Resource name: 'pe-ccsurvey-blob-001' 'Error: Invalid payload'."

From the Azure Portal, I navigated to the resource group deployment, where I was able to view the template and parameters file used.

Definition of the required private endpoint from the template file is shown below:

{
            "type": "Microsoft.DataFactory/factories/managedVirtualNetworks/managedPrivateEndpoints",
            "apiVersion": "2018-06-01",
            "name": "[concat(parameters('factoryName'), '/default/pe-ccsurvey-blob-001')]",
            "dependsOn": [
                "[concat(variables('factoryId'), '/managedVirtualNetworks/default')]"
            ],
            "properties": {
                "privateLinkResourceId": "[parameters('pe-ccsurvey-blob-001_properties_privateLinkResourceId')]",
                "groupId": "[parameters('pe-ccsurvey-blob-001_properties_groupId')]"
            }
        }

The relevant parameters from the parameters file are:

"pe-ccsurvey-blob-001_properties_privateLinkResourceId": {
    "value": "/subscriptions/correctSubscriptionIdHere/resourceGroups/rg-contactcentre-dev-001/providers/Microsoft.Storage/storageAccounts/storccsurveydev001"
},
"pe-ccsurvey-blob-001_properties_groupId": {
    "value": "blob"
}

The values in the parameters file look ok to me. The storage account it's pointing to does exist. Unfortunately, the error message isn't very specific - it just states "Invalid Payload".

Any ideas?

Upvotes: 0

Views: 8283

Answers (4)

José Júnior
José Júnior

Reputation: 601

For me, the fix was first:

  • delete manually private endpoints
  • run my ADF pipeline again

Upvotes: 0

Josh Sama
Josh Sama

Reputation: 33

I found a way to fix this if you have created already the private manage endpoints on dev and prod resource groups because it will be too sad re-creating data factory.

The steps I folllowed to fix this are below:

1- Created a new folder in the repository folder ( I've added this step to the release pipeline before the RMA deploy step)

2- Added a powershell code to copy the JSON files to the new folder

3-Added a powershelll code to replace the variable names with the desired ones ( just changed the JSON files in the new folder and did not change the original ones) to match the final RG-resource

4-Changed the source of the JSON files to point to the new folder where the JSON files match the manage private endpoint of the resource where is being deployed. (This needs to be done on the azure devops deploy RMA step)

Changes are happpening on the fly inside an agent so if you try to look for the copy files in the repository, they will not exist there, they only exists at the time the release pipeline gets executed inside the agent ( hard drive D on my case)

  copy-item -path "ADD THE FILENAME PATH" -destination " ADD NEW FILENAME PATH"
    
    $fileJson = get-content "NEW FILENAME PATH"
    
    $fileJson = $fileJson.replace('string you want to replace','new string value')
    
    set-content -path "NEW FILENAME PATH" -value  $fileJson

I spent a week trying to think how to do this because I was not able to delete resources due to Locks and policies so let me know any additional question and Ill help

Upvotes: 0

Urmo Arr
Urmo Arr

Reputation: 1

Had a similar problem. I fixed the issue by deleting the currently deployed private endpoint that was conflicting. And then created the endpoint template again manually, not in the adf UI.

Upvotes: 0

KarthikBhyresh-MT
KarthikBhyresh-MT

Reputation: 5034

Going through the official Best practices for CI/CD,

If a private endpoint already exists in a factory and you try to deploy an ARM template that contains a private endpoint with the same name but with modified properties, the deployment will fail.

Upvotes: 3

Related Questions