How to provision Azure Stream Analytics Job in one go?

Question

I need to provision an Azure Stream Analytics Job connected to a storage account for its job's sytem data. This requires that its managed identity has access to this storage account. This is done with a role assignment, which can only be done when the managed identity is present.

The azure-streamanalytics-cicd cli that Microsoft provides, generates the arm template and parameter file below which apparently cannot be deployed directly.

Can this be done in one deployment? It seems to be a chicken/egg problem: the managed identity is required to perform the role assignment, but the role assignment must be in place before the full job can be deployed.

Whenever I deploy the full job without the role assignment being in place, I get the error Failed to authenticate with the job storage account.

It seems I need to split up the deployment in three steps:

1. Provision skeleton job with the storage account configured, without the actual workload. This does not yield the error, but lights up the managed identity.
2. Perform role assignment for the managed identity onto the storage account.
3. Provision job workload.

Arm template:

{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "ASAApiVersion": {
      "type": "string"
    },
    "StreamAnalyticsJobName": {
      "type": "string",
      "minLength": 3,
      "maxLength": 63,
      "metadata": {
        "description": "Stream Analytics Job Name, can contain alphanumeric characters and hypen and must be 3-63 characters long"
      }
    },
    "Location": {
      "type": "string"
    },
    "OutputStartMode": {
      "type": "string",
      "allowedValues": [
        "JobStartTime",
        "CustomTime",
        "LastOutputEventTime"
      ]
    },
    "OutputStartTime": {
      "type": "string"
    },
    "DataLocale": {
      "type": "string"
    },
    "OutputErrorPolicy": {
      "type": "string",
      "allowedValues": [
        "Drop",
        "Stop"
      ]
    },
    "EventsLateArrivalMaxDelayInSeconds": {
      "type": "int"
    },
    "EventsOutOfOrderMaxDelayInSeconds": {
      "type": "int"
    },
    "EventsOutOfOrderPolicy": {
      "type": "string",
      "allowedValues": [
        "Adjust",
        "Drop"
      ]
    },
    "StreamingUnits": {
      "type": "int",
      "minValue": 1,
      "maxValue": 396,
      "metadata": {
        "description": "Number of Streaming Units"
      },
      "allowedValues": [
        1,
        3,
        6,
        12,
        18,
        24,
        30,
        36,
        42,
        48,
        54,
        60,
        66,
        72,
        78,
        84,
        90,
        96,
        102,
        108,
        114,
        120,
        126,
        132,
        138,
        144,
        150,
        156,
        162,
        168,
        174,
        180,
        186,
        192,
        198,
        204,
        210,
        216,
        222,
        228,
        234,
        240,
        246,
        252,
        258,
        264,
        270,
        276,
        282,
        288,
        294,
        300,
        306,
        312,
        318,
        324,
        330,
        336,
        342,
        348,
        354,
        360,
        366,
        372,
        378,
        384,
        390,
        396
      ]
    },
    "CompatibilityLevel": {
      "type": "string",
      "allowedValues": [
        "1.0",
        "1.1",
        "1.2"
      ]
    },
    "ContentStoragePolicy": {
      "type": "string",
      "allowedValues": [
        "SystemAccount",
        "JobStorageAccount"
      ]
    },
    "JobStorageAccountName": {
      "type": "string"
    },
    "JobStorageAuthMode": {
      "type": "string",
      "allowedValues": [
        "ConnectionString",
        "Msi"
      ]
    },
    "CustomCodeStorageAccountName": {
      "type": "string"
    },
    "CustomCodeStorageAccountKey": {
      "type": "string"
    },
    "CustomCodeContainer": {
      "type": "string"
    },
    "CustomCodePath": {
      "type": "string"
    },
    "Input_InputIoTHub_iotHubNamespace": {
      "type": "string"
    },
    "Input_InputIoTHub_consumerGroupName": {
      "type": "string"
    },
    "Input_InputIoTHub_endpoint": {
      "type": "string"
    },
    "Input_InputIoTHub_sharedAccessPolicyName": {
      "type": "string"
    },
    "Input_InputIoTHub_sharedAccessPolicyKey": {
      "type": "string"
    },
    "Output_outputmsgunfilteredcosmos_accountId": {
      "type": "string"
    },
    "Output_outputmsgunfilteredcosmos_accountKey": {
      "type": "string"
    },
    "Output_outputmsgunfilteredcosmos_database": {
      "type": "string"
    },
    "Output_outputmsgunfilteredcosmos_collectionNamePattern": {
      "type": "string"
    },
    "Output_outputmsgunfilteredcosmos_documentId": {
      "type": "string"
    }
  },
  "resources": [
    {
      "type": "Microsoft.StreamAnalytics/StreamingJobs",
      "apiVersion": "[parameters('ASAApiVersion')]",
      "name": "[parameters('StreamAnalyticsJobName')]",
      "location": "[parameters('Location')]",
      "identity": {
        "type": "SystemAssigned"
      },
      "properties": {
        "outputStartMode": "[parameters('OutputStartMode')]",
        "outputStartTime": "[if(equals(parameters('OutputStartMode'),'CustomTime'), parameters('OutputStartTime'), json('null'))]",
        "sku": {
          "name": "standard"
        },
        "jobType": "Cloud",
        "eventsOutOfOrderPolicy": "[parameters('EventsOutOfOrderPolicy')]",
        "outputErrorPolicy": "[parameters('OutputErrorPolicy')]",
        "eventsOutOfOrderMaxDelayInSeconds": "[parameters('EventsOutOfOrderMaxDelayInSeconds')]",
        "eventsLateArrivalMaxDelayInSeconds": "[parameters('EventsLateArrivalMaxDelayInSeconds')]",
        "dataLocale": "[parameters('DataLocale')]",
        "compatibilityLevel": "[parameters('CompatibilityLevel')]",
        "jobStorageAccount": {
          "accountName": "[parameters('JobStorageAccountName')]",
          "authenticationMode": "[parameters('JobStorageAuthMode')]"
        },
        "contentStoragePolicy": "[parameters('ContentStoragePolicy')]",
        "externals": {
          "storageAccount": {
            "accountName": "[parameters('CustomCodeStorageAccountName')]",
            "accountKey": "[parameters('CustomCodeStorageAccountKey')]"
          },
          "container": "[parameters('CustomCodeContainer')]",
          "path": "[parameters('CustomCodePath')]"
        },
        "transformation": {
          "name": "Transformation",
          "properties": {
            "streamingUnits": "[parameters('StreamingUnits')]",
            "query": "SELECT\r\n    GetMetadataPropertyValue(InputIoTHub, '[EventId]') AS Id,\r\n    GetMetadataPropertyValue(InputIoTHub, '[IotHub].[ConnectionDeviceId]') AS deviceId,\r\n    GetMetadataPropertyValue(InputIoTHub, '[IoTHub].[EnqueuedTime]') AS timeStamp,\r\n    InputIoTHub.*\r\nINTO\r\n    outputmsgunfilteredcosmos\r\nFROM\r\n    InputIoTHub\r\n"
          }
        },
        "inputs": [
          {
            "name": "InputIoTHub",
            "properties": {
              "type": "Stream",
              "datasource": {
                "type": "Microsoft.Devices/IotHubs",
                "properties": {
                  "iotHubNamespace": "[parameters('Input_InputIoTHub_iotHubNamespace')]",
                  "consumerGroupName": "[parameters('Input_InputIoTHub_consumerGroupName')]",
                  "endpoint": "[parameters('Input_InputIoTHub_endpoint')]",
                  "sharedAccessPolicyName": "[parameters('Input_InputIoTHub_sharedAccessPolicyName')]",
                  "sharedAccessPolicyKey": "[parameters('Input_InputIoTHub_sharedAccessPolicyKey')]"
                }
              },
              "compression": {
                "type": "None"
              },
              "serialization": {
                "type": "Json",
                "properties": {
                  "encoding": "UTF8"
                }
              }
            }
          }
        ],
        "outputs": [
          {
            "name": "outputmsgunfilteredcosmos",
            "properties": {
              "datasource": {
                "type": "Microsoft.Storage/DocumentDB",
                "properties": {
                  "accountId": "[parameters('Output_outputmsgunfilteredcosmos_accountId')]",
                  "accountKey": "[parameters('Output_outputmsgunfilteredcosmos_accountKey')]",
                  "database": "[parameters('Output_outputmsgunfilteredcosmos_database')]",
                  "collectionNamePattern": "[parameters('Output_outputmsgunfilteredcosmos_collectionNamePattern')]",
                  "partitionKey": null,
                  "documentId": "[parameters('Output_outputmsgunfilteredcosmos_documentId')]"
                }
              }
            }
          }
        ]
      }
    }
  ]
}

Arm template parameters:

{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "ASAApiVersion": {
      "value": "2017-04-01-preview"
    },
    "StreamAnalyticsJobName": {
      "value": "myasajob"
    },
    "Location": {
      "value": "Central US"
    },
    "OutputStartMode": {
      "value": "JobStartTime"
    },
    "OutputStartTime": {
      "value": "2019-01-01T00:00:00Z"
    },
    "DataLocale": {
      "value": "nl-NL"
    },
    "OutputErrorPolicy": {
      "value": "Stop"
    },
    "EventsLateArrivalMaxDelayInSeconds": {
      "value": 5
    },
    "EventsOutOfOrderMaxDelayInSeconds": {
      "value": 0
    },
    "EventsOutOfOrderPolicy": {
      "value": "Adjust"
    },
    "StreamingUnits": {
      "value": 1
    },
    "CompatibilityLevel": {
      "value": "1.2"
    },
    "ContentStoragePolicy": {
      "value": "JobStorageAccount"
    },
    "JobStorageAccountName": {
      "value": "mystorageaccount"
    },
    "JobStorageAuthMode": {
      "value": "Msi"
    },
    "CustomCodeStorageAccountName": {
      "value": "mystorageaccount"
    },
    "CustomCodeStorageAccountKey": {
      "value": null
    },
    "CustomCodeContainer": {
      "value": "43803218-0998-487b-9d49-4eb00ef41ca5"
    },
    "CustomCodePath": {
      "value": "UserCustomCode.zip"
    },
    "Input_InputIoTHub_iotHubNamespace": {
      "value": "myiothub"
    },
    "Input_InputIoTHub_consumerGroupName": {
      "value": "$Default"
    },
    "Input_InputIoTHub_endpoint": {
      "value": "messages/events"
    },
    "Input_InputIoTHub_sharedAccessPolicyName": {
      "value": "DPSRegistry"
    },
    "Input_InputIoTHub_sharedAccessPolicyKey": {
      "value": null
    },
    "Output_outputmsgunfilteredcosmos_accountId": {
      "value": "mycosmos"
    },
    "Output_outputmsgunfilteredcosmos_accountKey": {
      "value": null
    },
    "Output_outputmsgunfilteredcosmos_database": {
      "value": "mycosmosdb"
    },
    "Output_outputmsgunfilteredcosmos_collectionNamePattern": {
      "value": "unfiltered"
    },
    "Output_outputmsgunfilteredcosmos_documentId": {
      "value": ""
    }
  }
}

Answer 1

This is a chicken and egg problem. Currently you can't provision the job with that configuration in one go when using system-assigned MSI. Since that identity will not have been granted access to the storage account - it didn't exist before for you to do it.

For system-assigned MSI, you are right about your deployment steps, you need to:

• Create the job without any sub-resources (inputs, outputs, transformations...)
• Grant access to the storage account
• Create the sub-resources with the proper configuration

Alternatively, you could create the job with connection strings, then try to switch to MSI afterwards to work around the chicken/egg problem, but that's still more than 1 step, and require additional credentials, so not really solving much.

But if you use a user assigned MSI, it should just work as you would be able to grant access to the storage account before the job is created.