CODEWITHSUNDEEP
azureazure-web-app-serviceazure-resource-managerterraform-provider-azurease
Duffy Gillman
Duffy Gillman

Reputation: 11

Azure Isolated App Service Environment fails to create with Terraform with "An error has occurred."

I am attempting to create an Isolated App Service Environment (ASE) in Azure using Terraform. I have succeeded once and have an ASE running. Attempts to create a second ASE in the same subscription, but within a separate resource group, fail. The only error message available is "An error has occurred". Investigation within Azure Monitoring reveals that the create request which was delivered via an ARM template has encountered a 500 error (Internal Server Error) somewhere in the process of creating the ASE. However, no details are available in the log message to indicate where/when/how Azure encountered the 500 error.

Environment:

Details

I am creating multiple environments for my project: dev, test, and staging. They are each in separate resource groups within the same Azure subscription. All resources in my terraform are constructed with names unique to the environment/resource group. The intended lifecycle is for infrastructure changes to be deployed to dev, then test, then staging (and eventually a prod environment in a separate subscription). The initial configuration and deployment to dev has succeeded. Attempts to deploy to test, or to deploy a different ASE to dev, fail abjectly with very little feedback.

The original dev ASE is a v1 ASE. I have attempted to create a second ASE in test using the same terraform code. I have also tried creating a v3 ASE in dev (because the v3 will be cheaper). If the v3 ASE deploys successfully I will cut over to it in dev and will use it as the basis for test and stage instead of the v1 ASE. Regardless whether I try to deploy a v1 ASE to a separate resource group, or whether I try to deploy a v3 ASE to the same resource group as the v1 ASE, I get the same error.

This is the Terraform for the v1 ASE, including the subnet which will host it:

resource "azurerm_subnet" "subnet" {
  name                                           = "${local.prefix}-subnet"
  resource_group_name                            = var.resource_group_name
  virtual_network_name                           = var.vnet_name
  address_prefixes                               = var.cidrs
  enforce_private_link_endpoint_network_policies = var.enforce_private_link_endpoint_network_policies
  enforce_private_link_service_network_policies  = var.enforce_private_link_service_network_policies

  dynamic "delegation" {
    for_each = var.delegations

    content {
      name = "${local.prefix}-delegation-${delegation.key}"
      service_delegation {
        name    = delegation.value.name
        actions = delegation.value.actions
      }
    }
  }

  // List of Service endpoints to associate with the subnet.
  service_endpoints = var.service_endpoints
}

resource "azurerm_network_security_group" "nsg" {
  name                = "${local.prefix}-nsg"
  location            = var.resource_group_location
  resource_group_name = var.resource_group_name

  tags = merge(map("Name", "${local.prefix}-nsg"), local.tags)
}

resource "azurerm_subnet_network_security_group_association" "nsg_assoc" {
  subnet_id                 = azurerm_subnet.subnet.id
  network_security_group_id = azurerm_network_security_group.nsg.id
}

resource "azurerm_network_security_rule" "ase_mgmt" {
  name                        = "${local.prefix}-ase-mgmt"
  priority                    = 100 
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "*"
  source_port_range           = "*"
  source_address_prefix       = "AppServiceManagement"
  destination_port_range      = "454-455"
  destination_address_prefix  = var.subnet_cidr
  resource_group_name         = var.resource_group_name
  network_security_group_name = azurerm_network_security_group.nsg.name
}

resource "azurerm_network_security_rule" "ingress" {
  for_each                    = {
    for idx, cidr in var.ingress_cidrs : idx => cidr
  }

  name                        = "${local.prefix}-ingress-${each.key}"
  priority                    = 200 + each.key
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "*"
  source_port_range           = "*"
  source_address_prefix       = each.value
  destination_port_range      = "*"
  destination_address_prefix  = var.subnet_cidr
  resource_group_name         = var.resource_group_name
  network_security_group_name = azurerm_network_security_group.nsg.name
}

resource "azurerm_app_service_environment" "env" {
  name                         = "${local.prefix}-ase"
  subnet_id                    = azurerm_subnet.subnet.id
  pricing_tier                 = var.pricing_tier
  front_end_scale_factor       = var.front_scale_factor
  internal_load_balancing_mode = "Web, Publishing"
  allowed_user_ip_cidrs        = var.allowed_user_ip_cidrs

  cluster_setting {
    name  = "DisableTls1.0"
    value = "1"
  }

  depends_on = [
    azurerm_network_security_rule.ase_mgmt
  ]
}

The v3 ASE is configured identically, except for azurerm_app_service_environment.env, which is replaced with:

resource "azurerm_app_service_environment_v3" "env" {
  name                         = "${local.prefix}-ase-v3"
  resource_group_name          = var.resource_group_name
  subnet_id                    = azurerm_subnet.subnet.id

  cluster_setting {
    name  = "DisableTls1.0"
    value = "1"
  }

  depends_on = [
    azurerm_network_security_rule.ase_mgmt
  ]
}

Results

  1. Terraform generates this ARM request (identifiers have been redacted):
2021/07/19 09:07:44 [TRACE] dag/walk: vertex "root" is waiting for "meta.count-boundary (EachMode fixup)"
2021-07-19T09:07:45.121-0700 [DEBUG] plugin.terraform-provider-azurerm_v2.67.0_x5: AzureRM Request: 
PUT /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxx-dev-rg/providers/Microsoft.Web/hostingEnvironments/xxxxxxxx-dev-ase-v3?api-version=2020-06-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.16.3 (amd64-darwin) go-autorest/v14.2.1 Azure-SDK-For-Go/v55.4.0 web/2020-06-01 HashiCorp Terraform/0.14.11 (+https://www.terraform.io) Terraform Plugin SDK/2.7.0 terraform-provider-azurerm/2.67.0 pid-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Content-Length: 382
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Accept-Encoding: gzip

{
  "kind":"ASEV3",
  "location":"centralus",
  "properties":
  {
    "clusterSettings":[{
      "name":"DisableTls1.0",
      "value":"1"
    }],
    "name":"xxxxxxxx-dev-ase-v3",
    "virtualNetwork":{
      "id":"/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxx-dev-rg/providers/Microsoft.Network/virtualNetworks/xxxxxxxx-dev-vnet/subnets/xxxxxxxx-dev-ase-v3-ase-subnet",
      "subnet":"xxxxxxxx-dev-ase-v3-ase-subnet"
     }
  },
  "tags":{}
}
  1. The error eventually reported by Terraform looks like this in the debug output:
2021/07/19 09:13:53 [DEBUG] azurerm_app_service_environment_v3.env: apply errored, but we're indicating that via the Error pointer rather than returning it: creating App Service Environment: (Hosting Environment Name "xxxxxxxx-dev-ase-v3" / Resource Group "xxxxxxxx-dev-rg"): web.AppServiceEnvironmentsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="Failed" Message="The async operation failed." AdditionalInfo=[{"Message":"An error has occurred."}]: creating App Service Environment: (Hosting Environment Name "xxxxxxxx-dev-ase-v3" / Resource Group "xxxxxxxx-dev-rg"): web.AppServiceEnvironmentsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="Failed" Message="The async operation failed." AdditionalInfo=[{"Message":"An error has occurred."}]
  1. Reviewing the logs within Azure Monitor, I find a similarly vague error message. The message is summarized as InternalServerError. The JSON detail is included here for reference:
{
    "authorization": {
        "action": "Microsoft.Web/hostingEnvironments/write",
        "scope": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/resourceGroups/xxxxxxxx-dev-rg/providers/Microsoft.Web/hostingEnvironments/xxxxxxxx-dev-ase-v3"
    },
    "caller": "[email protected]",
    "channels": "Operation",
    "claims": {
      //REDACTED
    },
    "correlationId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx",
    "description": "",
    "eventDataId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx",
    "eventName": {
        "value": "EndRequest",
        "localizedValue": "End request"
    },
    "category": {
        "value": "Administrative",
        "localizedValue": "Administrative"
    },
    "eventTimestamp": "2021-07-19T15:51:45.4835627Z",
    "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/resourceGroups/xxxxxxxx-dev-rg/providers/Microsoft.Web/hostingEnvironments/xxxxxxxx-dev-ase-v3/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/ticks/637623067054835627",
    "level": "Error",
    "operationId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx",
    "operationName": {
        "value": "Microsoft.Web/hostingEnvironments/write",
        "localizedValue": "Create or Update App Service Environment"
    },
    "resourceGroupName": "xxxxxxxx-dev-rg",
    "resourceProviderName": {
        "value": "Microsoft.Web",
        "localizedValue": "Azure Web Sites"
    },
    "resourceType": {
        "value": "Microsoft.Web/hostingEnvironments",
        "localizedValue": "Microsoft.Web/hostingEnvironments"
    },
    "resourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/resourceGroups/xxxxxxxx-dev-rg/providers/Microsoft.Web/hostingEnvironments/xxxxxxxx-dev-ase-v3",
    "status": {
        "value": "Failed",
        "localizedValue": "Failed"
    },
    "subStatus": {
        "value": "InternalServerError",
        "localizedValue": "Internal Server Error (HTTP Status Code: 500)"
    },
    "submissionTimestamp": "2021-07-19T15:52:29.177138Z",
    "subscriptionId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx",
    "tenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx",
    "properties": {
        "statusCode": "InternalServerError",
        "serviceRequestId": null,
        "statusMessage": "{\"Message\":\"An error has occurred.\"}",
        "eventCategory": "Administrative",
        "entity": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/resourceGroups/xxxxxxxx-dev-rg/providers/Microsoft.Web/hostingEnvironments/xxxxxxxx-dev-ase-v3",
        "message": "Microsoft.Web/hostingEnvironments/write",
        "hierarchy": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx"
    },
    "relatedEvents": []
}

Upvotes: 1

Views: 1547

Answers (1)

Ricky Gummadi
Ricky Gummadi

Reputation: 5250

This to me looks like you have hit some subscription limitations. If you try create the same ASE via the Azure Portal does it provision for you? It will be good to know if you get an error trying to do the same thing via UI, if you did hit the same error in the GUI it will give you a better error message.

Upvotes: 0

Related Questions