Terraform remote exec task does not connect to remote server - last error: dial tcp i/o timeout

Question

I am running terraform on Azure DevOps, Azure Release pipelines. The end goal is to spin up a VM, install and run Ansible so that we can run playbooks to configure Windows server just provisioned.

However,I'm getting error as:

2019-04-30T14:27:13.9975581Z [0m[0mazurerm_virtual_machine.tf-vm-erx-builds (remote-exec): Connecting to remote host via SSH...
2019-04-30T14:27:13.9976968Z azurerm_virtual_machine.tf-vm-erx-builds (remote-exec):   Host: 10.112.4.11
2019-04-30T14:27:13.9977790Z azurerm_virtual_machine.tf-vm-erx-builds (remote-exec):   User: scmadmin
2019-04-30T14:27:13.9978268Z azurerm_virtual_machine.tf-vm-erx-builds (remote-exec):   Password: false
2019-04-30T14:27:13.9979371Z azurerm_virtual_machine.tf-vm-erx-builds (remote-exec):   Private key: true
2019-04-30T14:27:13.9979804Z azurerm_virtual_machine.tf-vm-erx-builds (remote-exec):   SSH Agent: false
2019-04-30T14:27:13.9980255Z azurerm_virtual_machine.tf-vm-erx-builds (remote-exec):   Checking Host Key: false
2019-04-30T14:27:21.1740015Z [0m[1mazurerm_virtual_machine.tf-vm-erx-builds: Still creating... (7m20s elapsed)[0m[0m
2019-04-30T14:27:31.1743030Z [0m[1mazurerm_virtual_machine.tf-vm-erx-builds: Still creating... (7m30s elapsed)[0m[0m
2019-04-30T14:27:32.9964209Z [31m
2019-04-30T14:27:32.9965326Z [1m[31mError: [0m[0m[1mError applying plan:
2019-04-30T14:27:32.9965547Z 
2019-04-30T14:27:32.9965890Z 1 error(s) occurred:
2019-04-30T14:27:32.9966284Z 
2019-04-30T14:27:32.9966897Z * azurerm_virtual_machine.tf-vm-erx-builds: timeout - last error: dial tcp 10.112.4.11:22: i/o timeout

However, if I run same terraform code from the Virtual machine from where I generated public and private key, it works like a charm. Please see the output below.

azurerm_virtual_machine.tf-vm-erx-builds: Provisioning with 'remote-exec'...
azurerm_virtual_machine.tf-vm-erx-builds (remote-exec): Connecting to remote host via SSH...
azurerm_virtual_machine.tf-vm-erx-builds (remote-exec):   Host: 10.112.4.11
azurerm_virtual_machine.tf-vm-erx-builds (remote-exec):   User: scmadmin
azurerm_virtual_machine.tf-vm-erx-builds (remote-exec):   Password: false
azurerm_virtual_machine.tf-vm-erx-builds (remote-exec):   Private key: true
azurerm_virtual_machine.tf-vm-erx-builds (remote-exec):   SSH Agent: true
azurerm_virtual_machine.tf-vm-erx-builds (remote-exec):   Checking Host Key: false
azurerm_virtual_machine.tf-vm-erx-builds (remote-exec): Connected!
azurerm_storage_account.tf-sa-erx-builds: Still creating... (2m40s elapsed)
azurerm_virtual_machine.tf-vm-erx-builds (remote-exec): Loaded plugins: fastestmirror, langpacks
azurerm_virtual_machine.tf-vm-erx-builds (remote-exec): epel-release-lat |  15 kB     00:00
azurerm_virtual_machine.tf-vm-erx-builds (remote-exec): Examining /var/tmp/yum-root-j6a5hG/epel-release-latest-7.noarch.rpm: epel-release-7-11.noarch
azurerm_virtual_machine.tf-vm-erx-builds (remote-exec): Marking /var/tmp/yum-root-j6a5hG/epel-release-latest-7.noarch.rpm to be installed
azurerm_virtual_machine.tf-vm-erx-builds (remote-exec): Resolving Dependencies
azurerm_virtual_machine.tf-vm-erx-builds (remote-exec): --> Running transaction check
azurerm_virtual_machine.tf-vm-erx-builds (remote-exec): ---> Package epel-release.noarch 0:7-11 will be installed

os_profile_linux_config {
  disable_password_authentication = true
  ssh_keys {
      path     = "/home/scmadmin/.ssh/authorized_keys"
      key_data = cat id_rsa.pub
  }
    }
    provisioner "remote-exec" {
    inline = [
      "sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -y",
      "sudo yum update -y",
      "sudo yum install git -y",
      "git clone https://'${var.git_username}':'${var.git_pat}'@'${var.git_url}'",
      "cd erxpreprd",
      "sudo yum install ansible -y",
      "sudo yum -y install python-pip",
      "sudo pip install pywinrm",
      "ansible all -vvv -i inventory/hosts -m win_ping",
      "ansible-playbook -vvv -i inventory/hosts playbooks/common.yml",
      "ansible-playbook -vvv -i inventory/hosts playbooks/apply-failoverclustering.yml"
    ]
     connection {
    type         = "ssh"
    user         = "${var.username}"
    private_key  = "${file("${var.private_key}")}" # private key id_rsa file
  }
  }
}

Not sure what the issue is and how to have it fixed.

One thing I can see is in Azure DevOps run SSH Agent: false where as on my local VM run SSH: Agent: true

I added code: agent="true" to the connection block but it gave me some other errors. P.S: NSG is as below:

resource "azurerm_network_security_group" "tf-nsg-erx-buildserver" {
  name                = "${var.buildserver_nsg}"
  location            = "${data.azurerm_resource_group.tf-rg-erx-external.location}"
  resource_group_name = "${data.azurerm_resource_group.tf-rg-erx-external.name}"
}
resource "azurerm_network_security_rule" "tf-nsr-erx-buildserver-22" {
  name                        = "Open Port 22"
  priority                    = 106
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "22"
  source_address_prefix       = "*"
  destination_address_prefix  = "*"
  resource_group_name         = "${data.azurerm_resource_group.tf-rg-erx-external.name}"
  network_security_group_name = "${azurerm_network_security_group.tf-nsg-erx-buildserver.name}"
}

Now, using the NSG to Network Interface as below:

resource "azurerm_network_interface" "tf-ni-erx-builds" {
 name                      = "${var.builds_base_hostname}${format("%02d",count.index+1)}-nic01"
 location                  = "${data.azurerm_resource_group.tf-rg-erx-external.location}"
 resource_group_name       = "${data.azurerm_resource_group.tf-rg-erx-external.name}"
 network_security_group_id = "${azurerm_network_security_group.tf-nsg-erx-buildserver.id}"

ip_configuration {
    name                          = "${var.builds_base_hostname}${format("%02d",count.index+1)}-iip01"
    subnet_id                     = "${azurerm_subnet.tf-sn-erx-builds.id}"
    private_ip_address_allocation = "static"
    private_ip_address            ="10.112.4.${count.index+11}"
}
}

And finally referencing, Network Interface in VM creation block as below:

resource "azurerm_virtual_machine" "tf-vm-erx-builds" {
  name                  = "${var.builds_base_hostname}${format("%02d",count.index+1)}"
  location              = "${data.azurerm_resource_group.tf-rg-erx-external.location}"
  resource_group_name   = "${data.azurerm_resource_group.tf-rg-erx-external.name}"
  network_interface_ids = ["${element(azurerm_network_interface.tf-ni-erx-builds.*.id, count.index)}"]
  vm_size               = "${var.builds_vm_size}"

Would like to have all executed from Azure DevOs

Answer 1

I also got the same error. In my case, it had to do with the Security Group. Essentially, I was working out of a new place and forgot to add my CIDR block to the Security Groups. Therefore, terraform was not able to reach the machine via SSH due to Security Group restrictions.

I modified my CIDR block list variable to include the IP of my current location and had no issues.

* The initial solution was founded in this issue on GitHub.

Answer 2

I suspect this,

key_data = cat id_rsa.pub

can you try this with something like,

ssh_keys {
  path     = "/home/myadmin/.ssh/authorized_keys"
  key_data = "${file("~/.ssh/demo_key.pub")}"
}