How can I use 2 providers in the same terraform config?

Question

My main.tf file starts with

terraform {
  required_version = ">= 0.13.7"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "= 2.32.0"
    }
    foobar = {
      source = "terraform.foo.com/foo/bar"
    }
  }
}

The catch here is that foo/bar is the module I'm developing locally so I also has this terraformrc file:

provider_installation {
  dev_overrides {
    "terraform.foo.com/foo/bar" = "/Users/appuser/foobar/bin/darwin-amd64"
  }
}

Here's the errors I run into when running ✗ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "2.32.0"...
- Finding latest version of terraform.foo.com/foo/bar...

Warning: Provider development overrides are in effect

The following provider development overrides are set in the CLI configuration:
 - "terraform.foo.com/foo/bar" = "/Users/appuser/foobar/bin/darwin-amd64"

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
no available releases match the given constraints 2.32.0


Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
"terraform.foo.com/foo/bar": no available releases
match the given constraints 

Update: when I remove terraformrc it does seem to work but I am not able to load the 2nd provider this way (since it relies on override):

terraform init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "2.32.0"...
- Finding latest version of hashicorp/foo/bar...
- Installing hashicorp/aws v2.32.0...
- Installed hashicorp/aws v2.32.0 (self-signed, key ID 34365D9472D7468F)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/plugins/signing.html

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
hashicorp/foo/bar: provider registry registry.terraform.io does not
have a provider named registry.terraform.io/hashicorp/foo/bar

Answer 1

The dev_overrides setting is, in spite of being placed inside the provider_installation block due to its thematic similarity, actually a runtime setting rather than an installation setting. Specifically, it asks Terraform to ignore whatever version of the provider terraform init previously selected and to use the given overridden provider instead.

Unfortunately, this model only works well if you're overriding a provider that already has at least one published version available, so that terraform init can select and install that version but then terraform apply (for example) can then ignore what terraform init installed, and use the override instead. (Terraform behaves this way because part of terraform init's responsibility is to update the Dependency Lock File, and so it needs to find at least one selectable version of each provider in order to produce a complete lock file.)

One way you could avoid this design quirk is to place a fake "release" of this provider in a local directory that you'd then configure as a filesystem mirror for that provider, in your .terraformrc file.

For example, if you create a directory /tmp/tf-workaround/terraform.foo.com/foo/bar/0.0.1/darwin_amd64 and place into it an empty file named terraform-provider-bar then that should be sufficient for terraform init to detect this as an available "published" version of the provider if given an CLI configuration like this:

provider_installation {
  dev_overrides {
    "terraform.foo.com/foo/bar" = "/Users/appuser/foobar/bin/darwin-amd64"
  }

  filesystem_mirror {
    path    = "/tmp/tf-workaround"
    include = ["terraform.foo.com/foo/bar"]
  }

  direct {
    exclude = ["terraform.foo.com/foo/bar"]
  }
}

terraform init should then find the placeholder empty file and "install" it into .terraform/providers as normal. That empty file won't actually work as a valid plugin, but that's okay because terraform apply will ignore it anyway and will use the directory given in dev_overrides instead.

However, the dependency lock file will contain an incorrect entry for terraform.foo.com/foo/bar after installation, so if you intend to commit this test configuration to version control then you may wish to manually remove that block to reduce confusion once there really is a release of this provider.

The less-complex way to work here is to test a provider during development with a configuration that only includes that provider, and wait until you've actually published the provider somewhere before you use it "for real" as part of a larger system. In that case, you'd typically skip running terraform init altogether because the only external dependency would be the overridden provider, and so nothing additional would need to be installed.

Answer 2

The fix (which was to add direct {}) was found in TF docs:

provider_installation {

  # Use /home/developer/tmp/terraform-null as an overridden package directory
  # for the hashicorp/null provider. This disables the version and checksum
  # verifications for this provider and forces Terraform to look for the
  # null provider plugin in the given directory.
  dev_overrides {
    "hashicorp/null" = "/home/developer/tmp/terraform-null"
  }

  # For all other providers, install them directly from their origin provider
  # registries as normal. If you omit this, Terraform will _only_ use
  # the dev_overrides block, and so no other providers will be available.
  direct {}
}

Actually I'm still getting

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
hashicorp/foo/bar: could not connect to hashicorp: Failed
to request discovery document: Get
"https://hashicorp/.well-known/terraform.json": dial tcp: lookup hashicorp on
100.217.9.1:53: no such host