Is there a way to know what calls Terraform AWS provider makes under the hood?

Question

Forgive the bad wording of the question title but effectively we are facing a situation when doing a plan in terraform where at the very end we get hit with a vague single-line access denied exception.

...
module.vpn_connection.aws_vpn_connection_route.default[0]: Refreshing state... [id=***]
2021/07/06 17:11:08 [INFO] backend/local: plan operation completed

Error: error describing AWS Organizations Account (***): AccessDeniedException: You don't have permissions to access this resource.[0m

make: *** [plan] Error 1

Now I'm not a terraform expert by any means (I'm a developer) but I have double checked multiple times and there is no where in our tf code that we attempt to make a call to explicitly describe accounts.

I have tried reinitialising and running the plan using TF_LOG_CORE=trace TF_LOG_PROVIDER=trace TF_LOG=trace and while everything else is getting logged in more detail, this line still continues to appear on its own without any further useful info.

We initially found out about this happening when our security folks made some changes to our AWS access including introduction of AWS SSO profiles (most probably irrelevant) as well as moving the state bucket from a legacy account to a new Ops account, so my best guess atm is the provider changes have caused this. Tho to be fair to them, the process of changing config for remote backend was done gracefully using terraform as well.

Upon googling, I've come across very little results except someone posted something similar here.

To make the matters more confusing, the API calls done before and after this point (logged in trace and for state locks I believe, like for instance POSTs to dynamodb/GetItem or Action=DescribeVpnConnections) all get 200 response codes back so nothing immediately appears to be leading to this error.

More on versions and system config:

MacOS Big Sur 11.2.3 (and Atlantis) Terraform v0.14.8. provider registry.terraform.io/hashicorp/aws v3.46.0 As state above, we are using remote S3 backend.

When compared with a 'healthy' account for example, I can see logs there around this point are as follows:

2021/06/24 19:33:13 [INFO] backend/local: plan operation completed
2021/06/24 19:33:13 [INFO] backend/local: writing plan output to: plan.out

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.[0m

I was wondering whether we could pinpoint which TF operation/AWS call is causing this and whether it's a call that terraform provider/modules make. Given how little info is given, I'm more after tips on what to do next in terms of troubleshooting rather than a final answer.

Completely out of ideas, any clues? Please let me know if you needed any code snippets or more piece of logs and I can provide, but really that's pretty much all I see too!

Cheers,

Answer 1

I realize this is a very late reply :-) but I would like to share a setup I found useful using TF_LOG_PROVIDER to track AWS API Calls made by the AWS Provider. I use a very simple filter based on the fact that most API calls appear in the log as "Action=..."

export TF_LOG_PROVIDER=DEBUG
export TF_LOG_PATH=/tmp/terraform.log
# Run in another terminal window the terraform command e.g. terraform apply -auto-approve
tail -f /tmp/terraform.log | grep "Action="

Below a screenshot of the calls when running terraform to destroy an EC2 instance and associated security group. Note the "polling" by the Terraform AWS provider after the TerminateInstances API call.

You can see the API calls "live" as Terraform does its thing :-)

For Azure, so far I am testing the following "filter" to attempt to see the API Calls:

grep -e '^DELETE\|^GET\|^PUT /subscriptions/' -e 'HTTP/2.0 201 Created'