Could not refresh state. Known issue, and known workaround is not working

Question

So, if you have been working for some time with Terraform, probably you have faced this error message more than once:

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Error refreshing state: state data in S3 does not have the expected content.

This may be caused by unusually long delays in S3 processing a previous state
update.  Please wait for a minute or two and try again. If this problem
persists, and neither S3 nor DynamoDB are experiencing an outage, you may need
to manually verify the remote state and update the Digest value stored in the
DynamoDB table to the following value: 

Usually, this is caused because something was screwed up during a destroy operation, and now there is a mismatch between the state and the lock.

The known solution for this is to delete the lock from DynamoDB and run terraform init again. And if it didn't resolve with that, also delete the tfstate from S3, which at this point doesn't have any data as the infrastructure was destroyed.

Surprisingly, neither is working now, and I don't have a clue why. There is no tfstate in the bucket, in fact, I even deleted every old version stored (bucket has versioning enabled). There is no lock in DynamoDB either.

Changing the tfstate name works without issues, but I can't change it as it would break the naming convention I'm using for all my tfstates.

So, any ideas what's going on here? On Friday, the infrastructure was deployed and destroyed without issues (as part of the destroy, I always check there is no lock left behind and delete the tfstate from S3). But today, I'm facing this error, and it's been a while already and can't figure it out..

Answer 1

Holy ****!

So, it turns out DynamoDB was being a serious troll here.

I searched for the key of the md5 of this tfstate, and it wasn't returned. But then I noticed there was a message about that there were more items to be returned... After clicking like 6 times on that button, it eventually returned a hidden lock for this tfstate.

Deleted it, and everything is back to normal again.

So, as summary, if you ever face this issue and you can't find the lock on DynamoDB... be sure that all items are returned in the query, as it can take many attempts to return them all.