CODEWITHSUNDEEP
amazon-web-servicesamazon-s3terraform
Mukund Jalan
Mukund Jalan

Reputation: 1339

How can I move Terraform state in an old S3 bucket to a new S3 bucket?

We have a few terraform configurations for which we use s3 as the backend. We have multiple AWS accounts, one for each of our environments.

In all the environments and across multiple region, we have different s3 bucket & dynamodb_table names used which as of now do not follow a valid convention and make it difficult to identify the purpose of buckets from its name. We now want to follow a convention based naming for all the s3 terraform state buckets. For this I will need to migrate the state of existing terraform resources to the new s3 buckets.

I am not sure what would be the best way to achieve this without having to run destroy on old state bucket and apply on the new one.

A few options that I could think of were as below but I am not sure if they are the right thing to do.

  1. Rename the s3 bucket: I understand this is not possible (or is it?)
  2. Move the s3 bucket: A solution to renaming is as mentioned here. Not sure if this approach will disturb the setup we have.
  3. State migration: The terraform source code is Git controlled but .terraform is not. So, I can make changes to the source code with new bucket, commit it and create a new git tag from this commit. Now when actually doing the migration, I would checkout old git tag being used in my environment, run terraform init, checkout the new git tag and run terraform init again which would ideally ask me if I want to do a migration and do the needful. This process is something similar to what terraform suggests here, but what I am not sure is, will this approach work in the kind of movement that I am expecting to do.

P.S.: I assume renaming the DynamoDB table or just using a new one instead of old would work out of the box as I would be making sure that when doing the state migration, I do not have any live terraform runs in progress.

Upvotes: 12

Views: 16522

Answers (5)

esch
esch

Reputation: 93

for me with Terraform 1.6 trying to move between GCS buckets just by changing bucket attribute of backend block didn't work; Terraform didn't offer to copy, and acted like a new backend. Solution was to:

  1. comment out the backend block, so by default will be local
  2. run terraform init - it'll offer to copy from gcs to local, say yes
  3. uncomment the backend block and change the bucket/path to point to the new one.
  4. run terraform init - it'll offer to copy from local to gcs, say yes

so essentially doesn't seem like terraform cli supports gcs --> gcs in one hop, so you must go gcs --> local --> gcs.

Not ideal if your Terraform state is sensitive, but in practice terraform cli is moving it to/from your machine anyways, so persisting it temporarily in filesystem not much different.

Upvotes: 3

Erick M. Sprengel
Erick M. Sprengel

Reputation: 2209

terraform has a smart command terraform init -migrate-state that will do the job for you.

STEPS:

1. Change the backend "s3":

  backend "s3" {
    region = "us-east-1"
    bucket = "banana"
    key = "banana/terraform.tfstate"
    profile = "my-profile"
  }

2. run init with -migrate-state:

terraform init -migrate-state

It'll detect that you moved your state between buckets and it'll ask you to migrate it:

the output displayed to me:

Initializing the backend...
Backend configuration changed!

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.

Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "s3" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

Upvotes: 8

Leo
Leo

Reputation: 1514

Make a local copy of the Terraform state:

terraform state pull > terraform.tfstate

Remove the .terraform directory:

rm -rf .terraform/

Update the bucket in your backend definition

Initialize Terraform again and answer yes to the prompt about importing your existing state:

$ terraform init

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

See this answer for more details: https://stackoverflow.com/a/71054509/476159

Upvotes: 4

Ermiya Eskandary
Ermiya Eskandary

Reputation: 23602

Move your Terraform state files to the new S3 buckets and then change the bucket parameter inside your S3 backend config to point to the bucket.

You do not need to do anything else if you just want to move the location of the state file - it's like changing the location of an application and then pointing the shortcut to the new location.

Upvotes: 3

Barak Haryati
Barak Haryati

Reputation: 146

Did you try to copy the StateFile From old bucket to new bucket and then change the S3 bucket in terraform backend configuration

Upvotes: 8

Related Questions