CODEWITHSUNDEEP
amazon-web-servicesamazon-s3terraform
Nathan
Nathan

Reputation: 4955

Terraform: How to migrate state between projects?

What is the least painful way to migrate state of resources from one project (i.e., move a module invocation) to another, particularly when using remote state storage? While refactoring is relatively straightforward within the same state file (i.e., take this resource and move it to a submodule or vice-versa), I don't see an alternative to JSON surgery for refactoring into different state files, particularly if we use remote (S3) state (i.e., take this submodule and move it to another project).

Upvotes: 66

Views: 51071

Answers (6)

mr.wolle
mr.wolle

Reputation: 1346

If you have the state already in a local file (here tfstate.json) you can initialize the new backend and push the state.

terraform init -backend-config=backend.config
terraform state push tfstate.json

Upvotes: 0

Nikolai R Kristiansen
Nikolai R Kristiansen

Reputation: 627

With terraform version 1.7+ you could solve this declaratively using removed and import blocks, for resources that are importable.

# In source state:
removed {
  from = random_uuid.example

  lifecycle {
    destroy = false
  }
}

# In destination state:
import {
   to = random_uuid.example
   id = "aabbccdd-eeff-0011-2233-445566778899"
}
resource "random_uuid" "example" {}

Upvotes: 1

Sang
Sang

Reputation: 4454

I use this script (not work from v0.12) to migrate the state while refactoring. Feel free to adopt it to your need.

src=<source dir>
dst=<target dir>
resources=(
    aws_s3_bucket.bucket1
    aws_iam_role.role2
    aws_iam_user.user1
    aws_s3_bucket.bucket2
    aws_iam_policy.policy2
)
cd $src
terraform state pull >/tmp/source.tfstate
cd $dst
terraform state pull >/tmp/target.tfstate
for resource in "${resources[@]}"; do
    terraform state mv -state=/tmp/source.tfstate -state-out=/tmp/target.tfstate "${resource}" "${resource}"
done
terraform state push /tmp/target.tfstate
cd $src
terraform state push /tmp/source.tfstate

Note that terraform pull is deprecated from v0.12 (but not removed and still works), and terraform push does not work anymore from v0.12.

Important: The terraform push command is deprecated, and only works with the legacy version of Terraform Enterprise. In the current version of Terraform Cloud, you can upload configurations using the API. See the docs about API-driven runs for more details.

==================

Below are unrelated to the OP:

If you are renaming your resources in the same project.

  • For version <= 1.0: use terraform state mv ....
  • For version >= 1.1, use the moved statement described: here or here.

There are several other useful commands that I listed in my blog

Upvotes: 8

RubberDuck
RubberDuck

Reputation: 12788

The least painful way I’ve found is to pull both remote states local, move the modules/resources between the two, then push back up. Also remember, if you’re moving a module, don’t move the individual resources; move the whole module.

For example:

cd dirA
terraform state pull > ../dirA.tfstate

cd ../dirB
terraform state pull > ../dirB.tfstate

terraform state mv -state=../dirA.tfstate -state-out=../dirB.tfstate module.foo module.foo

terraform state push ../dirB.tfstate

# verify state was moved
terraform state list | grep foo

cd ../dirA
terraform state push ../dirA.tfstate

Unfortunately, the terraform state mv command doesn’t support specifying two remote backends, so this is the easiest way I’ve found to move state between multiple remotes.

Upvotes: 143

Exequiel Barrirero
Exequiel Barrirero

Reputation: 5868

As mentioned in a related Terraform Q -> Best practices when using Terraform

  1. It is easier and faster to work with smaller number of resources:
    • Cmdsterraform plan and terraform apply both make cloud API calls to verify the status of resources.
    • If you have your entire infrastructure in a single composition this can take many minutes (even if you have several files in the same folder).

So if you'll end up with a mono-dir with every resource, never is late to start segregating them by service, team, client, etc.

Possible Procedures to migrate Terrform states between projects / services:

Example Scenario:

Suppose we have a folder named common with all our .tf files for a certain project and we decided to divide (move) our .tf Terraform resources to a new project folder named security. so we now need to move some resources from common project folder to security.

Case 1:

If the security folder still does not exists (which is the best scenario).

  1. Backup the Terraform backend state content stored in the corresponding AWS S3 Bucket (since it's versioned we should be even safer).
  2. With your console placed in the origin folder, for our case common execute make init to be sure your .terraform local folder it's synced with your remote state.
  3. If the security folder still does not exists (which should be true) clone (copy) the common folder with the destination name security and update the config.tf file inside this new cloned folder to point to the new S3 backend path (consider updating 1 account at a time starting with the less critical one and evaluate the results with terraform state list).

eg:

# Backend Config (partial)
terraform {
  required_version = ">= 0.11.14"

  backend "s3" {
    key = "account-name/security/terraform.tfstate"
  }
}
  1. Inside our newly created security folder, run terraform-init (without removing the copied .terraform local folder, which was already generated and synced in step 2) which, as a result, will generate a new copy of the resources state (interactively asking) in the new S3 path. This is a safe operation since we haven't removed the resources from the old .tfstate path file yet.
$ make init
terraform init -backend-config=../config/backend.config
Initializing modules...
- module.cloudtrail
- module.cloudtrail.cloudtrail_label

Initializing the backend...
Backend configuration changed!

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.

Acquiring state lock. This may take a few moments...
Acquiring state lock. This may take a few moments...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "s3" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes
...                                                             

Successfully configured the backend "s3"! Terraform will automatically                                                                        
use this backend unless the backend configuration changes.                                                                                    

Initializing provider plugins...                                                                                                              
...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
Terraform has been successfully initialized!                                                                                                                                                                                                                                             
...

enter image description here

  1. Selectively remove the desired resources from each state (terraform state rm module.foo) in order to keep the desired ones in /common and /security paths. Moreover, It's a must to carry out in parallel the necessary updates (add/remove) of the modules/resources from your .tf files in each folder to keep both your local code base declaration and your remote .tfstate in sync. This is a sensible operation, please start by testing the procedure in the less critical possible single resource.

As reference we can consider the following doc and tools:

Case 2:

If the security folder already exists and has it's associated remote .tfstate in its AWS S3 path you'll need to use a different sequence of steps and commands, possible the ones referenced in the links below: 1. https://www.terraform.io/docs/commands/state/list.html 2. https://www.terraform.io/docs/commands/state/pull.html 3. https://www.terraform.io/docs/commands/state/mv.html 4. https://www.terraform.io/docs/commands/state/push.html

Ref links:

Upvotes: 6

ydaetskcoR
ydaetskcoR

Reputation: 56997

Probably the simplest option is to use terraform import on the resource in the new state file location and then terraform state rm in the old location.

Terraform does handle some automatic state migration when copying/moving the .terraform folder around but I've only used that when shifting the whole state file rather than part of it.

Upvotes: 15

Related Questions