CODEWITHSUNDEEP
terraform
munchybunch
munchybunch

Reputation: 6153

Split terraform tfstate file

We started off with one tfstate file, and over time it's grown quite a bit.

Now, it's really slow to plan and I'd now like to split out into several tfstate files (one for our development environment, one for general shared infrastructure, and one per production environment).

Like how it's described at https://charity.wtf/2016/03/30/terraform-vpc-and-why-you-want-a-tfstate-file-per-env/ and Terraform Multiple State Files Best Practice Examples.

Is there any existing tool (built-in or not) to help with this? Sort of like terraform state mv but between tfstates?

Upvotes: 8

Views: 10119

Answers (3)

dimisjim
dimisjim

Reputation: 425

terraform state mv has the -state-out flag, where you can define another state to move your resources into.

However, I couldn't get it to work with version 0.11.14, thus I manually cut & pasted the modules from the state file I needed to move to the other state, which worked brilliantly.

EDIT: Here's a workaround, which basically dwonloads both statefiles, moves the states desired, and then reuploads them to the S3 buckets: https://github.com/hashicorp/terraform/pull/15652#issuecomment-410754814

Upvotes: 5

Iulian
Iulian

Reputation: 319

In a complex world, from what it happens I am part of, we went even further. Following a communal effort we successfully segregated not only the environments but all of the components that are part of the infrastructure.

Digging down, by components I mean a certain set of modules/outputs per component with no extra resources in the configuration but of course the dependencies problem had appeared... nothing that can't be solved with some data{} references to the tiny states and a few scripts.

The result is awesome. We have the same component instantiated with different parameters and each environment which is clean, easy to maintain and operate.

I do not recommend the approach in the case of existing chunky infrastructure where some additional transition related activities are required if you are sick and tired of a single monolithic state file, but it enables a lot of acceleration in a greenfield project.

Upvotes: -3

bitbrain
bitbrain

Reputation: 619

Having a separate state file each for infrastructure and application makes sense.

Is there any existing tool (built-in or not) to help with this? Sort of like terraform state mv but between tfstates?

No, as far as I can tell. Move out shared parts (e.g. ECS clusters, ALB, network configuration, iam roles etc.) into a separate project/repository.

When using S3 as state backend you can define different paths for your infrastructure and application states, for example:

  • /infrastructure/nonprod/terraform.tfstate
  • /infrastructure/prod/terraform.tfstate
  • /apps/app1/test/terraform.tfstate
  • /apps/app1/uat/terraform.tfstate
  • /apps/app1/prod/terraform.tfstate

When you want to deploy your application to TEST or UAT you simply call terraform init before terraform apply in your infrastructure project by providing the path to your non-prod S3 state. Then call terraform init on your app terraform config by providing the path to your TEST or UAT path.

Ideally, you can create your own shell scripts to provision and deploy your apps. Then in your favourite CI you can create a pipeline to provision infrastructure and deploy apps as you wish. Make sure you parameterize those scripts so you can pass which environment you want to provision or which app you want to deploy, for example:

./my-shared-infrastructure/provision-infrastructure.sh nonprod

./my-app-1/deploy-application.sh uat v1.0

Upvotes: 1

Related Questions