CODEWITHSUNDEEP
terraform
jagatjyoti
jagatjyoti

Reputation: 717

Backing up of Terraform statefile

I usually run all my Terraform scripts through Bastion server and all my code including the tf statefile resides on the same server. There happened this incident where my machine accidentally went down (hard reboot) and somehow the root filesystem got corrupted. Now my statefile is gone but my resources still exist and are running. I don't want to again run terraform apply to recreate the whole environment with a downtime. What's the best way to recover from this mess and what can be done so that this doesn't get repeated in future.

I have already taken a look at terraform refresh and terraform import. But are there any better ways to do this ?

Upvotes: 1

Views: 6829

Answers (3)

lambodar
lambodar

Reputation: 3783

and all my code including the tf statefile resides on the same server.

As you don't have .backup file, I'm not sure if you can recover the statefile smoothly in terraform way, do let me know if you find a way :) . However you can take few step which will help you come out from situation like this.

The best practice is keep all your statefiles in some remote storage like S3 or Blob and configure your backend accordingly so that each time you destroy or create a new stack, it will always contact the statefile remotely.

On top of it, you can take the advantage of terraform workspace to avoid the mess of statefile in multi environment scenario. Also consider creating a plan for backtracking and versioning of previous deployments.

terraform plan -var-file "" -out "" -target=module.<blue/green>

what can be done so that this doesn't get repeated in future.

Terraform blue-green deployment is the answer to your question. We implemented this model quite a while and it's running smoothly. The whole idea is modularity and reusability, same templates is working for 5 different component with different architecture without any downtime(The core template remains same and variable files is different).

module tree

We are taking advantage of Terraform module. We have two module called blue and green, you can name anything. At any given point of time either blue or green will be taking traffic. If we have some changes to deploy we will bring the alternative stack based on state output( targeted module based on terraform state), auto validate it then move the traffic to the new stack and destroy the old one.

Here is an article you can keep as reference but this exactly doesn't reflect what we do nevertheless good to start with.

Blue-Green Deployment

Upvotes: 5

Radhakrishnan Rk
Radhakrishnan Rk

Reputation: 561

If you are still unable to recover the terraform state. You can create a blueprint of terraform configuration as well as state for a specific aws resources using terraforming But it requires some manual effort to edit the state for managing the resources back. You can have this state file, run terraform plan and compare its output with your infrastructure. It is good to have remote state especially using any object stores like aws s3 or key value store like consul. It has support for locking the state when multiple transactions happened at a same time. Backing up process is also quite simple.

Upvotes: 1

Tiemen
Tiemen

Reputation: 894

Please see this blog post, which, unfortunately, illustrates import being the only solution.

Upvotes: 1

Related Questions