Link a state file to a module in Terraform

Question

My environment is too large/complex to be managed with a single Terraform state file - so we need to have multiple state files. Our current thinking is that we would have a state file per object type - there are very good reasons for structuring it like this so I'm looking for a solution that uses this structure, not a solution that suggests a different structure.

The logic for provisioning each object type is encapsulated in its own module i.e. there is a standalone module for creating AWS S3 buckets, a standalone module for creating Snowflake Storage Integration objects, etc.

There are also higher level modules that provision more complex infrastructure by calling the lower level modules i.e. because there is a common pattern where an S3 bucket is provisioned and then a Snowflake storage integration object is provisioned that references the S3 bucket, there would be a module that calls the S3 module and then calls the Snowflake Storage Integration module.

Is it possible that when an S3 bucket is provisioned using the S3 module, it always uses the same S3 state file - regardless of whether that S3 module is called directly or it is called from the higher level module?

So if the higher level module was called, the S3 provisioning would use the S3 state file and and the Storage Integration provisioning would use the Storage Integration state file

Answer 1

In Terraform, state files belongs to entire configurations rather than to individual modules.

If you call the same module multiple times in the same configuration, it will declare separate resources each time and each of those will be tracked separately in that configuration's overall state file.

Similarly, if you call the same module multiple times in different configurations, each one has a separate set of resources tracked in the configuration that each module refers to.

Modules in Terraform encapsulate declarations of objects, rather than the actual objects themselves. Shared modules are intended as a way to reuse the same definitions to declare many separate objects that are structured similarly, not to reuse the same objects in multiple locations.

---

The typical way to use the same object in multiple locations with Terraform is to decide on one single configuration that is responsible for managing that object, and then have your other configurations use data sources to represent the external dependency on an object managed elsewhere.

Your data blocks might either refer to the external object directly, if appropriate, or you can potentially introduce an indirection such as publishing the ID information about an object under a particular key in a generic data store such as AWS SSM Parameter Store and then having the other configurations read that key in order to learn the ID that they should use.

Data sources therefore serve as a way to create a sort of bridge to objects managed outside of one particular Terraform configuration, so that you can then use the information about them elsewhere in the configuration in a similar way to how you might refer directly to a resource object declared in the same configuration.