create terraform resource (S3 Bucket Object) if already doesn't exists

Question

Create the terraform resource based on the data source output condition.

Hi all, I want to achieve the below requirement in Terraform.

Requirement:

I want to create object (with key_name) in s3 bucket.
Before creating the object need to check whether the object with same key already exists or not using data source.
If it is already exists, do not create object. If not do create the object in s3 bucket.

Source Code

variable "key_name" {
  default = "test.txt"
}
variable "bucket_name" {
  default = "test-bucket-654321"
}

data "aws_s3_bucket_objects" "my_objects" {
  bucket = var.bucket_name
}

locals {
  key_present = anytrue([
    for key in data.aws_s3_bucket_objects.my_objects.keys: key == var.key_name
  ])
}

resource "aws_s3_bucket_object" "examplebucket_object" {
  count = local.key_present ? 0 : 1
  
  key     = var.key_name
  bucket  = var.bucket_name
  content = "Test Value"
}

output "my_objects" {
  value = data.aws_s3_bucket_objects.my_objects.keys
}
output "key_present" {
  value = local.key_present
}

Let's assume, key is does not exists in the bucket (For example, it is a new bucket with no objects)

1. On applying 1st time, it creates the object in bucket.
2. On applying 2ns time, it destroy the created object in bucket.
3. On applying 3rd time, it creates the object in bucket.

On 1st time, data source returns empty value. so it creates the object in bucket.
On 2nd time, data source returns created object name. so the key is already exists and it destroys the object in bucket.

Likewise every odd number of times it creates the objects and every even number of times it destroys the created object.

Output:

ubuntu@test-instance:~$ terraform apply --auto-approve
aws_s3_bucket_object.examplebucket_object[0]: Creating...
aws_s3_bucket_object.examplebucket_object[0]: Creation complete after 0s [id=test.txt]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:
key_present = false
my_objects = tolist([])

ubuntu@test-instance:~$ terraform apply --auto-approve
aws_s3_bucket_object.examplebucket_object[0]: Refreshing state... [id=test.txt]
aws_s3_bucket_object.examplebucket_object[0]: Destroying... [id=test.txt]
aws_s3_bucket_object.examplebucket_object[0]: Destruction complete after 0s

Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

Outputs:
key_present = true
my_objects = tolist([
  "test.txt",
])

Could anyone give me the solution for creating a terraform resource based on the data source output ?

Answer 1

I tried to approach it this way.

# Data source to check if the S3 bucket exists
data "aws_s3_bucket" "existing_bucket" {
  bucket = var.bucket_name
}

# Create the S3 bucket only if it doesn't already exist
resource "aws_s3_bucket" "my_s3_bucket" {
  count  = length(data.aws_s3_bucket.existing_bucket.id) == 0 ? 1 : 0
  bucket = var.bucket_name
}

resource "aws_s3_bucket_public_access_block" "my_s3_bucket" {
  count  = length(data.aws_s3_bucket.existing_bucket.id) == 0 ? 1 : 0
  bucket = aws_s3_bucket.my_s3_bucket[count.index].id. # will give you bucket URI

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_object" "folder_1" {
  count  = length(data.aws_s3_bucket.existing_bucket.id) == 0 ? 1 : 0
  bucket = aws_s3_bucket.my_s3_bucket[count.index].bucket # will give you bucket name
  key    = "my_folder_in_bucket_name/"
}

Explanation:
length(data.aws_s3_bucket.existing_bucket.id) ==0 ? 1 : 0: This expression uses the length function to determine the length of the id. If the length is 0, means bucket does not exist, and the expression returns 1 (indicating that TF should create the resource). If the length > 0, the bucket already exists, and the expression returns 0 (indicating that Tf should not create the resource).
The count.index is used to access the specific instance of the my_s3_bucket resource.

Answer 2

What your code above is doing is that in every even iteration the conditional meta-argument expression count = local.key_present ? 0 : 1 evaluates to 0 and thus you are commanding terraform to create explicitly 0 quantity of the resource.

Every even iteration, you terraform configuration for examplebucket_object effectively becomes:

resource "aws_s3_bucket_object" "examplebucket_object" {
  count = 0
  
  key     = var.key_name
  bucket  = var.bucket_name
  content = "Test Value"
}

The logic above is effectively commanding Terraform to remove examplebucket_object.

Terraform is designed to be idempotent, that it, if a resource is already under Terraform management (part of the state), Terraform will figure that out and it will not re-provision the same resource (assuming the same configuration is being applied, which is not in your case.)

So, in your case, your Terraform configuration is changing the intended target state depending on the previous state found, and this explains your conundrum.

In general you have to consider this situation:

1. If you're expecting to manage the lifecycle of a resource via Terraform from the beginning, then you don't need to do any data source queries and conditional checks.

2. If you are expecting to occasionally encounter existing resources and want to bring them under Terraform management, what you need to do is to import those resources into the state, via terraform import.

   • Currently aws_s3_bucket_object does not support terraform import, but there is a pull-request open.

---

Note:

I'd recommend you implement a Dependency Inversion approach as described in Conditional Creation of Objects from the official Terraform documentation:

Rather than trying to write a module that itself tries to detect whether something exists and create it if not, we recommend applying the dependency inversion approach: making the module accept the object it needs as an argument, via an input variable.

This is consistent with Terraform's declarative style: rather than creating modules with complex conditional branches, we directly describe what should already exist and what we want Terraform to manage itself.