CODEWITHSUNDEEP
terraform
red888
red888

Reputation: 31520

How do I get a list of s3 objects with the aws_s3_bucket_object data source?

My specific problem is the same as this guys, but I found his answer not detailed enough, also terraform has new features now that maybe can solve this better.

The problem is I'm using aws_elastic_beanstalk_application_version to register beanstalk version, but terraform removes the old version before registering the new one. This is because aws_elastic_beanstalk_application_version is replaced each time, what I need to do is generate a new one.

I'm attempting to do this with "count" and the aws_s3_bucket_object data source but I can't figure out how to get s3 objects as a list. I tried wildcards but that doesn't work:

data "aws_s3_bucket_object" "eb-bucket-data" {
  bucket = "mybucket"
  key = "*"
}

resource "aws_elastic_beanstalk_application_version" "default" {
  count       = "${length(data.aws_s3_bucket_object.eb-bucket-data.id)}"
  name        = "${element(data.aws_s3_bucket_object.eb-bucket-data.key, count.index)}"
  application = "myapp"
  bucket      = "mybucket"
  key        = "${element(data.aws_s3_bucket_object.eb-bucket-data.key, count.index)}"
}

Upvotes: 1

Views: 5163

Answers (2)

DarkerIvy
DarkerIvy

Reputation: 1587

There is a Pull Request for this data source, aws_s3_bucket_objects:

https://github.com/terraform-providers/terraform-provider-aws/pull/6968

Upvotes: 0

ydaetskcoR
ydaetskcoR

Reputation: 56839

The aws_s3_bucket_object data source currently only returns a single item. You could iterate through a list of items but that puts you back to your initial problem of needing to find the list of items in the first place.

Short of creating a pull request for an aws_s3_bucket_objects data source that returns a list of objects (as with things like aws_availability_zone and aws_availability_zones) you can maybe achieve this through shelling out using the external data source and calling the AWS CLI.

An (untested) example for this might look something like this:

data "external" "bucket_objects" {
  program = ["aws", "s3", "ls", "mybucket", "|", "awk", "'{print", "$4}'", "|", "jq", "-R", "-s", "-c", "'split(\"\n\")'", "|", "jq", "'.[:-1]'"]
}

This runs 

aws s3 ls mybucket | awk '{print $4}' | jq -R -s -c 'split("\n")' | jq '.[:-1]'

which lists the objects in the bucket, takes just the filename elements, splits them into a JSON array using jq and then removes the trailing newline element from the JSON array because the external data source expects a valid JSON object to be returned.

You should then be able to access that with something like:

resource "aws_elastic_beanstalk_application_version" "default" {
  count       = "${length(data.external.bucket_objects.result)}"
  name        = "${data.external.bucket_objects.result[count.index]}"
  application = "myapp"
  bucket      = "mybucket"
  key         = "${data.external.bucket_objects.result[count.index]"
}

Upvotes: 2

Related Questions