CODEWITHSUNDEEP
amazon-web-servicesgoamazon-s3
Stefan Radonjic
Stefan Radonjic

Reputation: 1598

AWS S3 Listing API - How to list everything inside S3 Bucket with specific prefix

I am trying to list all items with specific prefix in S3 bucket. Here is directory structure that I have:

Item1/
     Item2/
          Item3/
               Item4/
                     image_1.jpg
               Item5/
                     image_1.jpg
                     image_2.jpg

When I set prefex to be Item1/Item2, I get as a result following keys:

Item1/Item2/
Item1/Item2/Item3/Item4/image_1.jpg
Item1/Item2/Item3/Item5/image_1.jpg
Item1/Item2/Item3/Item5/image_2.jpg

What I would like to get is:

Item1/Item2/
Item1/Item2/Item3
Item1/Item2/Item3/Item4
Item1/Item2/Item3/Item5
Item1/Item2/Item3/Item4/image_1.jpg
Item1/Item2/Item3/Item5/image_1.jpg
Item1/Item2/Item3/Item5/image_2.jpg

Is there anyway to achieve this in golang?

Upvotes: 0

Views: 499

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 270124

Folders do not actually exist in Amazon S3. It is a flat object storage system.

For example, using the AWS Command-Line Interface (CLI) I could copy a command to an Amazon S3 bucket:

aws s3 cp foo.txt s3://my-bucket/folder1/folder2/foo.txt

This work just fine, even though folder1 and folder2 do not exist. This is because objects are stored with a Key (filename) that includes the full path of the object. So, the above object actually has a Key (filename) of:

folder1/folder2/foo.txt

However, to make things easier for humans, the Amazon S3 management console makes it appear as though there are folders. In S3, these are called Common Prefixes rather than folders.

So, when you make an API call to list the contents of the bucket while specifying a Prefix, it simply says "List all objects whose Key starts with this string".

Your listing doesn't show any folders because they don't actually exist.

Now, just to contradict myself, it actually is possible to create a folder (eg by clicking Create folder in the management console). This actually creates a zero-length object with the same name as the folder. The folder will then appear in listings because it is actually listing the zero-length object rather than the folder.

This is probably why Item1/Item2/ appears in your listing, but Item1/Item2/Item3 does not. Somebody, at some stage, must have "created a folder" called Item1/Item2/, which actually created a zero-length object with that Key.

Upvotes: 5

Related Questions