CODEWITHSUNDEEP
ruby-on-railsrubyamazon-s3
user648198
user648198

Reputation: 2020

How to list all files in an s3 folder using AWS-SDK gem in ruby on rails

I wanted to show a list of all files in an s3 folder so I can get all the last modified dates so I can determine what files has been changed.

I tried using objects.with_prefix('Folder1') it give me a full list but also contain Folder1.1 key

I don't know if i needed to use delimiter but I couldn't find anything how to use delimiter in aws sdk.

Thanks so much in advance!

I'm using 'aws-sdk' gem

Here is my bucket structure -Folder1 -File1 -File2 -Folder.1.1

Here is my code 

bucket = s3.buckets[bucket_name]
data = bucket.objects.with_prefix('Folder1/')
data.each do |object|
    puts "#{object.key}\t#{object.last_modified}";
end

Upvotes: 50

Views: 37515

Answers (5)

kebomix
kebomix

Reputation: 856

Too late answer but better than never.

You can do

s3_bucket.objects.with_prefix('folder_name').collect(&:key)

According to official documentation here

Updates: SDK V3

s3 = Aws::S3::Client.new
resp = s3.list_objects_v2({
  bucket: "BucketName", # required
  prefix: "FolderName", 
})

Upvotes: 44

helmutrs
helmutrs

Reputation: 270

Let's remember that the S3 is not a file system, so even the 'folder/' is an object.

Now if you want to get the files for a specific path, you can use the start_after. For example, if you have in your S3 bucket a list of objects like this.

pictures/
pictures/horse.jpg
pictures/dog.jpg
pictures/cat.jpg
lion.jpg

You can get all the pictures from the picture/ path by doing the following:

 s3 = Aws::S3::Resource.new(region: 'ap-southeast-1', access_key_id: ENV['AWS_ACCESS_KEY_ID'], secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'] )
 pictures = s3.bucket(bucket_name).objects(prefix: 'pictures/', delimiter: '', start_after: 'pictures/').collect(&:key)

The output is going to be only all the pictures keys without the folder/:

  • pictures/horse.jpg
  • pictures/dog.jpg
  • pictures/cat.jpg

Upvotes: 1

i0x539
i0x539

Reputation: 5393

https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#list_objects_v2-instance_method

SDK V3 has the prefix option for client!

resp = client.list_objects_v2({ bucket: "BucketName", # required prefix: "FolderName", })

Upvotes: 8

Mohit Jain
Mohit Jain

Reputation: 43949

You can use this small piece of code for getting list of files for a specific folder.

 s3 = Aws::S3::Resource.new(region: 'ap-southeast-1', access_key_id: ENV['AWS_ACCESS_KEY_ID'], secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'] )
 data_files = s3.bucket(bucket_name).objects(prefix: 'prefix/', delimiter: 'delimiter').collect(&:key)

Upvotes: 17

Adilbiy Kanzitdinov
Adilbiy Kanzitdinov

Reputation: 658

Currently I am also stuck with this problem. So far solution is to fetch all the objects and to filter them later:

data = bucket.objects(bucketname, prefix: 'Folder1')

data_without_folders = data.select { |obj| !(obj.key =~ /\/$/) }

For delimiter, you just have to pass it in bucket.objects call like:

data = bucket.objects(bucketname, prefix: 'prefix', delimiter: 'delimiter')

If better solution is available, I will let you know.

Upvotes: 3

Related Questions