CODEWITHSUNDEEP
c#amazon-web-servicesamazon-s3
Jameel
Jameel

Reputation: 328

C# List all files with filename under an amazon S3 folder

Using C# and amazon .Net SDK, able to list all the files with in a amazon S3 folder as below:

ListObjectsRequest request = new ListObjectsRequest();           
request.BucketName = _bucketName; //Amazon Bucket Name
request.Prefix = _sourceKey; //Amazon S3 Folder path           
do
{
    ListObjectsResponse response = _client.ListObjects(request);//_client - AmazonS3Client

Output

Folder
Folder/file1.pdf
Folder/file2.pdf
Folder/file3.pdf

But i wanted to achieve something like this:

Desired Output

file1.pdf
file2.pdf
file3.pdf

Thanks in advance

Upvotes: 26

Views: 61447

Answers (7)

Venkat S
Venkat S

Reputation: 43

When there are large number of objects you may need to use ListObjectsV2Request to read the objects in batches, similar to shown below

    private readonly IAmazonS3 _amazonS3; // Make sure you initialize this

    public async IAsyncEnumerable<IEnumerable<string>> ListObjects(string bucketName, string prefix)
    {
        ListObjectsV2Request objectsV2Request = new ListObjectsV2Request
        {
            BucketName = bucketName,
            Prefix = prefix
        };
        ListObjectsV2Response listObjectsV2Response;
        do
        {
            listObjectsV2Response = await _amazonS3.ListObjectsV2Async(objectsV2Request);
            objectsV2Request.ContinuationToken = listObjectsV2Response.NextContinuationToken;
            yield return listObjectsV2Response.S3Objects.Select(s => s.Key);
        } while (listObjectsV2Response.IsTruncated);
    }

Upvotes: 1

Ashutosh B Bodake
Ashutosh B Bodake

Reputation: 1412

_client = new AmazonS3Client(_awsAccessKeyId, _awsSecretAccessKey, _regionEndpoint);
var req = new ListObjectsV2Request
{
                BucketName = _bucketName,
                Prefix = "upper_folder_name/" + 
                "lower_folder_name/",
                MaxKeys = 100,
                Delimiter = "/",
                FetchOwner = false
            };
            var response = await _client.ListObjectsV2Async(req);
            foreach (var s3Object in response.S3Objects)
            {
                Console.WriteLine(s3Object.Key);
            }
 }

Upvotes: 0

novice2ninja
novice2ninja

Reputation: 471

Let's make it clear, simple, and with a little description.
As we all know, in S3 there is no concept of directories (folders). Ah, what? So everything inside S3 is nothing but objects.
Let's consider the below example s3 bucket - the bucket name is testBucket, the directory name is testDirectory and the directory contains two files testImage.jpg and testUserData.txt.

  • testBucket
    • testDirectory
      • testImage.jpg
      • testUserData.txt
AmazonS3Client s3Client = new AmazonS3Client();
string bucketName = "testBucket";
// to list out all the elements inside of a director [To know more][1]
string prefix = "testDirectory/";

ListObjectsRequest request = new ListObjectsRequest
                {
                    BucketName = bucketName,
                    Prefix = prefix,
                    // Use this to exclude your directory name
                    StartAfter = prefix,
                };
ListObjectsResponse response =  s3Client.ListObjects(request);

foreach (S3Object itemsInsideDirectory in response .S3Objects)
                {
                    Console.WriteLine(itemsInsideDirectory.Key);
                }

output:
If you use 'StartAfter', then output will be
testDirectory/testImage.jpg
testDirectory/testUserData.txt*

If you donot use 'StartAfter', then output will be
testDirectory/
testDirectory/testImage.jpg
testDirectory/testUserData.txt

Upvotes: 13

Marcelo Souza
Marcelo Souza

Reputation: 191

AmazonS3Client s3Client = new AmazonS3Client(S3_ACCESS_KEY, S3_SECRET_ACCESS_KEY, S3_REGIAO);
var lista = s3Client.ListObjectsAsync(S3_BUCKET, $"{S3_SUBDIRETORIO}/{produto}/{produto}.").Result;
var files = lista.S3Objects.Select(x => x.Key);
var arquivos = files.Select(x => Path.GetFileName(x)).ToList();

Upvotes: 19

Luis Fernando Camacho Camacho
Luis Fernando Camacho Camacho

Reputation: 1140

Also you can use the following c# code to retrieve the files information.

var bucketName = "your bucket";
var s3Client = new AmazonS3Client("your access key", "your secret key");
var dir = new S3DirectoryInfo(s3Client, bucketName, "your AmazonS3 folder name");
foreach (IS3FileSystemInfo file in dir.GetFileSystemInfos())
{
    Console.WriteLine(file.Name);
    Console.WriteLine(file.Extension);
    Console.WriteLine(file.LastWriteTime);
}

I am using amazon AWSSDK.Core and AWSSDK.S3 version 3.1.0.0 for .net 3.5. I hope it can help you

Upvotes: 33

Cindy
Cindy

Reputation: 384

    AmazonS3Client client = new AmazonS3Client();
    ListObjectsRequest listRequest = new ListObjectsRequest
    {
        BucketName = "your-bucket-name",
        Prefix = "example/path"
    };

    ListObjectsResponse listResponse;
    do
    {
        listResponse = client.ListObjects(listRequest);
        foreach (S3Object obj in listResponse.S3Objects)
        {
            Console.WriteLine(obj.Key);
            Console.WriteLine(" Size - " + obj.Size);
        }

        listRequest.Marker = listResponse.NextMarker;
    } while (listResponse.IsTruncated);

https://docs.aws.amazon.com/sdkfornet1/latest/apidocs/html/M_Amazon_S3_AmazonS3_ListObjects.htm

Upvotes: 9

Munesh Gupta
Munesh Gupta

Reputation: 21

string bucketName = your bucket name;
S3DirectoryInfo dir = new S3DirectoryInfo(client, bucketName, "folder name");
foreach (IS3FileSystemInfo file in dir.GetFiles())
{
    Console.WriteLine(file.Name);
    Console.WriteLine(file.Extension);
    Console.WriteLine(file.LastWriteTime);
}

Upvotes: 2

Related Questions