CODEWITHSUNDEEP
javascalaamazon-s3aws-sdk
user3313379
user3313379

Reputation: 489

S3 java API is not listing folders correctly.

I am trying to list the folders under the s3 bucket, The problem is that using the S3 browser tool, I can see four folders but when I use java/scala code to get the list of folders under the bucket, It returns only one folder. I have used the following code using simple AWS-JAVA-SDK.

val awsCreds: BasicAWSCredentials = new BasicAWSCredentials(accessKey, accessSecret)
val s3: AmazonS3 = new AmazonS3Client(awsCreds)
val listObjectsRequest = new ListObjectsRequest()
        .withBucketName(bucketName).withPrefix(prefix)
        .withDelimiter(delimiter);
val objectListing = s3.listObjects(listObjectsRequest);
val directories = objectListing.getCommonPrefixes
println(directories.mkString(","))

it prints only one folder /staging

I have also tried the awsScala library and used the following code

val bucket = s3.bucket("prod-tapp").get
val summaries=s3.ls(bucket, "/")
summaries.foreach(println(_))

But same result.

I can see the correct folders using desktop application of aws s3 browser on windows. Here is the screen-shot of the result. enter image description here

Any suggestion?

Thanks

Upvotes: 1

Views: 2291

Answers (2)

Ahmad Alobra
Ahmad Alobra

Reputation: 1

below code to get all objects from s3 (more than 1000 object)

List<S3ObjectSummary> keyList = new ArrayList<S3ObjectSummary>();
ObjectListing objects = s3.listObjects("bucket.new.test");
keyList.addAll(objects.getObjectSummaries());
while (objects.isTruncated()) {
    objects = s3.listNextBatchOfObjects(objects);
    keyList.addAll(objects.getObjectSummaries());
}

Upvotes: 0

Raniz
Raniz

Reputation: 11113

I'm guessing that you're not including the delimiter (/) in the prefix.

If I run the following code (Java, but doesn't really matter):

public class S3Prefix {

    private static final AmazonS3Client s3 = new AmazonS3Client();

    public static void main(String[] args) {
        Arrays.asList(null, "test1", "test1/").forEach(S3Prefix::listPrefix);
    }

    public static void listPrefix(String prefix) {
        System.out.println("Listing prefix '" + prefix + "'");
        final ListObjectsV2Result result = s3.listObjectsV2(new ListObjectsV2Request()
                .withPrefix(prefix)
                .withBucketName("raniz-prefix-test")
                .withDelimiter("/"));

        System.out.println("\tCommon prefixes");
        result.getCommonPrefixes().forEach(p -> System.out.println("\t\t" + p));

        System.out.println("\tKeys");
        result.getObjectSummaries().forEach(s -> System.out.println("\t\t" + s.getKey()));
    }
}

I get the following output:

Listing prefix 'null'
    Common prefixes
        test1/
        test2/
        test3/
    Keys
Listing prefix 'test1'
    Common prefixes
        test1/
    Keys
Listing prefix 'test1/'
    Common prefixes
    Keys
        test1/
        test1/bar.txt
        test1/foo.txt

As you can see, it's important to include the delimiter in the prefix if you want to list the contents of that "folder".

Upvotes: 2

Related Questions