CODEWITHSUNDEEP
pythonamazon-s3
little
little

Reputation: 3029

Python simples3 listdir limit the output to 1000 files?

I am using simples3 version 1.0. When calling listdir on s3 key that has a lot more than 1000 files, the generator return only the first 1000 files.

The method has an optional limit argument which is None by default. Setting the limit return an empty list. And there is no additional limit to listdir mentioned in the documentation.

Here is a code example, where bucket is simples3 S3Bucket and s3_key is a key in s3 that include over 5000 files under it:

count = 0
for file_info in bucket.listdir(prefix=s3_key):
   count += 1

>>> count
1000 ## Expected to be over 5000

Is there a limitation that is not documented? Is there a way with simples3 lib to get all the files under the s3 key?

Upvotes: 1

Views: 959

Answers (2)

lericson
lericson

Reputation: 1338

Author of simples3 here. I'm sorry this stopped working, but that's really on Amazon. As you can see in the implementation of S3Bucket.listdir there is support for pagination and it should be seamless. If it isn't, then that's because Amazon stopped returning the truncated flag or stopped returning markers to perform pagination.

Upvotes: 0

Dekel
Dekel

Reputation: 62626

The simples3 library is using aws's rest api, which by default limits the max keys to 1000.

It's not documented, but based on the code can use set the limit using the limit key:

for file_info in bucket.listdir(prefix=s3_key, limit=10000):

I didn't find any way to set it to unlimited.

Upvotes: 1

Related Questions