CODEWITHSUNDEEP
pythonamazon-s3boto3
martina.physics
martina.physics

Reputation: 9804

Read content of all files in S3 path in one go with boto

Using boto3, I can read the content of a file stored in S3 via

s3 = boto3.resource('s3')
s3.Object(bucket_name, path).get()['Body'].read()

I have multiple files in the same path ("directory") and I'd like to read them all building a global data variable storing their comprehensive content (say a dictionary or list).

Is there a way to accomplish this from boto without having to read them one by one and concatenate the bodies?

Upvotes: 0

Views: 948

Answers (1)

garnaat
garnaat

Reputation: 45906

No, there is nothing in the S3 API that supports reading multiple objects in a single API call. You could spin up multiple threads with each retrieving an object to get better throughput.

Upvotes: 1

Related Questions