CODEWITHSUNDEEP
pythonamazon-s3boto3
JMV12
JMV12

Reputation: 1045

Download S3 File Using Boto3

I'm currently writing a script in where I need to download S3 files to a created directory. I currently create a boto3 session with credentials, create a boto3 resource from that session, then use it to query and download from my s3 location. It looks something like the example below:

s3_session = boto3.Session(...)
s3_resource = s3_session.resource('s3')

mnt_loc = '/home/username/tmp/'

s3_loc = urlparse('s3://bucket_name/path1/path2/', allow_fragments=False)
s3_files = []
bucket = s3_resource.Bucket(s3_loc.netloc)
for elem in bucket.objects:
    s3_files.append(elem)

for elem in s3_files:
    s3_resource.Bucket(elem.bucket_name).download_file(elem.key, mnt_loc + elem.key.rsplit('/', 1)[-1])

In theory, I thought this would copy the specified file from the s3 location to my mount location while keeping the naming of the file. When executing, I'm getting an

[Errno 2] No such file or directory: /home/username/tmp/filename.csv.F2H1nxR0.

Why isn't this process working and also, why is the process appending these random strings to the end of my file name like the 'F2H1nxR0' shown?

Upvotes: 1

Views: 3792

Answers (1)

JMV12
JMV12

Reputation: 1045

I figured out a way to correct the error I was receiving. The issue was I was passing the s3.ObjectSummary parts into my download_file(). To fix this error, I converted all the parts to string as shown below:

s3_session = boto3.Session(...)
s3_resource = s3_session.resource('s3')

mnt_loc = '/home/username/tmp/'

s3_loc = urlparse('s3://bucket_name/path1/path2/', allow_fragments=False)
s3_files = []
bucket = s3_resource.Bucket(s3_loc.netloc)
for elem in bucket.objects:
    s3_files.append(elem)

for elem in s3_files:
    bucket = str(elem.bucket_name)
    path = str(elem.key)
    file_name = str(elem.key.rsplit('/', 1)[-1])
    s3_resource.Bucket(bucket).download_file(path, mnt_loc + file_name)

This eliminated the [Errno 2] No such file or directory: /home/username/tmp/filename.csv.F2H1nxR0 and gets rid of the 8 character string appended to the end of the file name. Thank you everyone for your help as it lead me to finding this fix.

Upvotes: 2

Related Questions