CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3botoboto3
user2426823
user2426823

Reputation:

Python/ Boto 3: How to retrieve/download files from AWS S3?

In Python/Boto 3, Found out that to download a file individually from S3 to local can do the following:

        bucket = self._aws_connection.get_bucket(aws_bucketname)
        for s3_file in bucket.list():
            if filename == s3_file.name:
                self._downloadFile(s3_file, local_download_directory)
                break;

And to download all files under one chosen directory: 

    else:
        bucket = self._aws_connection.get_bucket(aws_bucketname)
        for s3_file in bucket.list():
            self._downloadFile(s3_file, local_download_directory)

And helper function _downloadFile():

  def _downloadFile(self, s3_file, local_download_destination):
        full_local_path = os.path.expanduser(os.path.join(local_download_destination, s3_file.name))
        try:
            print "Downloaded: %s" % (full_local_path)
            s3_file.get_contents_to_filename(full_local_path)

But both don’t seem to be working. Using Boto 3 and Python, would like to be able to download all files, as a zip preferably, under a defined directory on S3 to my local.

What could I be doing wrong, and what’s the correct implementation of the parameters?

Thank you in advance, and will be sure to accept/upvote answer

UPDATE CODE: Getting an error: “AttributeError: 'S3' object has no attribute

import sys
import json
import os
import subprocess

import boto3
from boto.s3.connection import S3Connection

s3 = boto3.resource('s3')
s3client = boto3.client('s3')

#This works
for bucket in s3.buckets.all():
    print(bucket.name)

def main():
    #Getting an error: “AttributeError: 'S3' object has no attribute 'download’”
    s3client.download('testbucket', 'fileone.json', 'newfile')

if __name__ == "__main__": main()

Upvotes: 13

Views: 24291

Answers (1)

franklinsijo
franklinsijo

Reputation: 18290

To download files from S3 to Local FS, use the download_file() method

s3client = boto3.client('s3')
s3client.download_file(Bucket, Key, Filename)

If the S3 object is s3://mybucket/foo/bar/file.txt, then the arguments would be

Bucket   --> mybucket
Key      --> foo/bar/file.txt
Filename --> /local/path/file.txt

There aren't any methods to download the entire bucket. An alternative way would be to list all the objects in the bucket and download them individually as files.

for obj in s3client.list_objects(Bucket='mybucket')['Contents']:
    try:  
        filename = obj['Key'].rsplit('/', 1)[1]
    except IndexError:
        filename = obj['Key']

    localfilename = os.path.join('/home/username/Downloads/', filename)  # The local directory must exist.
    s3client.download_file('mybucket', obj['Key'], localfilename)

Note: The response of list_objects() is truncated to 1000 objects. Use the markers in the response to retrieve the remainder of objects in the bucket.

Upvotes: 22

Related Questions