CODEWITHSUNDEEP
pythonpandasamazon-web-servicescsvboto3
Omar M. Hussein
Omar M. Hussein

Reputation: 45

How to upload a data frame on AWS as CSV directly in python?

i tried boto3 but no luck

import boto3
from botocore.exceptions import NoCredentialsError

ACCESS_KEY = 'access_key'
SECRET_KEY = 'secret_key'


def upload_to_aws(local_file, bucket, s3_file):
    s3 = boto3.client('s3', aws_access_key_id=ACCESS_KEY,
                      aws_secret_access_key=SECRET_KEY)

    try:
        s3.upload_file(local_file, bucket, s3_file)
        print("Upload Successful")
        return True
    except FileNotFoundError:
        print("The file was not found")
        return False
    except NoCredentialsError:
        print("Credentials not available")
        return False


uploaded = upload_to_aws('local_file', 'information-arch', 's3_file_name')

print("Done! with the uploud")

Upvotes: 0

Views: 469

Answers (2)

Mayank Porwal
Mayank Porwal

Reputation: 34046

You can copy your dataframe directly to s3 like this:

Let's say you have a dataframe called df. You can use the to_csv option specifying your s3 path.

It will directly save the csv file on S3.

This works with pandas versions >= 0.24

df.to_csv(s3_path, index=False)

From pandas docs:

pandas now uses s3fs for handling S3 connections. This shouldn’t break any code. However, since s3fs is not a required dependency, you will need to install it separately, like boto in prior versions of pandas.

Upvotes: 1

Dipen Shah
Dipen Shah

Reputation: 2444

Hussein,

According to the boto3Documentation, you should upload your upload like this:

upload_file(Filename, Bucket, Key, ExtraArgs=None, Callback=None, Config=None)

Example:

import boto3
s3 = boto3.resource('s3')
s3.meta.client.upload_file('/tmp/hello.txt', 'mybucket', 'hello.txt')

Parameters

Filename (str) -- The path to the file to upload.

Bucket (str) -- The name of the bucket to upload to.

Key (str) -- The name of the key to upload to.

So on your upload_to_aws function called to pass the parameter like this way.

Thanks

Upvotes: 1

Related Questions