CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-lambdaaws-lambda-layers
dbeings
dbeings

Reputation: 85

Converting a python dataframe from AWS lambda to CSV in S3

I am trying to convert a write a python Dataframe and store it in S3. I have added the required layers in Lambda (s3fs & fsspec) and have provided requisite permissions to lambda to write to s3. But not I am getting the below error:

"errorMessage": "module 's3fs' has no attribute 'S3FileSystem'"

Below are the relevant lines of my code:

s3 = boto3.client('s3')
df.to_csv('s3://buckets/<my-bukcet-name>/mydata.csv')

Any pointers on what could be the reason for this?

Regards, Dbeings

Upvotes: 1

Views: 1620

Answers (1)

Mark B
Mark B

Reputation: 200446

Instead of including those layers, I would recommend including the Amazon provided AWS Data Wrangler layer.

Then you would use AWS Data Wrangler to write your dataframe directly to S3, like the following:

import awswrangler as wr

wr.s3.to_csv(df, 's3://buckets/<my-bukcet-name>/mydata.csv', index=False)

Upvotes: 4

Related Questions