CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-sagemaker
Stupid420
Stupid420

Reputation: 1419

memory error while writing a large dataframe to S3 AWS

I have created a dataframe with the following shape using amazon sagemaker.

10612611 rows × 4 columns

All are numeric values. When I am trying to write this dataframe into my S3 bucket as follows, I get memory error.

bytes_to_write = df.to_csv(None).encode()
with s3.open('aws-athena-query-results-xxxxxxx/query_result/xx.csv','wb') as f:
    f.write(bytes_to_write)

MemoryError:

I am using ml.t2.medium for sagemaker instance.

Upvotes: 0

Views: 580

Answers (1)

Stupid420
Stupid420

Reputation: 1419

I solved this issue by changing the instance type from ml.t2.medium to ml.t2.2xlarge and it worked perfectly.

The original issue was with the RAM of the instance type and not with S3.

Upvotes: 2

Related Questions