CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3lambdaaws-lambda
cgclip
cgclip

Reputation: 312

Use pickled files stored on S3 in AWS Lambda

I'm currently working on an Amazon Web Services Lambda function, where I need to utilize sklearn files that I've created locally but are now stored in S3. I've been stumped on how to actually do this, I've tried a few different methods from boto3's docs but this results in a timeout error:

import boto3
import pickle
s3 = boto3.client("s3", region_name="us-west-1")
response = s3.get_object(Bucket="mybucket", Key="bin_files/scaler.pkl")
loaded_file = pickle.load(response["Body"])

While doing this on the other hand results in a file not found error:

 with open(key, 'wb') as data:
     s3.Bucket(event['bucket']).download_fileobj(key, data)


"errorMessage": "[Errno 2] No such file or directory: 
'bin_files/scaler.pkl'"

Does anyone have any thoughts or tips on how to do this?

Upvotes: 1

Views: 2450

Answers (1)

Madhukar Mohanraju
Madhukar Mohanraju

Reputation: 2863

Correct code:

 with open('scaler.pkl', 'wb') as data:
     s3.Bucket('bucket').download_fileobj(key, data)

If you want to download the s3 file to current directory, then you should just pass the Filename to the file open() method and NOT the entire key. In your case you are passing bin_files/scaler.pkl as the Filename.

In your case, the python code will look for bin_files directory in the current working directory(directory where the boto script is getting executed) and try to write scaler.pkl inside the bin_files directory. Since the directory doesn't exist, it throws the [Errno 2] No such file or directory error.

If you add a PATH to the Filename, then you should ensure the entire PATH exists with appropriate directory permissions.

Upvotes: 2

Related Questions