CODEWITHSUNDEEP
amazon-web-servicesamazon-s3mxnetamazon-sagemaker
Lakshay Sharma
Lakshay Sharma

Reputation: 877

Training data in S3 in AWS Sagemaker

I've uploaded my own Jupyter notebook to Sagemaker, and am trying to create an iterator for my training / validation data which is in S3, as follow:

train = mx.io.ImageRecordIter(
        path_imgrec         = ‘s3://bucket-name/train.rec’ …… )

I receive the following exception: 

MXNetError: [04:33:32] src/io/s3_filesys.cc:899: Need to set enviroment variable AWS_SECRET_ACCESS_KEY to use S3

I've checked that the IAM role attached with this notebook instance has S3 access. Any clues on what might be needed to fix this?

Upvotes: 3

Views: 1452

Answers (1)

captainblack
captainblack

Reputation: 4425

If your IAM roles are setup correctly, then you need to download the file to the Sagemaker instance first and then work on it. Here's how:

# Import roles
import sagemaker
role = sagemaker.get_execution_role()

# Download file locally
s3 = boto3.resource('s3')
s3.Bucket(bucket).download_file('your_training_s3_file.rec', 'training.rec')

#Access locally
train = mx.io.ImageRecordIter(path_imgrec=‘training.rec’ …… )

Upvotes: 4

Related Questions