CODEWITHSUNDEEP
amazon-sagemaker
Stephan
Stephan

Reputation: 175

Where do I store my model's training data, artifacts, etc?

I'm trying to build and push a custom ML model with docker to Amazon SageMaker. I know things are supposed to follow the general structure of being in opt/ml. But there's no such bucket in Amazon S3??? Am I supposed to create this directory within my container before I build and push the image to AWS? I just have no idea where to put my training data, etc.

Upvotes: 0

Views: 2579

Answers (2)

Farrago-Alex
Farrago-Alex

Reputation: 120

When creating your custom model on AWS SageMaker, you can store your docker container with your inference code on ECR, while keeping your model artifacts just on S3. You can then just specify the S3 path to said artifacts when creating the model (when using Boto3's create_model, for example). This may simplify your solution so you don't have to re-upload your docker container every time you may need to change your artifacts (though you will need to re-create your model on SageMaker).

The same goes for your data sets. SageMakers' Batch Transform function allows you to feed any of your data sets stored on S3 directly into your model without needing to keep them in your docker container. This really helps if you want to run your model on many different data sets without needing to re-upload your image.

Upvotes: 1

Guy
Guy

Reputation: 12909

SageMaker is automating the deployment of the Docker image with your code using the convention of channel->local-folder. Everything that you define with a channel in your input data configuration, will be copied to the local Docker file system under /opt/ml/ folder, using the name of the channel as the name of the sub-folder.

{
"train" : {"ContentType":  "trainingContentType", 
           "TrainingInputMode": "File", 
           "S3DistributionType": "FullyReplicated", 
           "RecordWrapperType": "None"},
"evaluation" : {"ContentType":  "evalContentType", 
                "TrainingInputMode": "File", 
                "S3DistributionType": "FullyReplicated", 
                "RecordWrapperType": "None"},
"validation" : {"TrainingInputMode": "File", 
                "S3DistributionType": "FullyReplicated", 
                "RecordWrapperType": "None"}
}

to:

/opt/ml/input/data/training
/opt/ml/input/data/validation
/opt/ml/input/data/testing

Upvotes: 1

Related Questions