CODEWITHSUNDEEP
python-3.xamazon-web-servicesamazon-sagemakeraws-step-functions
Matias33
Matias33

Reputation: 1

make a comparison between two sagemaker-trained models using step functions

as the title says, I am trying to find a way to make a comparison between two models trained in sagemaker, using a step functions. The cycle currently looks like this: Start training, save the model, a lambda is run that checks if the accuracy was greater than 90 percent, if it is met: a batch job is started, if it is not met: the cycle is terminated. What I would like to be able to deploy is a solution that instead of being compared to 90% accuracy, that is compared to a previous training. I am currently using the Step Functions Data Science SDK.

Upvotes: 0

Views: 455

Answers (1)

Gili Nachum
Gili Nachum

Reputation: 5578

With Step Functions: Save a json file in S3 with previous model evaluation results, and have a Lambda step to compare current and previous results.

Also consider using SageMaker Pipelines instead of Step Functions. As SageMaker Pipelines is integrated with SageMaker Experiments, you'll add this conditional logic:

In the evaluation step (processing job) get the accuracy values of the previous executions and compute the best. This will be done via Python code in the processing script. Then, generate a JSON file that contains both the accuracy value for the current execution and the best value across the past executions. Define a conditional step that uses a PropertyFile and JsonGet expression to read from the JSON file both the left side and right side of the condition, in order to get a condition like [accuracy > previous_best_accuracy].

Upvotes: 1

Related Questions