CODEWITHSUNDEEP
pythonclassificationamazon-sagemakermultilabel-classification
Scorp
Scorp

Reputation: 247

AUC objective metric for SageMaker Hypertuning

I have trained the binary classification model using AWS built-in algorithm with SageMaker and want to evaluate the model using the AUC and confusion matrix. However, I see that SageMaker's Training and HyperTuner job just accepts the Accuracy metric.

  1. Is there a way in SageMaker to add the custom metric for a built-in image classification algorithm?
  2. As I understand AUC/Confusion Matrix/Precision/Recall/F1 are good metrics for a binary classifier, then Why these are missing in the AWS built-in image classification algorithm?
  3. Is there a way where I can batch transform my test data and get these metrics to evaluate the model as Accuracy alone is not good for evaluation?

Upvotes: 1

Views: 599

Answers (1)

Olivier Cruchant
Olivier Cruchant

Reputation: 4037

  1. SageMaker Built-in algorithms cannot accept custom metrics, they work only for the built-in metrics

  2. Confusion matrix is not a metric, it's a visualization. Also note that the image classifier is not a binary classifier, it's a general classifier that can have a large number of labels. Regarding the other metrics I can't speak on behalf of AWS teams :)

  3. Yes, using Batch Transform or real-time endpoints to create predictions to be used in your own custom analytics is a good idea. For example, in this blog post an ephemeral endpoint is created to produce predictions and a confusion matrix for the built-in linear classifier https://aws.amazon.com/blogs/machine-learning/build-multiclass-classifiers-with-amazon-sagemaker-linear-learner/

Upvotes: 0

Related Questions