CODEWITHSUNDEEP
amazon-sagemaker
user2770451
user2770451

Reputation: 27

Sagemaker: DeepAR Hyperparameter Tuning Error

Running into a new issue with tuning DeepAR on SageMaker when trying to initialize a hyperparameter tuning job - this error also occurs when calling the test:mean_wQuantileLoss. I've upgraded the sagemaker package, restarted my instance, restarted the kernel (using a juptyer notebook), and yet the problem persists. 

ClientError: An error occurred (ValidationException) when calling the 
CreateHyperParameterTuningJob operation: The objective metric type, [Maximize], that you specified for objective metric, [test:RMSE], isn’t valid for the [156387875391.dkr.ecr.us-west-2.amazonaws.com/forecasting-deepar:1] algorithm. Choose a valid objective metric type.

Code:

my_tuner = HyperparameterTuner(estimator=estimator,
                               objective_metric_name="test:RMSE",
                               hyperparameter_ranges=hyperparams,
                               max_jobs=20,
                               max_parallel_jobs=2)

# Start hyperparameter tuning job
my_tuner.fit(inputs=data_channels)

Stack Trace:
ClientError                               Traceback (most recent call last)
<ipython-input-66-9d6d8de89536> in <module>()
      7 
      8 # Start hyperparameter tuning job
----> 9 my_tuner.fit(inputs=data_channels)

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/tuner.py in fit(self, inputs, job_name, include_cls_metadata, **kwargs)
    255 
    256         self._prepare_for_training(job_name=job_name, include_cls_metadata=include_cls_metadata)
--> 257         self.latest_tuning_job = _TuningJob.start_new(self, inputs)
    258 
    259     @classmethod

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/tuner.py in start_new(cls, tuner, inputs)
    525                                                output_config=(config['output_config']),
    526                                                resource_config=(config['resource_config']),
--> 527                                                stop_condition=(config['stop_condition']), tags=tuner.tags)
    528 
    529         return cls(tuner.sagemaker_session, tuner._current_job_name)

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in tune(self, job_name, strategy, objective_type, objective_metric_name, max_jobs, max_parallel_jobs, parameter_ranges, static_hyperparameters, image, input_mode, metric_definitions, role, input_config, output_config, resource_config, stop_condition, tags)
    348         LOGGER.info('Creating hyperparameter tuning job with name: {}'.format(job_name))
    349         LOGGER.debug('tune request: {}'.format(json.dumps(tune_request, indent=4)))
--> 350         self.sagemaker_client.create_hyper_parameter_tuning_job(**tune_request)
    351 
    352     def stop_tuning_job(self, name):

~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    312                     "%s() only accepts keyword arguments." % py_operation_name)
    313             # The "self" in this scope is referring to the BaseClient.
--> 314             return self._make_api_call(operation_name, kwargs)
    315 
    316         _api_call.__name__ = str(py_operation_name)

~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    610             error_code = parsed_response.get("Error", {}).get("Code")
    611             error_class = self.exceptions.from_code(error_code)
--> 612             raise error_class(parsed_response, operation_name)
    613         else:
    614             return parsed_response

ClientError: An error occurred (ValidationException) when calling the CreateHyperParameterTuningJob operation: 
The objective metric type, [Maximize], that you specified for objective metric, [test:RMSE], isn’t valid for the [156387875391.dkr.ecr.us-west-2.amazonaws.com/forecasting-deepar:1] algorithm. 
Choose a valid objective metric type.

Upvotes: 2

Views: 1800

Answers (1)

Owen
Owen

Reputation: 919

It looks like you are trying to maximize this metric, test:RMSE can only be minimized by SageMaker HyperParameter Tuning.

To achieve this in the SageMaker Python SDK, create your HyperparameterTuner with objective_type='Minimize'. You can see the signature of the init method here.

Here is the change you should make to your call to HyperparameterTuner:

my_tuner = HyperparameterTuner(estimator=estimator,
                               objective_metric_name="test:RMSE",
                               objective_type='Minimize',
                               hyperparameter_ranges=hyperparams,
                               max_jobs=20,
                               max_parallel_jobs=2)

Upvotes: 2

Related Questions