TFX Evaluator seems it is not recognizing the baseline model output from the ResolverNode

Question

I want to use the validation capabilities (model diff or model comparison) of the model Evaluator component in TFX, so I used the base code of taxi template in TFX to do so.

The problem is that when the Evaluator component runs in Kubeflow on GCP throws the next error message within the logs:


    ERROR:absl:There are change thresholds, but the baseline is missing. This is allowed only when rubber stamping (first run).
    WARNING:absl:"maybe_add_baseline" and "maybe_remove_baseline" are deprecated,
            please use "has_baseline" instead.
    INFO:absl:Request was made to ignore the baseline ModelSpec and any change thresholds. This is likely because a baseline model was not provided: updated_config=
    model_specs {
      name: "candidate"
      signature_name: "my_model_validation_signature"
      label_key: "n_trips"
    }
    slicing_specs {
    }
    metrics_specs {
      metrics {
        class_name: "MeanSquaredError"
        threshold {
          value_threshold {
            upper_bound {
              value: 10000.0
            }
          }
        }
      }
    }
    INFO:absl:ModelSpec name "candidate" is being ignored and replaced by "" because a single ModelSpec is being used

Looking at the source code from the executor of the Evaluator component in TFX repo 1 line 138:


has_baseline = bool(input_dict.get(BASELINE_MODEL_KEY))

and then enters a function on line 141:


eval_config = tfma.update_eval_config_with_defaults(eval_config, has_baseline=has_baseline)

Then throws the cited error message only posible if the next condition is meet, from TFX repo 2

      if (not has_baseline and has_change_threshold(eval_config) and
          not rubber_stamp):
        # TODO(b/173657964): Raise an error instead of logging an error.
        logging.error('There are change thresholds, but the baseline is missing. '
                      'This is allowed only when rubber stamping (first run).')

And in fact thats the error that I get on the logs, the model is evaluated but not compared to a baseline even if I provide it the way indicated by sample code in for example:


      eval_config = tfma.EvalConfig(
          model_specs=[tfma.ModelSpec(name=tfma.CANDIDATE_KEY,label_key='n_trips', 
                                      signature_name='my_model_validation_signature'),
                       tfma.ModelSpec(name=tfma.BASELINE_KEY, label_key='n_trips', 
                                      signature_name='my_model_validation_signature', is_baseline=True)
                      ],
          slicing_specs=[tfma.SlicingSpec()],
          metrics_specs=[
              tfma.MetricsSpec(metrics=[
                  tfma.MetricConfig(
                      class_name="MeanSquaredError",#'mean_absolute_error',
                      threshold=tfma.MetricThreshold(
                          value_threshold=tfma.GenericValueThreshold(
                              upper_bound={'value': 10000}),
                                                                   
                          change_threshold=tfma.GenericChangeThreshold(
                              direction=tfma.MetricDirection.LOWER_IS_BETTER,
                              relative={'value':1})
                                                    )
                                  )
              ])
          ])
    
      evaluator = Evaluator(
          examples=example_gen.outputs['examples'],
          model=trainer.outputs['model'],
          baseline_model=model_resolver.outputs['model'],
          # Change threshold will be ignored if there is no baseline (first run).
          eval_config=eval_config)
    
      # TODO(step 6): Uncomment here to add Evaluator to the pipeline.
      components.append(evaluator)

And it continues...

TFX Evaluator seems it is not recognizing the baseline model output from the ResolverNode

Answers (1)

Related Questions