keras_tuner found dict_keys([]) after using custom Callback function

Question

I have a problem using the Keras tuner for hyperparameter tuning while using a custom callback function. In my callback function I basically want to evaluate every trained model and save some specific evaluation parameters as well as the model at the end of each training. I also want to save the training time, therefore I start a timer at the begin of each training. For this I use the methods on_train_begin and on_train_end. The model training runs fine and the results are also stored, but I always get the following error:

Search: Running Trial #1

Value             |Best Value So Far |Hyperparameter
64                |64                |filters_1
25                |25                |kernel_size_1
1                 |1                 |pool_size_1
2                 |2                 |strides_1
64                |64                |units

Epoch 1/3
6/6 [==============================] - 2s 215ms/step - loss: 0.0545 - accuracy: 0.1964 - val_loss: 0.0470 - val_accuracy: 0.2143
Epoch 2/3
6/6 [==============================] - 1s 120ms/step - loss: 0.0477 - accuracy: 0.4643 - val_loss: 0.0424 - val_accuracy: 0.4286
Epoch 3/3
6/6 [==============================] - 1s 126ms/step - loss: 0.0425 - accuracy: 0.5179 - val_loss: 0.0327 - val_accuracy: 0.5000
dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])
1/1 [==============================] - 0s 59ms/step - loss: 0.0178 - accuracy: 0.3000
1/1 [==============================] - 0s 172ms/step
INFO:tensorflow:Assets written to: ../models/hyperparametertuning1/with_background/100/Trial_0/model\assets
Results for trial 0 written to file.
dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])
Traceback (most recent call last):
  File "C:\Users\kle7ba\.conda\envs\ogs-ma\Lib\site-packages\keras_tuner\engine\base_tuner.py", line 270, in _try_run_and_update_trial
    self._run_and_update_trial(trial, *fit_args, **fit_kwargs)
  File "C:\Users\kle7ba\.conda\envs\ogs-ma\Lib\site-packages\keras_tuner\engine\base_tuner.py", line 257, in _run_and_update_trial
    self.oracle.update_trial(
  File "C:\Users\kle7ba\.conda\envs\ogs-ma\Lib\site-packages\keras_tuner\engine\oracle.py", line 107, in wrapped_func
    ret_val = func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\kle7ba\.conda\envs\ogs-ma\Lib\site-packages\keras_tuner\engine\oracle.py", line 364, in update_trial
    self._check_objective_found(metrics)
  File "C:\Users\kle7ba\.conda\envs\ogs-ma\Lib\site-packages\keras_tuner\engine\oracle.py", line 626, in _check_objective_found
    raise ValueError(
ValueError: Objective value missing in metrics reported to the Oracle, expected: ['val_loss'], found: dict_keys([])
Trial 1 Complete [00h 00m 10s]

If I do not use the custom callback function, but only early stopping, I do not get this error. Does someone have a solution for this problem?

This is how my callback function/class looks like:

class CustomCallback(keras.callbacks.Callback):

    trial_number = 0
    results = pd.DataFrame(columns=['Trial-ID', 'loss', 'val_loss', 'test_loss', 'accuracy', 'val_accuracy', 'test_accuracy', 'training_time', 'prediction_time'])

    def __init__(self, X_test, y_test, max_trials, results_directory):
        self.X_test = X_test
        self.y_test = y_test
        self.max_trials = max_trials
        self.t_training_start = None
        self.results_directory = results_directory
        self.base_results_directory = results_directory
        self.history = {"loss": [], "val_loss": [], "accuracy": [], "val_accuracy": []}

    def on_train_begin(self, logs=None):
        # starting the training time for each trial
        self.t_training_start = time.time()

    def on_train_end(self, logs):
        # Data evaluation triggered at the end of the model training
        # Create the export folders
        print(logs.keys())
        self.results_directory = self.results_directory + '/Trial_' + str(CustomCallback.trial_number)
        model_directory = self.results_directory + '/model'
        figure_directory = self.results_directory + '/plots'
        prediction_directory = self.results_directory + '/predictions'

        # Evaluating the training time
        t_training_end = time.time()
        training_time = t_training_end - self.t_training_start

        # Evaluate loss and accuracy
        loss = logs['loss']
        val_loss = logs['val_loss']
        accuracy = logs['accuracy']
        val_accuracy = logs['val_accuracy']

        # Evaluate the model using test data
        evaluation = self.model.evaluate(self.X_test, self.y_test)
        test_loss = evaluation[0]
        test_accuracy = evaluation[1]

        # Evaluate the prediction time
        t_prediction_start = time.time()
        predictions = self.model.predict(self.X_test)
        t_prediction_end = time.time()
        prediction_time = t_prediction_end - t_prediction_start
        prediction_time_single = (t_prediction_end - t_prediction_start)/len(self.X_test)

        ...

I am using the following code to perform the random search:

# Defining tuner to perform the search algorithm
tuner = kt.RandomSearch(
    build_model,
    objective='val_loss',
    max_trials=MAX_TRIALS,
    max_consecutive_failed_trials=MAX_TRIALS,
    overwrite=True,
    directory="tmp/",
    project_name='hyperparametertuning1'
)

# Perform the search algorithm to find the best combination of hyperparameters
tuner.search(X_train, y_train, epochs=EPOCHS, batch_size=10, validation_split=0.2, callbacks=[early_stopping, CustomCallback(X_test=X_test, y_test=y_test, max_trials=MAX_TRIALS, results_directory=RESULTS_DIRECTORY)])

I also tried to not implement the callback on the on_train_begin and on_train_end methods, but write own methods with a Lambda Callback. This works for some parts, but within the Lambda Callback I did not manage to get the model from each training. Therefore, I can't evaluate the model and generate predictions, which I want to save and use for generating some plots later.

# Define custom callback function
custom_callback = Callbacks.CustomCallback(X_test=X_test, y_test=y_test, max_trials=MAX_TRIALS, results_directory=RESULTS_DIRECTORY)

lambda_timer_start = lambda logs : custom_callback.timer_start(logs)
lambda_set_model = lambda logs: custom_callback.result_saving(logs)

timer_start_callback = keras.callbacks.LambdaCallback(on_train_begin=lambda_timer_start)
result_saving_callback = keras.callbacks.LambdaCallback(on_train_end=lambda_set_model)

But using the following function within the Callback class, it always tells me that model is not defined:

class CustomCallback(keras.callbacks.Callback):

    trial_number = 0
    results = pd.DataFrame(columns=['Trial-ID', 'loss', 'val_loss', 'test_loss', 'accuracy', 'val_accuracy', 'test_accuracy', 'training_time', 'prediction_time'])

    def __init__(self, X_test, y_test, max_trials, results_directory):
        self.X_test = X_test
        self.y_test = y_test
        self.max_trials = max_trials
        self.t_training_start = None
        self.results_directory = results_directory
        self.base_results_directory = results_directory
        self.history = {"loss": [], "val_loss": [], "accuracy": [], "val_accuracy": []}

    def timer_start(self, logs=None):
        # starting the training time for each trial
        self.t_training_start = time.time()

    def result_saving(self, logs=None):
        # Data evaluation triggered at the end of the model training
        # Create the export folders
        self.results_directory = self.results_directory + '/Trial_' + str(CustomCallback.trial_number)
        model_directory = self.results_directory + '/model'
        figure_directory = self.results_directory + '/plots'
        prediction_directory = self.results_directory + '/predictions'

        # Evaluating the training time
        t_training_end = time.time()
        training_time = t_training_end - self.t_training_start

        # Evaluate loss and accuracy
        loss = logs.get('loss')
        val_loss = logs.get('val_loss')
        accuracy = logs.get('accuracy')
        val_accuracy = logs.get('val_accuracy')

        # Evaluate the model using test data
        evaluation = self.model.evaluate(self.X_test, self.y_test)
        test_loss = evaluation[0]
        test_accuracy = evaluation[1]

        # Evaluate the prediction time
        t_prediction_start = time.time()
        predictions = self.model.predict(self.X_test)
        t_prediction_end = time.time()
        prediction_time = t_prediction_end - t_prediction_start
        prediction_time_single = (t_prediction_end - t_prediction_start)/len(self.X_test)

        ...

Answer 1

I stumbled upon the same problem and even though I cannot tell the reason behind, I found a workaround that solved it for me. Therefore, I decided to add it as an answer, which may come to late for OP but at least serves as part of my personal documentation should I ever encounter this error again.

The solution is to replace every occurrence of model.predict with model.__call__ in the on_train_end function. This however behaves a little different, like there is no batching any more, which may hinder its application in specific cases.

So the full line should look like this:

predictions = self.model(self.X_test)