Keras: Callbacks Requiring Validation Split?

Question

I'm working on a multi-class classification problem with Keras 2.1.3 and a Tensorflow backend. I have two numpy arrays, x and y and I'm using tf.data.Dataset like this:

dataset = tf.data.Dataset.from_tensor_slices(({"sequence": x}, y))
dataset = dataset.apply(tf.contrib.data.batch_and_drop_remainder(self.batch_size))
dataset = dataset.repeat()
xt, yt = dataset.make_one_shot_iterator().get_next()

Then I make my Keras model (omitted for brevity), compile, and fit:

model.compile(
    loss='categorical_crossentropy',
    optimizer='adam',
    metrics=['accuracy'],
)
model.fit(xt, yt, steps_per_epoch=100, epochs=10)

This works perfectly well. But when I add in callbacks, I run into issues. Specifically, if I do this:

callbacks = [
    tf.keras.callbacks.ModelCheckpoint("model_{epoch:04d}_{val_acc:.4f}.h5",
                    monitor='val_acc',
                    verbose=1,
                    save_best_only=True,
                    mode='max'),
    tf.keras.callbacks.TensorBoard(os.path.join('.', 'logs')),
    tf.keras.callbacks.EarlyStopping(monitor='val_acc', patience=5, min_delta=0, mode='max')
]
model.fit(xt, yt, steps_per_epoch=10, epochs=100, callbacks=callbacks)

I get:

KeyError: 'val_acc'

Also, if I include validation_split=0.1 in my model.fit(...) call, I'm told:

ValueError: If your data is in the form of symbolic tensors, you cannot use validation_split.`

What is the normal way to use callbacks and validation splits with tf.data.Dataset (tensors)?

Thanks!

Answer 1

Using the tensorflow keras API, you can provide a Dataset for training and another for validation.

First some imports

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import Dense
import numpy as np

define the function which will split the numpy arrays into training/val

def split(x, y, val_size=50):
    idx = np.random.choice(x.shape[0], size=val_size, replace=False)
    not_idx = list(set(range(x.shape[0])).difference(set(idx)))

    x_val = x[idx]
    y_val = y[idx]
    x_train = x[not_idx]
    y_train = y[not_idx]
    return x_train, y_train, x_val, y_val

define numpy arrays and the train/val tensorflow Datasets

x = np.random.randn(150, 9)
y = np.random.randint(0, 10, 150)

x_train, y_train, x_val, y_val = split(x, y)

train_dataset = tf.data.Dataset.from_tensor_slices((x_train, tf.one_hot(y_train, depth=10)))
train_dataset = train_dataset.batch(32).repeat()

val_dataset = tf.data.Dataset.from_tensor_slices((x_val, tf.one_hot(y_val, depth=10)))
val_dataset = val_dataset.batch(32).repeat()

Make the model (notice we are using the tensorflow keras API)

model = keras.models.Sequential([Dense(64, input_shape=(9,), activation='relu'),
                                 Dense(64, activation='relu'),
                                 Dense(10, activation='softmax')
                                ])
model.compile(optimizer='Adam', loss='categorical_crossentropy',
              metrics=['accuracy'])
model.summary()


model.fit(train_dataset,
          epochs=10, 
          steps_per_epoch=int(100/32)+1,
          validation_data=val_dataset,
          validation_steps=2)

and the model trains, kind of (output):

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 64)                640       
_________________________________________________________________
dense_1 (Dense)              (None, 64)                4160      
_________________________________________________________________
dense_2 (Dense)              (None, 10)                650       
=================================================================
Total params: 5,450
Trainable params: 5,450
Non-trainable params: 0
_________________________________________________________________
Epoch 1/10
4/4 [==============================] - 0s 69ms/step - loss: 2.3170 - acc: 0.1328 - val_loss: 2.3877 - val_acc: 0.0712
Epoch 2/10
4/4 [==============================] - 0s 2ms/step - loss: 2.2628 - acc: 0.2500 - val_loss: 2.3850 - val_acc: 0.0712
Epoch 3/10
4/4 [==============================] - 0s 2ms/step - loss: 2.2169 - acc: 0.2656 - val_loss: 2.3838 - val_acc: 0.0712
Epoch 4/10
4/4 [==============================] - 0s 2ms/step - loss: 2.1743 - acc: 0.3359 - val_loss: 2.3830 - val_acc: 0.0590
Epoch 5/10
4/4 [==============================] - 0s 2ms/step - loss: 2.1343 - acc: 0.3594 - val_loss: 2.3838 - val_acc: 0.0590
Epoch 6/10
4/4 [==============================] - 0s 2ms/step - loss: 2.0959 - acc: 0.3516 - val_loss: 2.3858 - val_acc: 0.0590
Epoch 7/10
4/4 [==============================] - 0s 4ms/step - loss: 2.0583 - acc: 0.3750 - val_loss: 2.3887 - val_acc: 0.0590
Epoch 8/10
4/4 [==============================] - 0s 2ms/step - loss: 2.0223 - acc: 0.4453 - val_loss: 2.3918 - val_acc: 0.0747
Epoch 9/10
4/4 [==============================] - 0s 2ms/step - loss: 1.9870 - acc: 0.4609 - val_loss: 2.3954 - val_acc: 0.1059
Epoch 10/10
4/4 [==============================] - 0s 2ms/step - loss: 1.9523 - acc: 0.4609 - val_loss: 2.3995 - val_acc: 0.1059

Callbacks

Adding callbacks also works,

callbacks = [ tf.keras.callbacks.ModelCheckpoint("model_{epoch:04d}_{val_acc:.4f}.h5",
                    monitor='val_acc',
                    verbose=1,
                    save_best_only=True,
                    mode='max'),
    tf.keras.callbacks.TensorBoard('./logs'),
    tf.keras.callbacks.EarlyStopping(monitor='val_acc', patience=5, min_delta=0, mode='max')
]


model.fit(train_dataset, epochs=10, steps_per_epoch=int(100/32)+1, validation_data=val_dataset,
          validation_steps=2, callbacks=callbacks)

Output:

Epoch 1/10
4/4
 [==============================] - 0s 59ms/step - loss: 2.3274 - acc: 0.1094 - val_loss: 2.3143 - val_acc: 0.0833

Epoch 00001: val_acc improved from -inf to 0.08333, saving model to model_0001_0.0833.h5
Epoch 2/10
4/4 [==============================] - 0s 2ms/step - loss: 2.2655 - acc: 0.1094 - val_loss: 2.3204 - val_acc: 0.1389

Epoch 00002: val_acc improved from 0.08333 to 0.13889, saving model to model_0002_0.1389.h5
Epoch 3/10
4/4 [==============================] - 0s 5ms/step - loss: 2.2122 - acc: 0.1250 - val_loss: 2.3289 - val_acc: 0.1111

Epoch 00003: val_acc did not improve from 0.13889
Epoch 4/10
4/4 [==============================] - 0s 2ms/step - loss: 2.1644 - acc: 0.1953 - val_loss: 2.3388 - val_acc: 0.0556

Epoch 00004: val_acc did not improve from 0.13889
Epoch 5/10
4/4 [==============================] - 0s 2ms/step - loss: 2.1211 - acc: 0.2734 - val_loss: 2.3495 - val_acc: 0.0556

Epoch 00005: val_acc did not improve from 0.13889
Epoch 6/10
4/4 [==============================] - 0s 4ms/step - loss: 2.0808 - acc: 0.2969 - val_loss: 2.3616 - val_acc: 0.0556

Epoch 00006: val_acc did not improve from 0.13889
Epoch 7/10
4/4 [==============================] - 0s 2ms/step - loss: 2.0431 - acc: 0.2969 - val_loss: 2.3749 - val_acc: 0.0712

Epoch 00007: val_acc did not improve from 0.13889

Answer 2

I think your problem is type of your datasets, where xt and yt is not a list or array, so it can not to slice or split.

Let assume x is array with shape (1000,2) fill with random number using numpy and y is class with 10 classes.

# Define x and y
import numpy as np
np.random.seed(2018)
n_cat = 10
x = np.random.rand(1000,2)  # Generate random 2 variables and 1000 rows
y = [np.random.randint(1,n_cat) for i in range(len(x))]     # Make 10 class

Then you assemble the model and compile

model.compile(
    loss='categorical_crossentropy',
    optimizer='adam',
    metrics=['accuracy'],
)

Create callbacks for checkpoint model

callbacks = [
    tf.keras.callbacks.ModelCheckpoint("model_{epoch:04d}_{val_acc:.4f}.h5",
                    monitor='val_acc',
                    verbose=1,
                    save_best_only=True,
                    mode='max'),
    tf.keras.callbacks.TensorBoard(os.path.join('.', 'logs')),
    tf.keras.callbacks.EarlyStopping(monitor='val_acc', patience=5, min_delta=0, mode='max')
]

fit the model

model.fit(x, y, batch_size=32, epochs=100, callbacks=callbacks, validation_split=0.1)

it's works for me.