AttributeError: 'SparseCategoricalCrossentropy' object has no attribute '_id'

Question

I've been trying to recreate a simple DNN using just the base Keras layer and writing everything from scratch. Everything seems to work just fine, but during the training loop I get this error: AttributeError: 'SparseCategoricalCrossentropy' object has no attribute '_id'

I've tried changing the loss function to either CategoricalCrossentropy and SparseCategoricalCrossentropy (with from_logits True or False), but the error always pops up.

Here's the code:

import numpy as np
import tensorflow as tf
from tensorflow import keras

from utils import plot_image, plot_mnist_results, plot_value_array


class Flatten(keras.layers.Layer):
    def __init__(self):
        super(Flatten, self).__init__()

    def build(self, input_shape):
        self.output_size = np.prod(input_shape)

    def call(self, X):
        return tf.reshape(X, shape=(-1, self.output_size))


class Dense(keras.layers.Layer):
    def __init__(self, units, activation):
        super(Dense, self).__init__()
        self.units = units
        self.activation = activation

    def build(self, input_shape):
        self.kernel = self.add_weight(
            name='kernel',
            dtype=tf.float64,
            initializer='glorot_normal',
            trainable=True,
            shape=(input_shape[-1], self.units)
        )
        self.bias = self.add_weight(
            name='bias',
            dtype=tf.float64,
            initializer=keras.initializers.Constant(0.1),
            trainable=True,
            shape=(1, self.units)
        )

    def call(self, X):
        return self.activation(tf.matmul(X, self.kernel) + self.bias)


class DNN(keras.models.Model):
    def __init__(self, units, activation):
        super(DNN, self).__init__()
        self.units = units
        self.activation = activation

    def build(self, input_shape):
        self.flatten = Flatten()
        self.hidden_layer = Dense(self.units, tf.nn.relu)
        self.output_layer = Dense(10, tf.nn.softmax)

    def call(self, X):
        print(self.hidden_layer(self.flatten(X)).shape)
        print(self.output_layer(self.hidden_layer(self.flatten(X))).shape)
        return self.output_layer(self.hidden_layer(self.flatten(X)))


# @tf.function
def train(model, loss, opt, X, y):
    with tf.GradientTape() as tape:
        gradients = tape.gradient(loss(model(X), y), model.trainable_variables)
        gradient_variables = zip(gradients, model.trainable_variables)
        opt.apply_gradients(gradient_variables)


mnist = keras.datasets.mnist

(train_images, train_labels), (test_images, test_labels) = mnist.load_data()

train_images = train_images / 255.0
test_images = test_images / 255.0

model = DNN(units=128, activation=tf.nn.relu)
opt = tf.optimizers.Adam(learning_rate=1e-3)

for epoch in range(3):

    for step in range(train_labels.shape[0]):
        loss = keras.losses.SparseCategoricalCrossentropy
        train(model, loss, opt, train_images[step, :, :], train_labels[step])

    train_loss = loss(model(train_images), train_labels)

    template = 'Epoch {}, Train loss: {:.5f}'
    print(template.format(epoch + 1, train_loss.numpy()))

I would expect for the model to train successfully, but it doesn't seem to be the case. What am I doing wrong?

Answer 1

You are saving keras.losses.SparseCategoricalCrossentropy class inside loss variable and then passing model(X), y inside the constructor which will give the error. Possible solutions include (but are not limited to) the following:

1. Put keras.losses.SparseCategoricalCrossentropy instance inside the loss variable by replacing loss = keras.losses.SparseCategoricalCrossentropy with loss = keras.losses.SparseCategoricalCrossentropy().
2. Put tf.keras.losses.sparse_categorical_crossentropy function inside the loss variable.

Basically, you mistakenly used Class instead of function. Remember that TF has both functional and Class implementation for losses.

Answer 2

From the given code, i could see that you are using tf and keras intermixed in places like given below.

opt = tf.optimizers.Adam(learning_rate=1e-3)

loss = keras.losses.SparseCategoricalCrossentropy

This could raise issues like this. For TensorFlow 2.0, you can use tf.keras uniformly in all places wherever you use keras directly.

Also i could find that, you are instantiating loss object inside the batch loop. which is not correct. You have to instantiate at the top of starting you epoch loop.

Rest all seems fine. Hope this helps!!!