Keras ValueError: Shapes (None, 1) and (None, 16) are incompatible

Question

I'm trying to train a multilabel classification from text input. I first tokenize the text

tokenizer = Tokenizer(num_words=max_words)
tokenizer.fit_on_texts(df['text'])
sequences = tokenizer.texts_to_sequences(df['text'])
data = pad_sequences(sequences, maxlen=maxlen)

getting the following shape:

Shape of data tensor: (1333, 100) Shape of label tensor: (1333,)

Then I split in train and validations

x_train = data[:training_samples]
y_train = labels[:training_samples]
x_val = data[training_samples: training_samples + validation_samples]
y_val = labels[training_samples: training_samples + validation_samples]

I use Glove for word representations

embeddings_index = {}
f = open(os.path.join(glove_dir, 'glove.6B.100d.txt'))
for line in f:
    values = line.split()
    word = values[0]
    coefs = np.asarray(values[1:], dtype='float32')
    embeddings_index[word] = coefs
f.close()

embedding_dim = 100

embedding_matrix = np.zeros((max_words, embedding_dim))
for word, i in word_index.items():
    if i < max_words:
        embedding_vector = embeddings_index.get(word)
        if embedding_vector is not None:
            embedding_matrix[i] = embedding_vector

I build the Keras model

model = Sequential()
model.add(Embedding(max_words, embedding_dim, input_length=maxlen))
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dense(16, activation='softmax'))
model.summary()

Ending up with

Model: "sequential_32"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_27 (Embedding)     (None, 100, 100)          1000000   
_________________________________________________________________
flatten_21 (Flatten)         (None, 10000)             0         
_________________________________________________________________
dense_56 (Dense)             (None, 64)                640064    
_________________________________________________________________
dense_57 (Dense)             (None, 16)                1040      
=================================================================
Total params: 1,641,104
Trainable params: 1,641,104
Non-trainable params: 0

I set the weigth of the emedding layer:

model.layers[0].set_weights([embedding_matrix])
model.layers[0].trainable = False

model.compile(optimizer='rmsprop',
    loss='categorical_crossentropy',
    metrics=['categorical_accuracy'])
history = model.fit(x_train, y_train, epochs=10, batch_size=32, validation_data=(x_val, y_val))

But I end up with the error

ValueError: Shapes (None, 1) and (None, 16) are incompatible

Everything works right if I do a single-label classification (using Dense(1) as last layer and sigmoid activation), but I can't understand why this is happening.

Answer 1

You should encode your labels into one-hot format if you use categorical_crossentropy.

Otherwise you can try with sparse_categorical_crossentropy as loss function which accept your format of labels (info).

https://stats.stackexchange.com/questions/326065/cross-entropy-vs-sparse-cross-entropy-when-to-use-one-over-the-other