The accuracy of neural network training remains unchanged

Question

When I was running a tensorflow model on python, the accuracy of my model can't be improved by training. Even if I change my training data to a quite regular one, the model still didn't work. What's the problem?

Code:

train_x = np.array([1] * 1000 + [2] * 1000 + [3] * 1000)
train_y = np.zeros((3000, 3))
train_y[:1000,0] = 1
train_y[1000:2000,1] = 1
train_y[2000:3000,2] = 1
val_x = train_x
val_y = train_y

model = tf.keras.Sequential()
model.add(layers.Dense(3, activation='relu'))
model.add(layers.Dense(3, activation='relu'))
model.compile(optimizer=tf.keras.optimizers.Adam(0.1),
             loss=tf.keras.losses.categorical_crossentropy,
             metrics=[tf.keras.metrics.categorical_accuracy])

model.fit(train_x, train_y, epochs = 10, batch_size = 32, verbose = 1,
          shuffle = False,
          validation_data=(val_x, val_y))

And training result

Epoch 1/10
94/94 [==============================] - 0s 2ms/step - loss: 10.7836 - categorical_accuracy: 0.3120 - val_loss: 10.7454 - val_categorical_accuracy: 0.3333

Epoch 2/10
94/94 [==============================] - 0s 1ms/step - loss: 10.7454 - categorical_accuracy: 0.3333 - val_loss: 10.7454 - val_categorical_accuracy: 0.3333

Epoch 3/10
94/94 [==============================] - 0s 1ms/step - loss: 10.7454 - categorical_accuracy: 0.3333 - val_loss: 10.7454 - val_categorical_accuracy: 0.3333

Epoch 4/10
94/94 [==============================] - 0s 1ms/step - loss: 10.7454 - categorical_accuracy: 0.3333 - val_loss: 10.7454 - val_categorical_accuracy: 0.3333

Epoch 5/10
94/94 [==============================] - 0s 2ms/step - loss: 10.7454 - categorical_accuracy: 0.3333 - val_loss: 10.7454 - val_categorical_accuracy: 0.3333

So where I should adjust to get better performance, and which thing I have done wrong?

Answer 1

1. The problem is that with 3 input neurons and 1 feature (i.e., 1 column), the neural network doesn't have enough if... then combinations to learn the pattern you're trying to teach it. If you one-hot encode your input, it will effectively learn to multiply every input column by one and it will give the right answer.
2. You have the wrong activation function. For multi-class problems, use 'softmax'.
3. Your optimizer's learning rate is a little too high, so the step is too high and jumping all over the cost function. Use 0.01 at most.

Fully-working example:

import numpy as np
import tensorflow as tf

train_x = np.array([1] * 1000 + [2] * 1000 + [3] * 1000)
train_x = tf.keras.utils.to_categorical(train_x - 1)
train_y = np.zeros((3000, 3))
train_y[:1000,0] = 1
train_y[1000:2000,1] = 1
train_y[2000:3000,2] = 1
val_x = train_x
val_y = train_y

model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(3, activation='relu'))
model.add(tf.keras.layers.Dense(3, activation='softmax'))
model.compile(optimizer=tf.keras.optimizers.Adam(0.01),
             loss=tf.keras.losses.categorical_crossentropy,
             metrics=[tf.keras.metrics.categorical_accuracy])

model.fit(train_x, train_y, epochs = 10, batch_size = 32, verbose = 1,
          shuffle = False,
          validation_data=(val_x, val_y))

Epoch 9/10
  32/3000 [..............................] - ETA: 0s - loss: 0.0067 - cat_acc: 1.0000
 608/3000 [=====>........................] - ETA: 0s - loss: 0.0063 - cat_acc: 1.0000
1184/3000 [==========>...................] - ETA: 0s - loss: 0.0244 - cat_acc: 1.0000
1760/3000 [================>.............] - ETA: 0s - loss: 0.0553 - cat_acc: 1.0000
2272/3000 [=====================>........] - ETA: 0s - loss: 0.0550 - cat_acc: 1.0000
2848/3000 [===========================>..] - ETA: 0s - loss: 0.0447 - cat_acc: 1.0000