CODEWITHSUNDEEP
pythonnumpytensorflow
KingBob
KingBob

Reputation: 37

Why is this neural network's loss always 0.0 and accuracy always 1.0?

I'm trying to train a basic feedforward neural network on a dataset of about 80 entries (mostly as a proof of concept, I'm aware my dataset is way too small). I based my code off of the MNIST dataset example. I've selected a batch size of 10 and am running it with 8 steps:

learning_rate = 0.01
num_steps = 8
batch_size = 10
display_step = 1

num_input = 16
n_hidden_1 = 8
n_hidden_2 = 8
num_classes = 1

X = tf.placeholder("float", [None, num_input])
Y = tf.placeholder("float", [None, num_classes])

weights = {
    'h1': tf.Variable(tf.random_normal([num_input, n_hidden_1])),
    'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
    'out': tf.Variable(tf.random_normal([n_hidden_2, num_classes]))
}
biases = {
    'b1': tf.Variable(tf.random_normal([n_hidden_1])),
    'b2': tf.Variable(tf.random_normal([n_hidden_2])),
    'out': tf.Variable(tf.random_normal([num_classes]))
}

layer_1 = tf.add(tf.matmul(X, weights['h1']), biases['b1'])
layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])
logits = tf.matmul(layer_2, weights['out']) + biases['out']

prediction = tf.nn.softmax(logits)
loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=Y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
train_op = optimizer.minimize(loss_op)
correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)
    for step in range(0, num_steps):
        batch_x, batch_y = manager.import_data()

        batch_x = batch_x[step * batch_size:(step + 1) * batch_size]
        batch_y = batch_y[step * batch_size:(step + 1) * batch_size]

        batch_x = np.reshape(batch_x, (batch_size, num_input))
        batch_y = np.reshape(batch_y, (batch_size, num_classes))
        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})
        if step % display_step == 0 or step == 1:
            loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x, Y: batch_y})
            print("Step " + str(step) + ", Minibatch Loss= " + \
                  "{:.4f}".format(loss) + ", Training Accuracy= " + \
                  "{:.3f}".format(acc))

manager.import_data() returns a list of numpy arrays. I know that I should be randomizing the selection of batches, and I'll eventually implement that -- however, the output is:

Step 0, Minibatch Loss= 0.0000, Training Accuracy= 1.000
Step 1, Minibatch Loss= 0.0000, Training Accuracy= 1.000
Step 2, Minibatch Loss= 0.0000, Training Accuracy= 1.000
Step 3, Minibatch Loss= 0.0000, Training Accuracy= 1.000
Step 4, Minibatch Loss= 0.0000, Training Accuracy= 1.000
Step 5, Minibatch Loss= 0.0000, Training Accuracy= 1.000
Step 6, Minibatch Loss= 0.0000, Training Accuracy= 1.000
Step 7, Minibatch Loss= 0.0000, Training Accuracy= 1.000

Obviously this shouldn't be the case. What am I doing wrong?

Upvotes: 1

Views: 1108

Answers (1)

Georgi Karadzhov
Georgi Karadzhov

Reputation: 771

I'd guess that in your training set all items are with the same label (for example 0).

The best course of action when dealing with neural networks would be to prepare 3 different set - train, val and test with roughly the same distribution between the classes. Train is used in training time, val is used at the end of each iteration to save or disregard the model. Test is something like a reality check for your model and you should not tune parameters based on your scores on test.

Upvotes: 1

Related Questions