CODEWITHSUNDEEP
pythonmachine-learningneural-networktensorflow
John
John

Reputation: 43

Simple TensorFlow Neural Network minimizes cost function yet all results are close to 1

So I tried implementing the neural network from:

http://iamtrask.github.io/2015/07/12/basic-python-network/

but using TensorFlow instead. I printed out the cost function twice during training and the error is appears to be getting smaller according yet all the values in the output layer are close to 1 when only two of them should be. I imagine it might be something wrong with my maths but I'm not sure. There is no difference when I try with a hidden layer or use Error Squared as cost function. Here is my code:

import tensorflow as tf
import numpy as np

input_layer_size = 3
output_layer_size = 1


x = tf.placeholder(tf.float32, [None, input_layer_size]) #holds input values
y = tf.placeholder(tf.float32, [None, output_layer_size]) # holds true y values

tf.set_random_seed(1)

input_weights = tf.Variable(tf.random_normal([input_layer_size, output_layer_size]))
input_bias = tf.Variable(tf.random_normal([1, output_layer_size]))

output_layer_vals = tf.nn.sigmoid(tf.matmul(x, input_weights) + input_bias)

cross_entropy = -tf.reduce_sum(y * tf.log(output_layer_vals))

training = tf.train.AdamOptimizer(0.1).minimize(cross_entropy)

x_data = np.array(
    [[0,0,1],
     [0,1,1],
     [1,0,1],
     [1,1,1]])

y_data = np.reshape(np.array([0,0,1,1]).T, (4, 1))


with tf.Session() as ses:
    init = tf.initialize_all_variables()

    ses.run(init)

    for _ in range(1000):
        ses.run(training, feed_dict={x: x_data, y:y_data})

        if _ % 500 == 0:
            print(ses.run(output_layer_vals, feed_dict={x: x_data}))
            print(ses.run(cross_entropy, feed_dict={x: x_data, y:y_data}))
            print('\n\n')

And this is what it outputs:

[[ 0.82036656]
 [ 0.96750367]
 [ 0.87607527]
 [ 0.97876281]]
0.21947 #first cross_entropy error 



[[ 0.99937409]
 [ 0.99998224]
 [ 0.99992537]
 [ 0.99999785]]
0.00062825 #second cross_entropy error, as you can see, it's smaller

Upvotes: 0

Views: 713

Answers (1)

daniel451
daniel451

Reputation: 11002

First of all: you have no hidden layer. As far as I remember basic perceptrons could possibly model the XOR problem, but it needed some adjustments. However, AI is just invented by biology, but it does not model real neural networks exactly. Thus, you have to at least build an MLP (Multilayer perceptron), which consits of at least one input, one hidden and one output layer. The XOR problem needs at least two neurons + bias in the hidden layer to be solved correctly (with a high precision).

Additionally your learning rate is too high. 0.1 is a very high learning rate. To put it simply: it basically means that you update/adapt your current state by 10% of one single learning step. This lets your network forget about already learned invariants quickly. Usually the learning rate is something in between 1e-2 to 1e-6, depending on your problem, network size and general architecture.

Moreover you implemented the "simplified/short" version of cross-entropy. See wikipedia for the full version: cross-entropy. However, to avoid some edge cases TensorFlow already has its own version of cross-entropy: for example tf.nn.softmax_cross_entropy_with_logits.

Finally you should remember that the cross-entropy error is a logistic loss function that operates on probabilities of your classes. Although your sigmoid function squashes the output layer into an interval of [0, 1], this does only work in your case because you have one single output neuron. As soon as you have more than one output neuron, you also need the sum of the output layer to be exactly 1,0 in order to really describes probabilities for every class on the output layer.

Upvotes: 1

Related Questions