CODEWITHSUNDEEP
tensorflow
cageman
cageman

Reputation: 333

tensorflow training on simple neural network does not improve accuracy

I got the network below and when I train it the accuracy remains 0.000. I tried to make it easy by including just 2 samples. The inputs are all zero's except for one of the samples. The difference between the samples is that in the case of zero's everywhere the output is something like 0.3 0.4 0.3 and in the other case its 0.4 0.3 0.3 (both sum up to 1). I would expect it to be easy to get at least 50% accuracy and probably 100% on just two training samples.

Question: is there something wrong in the configuration of my network? If not, any suggestions on how to proceed. So far tensorflow is not easy to debug for me.

Might have relevance: I first had the weights and bias initialized to zero and then got a 0.5 accuracy. When I print the content of the layers after training only the weight and bias of the out layer contain positive values.

self.session = tf.Session()
n_hidden_1 = 10 # 1st layer number of neurons
n_hidden_2 = 10 # 2nd layer number of neurons
self.num_input = 68 # data values
self.num_classes = 18

self.weights = {
    'h1': tf.Variable(tf.random_normal([self.num_input, n_hidden_1])),
    'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
    'out': tf.Variable(tf.random_normal([n_hidden_2, self.num_classes]))
}
self.biases = {
    'b1': tf.Variable(tf.random_normal([n_hidden_1])),
    'b2': tf.Variable(tf.random_normal([n_hidden_2])),
    'out': tf.Variable(tf.random_normal([self.num_classes]))
}
self.input = tf.placeholder(dtype=tf.float32, shape = [None, self.num_input])
self.output = tf.placeholder(dtype=tf.float32, shape = [None, self.num_classes])
layer_1 = tf.nn.relu(tf.add(tf.matmul(self.input, self.weights['h1']), self.biases['b1']))
# Hidden fully connected layer with 256 neurons
layer_2 = tf.nn.relu(tf.add(tf.matmul(layer_1, self.weights['h2']), self.biases['b2']))
# Output fully connected layer with a neuron for each class
self.out_layer = tf.nn.softmax(tf.matmul(layer_2, self.weights['out']) + self.biases['out'])

self.loss_op = tf.reduce_mean(tf.squared_difference(self.out_layer, self.output))
optimizer = tf.train.AdamOptimizer(learning_rate=0.1)
self.train_op = optimizer.minimize(self.loss_op)

# Evaluate model
correct_pred = tf.equal(tf.argmax(self.out_layer, 1), tf.argmax(self.output, 1))
self.accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
self.session.run(tf.global_variables_initializer())

def train(self,train_x,train_y):
    loss, acc = self.session.run([self.loss_op, self.accuracy], feed_dict={self.input: train_x, self.output: train_y})
    self.logger.info("Before training Loss= " + \
              "{:.4f}".format(loss) + ", Training Accuracy= " + \
              "{:.3f}".format(acc))

    self.session.run(self.train_op, feed_dict={self.input: train_x, self.output: train_y})
    loss, acc = self.session.run([self.loss_op, self.accuracy], feed_dict={self.input: train_x, self.output: train_y})
    self.logger.info("After training Loss= " + \
              "{:.4f}".format(loss) + ", Training Accuracy= " + \
              "{:.3f}".format(acc))

Upvotes: 0

Views: 172

Answers (1)

David Parks
David Parks

Reputation: 32111

It looks like you're running train(...) just once. You need to call session.run(train_op, feed_dict=...) in a loop.

That call will only make a single update to the parameters, which isn't going to be much better than random initialization.

Upvotes: 1

Related Questions