CODEWITHSUNDEEP
pythontensorflowmachine-learning
Fribble Guo
Fribble Guo

Reputation: 3

tensorflow GradientDescentOptimizer not updating variables?

I'm new to machine learning. I started with the simplest example of classification mnist handwritten images with softmax and gradient descent. By referencing some other examples, I came up with my own Logistic regression below:

import tensorflow as tf
import numpy as np


(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = np.float32(x_train / 255.0)
x_test = np.float32(x_test / 255.0)

X = tf.placeholder(tf.float32, [None, 28, 28])
Y = tf.placeholder(tf.uint8, [100])

XX = tf.reshape(X, [-1, 784])

W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))

def err(x, y):
    predictions = tf.matmul(x, W) + b
    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=tf.reshape(y, [-1, 1]), logits=predictions))
    # value = tf.reduce_mean(y * tf.log(predictions))
    # loss = -tf.reduce_mean(tf.one_hot(y, 10) * tf.log(predictions)) * 100.
    return loss

# cost = err(np.reshape(x_train[:100], (-1, 784)), y_train[:100])
cost = err(tf.reshape(X, (-1, 784)), Y)

optimizer = tf.train.GradientDescentOptimizer(0.005).minimize(cost)


init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)



# temp = sess.run(tf.matmul(XX, W) + b, feed_dict={X: x_train[:100]})

temp = sess.run(cost, feed_dict={X: x_train[:100], Y: y_train[:100]})
print(temp)
# print(temp.dtype)
# print(type(temp))

for i in range(100):
    sess.run(optimizer, feed_dict={X: x_train[i * 100: 100 * (i + 1)], Y: y_train[i * 100: 100 * (i + 1)]})
    # sess.run(optimizer, feed_dict={X: x_train[: 100], Y: y_train[:100]})

temp = sess.run(cost, feed_dict={X: x_train[:100], Y: y_train[:100]})
print(temp)


sess.close()

I tried to run the optimizer some iterations, feeding data with train image data and labeles. In my understanding, during the optimizer run, the variables of 'W' and 'b' should be update so the model would produce different result before and after training. But with this code, the printed costs of the model before and after optimizer run were the same. What can be wrong to make this happen?

Upvotes: 0

Views: 252

Answers (1)

Vlad
Vlad

Reputation: 8595

You are initializing the weights matrix W with zeros and as a result, all parameters receive the same gradient value at each weights update. For weights initialization use tf.truncated_normal(), tf.random_normal(), tf.contrib.layers.xavier_initializer() or something else, but not zeros.

This is a similar question.

Upvotes: 1

Related Questions